[author: Xuan-Phi Nguyen, Sharifah Magani Aljunied, Shafiq Joty, Lidong Bing] [full title: Democratizing LLMs for Low-Resource Languages by Leveraging Their English Dominant Abilities With Linguistically Diverse Prompts]

Literature Notes# 

TABLE rows.file.link AS "Literature Note", rows.file.cday AS "Date"
FROM #note/literature AND [Democratizing LLMs for Low-Resource Languages by Leveraging Their English Dominant Abilities With Linguistically Diverse Prompts](/notes/democratizing-llms-for-low-resource-languages-by-leveraging-their-english-domina/)
GROUP BY file.link
sort date ASCENDING

Highlights#

  • in low-resource languages, obtaining such hand-picked exemplars can still be challenging, where unsupervised techniques may be necessary (Page 1)
  • competent generative capabilities of LLMs are observed only in high-resource languages, while their performances among under-represented languages fall behind due to pre-training data imbalance (Page 1)
  • we propose to assemble synthetic exemplars from a diverse set of high-resource languages to prompt the LLMs to translate from any language into English. These prompts are then used to create intra-lingual exemplars to perform tasks in the target languages (Page 1)
  • fine-tuning a 7B model on data generated from our method helps it perform competitively with a 175B model (Page 1)
  • In non-English translation tasks, our method even outperforms supervised prompting by up to 3 chrF++ in many low-resource languages (Page 1) ^558469488
    • Note: Can possibly draw performance benchmarking ideas from here
  • While most LLMs were pre-trained with multilingual corpora in addition to the gigantic English corpus, and were shown to demonstrate impressive abilities in other languages (Brown et al., 2020; Chowdhery et al., 2022; Scao et al., 2022; Shi et al., 2022; Huang et al., 2023), they only excel in high-resource languages, such as French (Page 1)
  • In this work, we focus on unsupervised, and zero-shot, generative translation and summarization tasks in low-resource languages, where no supervised few-shot prompts are used (Page 1)
  • we propose LinguisticallyDiverse Prompting (LDP), a technique that prompts the models to locate the task of “translate any language X into English” by showing the model exemplar pairs between every language and English (En). Practically, we gather a small set of synthetic X -> En exemplars from a diverse set of high-resource languages using off-the-shelf unsupervised MT models (Page 2)
  • The most multilingual LLM is BLOOM (Scao et al., 2022), which was trained on 46 languages in the ROOTS corpus (Laurençon et al., 2022). This corpus includes 34 Indic and African languages regarded as low-resource, with each language having a pre-training 5% in Tumbuka for the African group coverage of less than 1% in Hindi for the Indic group, to 2e^-5 % in Tumbuka for the African group (Page 3)
  • Therefore, we use BLOOM as the main model to evaluate our methods and baselines in such 34 low-resource languages (Page 3)
  • the noteworthy gap between existing UMT and LLMs is that their language coverages do not overlap much, preventing us from using UMT models to enhance LLMs. (Page 3)
  • Hendy et al. (2023) show that GPT models can perform competitively alongside state-of-the-art MT models (Page 3)
    • Note: Get this paper
  • there is still limited research involving low-resource languages in completely zero-shot setups (Page 3)
  • linguistically-diverse prompting (LDP) method is inspired from three intuitive assumptions (Page 3)
  • LLMs have already learned most of the knowledge and task concepts implicitly during pre-training (Page 3) ^558469498
  • The second assumption is that the models intuitively learn to perform language encoding and understanding at an earlier time, before learning to generate language. (Page 4)
  • The third assumption is that LLMs can already exhibit near-human generative abilities in the dominant language E (mostly English) where pre-training data is often orders of magnitude larger than other (Page 4)
  • we argue that with respect to a minority language X, translation tasks between X and E are no longer symmetric and can be interpreted more broadly as follows (Page 4)
  • X -> E translation is a language understanding task (NLU) in X. This notion extends NLU beyond popular classification tasks, such as sentiment analysis or entailment (Page 4)
  • E -> X translation is a language generation task (NLG) in X, which is often harder to master than NLU (Page 4)
  • Specifically, while the input in E can be easy to encode, generating the intended results in X will be challenging if the model has not seen enough texts in X (Page 4)
  • we design in-context exemplars so that the model locates the task of “translate from any language X into E”, by demonstrating prompt pairs from “every language” to E. (Page 4)
  • This is because the target-side prompt distribution is now realistic and consistently close to the true target distribution we expect the model to generate, which has been shown to be crucial for in-context learning to work (Xie et al., 2021). (Page 5)
  • X -> E task. As mentioned above, we first gather n Z(sub)i -> X exemplar pairs (s(sub)Z, t^i(sub)E) with Z(sub)i ∈ Z(fancy) where Z(fancy) is a diverse set of languages with various writing systems, lexical and regional characteristics, such as French (Fr) and Chinese (Zh), and Zi ≠ {X, E}. Such exemplars can be collected by randomly selecting a single sentence from unlabeled data of the respective language Z(sub)i (Page 5)
    • Note: Convert to latex
  • E -> X task. We leverage [LDP prompt] to build intra-lingual prompts with unlabeled data from the target X language. Specifically, given m unlabeled texts s^j (sub) X ∈ D(sub)X with D(sub)X as monolinguial corpus in X, we produce synthetic back-translation (BT) target s^j(sub)E - L^(mt)(sub)X->E(s^j(sub)X). Then we use the BT synthetic pairs as in-context exemplars for E-> X translation tasks for input S(sub)E (insert formula -> L^(mtbt)) (Page 5)
    • Note: Convert to Latex
  • we can also use L^(mtbt) simply swapping the direction of the (s^1(sub)E, t^1(sub)X) to (s^1(sub)X, t^1(sub)E). Nonetheless, we found in the experiments that both L^(mt) and L^(mtbt) perform similarly and on par with supervised prompting for the X -> E task, suggesting that we do not need any supervised or unlabeled data to translate any language into English. … we can even omit these back-translation exemplars entirely with non-BT L^(mt) LDP by using native language tags (Page 5)
  • During training, we only compute loss on the [output] part to train the model to generate the right language (Page 5)
  • we empirically found that the model fails to learn to generate the low-resource languages unless we increase the learnable parameter counts significantly, which seems to defeat the purpose of using PEFT (Page 5)
    • Note: PEFT = parameter-efficient fine-tuning approach, e.g. LoRA
  • As the ROOTS corpus (Laurençon et al., 2022) that BLOOM (Scao et al., 2022) was pre-trained on offers the most diverse language coverage with open-sourced transparency, we tested our methods mainly with the BLOOM model (Page 6)
  • Figure 5a reveals one reason the models struggle to translate En->X when using LDP prompts L^mt (without intra-lingual BT data) is that the target-side distribution contains multiple languages, and the models struggle to recognize unfamiliar language tags, such as Marathi (Mr), and often generate wrong translations in the wrong languages (Page 9)
  • supplying synthetic intra-lingual prompts where the target-side is consistently in the intended language, as shown in Figure 5b with L^mtbt, is more important in getting the models to recognize language rather than the language tag. In fact, we found that removing the language tag entirely can help improve the performance slightly. (Page 9)
  • which high-resource languages should be selected as LDP exemplars. Table 6 examines which LDP language choice is optimal (Page 10)
  • choosing a single related language (Hindi), which is often called cross-lingual prompting (Zhang et al., 2023; Zhu et al., 2023), can be disastrous as the model tends to translate the prompt language rather than the test language (Page 10)
  • Choosing a single but distant language (Vi or Zh) yields better results, while choosing a wide variety of languages across different regions (e.g., Ar,Zh,Vi,Fr) may be the optimal choice. (Page 10)
  • We introduce linguistically-diverse prompting (LDP), which is designed to use synthetic high-quality in-context exemplars from high-resource languages to prompt LLMs to perform generative tasks, such as translation and summarization, in low-resource languages (Page 11)