Source on The Edge Cases

Notes for Backpropagation Through Space Time and the Brain

Mon, 15 Dec 2025 18:11:24 +0000

Summary

What are the key ideas? How can I apply this knowledge that I learned? How do these ideas relate to what I already know?

💡Ideas

Note: These should become Project Idea fleeting notes with a link back to this source.

Fleeting Notes

Note: Conceptual notes that aren’t as tied to the source material. These are fodder for processing into evergreen notes.

Literature Notes

Note: Commentary on the book or things you don’t want to forget. Maybe for use in a public mind garden. Should have a visible source reference. These are “weakly evergreen” because they’re tied heavily to a single piece. Must be in your own words. These should be processed into Evergreen notes, or used as fodder for them. ““Literature notes”, titled after a single work and meant primarily as linkages to other more durable notes, and as targets for backlinks. I write these roughly as “outline notes,” except for someone else’s ideas.” These should be regularly refactored into individual notes in Sources/! Literature Notes/

Designing Machine Learning Systems

Wed, 04 Oct 2023 00:41:17 +0000

[author: Chip Huyen]

Summary

What are the key ideas? How can I apply this knowledge that I learned? How do these ideas relate to what I already know?

💡Ideas

Note: These should become Project Idea fleeting notes with a link back to this source.

Fleeting Notes

Note: Conceptual notes that aren’t as tied to the source material. These are fodder for processing into evergreen notes.

Literature Notes

Democratizing LLMs for Low-Resource Languages by Leveraging Their English Dominant Abilities With Linguistically Diverse Prompts

Tue, 04 Jul 2023 20:05:00 +0000

[author: Xuan-Phi Nguyen, Sharifah Magani Aljunied, Shafiq Joty, Lidong Bing] [full title: Democratizing LLMs for Low-Resource Languages by Leveraging Their English Dominant Abilities With Linguistically Diverse Prompts]

Literature Notes

TABLE rows.file.link AS "Literature Note", rows.file.cday AS "Date"
FROM #note/literature AND [Democratizing LLMs for Low-Resource Languages by Leveraging Their English Dominant Abilities With Linguistically Diverse Prompts](/notes/democratizing-llms-for-low-resource-languages-by-leveraging-their-english-domina/)
GROUP BY file.link
sort date ASCENDING

Highlights

in low-resource languages, obtaining such hand-picked exemplars can still be challenging, where unsupervised techniques may be necessary (Page 1)
competent generative capabilities of LLMs are observed only in high-resource languages, while their performances among under-represented languages fall behind due to pre-training data imbalance (Page 1)
we propose to assemble synthetic exemplars from a diverse set of high-resource languages to prompt the LLMs to translate from any language into English. These prompts are then used to create intra-lingual exemplars to perform tasks in the target languages (Page 1)
ﬁne-tuning a 7B model on data generated from our method helps it perform competitively with a 175B model (Page 1)
In non-English translation tasks, our method even outperforms supervised prompting by up to 3 chrF++ in many low-resource languages (Page 1) ^558469488
- Note: Can possibly draw performance benchmarking ideas from here
While most LLMs were pre-trained with multilingual corpora in addition to the gigantic English corpus, and were shown to demonstrate impressive abilities in other languages (Brown et al., 2020; Chowdhery et al., 2022; Scao et al., 2022; Shi et al., 2022; Huang et al., 2023), they only excel in high-resource languages, such as French (Page 1)
In this work, we focus on unsupervised, and zero-shot, generative translation and summarization tasks in low-resource languages, where no supervised few-shot prompts are used (Page 1)
we propose LinguisticallyDiverse Prompting (LDP), a technique that prompts the models to locate the task of “translate any language X into English” by showing the model exemplar pairs between every language and English (En). Practically, we gather a small set of synthetic X -> En exemplars from a diverse set of high-resource languages using off-the-shelf unsupervised MT models (Page 2)
The most multilingual LLM is BLOOM (Scao et al., 2022), which was trained on 46 languages in the ROOTS corpus (Laurençon et al., 2022). This corpus includes 34 Indic and African languages regarded as low-resource, with each language having a pre-training 5% in Tumbuka for the African group coverage of less than 1% in Hindi for the Indic group, to 2e^-5 % in Tumbuka for the African group (Page 3)
Therefore, we use BLOOM as the main model to evaluate our methods and baselines in such 34 low-resource languages (Page 3)
the noteworthy gap between existing UMT and LLMs is that their language coverages do not overlap much, preventing us from using UMT models to enhance LLMs. (Page 3)
Hendy et al. (2023) show that GPT models can perform competitively alongside state-of-the-art MT models (Page 3)
- Note: Get this paper
there is still limited research involving low-resource languages in completely zero-shot setups (Page 3)
linguistically-diverse prompting (LDP) method is inspired from three intuitive assumptions (Page 3)
LLMs have already learned most of the knowledge and task concepts implicitly during pre-training (Page 3) ^558469498
The second assumption is that the models intuitively learn to perform language encoding and understanding at an earlier time, before learning to generate language. (Page 4)
The third assumption is that LLMs can already exhibit near-human generative abilities in the dominant language E (mostly English) where pre-training data is often orders of magnitude larger than other (Page 4)
we argue that with respect to a minority language X, translation tasks between X and E are no longer symmetric and can be interpreted more broadly as follows (Page 4)
X -> E translation is a language understanding task (NLU) in X. This notion extends NLU beyond popular classiﬁcation tasks, such as sentiment analysis or entailment (Page 4)
E -> X translation is a language generation task (NLG) in X, which is often harder to master than NLU (Page 4)
Speciﬁcally, while the input in E can be easy to encode, generating the intended results in X will be challenging if the model has not seen enough texts in X (Page 4)
we design in-context exemplars so that the model locates the task of “translate from any language X into E”, by demonstrating prompt pairs from “every language” to E. (Page 4)
This is because the target-side prompt distribution is now realistic and consistently close to the true target distribution we expect the model to generate, which has been shown to be crucial for in-context learning to work (Xie et al., 2021). (Page 5)
X -> E task. As mentioned above, we ﬁrst gather n Z(sub)i -> X exemplar pairs (s(sub)Z, t^i(sub)E) with Z(sub)i ∈ Z(fancy) where Z(fancy) is a diverse set of languages with various writing systems, lexical and regional characteristics, such as French (Fr) and Chinese (Zh), and Zi ≠ {X, E}. Such exemplars can be collected by randomly selecting a single sentence from unlabeled data of the respective language Z(sub)i (Page 5)
- Note: Convert to latex
E -> X task. We leverage [LDP prompt] to build intra-lingual prompts with unlabeled data from the target X language. Specifically, given m unlabeled texts s^j (sub) X ∈ D(sub)X with D(sub)X as monolinguial corpus in X, we produce synthetic back-translation (BT) target s^j(sub)E - L^(mt)(sub)X->E(s^j(sub)X). Then we use the BT synthetic pairs as in-context exemplars for E-> X translation tasks for input S(sub)E (insert formula -> L^(mtbt)) (Page 5)
- Note: Convert to Latex
we can also use L^(mtbt) simply swapping the direction of the (s^1(sub)E, t^1(sub)X) to (s^1(sub)X, t^1(sub)E). Nonetheless, we found in the experiments that both L^(mt) and L^(mtbt) perform similarly and on par with supervised prompting for the X -> E task, suggesting that we do not need any supervised or unlabeled data to translate any language into English. … we can even omit these back-translation exemplars entirely with non-BT L^(mt) LDP by using native language tags (Page 5)
During training, we only compute loss on the [output] part to train the model to generate the right language (Page 5)
we empirically found that the model fails to learn to generate the low-resource languages unless we increase the learnable parameter counts signiﬁcantly, which seems to defeat the purpose of using PEFT (Page 5)
- Note: PEFT = parameter-efficient fine-tuning approach, e.g. LoRA
As the ROOTS corpus (Laurençon et al., 2022) that BLOOM (Scao et al., 2022) was pre-trained on offers the most diverse language coverage with open-sourced transparency, we tested our methods mainly with the BLOOM model (Page 6)
Figure 5a reveals one reason the models struggle to translate En->X when using LDP prompts L^mt (without intra-lingual BT data) is that the target-side distribution contains multiple languages, and the models struggle to recognize unfamiliar language tags, such as Marathi (Mr), and often generate wrong translations in the wrong languages (Page 9)
supplying synthetic intra-lingual prompts where the target-side is consistently in the intended language, as shown in Figure 5b with L^mtbt, is more important in getting the models to recognize language rather than the language tag. In fact, we found that removing the language tag entirely can help improve the performance slightly. (Page 9)
which high-resource languages should be selected as LDP exemplars. Table 6 examines which LDP language choice is optimal (Page 10)
choosing a single related language (Hindi), which is often called cross-lingual prompting (Zhang et al., 2023; Zhu et al., 2023), can be disastrous as the model tends to translate the prompt language rather than the test language (Page 10)
Choosing a single but distant language (Vi or Zh) yields better results, while choosing a wide variety of languages across different regions (e.g., Ar,Zh,Vi,Fr) may be the optimal choice. (Page 10)
We introduce linguistically-diverse prompting (LDP), which is designed to use synthetic high-quality in-context exemplars from high-resource languages to prompt LLMs to perform generative tasks, such as translation and summarization, in low-resource languages (Page 11)

Machine Learning System Design Interview

Sat, 15 Apr 2023 02:43:54 +0000

Note: See here for valid statuses

[author: Ali Aminian Alex Xu]

Summary

What are the key ideas? How can I apply this knowledge that I learned? How do these ideas relate to what I already know?

💡Ideas

Note: These should become Project Idea fleeting notes with a link back to this source.

Fleeting Notes

Note: Conceptual notes that aren’t as tied to the source material. These are fodder for processing into evergreen notes.

System Design Interview

Mon, 31 Oct 2022 03:03:50 +0000

Note: See here for valid statuses

[author: ]

What is the Book About?

Note: See How To Read a Book What kind of book is it?

Non-fiction, practical book. Examples of system design interview questions. What is the book about in 1-2 sentences at most?
System design interview questions and how to answer them.

Summaries

Chapter 1: Scale from Zero to Millions of Users

Single Server Setup

Request flow:
1. Users access websites through domain names, which are resolved by DNS
2. Internet protocol (IP) address is returned to the browser.
3. Once IP address is obtained, HTTP requests are sent directly to server
4. The web server returns HTML pages or JSON response.

Database

Separating web/mobile traffic (web tier) from database (data tier) servers allows them to be scaled independently.
Non-relational DBs might be right choice if:
- Application requires super-low latency
- Data are instructured
- You only need to serialize and deserialize data
- You need to store massive amount of data.

Vertical vs. Horizontal Scaling

Vertical scaling - adding more power (CPU, memories, etc.) to servers. “Scale up”
- Limitations:
  - Hard limit - can’t add unlimited CPU and memory
  - No failover or redundancy
Horizontal scaling - adding more serviers to pool of resources

Load Balancer

A load balancer evenly distributes traffic among web servers.
The load balancer communicates with web serves via private IPs.
Load balancers can redirect traffic away from offline servers
They also can handle rapid gains in traffic, allowing you to simply add more servers behind the load balancer to accomodate the growing traffic.

Database Replication

Master/slave terminology: master only handles writes, slave databases only support reads and get copies of databases from the master.
Advantages
- Better performance
- More reliable
If one slave DB is available then goes offline, read operations get directed to master database
If master goes offline, after some time a slave will be promoted to the new master.

Cache

A cache is a temporary storage area that stores the result of expensive responses/frequently acccesed data in memory to serve them more quickly.

FourthBrain - ML Model Drift and Decay

Tue, 22 Mar 2022 21:43:08 +0000

FourthBrain - ML Model Drift and Decay

Metadata

URL: https://www.youtube.com/watch?v=C1yiOonTYqM Author: FourthBrain

Literature Notes

TABLE rows.file.link AS "Literature Note", rows.file.cday AS "Date"
FROM #note/literature AND Example Title
GROUP BY file.link
sort date ASCENDING

Note: there may not be many literature notes for this, because it’s a how-to type of talk, where there won’t be notes on the piece itself, but on what it teaches.

Fleeting Notes

Supervised ML is interpolation geometry. “All training data exists as points in a Euclidean space…we can create an enclosure. Then, inside that enclosure, we can interpolate.” Global problem space -> regional problem space -> training domain -> model. When we get data, there are quantity and quality gaps which are smaller than the ideal training domain. This means that a model won’t work well at trying to predict on observations that fall outside the training domain because machine learning models for non-linear problems are interpolation machines. They can not extrapolate. Active learning can fix gaps between the training domain and regional problem space. This flags and prioritizes confusing inputs for human ground-truthing and augments the training set with newly labeled observations to fill gaps in the training domain. Most machine learning methods are based on the assumption that the problem space does not change over time. In the real world, though, the problem space does indeed change - this is called non-stationarity. In other words, the problem space drifts out of alignment with the existing training domain and model (data drift). Addressing data drift can take the form of filling gaps with active learning or pruning old or obsolete observations by using a weighting scheme (among other strategies). MLOps and Pipelines are beneficial for monitoring models for appreciable drift and refitting the model on augmented training data when drift is detected. Two kinds of drift: - concept drift - or real drift; a shift in the relationship between model inputs and outputs (changes p(y | x)). This always causes model decay. - data drift - change to the statistical distributions within the input data (also known as input drift, feature drift, covariate shift). This can cause model decay by driving a change in p(y | x). If it doesn’t cause that change, then it’s called virtual drift. Both kinds of drift can manifest in 4 modes: 1. Abrupt 2. Incremental 3. Gradual 4. Reoccurring (like seasonality)

Learn Functional Programming With Elixir

Thu, 19 Aug 2021 21:35:26 +0000

Metadata

Author: Ulisses Almeida
Full Title: Learn Functional Programming With Elixir
Category: #books

Literature Notes

TABLE rows.file.link AS "Literature Note", rows.file.cday AS "Date"

FROM #note/literature AND [Learn Functional Programming With Elixir](/notes/learn-functional-programming-with-elixir/)

GROUP BY file.link

sort date ASCENDING

Summaries

Chapter 1: Thinking Functionally

In functional programming, functions are the basic building blocks of a program, values are immutable, and the code is declarative, rather than imperative. Functions, immutability, and declarative code are the core principles of functional programming.