Improving Large Language Models for Yiddish
Listen to the summary
Uses a voice available on your device
Audio options
On this page
Key Takeaways
- MameLoshnLM outperformed baseline models like Gemma-2 9B and Llama 3.1 8B on the Kashes benchmark with an average score of 62.6.
- The team created Oytser, a high-quality pretraining corpus, to replace existing noisy web-scale data.
- The authors identified that existing mC4 Yiddish data is low quality, with nearly 30 percent being machine-translated and over 21 percent being misclassified Hebrew.
- The model demonstrated strong performance in English to Yiddish translation, exceeding the nearest competitor by more than 11 COMET points.
Summary & Methodology Analysis
The researchers addressed the scarcity of high-quality Yiddish data by building Oytser, a corpus that combines contemporary web sources with OCR-digitized literary materials processed via the Jochre 3 tool. This approach aimed to mitigate the issues found in standard web-scale datasets like mC4, where an audit revealed that only 42.2 percent of documents were genuine Yiddish content. The remaining portions consisted of 29.8 percent machine-translated text and 21.9 percent misidentified Hebrew content. By filtering these sources, the team established a cleaner foundation for training.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is MameLoshnLM?
It is a language model designed specifically to improve support for the Yiddish language.
Q2. Why is this research important?
Yiddish is a low-resource language that has been historically underserved by language models due to a lack of quality data and evaluation benchmarks.
Q3. How did the model perform compared to others?
MameLoshnLM achieved an average score of 62.6 on the Kashes benchmark, outperforming Gemma-2 9B at 57.0, Llama 3.1 8B at 56.8, and Qwen3 8B at 54.7.
Q4. What architecture does MameLoshnLM use?
The researchers performed continued pretraining on Llama 3.1 8B using a causal language modeling objective, which is a process where a model is trained to predict the next token in a sequence.
Q5. What is Kashes?
Kashes is a multi-task evaluation benchmark created by the authors to measure performance across translation, linguistic analysis, information extraction, and language understanding.
Q6. How was the training data assembled?
The Oytser corpus was assembled using web-native contemporary sources and literature digitized from the Yiddish Book Center using Jochre 3 software.
Q7. What were the findings regarding existing datasets?
The audit of the mC4 Yiddish split showed that less than half of the documents were high-quality Yiddish, with significant contamination from machine-translated content and mislabeled Hebrew.
Q8. Does this model improve machine translation?
Yes, it showed strong improvements in English to Yiddish translation on the FLORES+ benchmark, outperforming the closest competitor by more than 11 COMET points.
Q9. What are the primary limitations of the current research?
The paper notes that publicly available Yiddish pretraining text is limited in amount and quality, and existing benchmarks are often based on automatic translation rather than tasks specifically designed for the language.