Mwydryn fine-tuned tested
A small, open experiment from Bangor showing how a relatively small model can be adapted for Welsh.
The facts
| Developer | BangorAI, an experimental account within Bangor’s language technology work |
|---|---|
| Models | BangorAI/phi2-mwydryn-1 and BangorAI/mwydryn-7b-fersiwn-2 |
| Method | Phi-2 (2.7B, Microsoft) → continued pre-training on Welsh (BangorAI/phi2-cy-100k dataset) → fine-tuning on alpaca-cleaned translated into Welsh |
| Stated status | Experimental — the model card notes weaknesses in factual accuracy and consistency |
Why include it on the leaderboard?
It does not compete with the largest models, and its creators do not claim it does. It appears on the leaderboard for two reasons. First, transparency: this is a small Welsh project documented openly, step by step, including its data, method and weaknesses. Second, it provides a starting point. To measure AI’s progress in Welsh over the coming years, we need to know what a small model can do in 2026, rather than looking only at the largest models.
BangorAI has also published 7B models: Mistral-7B-Cymraeg-Welsh-v2 and Mwydryn 7B version 2. The base model underwent further pre-training on Welsh data and then instruction fine-tuning. With these models, BangorAI has demonstrated the complete process — continued pre-training followed by supervised fine-tuning — at a scale a university lab can sustain.
CymraegBench v0.1 results
The three models were tested on the same 19 core suites, with up to 100 cases per suite, temperature 0 and random seed 1. Each model also received the 232 CCC Language Audit cases. Empty output or output reaching the token limit scores zero.
| Model | Overall score | Reasoning | Practical Welsh | Translation BLEU | Invalid |
|---|---|---|---|---|---|
| Mwydryn 7B version 2 | 30.37 | 39.96 | 20.78 | 41.89 | 0.83% |
| Mistral-7B-Cymraeg-Welsh-v2 | 30.09 | 40.85 | 19.33 | 31.39 | 0.49% |
| Mwydryn Phi-2 | 26.90 | 36.47 | 17.33 | 9.11 | 2.83% |
Mwydryn 7B v2 scored 30.37 and Mistral Cymraeg v2 30.09. Mistral scored 40.85 on reasoning, while Mwydryn 7B scored 20.78 on practical Welsh and 41.89 on translation. Mwydryn Phi-2 scored 26.90 overall and 9.11 on translation.
See the analysis of the seven open or local models and the full data on GitHub.