Open-weight models: the Welsh results
Twenty open or local models, tested on the same core suites as the leading models.
Muse Spark 1.2, the three Qwen3.8 models, MiMo V2.6 Pro, Kimi K3, Muse Glimmer 30B, GLM-5.3 Flash, GLM-5.3 and GLM-5.2 were tested through API services; DeepSeek V4.1 Flash was tested locally on eight GPUs, Jupiter-N-120B on a DGX Spark cluster, and the smaller models on an RTX 4090. There were 108,374 responses: 33,657 each from Jupiter and Caernarfon Cymraeg Instruct, and 2,053 each from twenty other models. Temperature 0 was used where supported by the API, with the prompt format specified by each model’s developer; random seed 1 was used where supported. The overall score uses the same 19 core suites as the main CymraegBench leaderboard; the 232 cases of the CCC Language Audit do not contribute to this score.
| Extended rank | Model | Overall score | Welsh reasoning | Practical Welsh | Translation BLEU | Invalid |
|---|---|---|---|---|---|---|
| 7 | Muse Spark 1.2 open weights | 83.01 | 85.68 | 80.33 | 71.46 | 0.00% |
| 11 | Qwen3.8 2.4T-A95B open weights | 81.09 | 86.28 | 75.89 | 60.84 | 0.00% |
| 12 | MiMo V2.6 Pro open weights | 80.73 | 83.79 | 77.67 | 61.98 | 0.00% |
| 14 | Qwen3.8 Flash open weights Official API | 79.81 | 84.06 | 75.56 | 58.79 | 0.05% |
| 16 | Kimi K3 open weights | 75.12 | 77.92 | 72.33 | 67.62 | 0.00% |
| 17 | GLM-5.3 Flash open weights | 74.38 | 76.21 | 72.56 | 58.41 | 0.00% |
| 18 | GLM-5.3 open weights | 73.38 | 77.31 | 69.44 | 58.05 | 0.00% |
| 19 | Muse Glimmer 30B open weights | 70.64 | 77.50 | 63.78 | 60.71 | 0.00% |
| 20 | Qwen3.8 27B open weights | 70.47 | 79.83 | 61.11 | 46.03 | 0.00% |
| 21 | DeepSeek V4.1 Flash open weights | 69.45 | 76.23 | 62.67 | 69.03 | 0.00% |
| 22 | GLM-5.2 open weights | 68.42 | 68.62 | 68.22 | 57.07 | 0.00% |
| 26 | Qwen3.5 9B open weights | 55.10 | 57.98 | 52.22 | 37.06 | 0.00% |
| 27 | Gemma 4 12B Cymraeg v6 (Q8_0) open weights | 52.70 | 54.52 | 50.89 | 52.30 | 0.00% |
| 28 | Gemma 4 E4B Cymraeg v4 (Q8_0) open weights | 49.70 | 51.39 | 48.00 | 40.23 | 0.00% |
| 29 | Jupiter-N-120B open weights | 48.87 | 52.70 | 45.04 | 43.85 | 0.37% |
| 30 | Mwydryn 7B version 2 open weights | 30.37 | 39.96 | 20.78 | 41.89 | 0.83% |
| 31 | Mistral-7B-Cymraeg-Welsh-v2 open weights | 30.09 | 40.85 | 19.33 | 31.39 | 0.49% |
| 32 | Mwydryn Phi-2 open weights | 26.90 | 36.47 | 17.33 | 9.11 | 2.83% |
| 33 | Machynlleth 3B Instruct open weights | 16.44 | 8.44 | 24.44 | 41.03 | 1.80% |
| 34 | Caernarfon 3B Cymraeg Instruct v0.1 fine-tuned | 10.83 | 16.80 | 4.86 | 22.00 | 3.52% |
| 35 | Techiaith Llama 3.2 1B Welsh SFT open weights | 9.87 | 17.18 | 2.56 | 1.05 | 34.53% |
| 36 | BritLLM Caernarfon 3B (base model) open weights | 0.00 | 0.00 | 0.00 | 14.94 | 56.84% |
Invalid means empty output, output reaching the token limit or a case exceeding the model’s context window; it scores zero. The percentage is calculated across all test cases, including CCC. Every model was scored on 100% of cases, with no unresolved infrastructure errors.
What do the results tell us?
Muse Spark 1.2 is the highest open-weight model, seventh overall at 83.01. Qwen3.8 2.4T-A95B follows, eleventh at 81.09, then MiMo V2.6 Pro twelfth at 80.73. Kimi K3, both GLM-5.3 models, Muse Glimmer 30B, Qwen3.8 27B, DeepSeek V4.1 Flash and GLM-5.2 all score between 68.42 and 75.12. Qwen3.5 9B scored 55.10 on a single RTX 4090, Jupiter 48.87, Mwydryn 7B v2 30.37, and Mistral Cymraeg v2 30.09.
Techiaith 1B’s result reveals a different problem: not just limited knowledge, but difficulty following a short instruction and finishing an answer. The exact Alpaca format from Techiaith’s official demo was used, with its setting of repetition_penalty = 1.15; nevertheless, over a third of outputs reached the limit. That is a useful finding for the next fine-tuning round.
The effect of fine-tuning a 3B model: the BritLLM Caernarfon base model scored 0.00, with 56.84% invalid. The Caernarfon Cymraeg Instruct adapter raises the score to 10.83, while Machynlleth’s Celtic pre-training and instruction fine-tuning raise it to 16.44 and reduce the invalid rate to 1.80%. The comparison shows that instruction format and data are essential; the ability to continue Welsh text alone is not enough.