Machynlleth 3B Instruct open weights tested
A small 2.67B Celtic model that performs better on practical Welsh than on reasoning tests.
The model
| Developer | jonpojonpo |
|---|---|
| Base model | britllm/britllm-3b-v0.1 |
| Size | 2.67 billion parameters |
| Additional training | 22.4 million Celtic and Scots tokens, then 41,025 ChatML instruction conversations |
| Context | 2,048 tokens |
| Licence | CC BY-NC 4.0 |
Source: Machynlleth 3B Instruct’s Hugging Face card.
CymraegBench result
| Overall score | Welsh reasoning | Practical Welsh | Translation BLEU |
|---|---|---|---|
| 16.44 | 8.44 | 24.44 | 41.03 |
Up to 100 cases from each of the 19 core suites were tested, using the original BF16 checkpoint through vLLM 0.27.1 on an RTX 4090. Temperature zero, seed 1, the model’s ChatML template and its native 2,048-token context window were used. The 232 CCC cases do not contribute to the overall score.
The model completed 2,016 of the 2,053 cases; 37 outputs hit the token limit and were recorded as invalid with a zero score. There were no unresolved infrastructure errors. Its score of 16.44 places it fourth among open models up to 7B and 31st out of all 34 models.
An uneven profile. Machynlleth scored 90.00 on vocabulary, 72.00 on obscenities and 47.00 on PIQA, but zero on MGSM, XNLI, WNLI, CEFR, grammar and registers. BLEU 41.03 is strong relative to its overall score, while the reasoning score of 8.44 shows a clear limitation of the small model.
CCC Language Audit
| Subset | Score |
|---|---|
| Correction and explanation | 0.00 |
| Idiom | 0.00 |
| Terminology | 40.63 |
| Summarisation and quotation | 4.79 |
| References | 25.00 |
| Welsh prompt | 20.00 |
| English prompt | 20.00 |
| Average across the seven subsets | 15.77 |
On the CCC audit, the strongest result was terminology. The model failed to select the correct answer on correction and explanation, although it did not invent a source reference in any references case.