Llandudno-9B open weights
Independent evaluation of Jonathan Poole’s Welsh model, fine-tuned from Qwen3.5-9B.
Our CymraegBench result
| Overall | Welsh reasoning | Practical Welsh | BLEU |
|---|---|---|---|
| 58.31 | 63.51 | 53.11 | 49.67 |
2,053 cases scored. Invalid outputs: 0 (0.00%). No unresolved infrastructure errors.
Protocol
CymraegBench v0.1, BF16 weights without low-bit quantisation, thinking off, temperature 0 and seed 1. One run: up to 100 cases per core suite, plus 232 CCC cases. Overall is the mean of reasoning and practical scores; BLEU and CCC are reported separately.
Initial output floor of 512 tokens; only failures were retried with a 2,048-token floor. Persistent invalid outputs score zero. The benchmark is not a native-speaker assessment or a safety evaluation.
CCC Language Audit
| Correction and explanation | 30.78 |
|---|---|
| Idioms | 84.38 |
| Terminology | 65.62 |
| Summarisation and quotation | 94.94 |
| References | 70.94 |
| Welsh instructions | 100.00 |
| English instructions | 93.33 |
Mean of the seven subsets: 77.14. Compare all CCC results
Comparison with the developer
The developer reports an overall score of 58.07. Our independent result is close, but runtime settings differ; this is not an identical replication.