Llandudno-4B open weights
Independent evaluation of Jonathan Poole’s Welsh model, fine-tuned from Qwen3.5-4B.
Our CymraegBench result
| Overall | Welsh reasoning | Practical Welsh | BLEU |
|---|---|---|---|
| 55.64 | 54.39 | 56.89 | 47.70 |
2,053 cases scored. Invalid outputs: 1 (0.05%). No unresolved infrastructure errors.
Protocol
CymraegBench v0.1, BF16 weights without low-bit quantisation, thinking off, temperature 0 and seed 1. One run: up to 100 cases per core suite, plus 232 CCC cases. Overall is the mean of reasoning and practical scores; BLEU and CCC are reported separately.
Initial output floor of 512 tokens; only failures were retried with a 2,048-token floor. Persistent invalid outputs score zero. The benchmark is not a native-speaker assessment or a safety evaluation.
CCC Language Audit
| Correction and explanation | 24.45 |
|---|---|
| Idioms | 68.75 |
| Terminology | 71.88 |
| Summarisation and quotation | 93.81 |
| References | 55.94 |
| Welsh instructions | 76.67 |
| English instructions | 80.00 |
Mean of the seven subsets: 67.36. Compare all CCC results
Comparison with the developer
The developer reports an overall score of 55.36. Our independent result is close, but runtime settings differ; this is not an identical replication.