BritLLM Caernarfon 3B open weights tested
A multilingual 3B base model capable of continuing Welsh text, but not fine-tuned to follow instructions.
The model
| Developer | BritLLM, led by University College London’s NLP group |
|---|---|
| Checkpoint | britllm/britllm-3b-v0.1, also known as Caernarfon 3B |
| Type | Raw base model; not a chat or instruction model |
| Size and precision | 3 billion parameters, FP16 |
| Pre-training | Over 1.4 trillion tokens; English alongside Welsh, Irish, Scottish Gaelic and Scots data |
| Context | 2,048 tokens |
| Licence | Open Data Commons Attribution (ODC-By) 1.0 |
Sources: model card and official BritLLM website.
CymraegBench result
| Overall score | Welsh reasoning | Practical Welsh | Translation BLEU |
|---|---|---|---|
| 0.00 | 0.00 | 0.00 | 14.94 |
Up to 100 cases from each of the 19 core suites were tested, alongside the 232 CCC cases, using the original FP16 checkpoint through vLLM 0.27.1 on an RTX 4090. Temperature zero, seed 1 and the native 2,048-token context window were used.
The model has no native chat template. To use the same harness, each request was formatted as plain text with the labels Defnyddiwr: and Cynorthwyydd:. Every case received at least 512 output tokens before an incomplete response was recorded. This is a fair measure of instruction following before fine-tuning, not a multiple-choice likelihood test.
The model completed 886 of the 2,053 cases naturally; 1,167 reached the output limit and were recorded as invalid with a zero score. There were no unresolved infrastructure errors. None of the core answers earned points, although the translation corpus BLEU was 14.94.
What does zero mean here? The model often generates or repeats Welsh and English text, but does not return the requested short answer. BritLLM’s published Welsh BritEval score of 55.2 uses a different multiple-choice protocol and does not contradict this generation result. Machynlleth 3B Instruct, derived from this checkpoint, rises to 16.44 and reduces the invalid rate to 1.80% after Celtic pre-training and instruction fine-tuning.
CCC Language Audit
| Subset | Score |
|---|---|
| Correction and explanation | 0.00 |
| Idiom | 0.00 |
| Terminology | 0.00 |
| Summarisation and quotation | 0.00 |
| References | 7.19 |
| Welsh prompt | 0.00 |
| English prompt | 12.50 |
| Average across the seven subsets | 2.81 |
The small scores on references and English prompts mainly come from not inventing unacceptable sources, rather than completing the task. The model earned no points for an answer, correction, explanation, idiom, term, summary or source selection.