Caernarfon 3B Cymraeg Instruct v0.1 open tested
Bangor AI’s experimental Welsh LoRA adapter, turning the Caernarfon base model into an instruction-following model.
Training
| Adapter developer | Bangor AI — an independent experiment, not an official BritLLM or UCL release |
|---|---|
| Base model | britllm/britllm-3b-v0.1: 3 billion parameters, pre-trained on Isambard-AI |
| Instruction data | BangorAI/alpaca-cy-1: 50,867 unique examples; 50,613 for training and 254 for validation |
| Method | One pass through the training data (epoch); LoRA r=32; learning rate 2×10⁻⁴; training loss 0.7981 and validation loss 0.7368 |
| Context | 2,048 tokens — the base model’s original limit |
CymraegBench result: all 33,657 cases
The model completed 32,473 cases; 1,184 (3.52%) were recorded as invalid, and no infrastructure errors remained at the end of testing. The overall score is 10.83, placing the model twenty-second on the 23-model leaderboard.
| Score | Result |
|---|---|
| Overall score | 10.83 |
| Welsh reasoning | 16.80 |
| Practical Welsh | 4.86 |
| Legislation translation — BLEU | 22.00 |
Fine-tuning has substantially improved instruction following: the model scored 49.80 on COPA and 47.84 on PIQA. However, many practical Welsh tests remain at zero, and the gap between the reasoning score of 16.80 and practical score of 4.86 shows this is a starting point, not a finished Welsh model.
Three CEFR prompts were too long for the model’s 2,048-token context window even after reducing the output limit to eight tokens. These, and any other output that hit the model’s limit, were recorded as invalid with a zero score. The scores by suite, CCC dimensions and metadata are openly available.
What happened to the old zero score? That was the test result for the BritLLM Caernarfon 3B base model before any instruction fine-tuning. A new test with a 512-token output allowance confirms the overall score of 0.00, but reduces the invalid rate from 94.59% to 56.84% and distinguishes failure to follow instructions from an excessively short output limit.
The published BritEval result of 55.2 for the base model uses a different protocol and cannot be compared directly with CymraegBench’s overall score. See the full comparison of open models.