Gemma 4 12B Cymraeg v6 open weights
Independent evaluation of an EryriLabs Welsh model on RTX 4090.
The model
| Developer | EryriLabs |
|---|---|
| Base | google/gemma-4-12b-it |
| Size | 12 billion parameters |
| Training | Around 307 million Welsh CPT tokens in two sequential blocks, followed by conversational fine-tuning with replay data. |
| Developer-stated licence | Apache-2.0 |
Merged GGUF and an SFT adapter are available. The current card says merged BF16 weights and both CPT adapters are not public, so the complete merge cannot be reproduced from public sources.
The developer warns that long greedy-decoded answers can repeat or drift into English. This temperature-zero protocol differs from the recommended conversational settings.
Our CymraegBench result
| Overall | Welsh reasoning | Practical Welsh | BLEU |
|---|---|---|---|
| 52.70 | 54.52 | 50.89 | 52.30 |
2,053 cases scored; 0 invalid outputs (0.0%), 0 refusals. No unresolved infrastructure errors.
6 first-pass failures were retried; every case then produced a complete answer.
Protocol and reliability
CymraegBench v0.1: up to 100 cases per core suite, seed 1, temperature 0, a 512-token output floor, thinking off and the model’s chat template. There are 19 core suites and 232 CCC cases; neither CCC nor BLEU contributes to the overall score.
RTX 4090 on b4090; GGUF Q8_0 via llama.cpp b12019; four parallel requests and 16,384 context tokens per slot. A pre-retry snapshot is retained. Only failures are retried, with a 2,048-token floor and server-enforced repeat_penalty 1.05. Persistent output failures score zero; infrastructure errors block completion.
Weight revision: 748c37bb3b9d088befb17b7196ff9f2010e3df5d
CCC Language Audit
| Summarisation and quotation | 81.08 |
|---|---|
| Welsh instructions | 100.00 |
| English instructions | 86.67 |
| References | 48.44 |
| Correction and explanation | 34.22 |
| Idioms | 71.88 |
| Terminology | 65.62 |
Mean of the seven subsets: 69.70
Developer-reported results — separate
| Overall | Welsh reasoning | Practical Welsh | BLEU |
|---|---|---|---|
| 52.70 | 54.74 | 50.67 | 52.45 |
These are the figures submitted in issue #3, not our run’s results. Harness version, context per slot and the effective retry setting can affect the result.