Gemma 4 E4B Cymraeg v4 open weights
Independent evaluation of an EryriLabs Welsh model on RTX 4090.
The model
| Developer | EryriLabs |
|---|---|
| Base | google/gemma-4-E4B-it |
| Size | 6.9B total; around 4B effective |
| Training | 43.8 million Welsh CPT tokens, then 19,443 instruction examples with replay data. |
| Developer-stated licence | Apache-2.0 |
Merged BF16 weights, CPT/SFT adapters and GGUF versions are available. This test uses Q8_0, not BF16.
The developer notes repetition in long answers, factual errors and excessive quotation. This benchmark is not a safety assessment or native-speaker review.
Our CymraegBench result
| Overall | Welsh reasoning | Practical Welsh | BLEU |
|---|---|---|---|
| 49.70 | 51.39 | 48.00 | 40.23 |
2,053 cases scored; 0 invalid outputs (0.0%), 0 refusals. No unresolved infrastructure errors.
9 first-pass failures were retried; every case then produced a complete answer.
Protocol and reliability
CymraegBench v0.1: up to 100 cases per core suite, seed 1, temperature 0, a 512-token output floor, thinking off and the model’s chat template. There are 19 core suites and 232 CCC cases; neither CCC nor BLEU contributes to the overall score.
RTX 4090 on b4090; GGUF Q8_0 via llama.cpp b12019; four parallel requests and 16,384 context tokens per slot. A pre-retry snapshot is retained. Only failures are retried, with a 2,048-token floor and server-enforced repeat_penalty 1.05. Persistent output failures score zero; infrastructure errors block completion.
Weight revision: b415f5c752bb4bf91ede4bbaf9236d45cb948837
CCC Language Audit
| Summarisation and quotation | 96.07 |
|---|---|
| Welsh instructions | 93.33 |
| English instructions | 100.00 |
| References | 69.06 |
| Correction and explanation | 27.81 |
| Idioms | 56.25 |
| Terminology | 50.00 |
Mean of the seven subsets: 70.36
Developer-reported results — separate
| Overall | Welsh reasoning | Practical Welsh | BLEU |
|---|---|---|---|
| 49.64 | 51.39 | 47.89 | 39.80 |
These are the figures submitted in issue #3, not our run’s results. Harness version, context per slot and the effective retry setting can affect the result.