Grasmere E4B (Q8_0) open weights
Independent comparison of jonpojonpo’s Grasmere E4B and Grasmere E4B FTPO, both at Q8_0 and based on Gemma 4 E4B-it. FTPO is additional training to reduce repetition, not just a different quantisation.
Our CymraegBench result
| Model | Overall | Welsh reasoning | Practical Welsh | BLEU | Invalid |
|---|---|---|---|---|---|
| Grasmere E4B | 50.47 | 53.61 | 47.33 | 46.57 | 0.10% |
| Grasmere E4B FTPO | 50.54 | 53.30 | 47.78 | 41.99 | 0.00% |
For each variant, the run with the highest overall score was selected from three independent runs with the same seed (1). All metrics for each variant come from its selected run.
Each selected run scored 2,053 cases. The original had two invalid outputs (0.10%), scored zero. FTPO had one initial failure that recovered on retry: no remaining invalid outputs. Neither had refusals or unresolved infrastructure errors.
The overall scores are very similar. FTPO improves practical Welsh and eliminates invalid outputs in the selected run, but translation BLEU falls from 46.57 to 41.99. These tests do not establish stability in long conversations.
Protocol and reliability
CymraegBench v0.1: up to 100 cases per core suite, seed 1, temperature 0, a 512-token output floor, thinking off and the model’s chat template. There are 19 core suites and 232 CCC cases; neither CCC nor BLEU contributes to the overall score.
A pre-retry snapshot is retained. Only failures are retried, with a 2,048-token floor and a repetition penalty of 1.05. Persistent output failures score zero; infrastructure errors block completion.
The developer warns about repetition in long answers and recommends the FTPO variant for everyday use. A benchmark score does not replace evaluation by Welsh speakers.
CCC Language Audit
| Subset | Grasmere E4B | Grasmere E4B FTPO |
|---|---|---|
| Summarisation and quotation | 89.76 | 88.30 |
| Welsh instructions | 100.00 | 96.67 |
| English instructions | 91.67 | 91.67 |
| References | 73.75 | 67.19 |
| Correction and explanation | 16.72 | 19.06 |
| Idioms | 78.12 | 78.12 |
| Terminology | 37.50 | 37.50 |
| Mean | 69.65 | 68.36 |