Methodology
Fair measurement requires clear results and an explanation of the method so others can repeat the tests. This is how we do it.
Two levels of evaluation
Level 1 — standard community benchmarks
We use the existing academic test suites exactly as published: BritEval (arc_welsh, piqa_welsh, truthfulqa_welsh and xnli_brit — UCL); mgsm_cy and COPA-cy (Techiaith, Bangor); and Techiaith’s package llm-evals-cy , which includes ten tests covering vocabulary, grammar, place-name mutations, registers, legislation translation and more. Details of each suite are in the catalogue.
On choice tasks, the answer label is scored rather than requiring the entire response to match the label exactly. On the binary Grammar and ToxiGen tests, Y or N must appear clearly at the start of the answer; an explanation may follow, but the scorer does not search prose for a letter that happens to match the target. Raw output is retained so every decision can be audited.
Level 2 — Bangor AI’s cynghanedd testing protocol
This is Bangor AI’s original contribution, developed over a year of observing how models find shortcuts in tests. The checker’s repository is BangorAI/cynghanedd:
- The task: compose a seven-syllable line containing cynghanedd and responding to a specified topic.
- The protocol: 40 hidden topics dissimilar to the training material; five attempts per topic; a single turn, without feedback or rhyming hints. A fixed seed is used.
- The judge: a rules-based engine — not another model — checks each line for cynghanedd groes, draws, sain or lusg, and names the errors (false cynghanedd, heavy and light consonants, excessive rhyme, and so on). How it works →
- The criteria, each stricter than the previous one:
- valid, 7 syllables: a valid line containing exactly seven syllables;
- original: a line not appearing verbatim in the known training data (compared with a snapshot from when the model was trained, since training data changes over time);
- clean and original: an original line in which every word is valid Welsh, rather than an invented word the phonological system can pronounce but which does not appear in a dictionary;
- appropriate: a line an independent judge — not the checker — considers genuinely related to the topic. A valid line unrelated to the theme is the most common way to score without answering the task.
Testing voice technology
Recognising spoken Welsh and generating Welsh speech require different protocols from text. For speech recognition, we will record word error rate and test place names, accents and code-switching. For text-to-speech, listeners will assess clarity, naturalness and accuracy. The full plan and tools are on the Voice.
How we prevent models from avoiding the task
Each of these patterns emerged during Bangor AI’s experiments:
| How the task is avoided | Real example | How we prevent it |
|---|---|---|
| Rhyming a word with its mutated form | teulu ~ deulu (six times in one ‘valid’ poem) | Compare mutated forms with the base word and name the error |
| Inventing pronounceable pseudo-words | gwinoth, rhyhwyt, hudod, llonells | Check the vocabulary: every word must be in a dictionary |
| Repeating memorised lines | 89.5% ‘valid’ — and three unique lines | Check whether lines are original by comparing them with a snapshot of the training data |
| Valid lines unrelated to the theme | The same ‘hedd yr heddiw’ for every topic | Measure appropriateness assessed by an independent judge |
Reporting principles
- Source every number. If something has not been measured, the record says ‘not yet measured’. In the documents we have reviewed, many major labs have published no Welsh measurement.
- Publish the failures. A failed experiment can tell us more than a successful one; both are recorded.
- Reproducibility. Protocols are published in full so anyone can repeat the tests — and correct us.