Welsh test suites
The tests available to measure machines’ Welsh — and the areas where data is still missing.
Welsh benchmarks are scarce. While hundreds of tests are available for English, the main Welsh resources can be summarised on one page. Three academic sources sustain the field: UCL (BritLLM/UK-LLM), Bangor University (Techiaith), and Cardiff (Cardiff NLP). Bangor AI’s cynghanedd testing protocol adds to these.
BritEval (BritLLM / UK-LLM, UCL)
Standard benchmarks adapted through machine translation into Welsh, Irish, Scots and Scottish Gaelic. These are the Welsh suites:
| Suite | What it measures | Size | Link |
|---|---|---|---|
| arc_welsh | Multiple-choice science questions (ARC-Easy + ARC-Challenge) | 7,722 rows | HF |
| piqa_welsh | Physical common sense | — | HF |
| truthfulqa_welsh | Truthfulness (mc1/mc2 multiple-choice format) | 750 rows | HF |
| xnli_brit | Textual inference in the languages of Britain | 9,690 rows | HF |
Published scores comparing models in Welsh include the BritEval averages at llm.org.uk: Caernarfon 3B 55.2, Mistral 7B 63.0, Llama 3 8B 62.7.
llm-evals-cy (Techiaith, Bangor University)
An evaluation harness with ten tests, built on DeepEval and LiteLLM, supporting OpenAI, Anthropic and Ollama. It is available on GitHub. These are the ten tests:
welsh-lexicon · welsh-grammar · welsh-yes-no · welsh-obscenities · welsh-bilingual-placenames · welsh-legislation-translation (BLEU) · welsh-registers · welsh-mmlu-lite · welsh-toxigen · welsh-arc-easy-mini-cy
Two of the tests are particularly specific to Welsh: welsh-yes-no — since there is no single simple pair of words for ‘yes’ and ‘no’, a choice must be made between forms such as ydy, oes, do and gwnaf — and welsh-bilingual-placenames. The repository currently has no public leaderboard, and we intend to fill that gap.
Other Techiaith datasets
| Suite | What it measures | Size | Licence | Link |
|---|---|---|---|---|
| mgsm_cy | School mathematics word problems, with worked solutions | 258 rows (8 training / 250 test) | CC-BY-SA-4.0 | HF |
| COPA-cy | Cause-and-effect reasoning (choosing the likely explanation) | 1,500 rows (1,000 / 500) | CC-BY-4.0 | HF |
| wnli-cy | Welsh textual inference | see the card | see the card | HF |
The Techiaith and Dewi Bryn Jones catalogue also lists ToxiGen, Macsen and speech evaluation datasets.
welsh-cefr (Cardiff NLP)
2,658 rows of Welsh learning materials (dialogues, exercises and reading passages from learnwelsh.cymru) labelled by CEFR proficiency level, alongside two ready-made classifiers: WelshCEFR-BritLLM (3B) and WelshCEFR-EuroBert (0.2B). Funded through the Welsh Government’s ‘CEFR Analyser’ grant. HF link.
The cynghanedd protocol (Bangor AI)
One of Bangor AI’s contributions is a published measure of model-generated strict-metre poetry. The checker’s project repository is BangorAI/cynghanedd, and the protocol has two parts:
- The hidden-set protocol: 40 themes unseen during training, five attempts per theme, and a single chance to respond. Each line is judged by the cynghanedd rules engine — croes, traws, sain, llusg and the faults the checker is programmed to detect — and then by a semantic model measuring its appropriateness to the theme. The measures are valid-seven-syllable raw, appropriate+valid+original, and the number of unique lines. We check originality by comparing lines with a snapshot of the training data taken when the model was trained. Every model needs rescoring when checker rules change, and machine judgement does not replace human literary review.
- The Barddas benchmark: 188 labelled strict-metre poems — englynion, cywyddau, gwawdodynnau, hir-a-thoddeidiau and more, totalling 1,092 lines — from nine issues of Barddas magazine, with a strict training/benchmark split so no model is tested on material it has seen.
The gaps
Several important gaps remain in the available tests:
- No study of tokenisation efficiency for Welsh — how many more tokens does Welsh text require than its English translation with the major labs’ tokenisers? Similar studies exist for Greek, Polish and Thai, but not Welsh.
- No WJEC or GCSE examination evaluation — no public study has a model sit a real Welsh paper.
- Few public leaderboards — AIsteddfod publishes one contribution that can be repeated and compared.
See the leaderboard for model results, and the section on Bangor AI research for methods.