What does AIsteddfod contribute?
In our review of public material in summer 2026, we found no Welsh benchmark from the major labs — OpenAI, Anthropic, Meta, xAI, Moonshot, Zhipu or DeepSeek. Their multilingual tables typically cover around 10–15 languages, and Welsh is not among them. Valuable evaluation work is already taking place at UCL and the universities of Bangor and Cardiff. AIsteddfod aims to add to that work by maintaining an open leaderboard, publishing transparent evaluation methods, and regularly asking — what can a machine create in Welsh?
News
The latest developments in AI and Welsh: models, research, policy and new projects.
Evaluations
36 models evaluated — from leading models to small Welsh models that can run locally.
Language and translation
The quality of machine-generated Welsh: the Coleg Cymraeg report, translation tools and advice on giving better instructions.
Voice
Try Welsh text-to-speech and speech recognition, and see how we will measure voice quality.
Bangor AI research
Bangor AI research: a cynghanedd checker, experiments to improve models through feedback on CUDA GPUs, and fine-tuning Jupiter‑N on a cluster of DGX Spark computers.
Resources
Welsh corpora, test suites, models and papers — collected in one place, with direct links.
Bangor AI research
Improving Welsh AI through open research
Bangor AI research is testing an automated approach to improving models, using the results of each experiment to plan the next: a model generates Welsh, test suites and checkers assess the output, the evidence informs the next experiment or training stage, and the cycle begins again. This approach resembles the idea of ‘continuous exploration’ in Discovery Loop — automating the cycle of proposing, implementing, measuring and learning to test more ideas. We are investigating whether the quality of data, tests, checkers and feedback matters as much as the choice of model, particularly as open-weight models improve. Open infrastructure in Wales could help researchers and communities run, examine and improve Welsh AI on their own terms, and contribute what they learn to wider research.
The leaderboard at a glance
In the main CymraegBench v0.1 sample, the top twenty models have generated 40,132 responses across 19 Welsh test suites, and all infrastructure errors have been resolved.
| Rank | Model | Overall score | Welsh reasoning | Practical Welsh |
|---|---|---|---|---|
| 1 | GPT-6 Astra | 90.13 | 89.49 | 90.78 |
| 2 | GPT-6.1 Sol | 89.05 | 88.11 | 90.00 |
| 3 | Claude Opus 5.5 | 87.55 | 89.55 | 85.56 |
| 4 | Gemini 4 Argon | 87.41 | 90.15 | 84.67 |
Bold indicates the highest score in each column; all models tied for the lead are shown in bold.
GPT-6 Astra scored 90.13, 1.08 points ahead of GPT-6.1 Sol at 89.05. Opus 5.5 leads the CCC Language Audit with an average of 96.76; GPT-6.1 Sol scored 84.20 there. View the leaderboard, methods and open results →
Cynghanedd: form and meaning
Bangor AI’s specialist test (40 hidden topics, five attempts at each, with no second chance) reveals a different challenge:
| Model | Form: valid lines, 7 syllables | Meaning: valid, appropriate and original | What it shows |
|---|---|---|---|
| GPT-5.6 Sol (High/Flex) | 23.0% | 16.0% | Rescored; generated 42.5% of the requested lines |
| Claude Opus 4.8 (leading model) | 20.0%† | 13.5%† | Previous result; rescoring needed |
| Our fine-tuned 9B model (DPO r3) | 65.0%† | 1.0%† | Previous result; rescoring needed |
Sol returned 85 of the 200 requested lines; the remainder scored zero in both percentages. Its result has been rescored with the corrected checker. † The Opus and 9B rows were calculated using an earlier version of the checker, so the small differences between them should not be used to establish a definitive ranking until the raw outputs have been rescored.
Four examples of Sol’s output
«Distaw'r lli, dyst yw'r lleuad» — croes · «Marchnad y dre: lle i'r llu» — sain · «Ar draeth, heb don, mae'n llonydd» — llusg · «Trwm yw llaw y distawrwydd» — llusg.
All four are seven-syllable lines that passed the corrected checker. The first line received a relevance score of 1; the other three scored 2. In the full High/Flex test, 46 of the 200 lines passed the form test (23.0%), and 32 were also clean and appropriate to the theme (16.0%). View the themes and context →
Latest news
Behind the score: five new models, five different profiles
Five new CymraegBench runs show that a high overall score alone does not guarantee consistency, instruction following or neutrality.
Welsh punctuation on your device: an open model, a focused test
Two new open models restore Welsh punctuation locally; new research shows how to test tokenisation, language switching and on-device use.
From Kolibri to Wales: how much would a sovereign Welsh AI model cost?
A Welsh model the size of Kolibri would cost around £2.5m–£3.5m in computing; a practical sovereign programme, including data, people, testing and deployment, would be closer to £10m.
From principle to testing: building Welsh into AI
A UNESCO toolkit and a new test of models’ grammatical ability show why Welsh needs to be built in from the start, and every checkpoint tested.
Four new models, one lesson: the details matter more than the ranking
Opus 5.5 ranks second on CymraegBench, but four new evaluations show why the profile behind the score matters more than the ranking.