Evaluating AI in Welsh

How well do machines use Welsh? With no other public leaderboard, we record the results here.

This is the gap that prompted us to establish AIsteddfod: none of the major labs — OpenAI, Anthropic, Meta, xAI, Moonshot, Zhipu or DeepSeek — publishes benchmark results for Welsh. Their multilingual tables usually cover only around 10–15 languages, and Welsh is not among them. Almost all quantitative resources for evaluating Welsh come from academic work in Britain: BritLLM/UK-LLM at UCL, Techiaith at Bangor University, and Cardiff NLP. A full catalogue is available on the test suites page, with details of the protocols and quality assurance methods in our methodology.

The size of the gap is clear in training data. Only around 0.065% of PaLM’s data was Welsh, and around 0.00177% of GPT-3’s training data, according to figures from Prys and Jones via Techiaith. Very little Welsh material therefore underpins the abilities measured below.

CymraegBench v0.1: the twenty leading models

The main sample now contains 40,132 responses from the top twenty models across 19 Welsh test suites, with every infrastructure error resolved. Up to 100 cases were used per suite, selected deterministically to return the same cases each time, with random seed 1 where supported by the provider, and the same harness for every model. Aggregate results, scores for each suite and the details needed to repeat the tests are openly available in BangorAI/cymraeg-bench.

Bold indicates the highest score in each column; all models tied for the lead are shown in bold. How the overall score is calculated: The average of the nine Welsh reasoning tests receives 50% weight, and the average of the nine practical Welsh tests 50%. Each test carries equal weight within its group. The legislation translation test’s BLEU is a separate corpus measure and is not part of the overall score. Every model was scored on 100% of cases; WNLI contains 71 cases and ARC-Easy mini 50. Refusal or invalid output scored zero. GPT-6 Astra, GPT-6.1 Sol, Opus 5.5, Gemini 4 Argon, Muse Wonder, Opus 5, Grok 4.7, Fable 5.1, Fable 5, Muse Spark 1.2, Muse Glimmer 30B, GLM-5.3 Flash and GLM-5.3 were run with effort=low. The setting effort=none was used for Mistral Large 4 because its route does not offer low; the parameter temperature was omitted for GPT-6, GPT-6.1 and Gemini because their routes do not accept it. On the two binary suites, a clear label of Y/N at the beginning of an answer is accepted even when followed by an explanation. Only the final answer was scored for reasoning models.

The rest of the leaderboard

The extended leaderboard now contains 36 models and 135,492 responses. Jupiter-N-120B and Caernarfon 3B Cymraeg Instruct v0.1 were tested on the full public suite — 33,657 cases each — while the other 34 models were tested on samples of up to 100 cases per suite. The same 19 core suites and formula are used in every test; random seed 1 was used where the provider supported it. Open-weight models with up to 7 billion parameters appear in a separate table below. The 232 cases of the CCC Language Audit are reported separately.

Small open models: up to 7B

Open-weight models with up to 7 billion parameters have been separated from the bottom of the main table so they can be compared as a practical class for local use. The overall rank is retained to show their position among all 36 models; ‘Small-model rank’ orders them within this group only.

Bold in the small-model table indicates the best result in each score column; for ‘Invalid’, the lowest percentage is best. Empty output, output reaching the token limit, or a prompt longer than the model’s context window is invalid and scores zero. The two full-suite tests provide more stable estimates than the earlier samples, so detailed comparisons with samples should be treated as indicative. The new BritLLM base-model test uses a fairer output limit than the archived run, but confirms that the raw checkpoint does not follow instructions before fine-tuning. Read the analysis and download the data →

CCC Language Audit: beta results

We have also tested 30 models on 232 cases turning the Coleg Cymraeg report’s findings about AI, Welsh and higher education into repeatable tests. Claude Opus 5.5 had the highest average across the seven subsets, at 96.76; Gemini 4 Argon follows at 96.45, then Claude Fable 5 at 95.60. GPT-6.1 Sol averaged 84.20. The capability profile, rather than a single average, remains the audit’s main result. View the results, dimensions and downloadable data →

Older Welsh BritEval reference measures (average, from llm.org.uk): Caernarfon 3B 55.2; Mistral 7B 63.0; Llama 3 8B 62.7. These cannot be compared directly with CymraegBench’s overall score, which combines 18 suites. For Jupiter-N, the published improvement was +18 on Welsh ARC-Easy and +5.25 on Welsh MMLU-Lite, rather than a BritEval average.

The cynghanedd leaderboard

Bangor AI’s protocol contributes a published measure of models’ ability to compose strict-metre Welsh poetry. It uses 40 themes unseen during training, five attempts per theme, and a single chance to answer — without feedback loops or rhyming hints. Each line is judged by the cynghanedd rules engine, and the project repository is BangorAI/cynghanedd.

* Sol returned 85 of the 200 requested lines, or 42.5%; all 85 were unique. The 115 missing lines received zero in both percentage columns. Claude Opus 4.8 judged the relevance of clean, valid lines independently of the model that generated them. † The Opus and 9B rows were calculated using an earlier checker version and await rescoring from the raw output.

Models we are tracking

Leading models

Open models

Models trained on Welsh