25 September 2026
Four new models, one lesson: the details matter more than the ranking
Claude Opus 5.5 is second on CymraegBench, but the differences between reasoning, practical Welsh, correction and explanation say more than a single score.
Second, but not second at everything
Anthropic released Claude Opus 5.5 on 22 September. There was no Welsh result in the company's tables, so the model was run on the same CymraegBench protocol as the other models on the site.
According to the full evaluation page, Opus 5.5 scored:
- 87.55 overall;
- 89.55 for Welsh reasoning;
- 85.56 for practical Welsh;
- and 72.52 BLEU on legislative translation.
It completed all 2,053 cases without refusals or invalid output. It is 2.58 points behind GPT-6 Astra overall. Yet its reasoning score is 0.06 points higher than Astra's; Astra's advantage in practical Welsh keeps it on top.
That is a warning against turning the leaderboard into a one-number race. Compared with Opus 5, Opus 5.5 has gained 2.39 points overall and five points in practical Welsh, but its reasoning score is 0.22 points lower. The new version is better across the full protocol, but not on every dimension.
The score also needed correcting before publication. An early estimate was 87.61, but an audit of the binary scorer found one answer ending with the wrong label. The final result is 87.55. The difference is small; the principle is important. It must be possible to reopen the data and check the measurement itself.
Correcting can be easier than explaining
Alongside CymraegBench, the models also face the CCC Language Audit: 232 cases turning research findings about AI and Welsh in higher education into repeatable tests.
Opus 5.5 currently has the highest mean, at 96.76. It scored 100 for correction, but 64.84 for explaining the correction. The same pattern appears in other models. Claude Fable 5.1 scored 96.88 for correction and 63.28 for explanation; MiMo V2.6 Pro scored 75.00 and 42.19; Muse Glimmer 30B scored 65.63 and 35.94.
A model can therefore reach the correct sentence without giving a reliable reason for it. For someone polishing text, the answer may be useful. For a learner or tutor trying to understand the rule, the gap is much more serious. Editing and teaching are not the same capability.
Three other distinct profiles
Claude Fable 5.1 is ninth with 80.53. Its reasoning score of 86.05 is strong; its practical Welsh score is 75.00. It completed 2,052 of 2,053 cases, with one PIQA answer still empty after a retry.
MiMo V2.6 Pro is eighth with 80.73: only 0.20 points ahead of Fable, but with a very different profile. Its scores are 83.79 for reasoning, 77.67 for practical Welsh and 61.98 BLEU. Xiaomi published the MiMo V2.6 family's weights this week, and the official model card specifies an MIT licence. That creates an opportunity for researchers to run and adapt it on their own infrastructure.
But there is an important reliability note. Fifty-five answers hit the original output cap; 38 were recovered with a larger cap, but 17 remained invalid. Eleven were correction-and-explanation cases, four involved place names, and the other two were reasoning tests. The weights are open; that alone does not guarantee reliable output.
Muse Glimmer 30B, a model with Apache-2.0 weights, is fifteenth with 70.64. It completed every case, but its practical Welsh score of 63.78 and explanation score of 35.94 show a clear gap behind the leading models.
‘Open’ and ‘good at Welsh’ are not synonyms. Open access enables adaptation, inspection and retesting. Output quality is another question, and needs to be measured.
What does real-world speech reveal?
The strongest lesson from the international comparison came from Vimarsha, a new preprint testing ten speech recognition systems across 22 Indian languages. The 100-hour set includes field and ‘in-the-wild’ recordings from 5,099 speakers in 356 districts, alongside human-verified valid spelling variants.
The best systems for Indian languages move from around 10–15% word errors on controlled speech to 27–34% in the wild. GPT-4o Transcribe averaged 89.0% in the wild in this test. Differences by demographics, speaking style and speed were also observed.
This is not a Welsh result, and does not prove that the same decline would occur here. But it demonstrates that how a test is collected can completely change the conclusion. A Welsh speech test should include local accents, ages, fluent speakers and learners, switching between Welsh and English, spontaneous speech and everyday audio conditions.
When language structure is part of the model
Morpho-VITS, a preprint on Kinyarwanda text-to-speech, adds morpheme encoding and attention between phonemes and morphemes. Word error rate fell from 13.4% to 9.4%, and character error rate from 3.0% to 1.7%. In 2,100 comparisons by 21 native listeners, naturalness rose from 3.33 to 3.55.
The limitation is equally interesting: the error rate on unseen morphemes was 19.3%, compared with 8.3% on seen ones. For Welsh, that suggests a concrete experiment on mutations, endings and rare forms. Showing that a model can reproduce patterns already in the data is not enough.
BanglaKontho provides a second lesson. The Bangla text-to-speech set contains 20 hours of audiobook audio, 7,050 segments, verified transcripts and a text normaliser for numbers, currency, dates, punctuation and Unicode. A model trained on it achieved a word error rate of 9.5% and naturalness score of 4.46, compared with 16.0% and 3.16 using the same architecture on another, shorter corpus.
The lesson is that more than ‘more audio’ is needed. Length, domain, speaker consistency and text normalisation all affect the result. A long-form Welsh corpus—with a public normalisation pipeline—could fill a different gap from a large collection of short clips.
Both papers are preprints. The results have not been independently replicated, and no public checkpoint was found for Morpho-VITS. They should be treated as promising evidence for designing an experiment, not a finished solution for Welsh.
Measuring the Welsh people use
No new Welsh model or dataset was found on the wider web during the reporting window. Nation.Cymru was searched specifically. Its most prominent AI item on 25 September concerned businesses in Wales using AI, not the technology's Welsh-language ability. That is a valid economic story, but not evidence about the language, and it has not been used to pad out a quiet week.
The concrete development is the four new evaluations. They show that the best models can do a great deal in Welsh; they also show where an overall score hides a problem. Reasoning can be stronger than practical Welsh. Correction can be stronger than explanation. An open model can be easy to inspect but less accurate. Output can look complete while a small number of cases have failed entirely.
The next step is to broaden what we mean by a ‘Welsh test’: not just more clean questions, but real-world speech, dialect, code-switching, local names, morphology, long text and mistakes that change meaning. Opus 5.5's ranking is news. Building a test that properly challenges it will be more important still.