11 September 2026
BuzzASR: a new Welsh speech model to test
A specialist Welsh speech recognition model is available to download, with a large reduction in errors on two public test sets—but testing on real-world language is still to come.
One model for each language
The BuzzASR paper was submitted on 9 September and has been accepted to Findings of EMNLP 2026. Instead of relying on one multilingual model to do everything, the project creates a specialist model for each of the 102 FLEURS languages.
The Welsh model was fine-tuned from Whisper-large-v3 and is available under the MIT licence. The card says it used FLEURS and Common Voice 25, alongside Goldfish text data. The weights can be downloaded and the model run locally.
On the FLEURS test, the card records a character error rate of 7.31% and a word error rate of 17.45%. On Common Voice, the figures were 2.25% and 5.98%. On the combined test, the character error rate was 5.24%—around 2.5 times lower than the unmodified Whisper-large-v3's 12.88%.
Across the project, the team says the best BuzzASR model beat Whisper on 77 of the 102 languages in the combined test. The median character error rate in the paper falls from 15.5% to 7.6%. That is encouraging evidence for specialising a large model for one language.
A strong result, but an unresolved question
There is a small but important difference between the sources. The Welsh model card describes the model as fully fine-tuned with a native tokeniser, reporting 5.24% character errors and 12.57% word errors on the combined test.
In the first version of the paper's table, however, the simple fine-tuning recipe appears to give the best Welsh result: 5.27% character errors and 15.15% word errors. The figures for the full recipe are higher there.
This could mean the checkpoint was retrained or finalised after the paper's table was frozen. No public explanation reconciling the two was available by the end of the search. When testing the model, therefore, record the exact weight version, decoding settings and card date—not just the model name.
More importantly, the published test includes only FLEURS and Common Voice. It does not yet tell us how well the model handles natural conversation, bilingual meetings, dialects, place names, numbers or noisy audio. The card itself warns that performance in other domains may vary.
Two new text models on the scoreboard
There was also a major move in AIsteddfod's text benchmark. On 9 September, GPT-6 Astra completed all 2,053 CymraegBench cases and scored 90.13 overall. That is 4.97 points ahead of Claude Opus 5, with 89.49 for Welsh reasoning, 90.78 for practical Welsh and 74.10 BLEU for legislative translation.
The average is not the whole story. The model scored only 39% on CEFR classification. In the Coleg Cymraeg Language Audit, it corrected 96.88% of targets but scored only 55.47% for explaining the corrections. That is a good example of why we should look beneath one headline score.
On 10 September, DeepSeek released V4.1 Flash, an open-weight model. In our local test, it scored 69.45 overall—7.82 points above V4-Flash-0731. Its reasoning score rose by 13.09 points, but practical Welsh improved by only 2.56 points.
The detail shows a mixed pattern. DeepSeek V4.1 achieved 69.03 BLEU on legislative translation and 93.75% on idioms, but 30% on vocabulary, 33% on CEFR and 42% on bilingual place names. This test shows a better model than its predecessor, but the improvement is not spread evenly across Welsh.
The three results are not direct competitors. BuzzASR is a speech model; GPT-6 Astra and DeepSeek V4.1 Flash are general text models. Together, they show the value of measuring the specific capability needed, rather than relying on a ‘multilingual’ label.
An important lesson from code-switching
A particularly useful warning came from a new study of English–Yoruba speech. The researchers tested eleven systems on 2,000 utterances. The best system by overall word error rate appeared comparable to a leading audio model.
But measuring the words around language switches revealed a different picture. Error on Yoruba tokens was around 0.97 for almost every faithful system, with failures concentrated at switches into Yoruba. Some audio models performed better at the boundary, but could translate instead of transcribing, add words or drop parts of the instruction.
This is highly relevant to Welsh. A meeting transcript can look good on average because many English words are correct, while the system fails precisely when someone switches to Welsh. A Welsh test should measure switches into Welsh separately.
Other developments this week reinforce the same lesson. CantoneseLLM v2 uses language and script rules during reinforcement learning to stop answers drifting into standard Mandarin. An audit of Urdu stories finds grammatical errors, repetition and a lack of cultural depth that giving the model a few examples did not resolve. And an audit of a Central Kurdish speech resource shows how metadata, licensing and tests overlapping with training can undermine a resource that otherwise looks open.
The test Wales needs
The sensible next step is to compare BuzzASR with Whisper and existing Welsh services on an independent test set. It should include:
- clear speech and natural conversation, meetings, broadcasting and noisy phone recordings;
- northern and southern varieties, different ages and speaking speeds;
- switches between Welsh and English, with switches into Welsh scored separately;
- personal and place names, mutations, numbers, acronyms and borrowed terms;
- a measure of the transcript's meaning, not just character and word errors;
- fixed model versions, published normalisation rules and a test set unseen during training.
There is no need to wait for an enormous corpus to begin. What is needed is a balanced sample, human-checked transcripts and a protocol others can repeat.
Wrapping up the week
BuzzASR is a concrete step: a specialist Welsh model, downloadable weights, a clear licence and results by test set. GPT-6 Astra sets a new ceiling for text models on CymraegBench, and DeepSeek V4.1 Flash shows genuine progress in an open-weight model.
But the best test reflects actual use. For Welsh speech, that means dialect, noise, names and code-switching. The new model gives us a strong candidate. The next contribution for Wales is to build the independent test that tells us where we can trust it.