Testing, measuring and developing Welsh AI
News → Week 35

28 August 2026

High correction scores, but explanation is another matter

Three new models have achieved strong Welsh scores — but an audit of 3,712 answers shows correcting a sentence is much easier than explaining the correction.

Three new models completed AIsteddfod’s Welsh tests this week. The strongest, Qwen3.8 2.4T-A95B, reaches 81.09 — placing it among the top five. But a new Welsh audit of 3,712 answers suggests the less impressive number is more important: a model can correct a sentence without reliably explaining the correction.

Paper illustration of a model repairing a language vessel above complex layers of explanations
The final answer can look correct while its accompanying explanation is unreliable.

Results for three new models

On 22 August, Qwen3.8 2.4T-A95B achieved an overall score of 81.09 in CymraegBench: 86.28 on reasoning, 75.89 on practical Welsh and 60.84 BLEU on legislation translation. It completed all 2,053 cases without refusal, invalid output or infrastructure errors.

The model is very large. According to its official model card, it has 2.4 trillion parameters in total, with 95 billion active at a time, and a native 262,144-token context window. Its weights can be downloaded, but under a bespoke Qwen licence; ‘available weights’ is therefore more accurate than calling it open source.

On 27 August, Qwen3.8 Flash reached 79.81: 84.06 on reasoning, 75.56 on practical tests and 58.79 BLEU. It completed 2,052 of the 2,053 cases. The failed case used the full allowance of 131,072 reasoning tokens without a final answer. One case does not prove unreliability, but shows why evaluation should include answer length, cost and completion rate alongside accuracy.

On the same day, GLM-5.3-Flash scored 74.38: 76.21 on reasoning, 72.56 on practical tests and 58.41 BLEU. It completed every case. Its MIT-licensed weights make local testing and adaptation easier, although an open licence alone does not guarantee good Welsh.

The three results build on last week’s Qwen3.8 27B result. A95B is 10.62 points higher than 27B on the same harness, but the difference cannot be attributed to size alone: architecture, training and the serving system also change.

The gap between doing and understanding

The most useful development, however, is the beta audit of models’ ability to correct and explain Welsh. It turns the finding in the Coleg Cymraeg Cenedlaethol report— that models often correct better than they explain — into a public, repeatable test.

Sixteen models were tested on 232 cases: 3,712 answers, with every infrastructure error resolved. Fable led at 95.60, followed by Sol at 93.01, Opus 5 at 92.18 and Qwen3.8 Flash at 91.92.

But separating correction and explanation is more revealing. Among the top 14 models, correction scores were between 8.59 and 46.87 points higher than explanation scores. A95B scored 79.69 for correction but 56.25 for explanation. Qwen Flash scored 89.06 for correction but 58.59 for explanation.

This matters in classrooms, editing and language learning. If a model correctly changes a mutated form but invents a grammatical rule to justify it, the confident explanation may do more harm than an obvious error.

Two other patterns emerged. Prompt language affected each model differently: Qwen3.8 27B scored 16.67 points higher with Welsh instructions, Kimi 13.33 and Grok 8.33, while DeepSeek scored 11.67 points higher with English instructions. Avoiding a fake source ID also did not guarantee correct source content. Citation format and citation truth are different tests.

The results have a clear limitation. This is a beta audit, with deterministic scorers, awaiting independent review by at least two Welsh-language editors. The results are not yet a definitive measure of naturalness, dialect or educational value. The data and scorers are available for others to inspect.

Three lessons from other languages

This week’s developments in lower-resource languages offer three practical warnings for Welsh.

First, a new paper on Baniwa speech recognition, an Indigenous Arawakan language, shows what can be done with very little data. Whisper Small was fine-tuned on 1,373 transcribed recordings — just 32.4 minutes of audio. The best validation result was 37.5% WER and 7.45% CER; longer training worsened WER to 40%, a likely sign of overfitting.

This is a credible starting point, not a ready-made conversational system. The data consisted of isolated words and short phrases. Recordings are not public because of community ownership and access; requests can be made to the custodians. Community data is not always ‘missing data’. Sometimes restricting access is part of responsible stewardship.

Second, a study of model quantisation on Bangla tasks tested three model families in several formats. One model lost up to 57.35% of reasoning accuracy under GGUF-W8A16, while Qwen and Llama were more stable under GPTQ. Bit count alone does not determine accuracy loss; the result also depends on the model, compression method and task.

This is directly relevant to locally running Welsh technology. A full model’s score on a remote server does not guarantee the score of a quantised file on a laptop or school computer. The exact version used in practice must be tested.

Third, TabuLM adds row, column and cell information to a Kinyarwanda model for table question-answering. It reported an exact-match score of 62.0, 5.7 points above KinyaBERT, and fine-tuned models were much better than GPT-4o and GPT-4o mini at combining data.

Giving the model task structure is promising for Welsh government and service tables. But only 50 of the planned 106 development questions were evaluated, and the paper’s GitHub and Hugging Face pages did not open when checked. This is a method to test, not a ready system to adopt.

Paper illustration linking a small speech dataset, a structured table and a compressed model across a landscape
Limited data, task structure and the exact deployed version: three things a single overall score can hide.

What changed since last week

The project TranslatePsy-AfriSLM, a translation series for 19 African languages, appeared in last week’s article. Its headline results have not changed, so need not be repeated. Two concrete updates: the new paper version says it has been accepted at the EMNLP 2026 main conference, and the Hugging Face collection that previously failed to open now opens and contains models and a 206M synthetic mixture.

No other Welsh model, dataset, speech system or translation engine was found during this period. Nation.Cymru/Nation.Wales was searched specifically alongside other news, institutional and specialist sources. General AI workshops in Wales and Welsh-language stories without a language-technology result were excluded. Nor did new technical evidence or a formal development emerge in the case of Y Dderwen after the 7 August record.

More than one score is needed

A95B is a measurable step forward for Welsh in these tests. Flash shows that an architecture using fewer active parameters can come very close. GLM adds an MIT-licensed comparison. But the correction-and-explanation audit prevents the leaderboard becoming an advertisement.

The next test should ask more than ‘did the model get the right answer?’ Can it explain the answer in reliable Welsh? Does it finish within cost and token limits? Is the quantised version equally strong? What type of voice or table was it tested on? Who controls the data? Which licence permits use?

‘Supports Welsh’ is not the result. What matters is the Welsh task the model can complete, the version used, its data and licence, and how it fails. This week, the gap between correction and explanation is the weakness we should not hide.

Sources and further reading