21 August 2026
Qwen3.8 27B: what lies behind the score?
Qwen’s new open-weight model scores 70.47 on CymraegBench — but the gap between Welsh reasoning and practical Welsh shows why the full profile matters.
The week’s main story changed when a new Welsh result arrived on 21 August. Qwen3.8 27B is a dense multimodal model with 27 billion parameters, Apache 2.0 weights and a native 262,144-token context window. It understands text, images and video, with controllable reasoning depth.
Qwen3.8 27B: 70.47 in Welsh
In AIsteddfod’s CymraegBench v0.1 test, Qwen3.8 27B achieved an overall score of 70.47. That puts it seventh overall, between Kimi K3 and GLM-5.2, and second among open-weight models after Muse Spark 1.2.
It completed all 1,821 cases without refusing a request, producing invalid output or suffering an infrastructure error. The model was tested through OpenRouter with temperature 0, random seed 1 and reasoning_effort=low; reasoning was kept separate and the final answer scored.
But 70.47 is not the whole story. The model scored 79.83 on Welsh reasoning, including 96 on Welsh ARC-Easy, 93 on COPA, 91 on Welsh MGSM, 84.51 on Welsh WNLI and 83 on Welsh XNLI. Its practical Welsh score was 61.11 ; on legislation translation, 46.03 BLEU.
The practical tests show where work is needed: 12 on bilingual place names, 27 on CEFR levels, 28 on obscenity classification, 47 on vocabulary and 50 on grammar. Some classification tests are skewed, so no subscore should be treated as a final verdict. But the pattern is clear: the model is strong on structured reasoning and much less certain on specifically Welsh knowledge and usage.
Compared with Qwen3.5 9B on the same harness, the overall score is 15.37 points higher, reasoning 21.85 higher, practical Welsh 8.89 higher and BLEU 8.97 higher. This is not a controlled size experiment — the model, generation software and serving system differ — but it is substantial, measurable progress.
No other Welsh model, dataset, speech system or translation engine was found during this period. Nor did new technical evidence or a formal outcome emerge in the case of Y Dderwen after our record on 7 August.
A current Newyddion S4C story about AI in Welsh sport discusses athlete data, the Pelly app and the Egni.AI carbon tool at M-SParc. It is legitimate AI news in Wales, but introduces no Welsh language technology, bilingual result or Welsh dataset. A story does not become a Welsh-language AI story merely by being published in Welsh.
Qwen3.8’s uneven profile makes this week’s comparative work more relevant, not less. Three projects in other languages reveal how incorrect measurement can make progress disappear — or make failure look better than it is.
The model changed; the score stayed
The preprint Thinking in a Low-Resource Language, submitted on 18 August, post-trains four mixture-of-experts models to reason in Greek. The authors used 118,092 training rows and tested not just accuracy but reasoning-trace language, token count, answer completion and format compliance.
Headline accuracy barely changed. In one comparison the trained model scored 76.5 against the base model’s 77.2. Reading only those two numbers would suggest little happened.
But before training, the models reasoned in Greek in none of 1,000 inspected traces. After supervised training, around 98% of traces used the question’s language. Something fundamental changed for users: a process hidden in another language became something a Greek speaker could read and challenge.
A second result was more troubling. Repeating the same training with only the random seed changed shifted the score by 7.7 points. That exceeded every difference attributed to data choice or training recipe. Some ‘improvements’ were smaller than experimental noise.
The lesson for Welsh benchmarks is repeated testing. Records must also capture what matters to users: the language in which the model plans, the path’s length, format compliance and answer cost. A model can retain the same accuracy yet become much more transparent — or much less.
When a space looks like a wrong word
The second paper moves from text to audio. A Mizo team collected 17.62 hours of speech from 200 speakers, using news sentences and court judgements translated and checked by native speakers. They fine-tuned Whisper small, medium and large-v3, alongside the multilingual SraVaani 1.0 model.
Whisper-large-v3 achieved a conventional word error rate of 18.08%. But Mizo writers sometimes join or separate morphological elements without changing the letter sequence or the meaning for readers. The team created an additional measure accepting a ni and ani as the same sequence when characters match exactly. Under this measure, the error rate was 7.22%.
This does not mean ignoring standard orthography or always choosing the lower number. The paper’s strength is publishing both numbers and then auditing outputs with a native Mizo speaker.
Six real error types emerged: wrong script, personal and place names, glottal stops, numbers, English mixed with Mizo, and a language-specific sound–spelling distinction. Even the best Whisper model produced four sentences with Hebrew-script fragments. A generous word-boundary measure cannot hide that.
The distinction is directly relevant to Welsh. A speech test should report unadjusted word error rate, character error rate and a result under a published Welsh normalisation policy. Speakers should then examine mutations, apostrophes, dialect spellings, place names, numbers and English code-switching. Model error and reasonable writing choices must be distinguished without hiding either.
The paper says most of the corpus is available on AI-Kosh under the ANRF Open Licence, with the remaining data forthcoming, while the models are available to researchers on request. This is the authors’ preprint, with only 192 test sentences from five speakers. The result is promising, not the final word on Mizo.
Less data, more to learn
TranslatePsy-AfriSLM tackles translation in 19 African languages. The authors built parallel and synthetic mixtures, used three quality estimators, and scored each pair in the same direction as the final training.
The most practical result concerned waste. An unfiltered open-source mixture scored 0.528 SSA-COMET on BOUQuET after using 44.93 billion tokens. A filtered mixture scored 0.530 using just 1.76 billion — 96% less. More useful data, rather than more data, was the answer.
In the authors’ tables, the smallest model, 0.8B, surpassed every compared general, specialist and African model on FLORES-200, BOUQuET and Smol under SSA-COMET. It scored 0.5944, 0.6223 and 0.4973 respectively. These are preprint results from the authors’ experiments, not an independent retest.
One important limitation remains. The paper calls the models and data open and links to a Hugging Face collection, but that collection did not open when checked on 21 August. Without accessible model cards, data cards and licences, the open release cannot yet be verified.
For Welsh, the method suggests a clear test. Before pouring every translation memory, subtitle and synthetic pair into training, score quality in the required Welsh→English or English→Welsh direction; calibrate the filter on human Welsh translations; remove repetition and keep test data out of training; and retain a separate set for translators’ judgement.
Safety and hidden cost
Two other developments apply the same principle to safety and tokenisation.
BabelSteering uses 128 English examples to create an internal direction encouraging a model to refuse harmful requests in eight languages. On Gemma 7B, refusal rose by 11 points on average without a GlobalMMLU decline.
But refusal of suspicious-looking harmless requests also rose: from 9% to 21% on average. In Bengali, harmful-request refusal rose 17 points, but over-refusal rose 23. ‘Welsh safety’ should therefore not be measured by counting refusals alone. Harmful and harmless requests must be submitted in natural Welsh, dialect and cultural context, measuring both kinds of mistake.
TokEval, a COLM 2026 paper with MIT code, examines the stage before the model sees words at all. It measures information efficiency, morpheme boundaries, UTF-8 integrity, number handling and language cost disparities. In the author’s experiments, some theoretical measures predicted language-modelling ability with Spearman correlations up to 0.80.
This tool can be used now. Tokenisers could be compared on real Welsh: prose, names, mutations, compounds, numbers, code and bilingual sentences. If the same meaning costs twice as many tokens in Welsh as in English, that is a matter of capability, price and access — not a minor technical detail.
What should a Welsh test tell us?
It is tempting to compress everything into one number, easy to put in a table and share. But this week reveals at least seven different questions:
- Is the answer correct?
- In which language does the model reason and use tools?
- Is the difference greater than random variation between training runs?
- What real linguistic errors lie behind the average rate?
- Which normalisation changes lower the score, and which raise it?
- Does filtering make data more useful without removing dialect or variety?
- Does improved safety make the model refuse harmless Welsh?
No single measure answers all seven. A credible Welsh evaluation should publish several results side by side: raw and language-aware scores; speaker-assessed error categories; variation between runs; dialect, names and code-switching; tokens and cost; trace language; safety and over-refusal; and links to data, code and licences.
Qwen3.8 27B is a real step forward: a relatively compact open-weight model that reasons strongly in the Welsh tests and completes every case. The same profile reveals the next work — place names, language level, vocabulary, grammar and translation — making the result useful rather than merely impressive.
‘Supports Welsh’ is not a result. It is a claim. Measurement that understands the language — and shows enough of its workings for others to check — turns the claim into evidence.