Behind the score: five new models, five different profiles

Five new CymraegBench runs have changed the leaderboard. GPT-6.1 Sol is now second, but a supplementary test shows that a high overall score does not guarantee consistent instruction-following.

Five abstract machines measuring the same Welsh ribbon, with a side path beginning to fray — A high overall score can hide a specific weakness, such as failing to stick to permitted sources.

Five runs, the same test

On 14 and 15 October, five new models completed the same CymraegBench core sample: up to 100 cases in each of 19 sets, producing 2,053 responses per model. There were no refusals or invalid outputs in any of the five runs.

Here are the results in the order of the live leaderboard at the time of the scan:

Rank	Model	Score	Reasoning	Practical Welsh	BLEU
2	GPT-6.1 Sol	89.05	88.11	90.00	72.16
4	Gemini 4 Argon	87.41	90.15	84.67	73.66
5	Muse Wonder	86.23	86.68	85.78	71.41
9	Grok 4.7	82.34	84.90	79.78	69.71
24	Mistral Large 4	59.93	66.08	53.78	59.28

Only 1.08 points separate GPT-6.1 Sol and GPT-6 Astra at the top. Gemini 4 Argon is particularly strong on reasoning, and has the highest translation BLEU score of the five. Muse Wonder and Grok 4.7 are also competitive; the gap between them and Mistral Large 4 is much larger.

That is useful, but not the whole story.

When a high ranking does not mean consistent instruction-following

CCC Language Audit measures more specific tasks: correction and explanation, idioms, terms, summarisation and quotation, references, and following source-related instructions in Welsh and English.

GPT-6.1 Sol's profile is uneven here. Its average across the seven subsets is 84.20. It scored 96.88 for correction, but 57.81 for explanation; on the two source-use tests, it scored 64.17 with Welsh instructions and 56.67 with English instructions. A simple lack of Welsh understanding is not the cause, since its other results are strong. Rather, it is a specific failure to respect the evidence boundaries set.

Gemini 4 Argon changes the picture. Although 1.64 points behind GPT-6.1 Sol on the main score, its CCC audit average is 96.45, with 100 on both instruction-following tests. Grok 4.7 also reaches 90.17 in the CCC audit, despite being ninth on the main list.

The opposite is true of Muse Wonder: 100 for terms and 90.16 for references, but 65.00 and 73.33 for following source-related instructions. Mistral Large 4 reaches 96.07 for summarisation and quotation, but only 53.13 for terms.

There is therefore no single ‘best model’ for every use. The order depends on the task, and an average can hide precisely the weakness that matters in a real decision.

Would AI-assisted appraisal be more neutral?

It could be more consistent in a narrow sense. If every application were assessed against the same rubric, using the same evidence and checking steps, a system could help record why each score was awarded. It could also highlight inconsistency between two human assessors.

But consistency is not neutrality. A model can consistently repeat the same bias; fail to follow a source-related instruction; or hide poor interpretation behind a persuasive explanation. These tests do not measure fairness between groups of applicants, or establish that any model is neutral.

The safer approach is limited AI assistance: extracting evidence from permitted material, linking it to a public rubric, showing uncertainty, and retaining a record so a person can check and challenge the result. The model should not set hidden criteria or make the final decision alone.

This aligns with the boundary set by the Arts Council of Wales in the policy we discussed last week: AI will not be used to decide or evaluate funding applications. Local AI may reduce some concerns about sending data to an external company, but does not remove the need for a rubric, Welsh tests, an audit trail, human oversight or a right of appeal.

The test language can change a safeguard's decision

Two preprints published late on 15 October add an important warning. In LinguaGuardBench, researchers kept tool calls, observations and labels for 1,220 steps identical, translating only the user request into German, French, Turkish, Swahili and Amharic. Across 19 guard systems, many smaller models changed their verdict more often as the language became scarcer in the data. Among nine systems below 13 billion parameters, the adjusted flip rate rose from 1.5 percentage points in French to 4.4 in Amharic.

another study in five Indian languages shows that how a test is presented matters too. When four models were explicitly told they were in a safety evaluation, harmful-request refusals rose by an average of 14.4 percentage points in the Indian languages. But benign-request refusals also rose, from 6.9% to 11.3%.

These are not Welsh tests. But they suggest two clear experiments for Wales: change only the request language when testing a safeguard, and compare ordinary use with instructions telling the model it is being tested. English safety measurements should not be assumed to transfer neatly to Welsh.

A similar lesson from Bavaria: test the version actually running

new preprint on Bavarian, standard German and English, submitted on 13 October, offers another practical warning. Researchers compared 14 open-weight models in BF16, eight-bit GPTQ and two four-bit formats, using the same set of 206 aligned questions.

On average, eight-bit GPTQ was almost identical to BF16: a difference of +0.03 points. The two four-bit formats lost around 1.4–1.5 points, but changed 11.9–16.0% of decisions. Bavarian was not always the worst-affected language. The outcome depended on the model, quantisation method, calibration language and test type.

This is a genuine development since our earlier article about the BAVGROUND benchmark. The lesson for Welsh is to test the final artifact—the four- or eight-bit version on the device—rather than quoting the full parent model's score. The test should include Welsh, English and mixed calibration data.

Waveforms from language communities passing through acoustic gates into a modular listening instrument — The whole speech pipeline is part of the test: audio, transcript, instructions and final answer.

When speech adds another gap

SEA-HELM Audio, a preprint submitted on 14 October, compares performance on the same tasks in text and synthesised speech. Passing speech through recognition before sending it to a language model reduced multiple-choice accuracy by 9.00–18.50 points. On instruction-following tasks the loss was 20.00–42.86 points; Tamil lost 40 points in both pipelines tested.

The paper does not test Welsh, and synthetic speech is a limitation. Nevertheless, the method is directly relevant. A text score cannot predict a Welsh voice assistant's reliability. Three numbers are needed: original-text performance, transcription accuracy and end-to-end performance.

Nufo, a new Ewe benchmark, shows how to build a speech test more carefully. The authors combined existing resources into a 174.82-hour corpus, using audio hashes and normalised text to remove training–test overlap. Whisper-medium achieved a word error rate of 37.98%; removing punctuation and combining marks reduced it to 30.95%. The difference reminds us that orthography and scoring are part of the result, not trivial details to address afterwards.

Do not create a language nobody speaks

Masewalkopa translation project, for a Nawatl variety spoken in Santa Ana Tlacotenco, offers a more fundamental data lesson. The authors warn that pooling text from related indigenous varieties to enlarge a corpus can create an artificial register belonging to no particular community.

Starting with 70,000 multi-variety pairs, they used 2,665 Masewalkopa pairs and 192 targets transcribed by elders to specialise an mT5-large model for the variety. A morphological analyser reduced unknown forms from 82.0% to 19.9%, and the model reached 58.20 and 57.90 chrF++ in the two test directions. After int8 quantisation, CPU processing time fell from 137.5 seconds to 1.8–2.6 seconds per request.

For Welsh, this is not an argument against sharing data. It is an argument for naming the variety, retaining text provenance, involving the target community, and testing whether output sounds like real people's language—not a convenient average of every available corpus.

A large resource, but major rights questions too

ELARSpeech claims 30,202 hours of speech in 675 lower-resource languages, curated from the Endangered Languages Archive. The authors say they are releasing the data, ASR and TTS models, and construction pipeline.

That could be a very significant development. But at scan time it was not possible to confirm the dataset's overall licence, individual recording terms, or how much Welsh—if any—it contains. ‘Available for research’ does not necessarily mean every community has authorised generative-model training. Before promoting ELARSpeech as a pattern for Wales, consent, access, attribution and governance need checking collection by collection.

The conclusion

The five Welsh runs give us more models to compare, but also show that the choice depends on the task. GPT-6.1 Sol has the highest overall score of the five. Gemini 4 Argon has the most consistent CCC profile. Grok 4.7 shows that a model lower on the main list can perform better at following some evidence-related instructions. Muse Wonder and Mistral Large 4 show that strength in one area can accompany weakness in another.

The comparative research reinforces the same principle: test the exact version, complete pipeline and specific behaviour at issue. If AI assists appraisal, the work should be repeatable, visible and open to challenge. Neither a high ranking nor running locally makes a decision neutral.

Timing note: the final scan was completed at 09:01 BST on 16 October. Material published later on 16 October will be included in the next scan.