Evaluations → CCC Language Audit

CCC Language Audit beta v0.1

What happens when we ask models not only to correct Welsh, but to explain the rule, choose the natural term, preserve a quotation exactly and base an answer on evidence?

This audit turns the types of failure described in Kara Lewis’s report, Artificial Intelligence (AI), Welsh and Higher Education, into 232 cases with specific scoring criteria. Every example was newly written; they do not reproduce the report’s test material. Thirty models were tested with the same harness and rule-based scorers.

30models

232cases per model

6,960model responses

0unresolved infrastructure errors

The overall picture: Claude Opus 5.5 has the highest average across the seven subsets, at 96.76, followed by Gemini 4 Argon at 96.45, Claude Fable 5 at 95.60, Claude Fable 5.1 at 93.88 and GPT-5.6 Sol at 93.01. This average is not an official overall score: the capability profile matters more than a single number.

The seven subsets

The ‘Average’ column weights each subset equally. It is offered only as a quick summary of the beta test results; it is not combined with the CymraegBench overall score. ‘Cases scored’ shows how many cases received a score.

Model	Average	Correction and explanation	Idiom	Terminology	Summarisation and quotation	References	Welsh instructions	English instructions	Cases scored
Claude Opus 5.5	96.76	96.48	100.00	100.00	94.46	86.41	100.00	100.00	232/232
Gemini 4 Argon	96.45	92.73	100.00	96.88	94.94	90.63	100.00	100.00	232/232
Claude Fable 5	95.60	94.06	100.00	93.75	92.92	88.44	100.00	100.00	232/232*
Claude Fable 5.1	93.88	93.91	100.00	93.75	92.92	76.56	100.00	100.00	232/232
GPT-5.6 Sol	93.01	88.67	96.88	84.38	94.29	86.88	100.00	100.00	232/232
Claude Opus 5	92.18	91.48	96.88	96.88	87.71	72.34	100.00	100.00	232/232
Qwen3.8 Flash	91.92	80.94	93.75	84.38	95.18	89.22	100.00	100.00	232/232†
Claude Opus 4.8	90.82	81.72	93.75	81.25	96.96	88.75	96.67	96.67	232/232
Grok 4.7	90.17	75.00	93.75	90.63	90.60	84.53	96.67	100.00	232/232
Qwen3.8 2.4T-A95B	89.99	74.38	84.38	90.63	89.91	90.63	100.00	100.00	232/232
Muse Glimmer 30B	87.66	61.72	96.88	87.50	91.07	90.63	100.00	85.83	232/232
Muse Spark 1.2	87.16	84.22	100.00	96.88	90.83	87.34	73.33	77.50	232/232
Kimi K3	85.67	66.13	87.50	81.25	93.12	85.00	100.00	86.67	230/232
Muse Wonder	85.34	87.66	90.63	100.00	90.60	90.16	65.00	73.33	232/232
GLM-5.3 Flash	84.96	66.95	78.13	93.75	90.30	75.63	93.33	96.67	232/232
MiMo V2.6 Pro	84.66	70.70	81.25	78.13	87.89	74.69	100.00	100.00	221/232‡
GPT-6 Astra	84.26	92.42	96.88	96.88	92.26	89.69	60.83	60.83	232/232
GPT-6.1 Sol	84.20	92.42	96.88	96.88	92.68	89.69	64.17	56.67	232/232
GLM-5.3	82.94	67.58	81.25	75.00	93.33	78.44	89.17	95.83	232/232
DeepSeek V4.1 Flash	82.06	72.73	93.75	75.00	96.25	70.00	76.67	90.00	232/232
Mistral Large 4.0	79.40	59.30	84.38	53.13	96.07	73.75	93.33	95.83	232/232
Grok 4.20	78.03	72.27	75.00	62.50	99.64	78.44	83.33	75.00	232/232
GLM-5.2	77.57	67.03	81.25	53.12	91.67	65.78	96.67	87.50	232/232
DeepSeek V4-Flash-0731	75.50	73.12	74.19	75.00	94.05	55.47	72.50	84.17	231/232
Qwen3.8 27B	73.53	40.94	56.25	71.88	91.19	87.81	91.67	75.00	232/232
Qwen3.5 9B	71.10	37.97	84.38	59.38	90.54	32.97	95.83	96.67	232/232
Gemma 4 E4B Cymraeg v4	70.36	27.81	56.25	50.00	96.07	69.06	93.33	100.00	232/232
Gemma 4 12B Cymraeg v6	69.70	34.22	71.88	65.62	81.08	48.44	100.00	86.67	232/232
Jupiter-N-120B	67.66	17.03	75.00	46.88	90.00	59.69	89.17	95.83	232/232
Machynlleth 3B Instruct	15.77	0.00	0.00	40.63	4.79	25.00	20.00	20.00	232/232
Caernarfon 3B Cymraeg Instruct v0.1	10.85	0.00	3.13	12.50	5.00	22.81	17.50	15.00	232/232
BritLLM Caernarfon 3B (base model)	2.81	0.00	0.00	0.00	0.00	7.19	0.00	12.50	232/232

Bold indicates the highest score in each column; all models tied for the lead are shown in bold. * Fable refused one references case; the refusal scored zero under the protocol. † One Qwen3.8 Flash response used the 131,072-token limit without producing a final answer and scored zero. ‡ Eleven MiMo responses on correction and explanation remained incomplete after retries with up to 16,384 tokens; these scored zero. Two Kimi responses and one DeepSeek V4-Flash-0731 response ended at the token limit. In the GLM-5.3 test, 17 requests returning empty or incomplete answers were repeated with the limit raised to 65,536 tokens. All then returned complete answers. GPT-6 Astra, GPT-6.1 Sol and DeepSeek V4.1 Flash completed every case on the first attempt. Under the current publication protocol, output remaining empty or incomplete after a retry is recorded as invalid with a zero score, not as an infrastructure error.

The relationship between correction and explanation

Among the 26 leading models, correction scores were between 8.59 and 46.87 percentage points higher than explanation scores; GPT-6 Astra’s gap was 41.41 points. Qwen3.8 27B had the smallest gap, while Opus 5.5’s explanation score of 64.84 was the highest. The same pattern did not apply to Jupiter or the three 3B models: Jupiter scored 12.50 for correction and 13.28 for explanation, while Machynlleth, Caernarfon Instruct and the BritLLM base model scored zero on both. For these models, the results show that the correction task itself is a problem before the explanation stage.

Model	Correction	Explanation	Gap
Claude Opus 5.5	100.00	64.84	35.16
Gemini 4 Argon	95.31	59.38	35.94
GPT-6 Astra	96.88	55.47	41.41
GPT-6.1 Sol	96.88	57.81	39.06
Claude Fable 5	96.88	62.50	34.38
Claude Fable 5.1	96.88	63.28	33.59
Claude Opus 5	96.88	60.94	35.94
GPT-5.6 Sol	92.19	53.91	38.28
Claude Opus 4.8	85.94	58.59	27.35
Qwen3.8 2.4T-A95B	79.69	56.25	23.44
Qwen3.8 Flash	89.06	58.59	30.47
Muse Spark 1.2	85.94	63.28	22.66
Muse Wonder	90.63	62.50	28.13
Muse Glimmer 30B	65.63	35.94	29.69
Kimi K3	75.81	45.97	29.84
MiMo V2.6 Pro	75.00	42.19	32.81
GLM-5.3 Flash	71.88	43.75	28.13
GLM-5.3	70.31	44.53	25.78
DeepSeek V4.1 Flash	79.69	48.44	31.25
Grok 4.7	78.13	50.78	27.34
Mistral Large 4.0	71.88	42.97	28.91
Grok 4.20	78.12	49.22	28.90
GLM-5.2	78.12	31.25	46.87
DeepSeek V4-Flash-0731	76.56	47.66	28.90
Qwen3.8 27B	39.06	30.47	8.59
Qwen3.5 9B	40.63	21.88	18.75
Gemma 4 E4B Cymraeg v4	32.81	15.62	17.19
Gemma 4 12B Cymraeg v6	40.62	25.78	14.84
Jupiter-N-120B	12.50	13.28	−0.78
Machynlleth 3B Instruct	0.00	0.00	0.00
Caernarfon Cymraeg Instruct	0.00	0.00	0.00

Three further findings

The effect of prompt language depends on the model

Qwen3.8 27B performed 16.67 points better with the Welsh prompt, Kimi 13.33 points better, and Grok 8.33 points better. But DeepSeek V4.1 Flash performed 13.33 points better with the English prompt, the older DeepSeek 11.67 points better, and GLM-5.3 6.66 points better in English. GPT-6 scored 60.83 in both languages, while Qwen3.8 Flash and Qwen3.8 2.4T-A95B scored 100 in both.

Avoiding invented sources is not the same as finding the answer

The models scored between 96.88 and 100 for not inventing source IDs. Yet scores for answer content ranged from 39.06 to 71.88. A model can avoid inventing references and still fail to answer fully.

Models successfully preserved facts and quotations

The models generally preserved the selected facts accurately: GPT-6, Qwen3.8 27B, Qwen3.8 Flash, GLM-5.3 Flash and GLM-5.3 scored 100, and Qwen3.8 2.4T-A95B scored 95.83. These models preserved the quotation exactly; Grok was strongest across the whole subset, at 99.64.

Data, method and status

The 232 cases, scorers and harness are openly available in BangorAI/cymraeg-bench. The scorers follow the same explicit rules each time: no other model was used as a judge.

These are beta research results, not a final verdict. The suite does not claim to represent every kind of Welsh or every higher education task. It will remain at 0.1.0-beta status until at least two independent Welsh-language editors have reviewed every case. Before publication, the quotation scorer was corrected to recognise typographic quotation marks; 30 responses were rescored without any new model calls.

Download the results

Combined results from published tests · up to 4 November 2026 · 1,933,481 input tokens and 1,807,130 output tokens in the downloadable data.