Qwen3.5 9B open weights tested
A compact 9B model run locally in BF16 on one RTX 4090: reasonably strong on general knowledge in Welsh, but with clear gaps in place names, mathematics and yes/no forms.
CymraegBench v0.1 result
| Measure | Score |
|---|---|
| Overall score | 55.10 |
| Welsh reasoning | 57.98 |
| Practical Welsh | 52.22 |
| Legislation translation | 37.06 BLEU |
Qwen3.5 9B placed seventeenth on the extended leaderboard, between Grok 4.20 and Jupiter-N-120B. It completed all 2,053 cases without refusal, invalid output or infrastructure errors.
What does the profile show?
Strengths
Welsh ARC-Easy 91, ARC-Challenge 73, COPA 78, vocabulary 75 and Welsh ToxiGen 73.
Weaknesses in Welsh tasks
Bilingual place names 12, Welsh MGSM 14, yes/no forms 31 and CEFR levels 32.
CCC audit
Average 71.10: strong on instructions and summarisation, but weaker on references, correction and explanation.
How was the model tested?
The model was tested locally using BF16 weights through vLLM 0.26 on one NVIDIA RTX 4090, with an active 4,096-token window and four concurrent requests. Reasoning was disabled in the chat template so the test measured the Welsh answer itself. Random seed 1 and up to 100 cases per suite were used, just as in the other leaderboard samples. One CCC case needed its output limit raised to 2,048 tokens and a repetition penalty of 1.05; tests that had already succeeded were not repeated.
Download the results
Model: Qwen/Qwen3.5-9B · test cymraegbench-qwen3-5-9b-local-bf16-v0.1 · 10 August 2026.