Qwen3.8 27B open weights tested
Qwen’s 27B model tested through OpenRouter: strong Welsh reasoning and a complete response to every case, but clear gaps in some practical language tests.
CymraegBench v0.1 result
| Measure | Score |
|---|---|
| Overall score | 70.47 |
| Welsh reasoning | 79.83 |
| Practical Welsh | 61.11 |
| Legislation translation | 46.03 BLEU |
Qwen3.8 27B placed twelfth on the extended leaderboard, between GLM-5.3 and GLM-5.2. It completed all 2,053 cases — 1,821 core cases and 232 CCC Language Audit cases — without refusal, invalid output or infrastructure errors.
What does the profile show?
Strengths
Welsh ARC-Easy 96, ARC-Challenge 90, COPA 93, Welsh MGSM 91, Welsh ToxiGen 95 and ARC-Easy mini 96.
Testing comprehension
Welsh WNLI 84.51, Welsh XNLI 83 and Welsh MMLU-lite 81, leading to a reasoning score of 79.83.
Weaknesses in Welsh tasks
Bilingual place names 12, CEFR levels 27, obscenities 28, vocabulary 47 and grammar 50.
Its legislation translation BLEU of 46.03 is higher than Qwen3.5 9B’s score, but lower than the leading models’ scores. As the vocabulary and obscenity tests have skewed class distributions, these scores should be read alongside the full profile rather than in isolation.
CCC Language Audit
The model achieved a beta average of 73.53 across the seven CCC subsets. It was strong on summarisation and quotation (91.19), Welsh instructions (91.67) and references (87.81), but weaker on correction and explanation (40.94). Its correction score of 39.06 was only eight and a half points above its explanation score of 30.47 — the smallest gap among the fourteen leading models.
How was the model tested?
The model was tested as qwen/qwen3.8-27b through OpenRouter, with temperature 0, random seed 1, seven concurrent requests and up to 100 cases per suite. The setting was reasoning_effort=low; reasoning was kept separate and only the final answer was scored. Each response had a 32,768-token limit to avoid cutting off reasoning before completion, although the longest actual response was 1,922 tokens. Every request ended with stop.
Download the results
- Score summary (CSV)
- Scores for every suite (CSV)
- CCC Language Audit dimensions (CSV)
- Test details (JSON)
Model: Qwen/Qwen3.8-27B · API service: OpenRouter · test cymraegbench-qwen3-8-27b-openrouter-v0.1 · 22 August 2026.