GPT-6 Astra commercial tested
OpenAI’s leading model, tested through OpenRouter on the core suite and CCC Language Audit.
CymraegBench v0.1 result
| Measure | Score |
|---|---|
| Overall score | 90.13 |
| Welsh reasoning | 89.49 |
| Practical Welsh | 90.78 |
| Legislation translation | 74.10 BLEU |
GPT-6 Astra leads the extended leaderboard, 4.97 points ahead of Claude Opus 5. It completed all 2,053 cases without refusal, invalid output, retry or infrastructure errors.
What does the profile show?
Strong reasoning
It scored 100 on Welsh ARC-Easy and Welsh MGSM, and 99 on Welsh ARC-Challenge and Welsh COPA. The reasoning category score was 89.49.
Practical Welsh
It scored 100 on vocabulary, 97 on obscenities, 96 on ARC-Easy mini and 95 on MMLU-Lite. Its practical score of 90.78 is the highest in the table.
The main weakness
CEFR levels was the weakest core test at 39. This shows why a profile by suite is more useful than a single score alone.
CCC Language Audit
The model achieved a beta average of 84.26 across the seven subsets, the tenth-highest result. It scored 96.88 on idiom and terminology, 92.42 on correction and explanation, 92.26 on summarisation and quotation, and 89.69 on references. Following Welsh and English instructions was the specific weakness: 60.83 in both languages. Within the correction test, it scored 96.88 for correction but 55.47 for explanation.
How was the model tested?
The model was tested as openai/gpt-6-astra through OpenRouter, with random seed 1, reasoning_effort=low, seven concurrent calls and up to 100 cases per core suite. The GPT-6 API does not accept temperature, so that parameter was omitted. Each response had a 32,768-token limit to let hidden reasoning finish, and only the final answer was scored.
The test used 248,950 input tokens and 64,754 output tokens. Of the output, 37,821 were hidden reasoning tokens and 26,933 were visible answer tokens. The 2,053 calls cost $5.7272; this total matches the recorded change on the OpenRouter key.
Download the results
- Score summary (CSV)
- Scores for every suite (CSV)
- CCC Language Audit dimensions (CSV)
- Test details (JSON)
Model: GPT-6 Astra · API service: OpenRouter · test cymraegbench-gpt-6-astra-openrouter-v0.1 · 9 September 2026.