Claude Opus 4.8 commercial tested
Published results for strict-metre Welsh poetry.
The facts
| Developer | Anthropic |
|---|---|
| Released | 28 May 2026 |
| Licence | Proprietary (API) |
| Multilingual | A table of 15 languages in Anthropic’s documentation (Spanish 98% → Yoruba around 80% of English performance) — Welsh is not included. Commentary on the system card identified multilingual tasks as a relative weakness. |
What we measured: cynghanedd
Opus 4.8 was tested with our full hidden-set protocol: 40 themes unseen during training, five attempts per theme, and a single chance to respond, without a feedback loop or rhyming hints. Each line was judged by the cynghanedd rules engine:
| Measure | Opus 4.8 | Our 9B (DPO r3) |
|---|---|---|
| Valid seven-syllable lines (raw) | 20.0% | 65.0% |
| Appropriate + valid + original | 13.5% | 1.0% |
| Unique lines | 100% | 42% |
| Croes lines | 8 | 1 |
The local 9B model scored 65.0% on form, compared with 20.0% for Opus. On the combined measure of valid, appropriate and original lines, Opus scored 13.5% and the 9B model 1.0%. Every Opus line was unique, and it produced eight examples of cynghanedd groes, compared with one from the 9B model. The results suggest different strengths in meaning and formal control.
Some Opus lines used a more contemporary register than the small model’s lines, for example:
Gwylan wen ar y glan wag
— valid lines obtained using the protocol, July 2026
The Coleg Cymraeg report also noted that Claude was among the tools tested on Welsh academic texts — a useful corrector, but occasionally inventing terms (‘hunanganolig’ instead of ‘hunanganolog’).
CymraegBench v0.1 result
| Overall score | Welsh reasoning | Practical Welsh | Translation BLEU |
|---|---|---|---|
| 81.56 | 85.02 | 78.11 | 69.29 |
Opus 4.8 is fifth on the extended leaderboard after GPT-6 Astra, Opus 5, Muse Spark 1.2 and GPT-5.6 Sol. Only one request out of 1,821 was refused. See the full leaderboard and the open results.