Claude Fable 5.2
CymraegBench v0.1 — low effort, adaptive thinking on.
| Overall | Welsh reasoning | Practical Welsh | BLEU |
|---|---|---|---|
| 89.45 | 88.34 | 90.56 | 74.64 |
2,053 cases; no invalid outputs or unresolved infrastructure errors after retrying one timed-out request.
Protocol and scoring
Fable 5.2 uses the corrected binary-answer extractor: an explicit Y/N at the start or on the final line is accepted. Not all older results have been rescored yet.
85 answers were rescored without generating new answers: the score rose from 84.73 to 89.45. Original results and the audit are preserved.
Up to 100 cases per suite, sampling seed 1, and 232 CCC cases. A 2,048-token output floor shared between thinking and answer. Thinking is always on; the model does not accept temperature or generation-seed settings. This is not a thinking-off run.
CCC Language Audit
96.21 — mean of the seven subsets, separate from the overall score.
| ccc-cywiro-esbonio | 95.47 |
|---|---|
| ccc-priod-ddull | 100.00 |
| ccc-termau | 100.00 |
| ccc-crynhoi-dyfynnu | 91.07 |
| ccc-cyfeiriadau | 91.09 |
| ccc-cyfarwyddyd-cymraeg | 95.83 |
| ccc-cyfarwyddyd-saesneg | 100.00 |