Testing, measuring and developing Welsh AI
Evaluations → Claude Fable 5

Claude Fable 5 commercial tested

Eighth on the extended leaderboard, but with an unusual refusal rate on biology questions.

Four evaluators studying abstract models in a transparent testing arena

The facts

DeveloperAnthropic
Released9 June 2026
ClassA ‘Mythos’-class model, positioned by Anthropic above Opus. Fable 5 is the generally available version; Mythos 5 is the restricted-access version with fewer safeguards.
Price$10 / $50 per million tokens (input/output)
LicenceProprietary (API)

Welsh: no data from Anthropic

The multilingual result quoted in the announcement is around 96% on MGSM, a mathematics benchmark covering ten languages, but not Welsh. Anthropic’s announcement does not mention Welsh, and the Claude family’s 15-language documentation table does not include it either. As with the other models on the leaderboard, this is not evidence of inability; it is evidence of a lack of measurement.

The question is relevant for Fable because its predecessor has already been measured. Opus 4.8 reached 20.0% for valid seven-syllable lines and 13.5% for appropriate+valid+original in Bangor AI’s cynghanedd protocol. Repeating the test with Fable can help measure whether its Welsh ability has changed.

CymraegBench v0.1 result

Overall scoreWelsh reasoningPractical WelshTranslation BLEU
78.6483.7373.5669.46

Fable is eighth after the latest results were added, but refused 79 of 1,821 requests (4.34%). Many were school biology questions classified as ‘bio’ requests by Anthropic’s safety system; each refusal scored zero. See the full leaderboard and the open results.

What comes next

The cynghanedd protocol and qualitative tests Coleg Cymraeg report will be the next steps.