Muse Spark 1.2 open weights tested
Third on the extended CymraegBench leaderboard with an overall score of 83.01.
The facts
| Developer | Meta |
|---|---|
| Service used for the test | OpenRouter |
| API ID | meta/muse-spark-1.2 |
| Context | 1,048,576 tokens |
| Classification | Open weights |
Model and API service source: OpenRouter catalogue.
CymraegBench result
| Overall score | Welsh reasoning | Practical Welsh | Translation BLEU |
|---|---|---|---|
| 83.01 | 85.68 | 80.33 | 71.46 |
The same sample was used as for the other leading models, selected deterministically to return the same cases each time: up to 100 cases from each of the 19 core suites. The setting effort=low was specified explicitly. Meta’s API does not support the parameter seed, so it was omitted and this was recorded in the metadata. An output limit of up to 8,192 tokens allowed hidden reasoning to finish without truncating the answer. The 232 CCC cases do not contribute to the overall score.
Muse Spark 1.2 is third on the extended leaderboard: 0.07 points ahead of GPT-5.6 Sol, 2.15 points behind Claude Opus 5, and 7.12 behind GPT-6 Astra. It completed all 2,053 cases without refusal or invalid output.
CCC Language Audit
| Subset | Score |
|---|---|
| Correction and explanation | 84.22 |
| Idiom | 100.00 |
| Terminology | 96.88 |
| Summarisation and quotation | 90.83 |
| References | 87.34 |
| Welsh prompt | 73.33 |
| English prompt | 77.50 |
| Average across the seven subsets | 87.16 |
Corrections were strong (85.94), but the explanation score was 63.28. Muse scored 100 for not inventing a source ID, and 70.31 for answer content. Its English-prompt score was 4.17 points higher than its Welsh-prompt score in this test.
See the full leaderboard, CCC Language Audit and the open results.