From principle to testing: building Welsh into AI

A UNESCO toolkit and a new test of models’ grammatical ability show why Welsh needs to be built in from the start, and every checkpoint tested.

Hands from different generations building a multilingual digital system along a path through a Welsh landscape — Welsh should be part of the idea, specification and service—not a layer added at the end.

Welsh as part of the specification

UNESCO published a new resource, Practical Toolkit for Supporting Multilingual Digital Services, on 29 September. It adapts the Welsh Government's experience of bilingual technology design for governments, organisations and digital teams in other countries.

The main principle of UNESCO's publication is that a language should not be added at the end of a project. Multilingual requirements should be part of the idea, specification, interface, content and functionality from the outset. The Welsh Government said on 30 September that Welsh experience was being presented as an international example, particularly for indigenous and lower-resource languages.

This is not a new AI model. The toolkit contains no new dataset, weights or Welsh score. Its significance lies in the stage before choosing a model: who writes the requirements, what Welsh input the service must accept, how equality is tested, and what happens when the language changes within the same conversation.

If an AI service treats Welsh as a translation layer added at the end, failure can appear everywhere else: speech recognition, local names, switching between Welsh and English, error messages and the route to human support. The toolkit makes the principle clear. This week's new research shows how thoroughly the result needs testing.

The same family, a different result

The a new study on arXiv on 30 September. It compares base models with post-trained versions—the stage intended to improve instruction-following and conversation—on MultiBLiMP, a test of sentence pairs across 101 languages.

The paper's tables contain 1,120 Welsh pairs. The results vary widely:

Gemma 3 1B fell from 78.2% to 65.4% after post-training;
Gemma 4 31B fell from 94.5% to 85.1%;
Qwen3 8B rose from 69.5% to 86.2%;
and TinyAya 3.35B rose from 85.1% to 90.5%.

Averaged across languages and larger models, the paper finds a loss after post-training, with the greatest cost for lower-resource languages. But Welsh shows why that average should not become a simple rule: the same stage costs one model 12.8 points and adds 16.7 to another.

‘Qwen’, ‘Gemma’ or ‘multilingual model’ is therefore not enough detail. The exact checkpoint, test method and way the question was asked must be named. In this study, a native-language prompt—‘Dyma frawddeg yn y Gymraeg’ (‘Here is a sentence in Welsh’)—also recovered some ability lost by a direct approach.

evaluation code is available, but this paper is a preprint. The arXiv page states ‘EMNLP Main 2026’; a separate proceedings record was not confirmed during this review. An OpenReview copy was already public before this week, so the new development is the arXiv version, not the research's first appearance.

Do not trust a single run

MGhana-ST, a speech translation dataset and benchmark for Ga, Twi, Ewe and Fante, adds another warning. The researchers used 16.1 hours of audio, and 37 native speakers revised the English translations directly from the audio.

In an earlier single-run analysis, pooling the languages during training appeared to help three of the four. After repeating the tests with three random seeds, that story disappeared. Ga's gain of 0.51 BLEU and Twi's 0.06 were within individual-run variation; Ewe lost 6.99 and Fante 5.11.

That is a direct lesson for Welsh experiments. If the gain is smaller than variation between runs, one positive number is insufficient evidence. Publish the mean, variation and seed details, not just the best run. The MGhana-ST annotations are available under CC BY 4.0, although Ewe audio retains its original source's non-commercial and no-derivatives restrictions.

Testing speech, not just a language label

a new test of 11 speech recognition systems across Basque, Catalan, Galician, Portuguese and Spanish shows what an average can hide. For Basque, word error rate ranged from 7.4% with Scribe v2 to over 100% in several systems without explicit language support. Whisper-large-v3 scored 51.9%, while seamless-m4t-v2-large scored 13.0%.

Most systems also showed a difference between male and female speakers; the largest pooled gap was 17.1 points. The authors warn that the sets are unbalanced and drawn from different domains, so the numbers do not establish every service's real-world behaviour. They do show, however, that a ‘multilingual’ label does not guarantee equal performance.

For Welsh, a useful speech test would break results down by dialect, gender, age, fluency, domain, audio quality and code-switching. It would also record model speed and licensing. One average WER cannot answer those questions.

A diverse group of speakers sending ribbons of speech through several checkpoints and different measurement paths — One score is not enough: tests need repeating across different speakers, domains and checkpoints.

Choosing a partner through evidence

RunyaNER introduces the first publicly available named-entity benchmark for Runyankore: 30,007 sentences, 237,076 tokens and 5,511 names of people, places, organisations and dates. Every label was checked by a first-language speaker.

The question was which other language best helps when Runyankore data is scarce. In zero-shot transfer, four languages selected by similarity of labelled examples gave a mean F1 of 0.641. Four languages selected using URIEL typological features scored 0.580. But once a little labelled Runyankore data was added, the methods converged around 0.814–0.818.

This is not a data-free method: it needs a seed of labelled examples in the target language. That is precisely the lesson for Welsh. Rather than assuming English, Irish or Breton is the best partner for every task, a small but carefully prepared Welsh sample can show which transfer actually works.

From public principle to public evidence

No new Welsh model or dataset emerged during this window. Nation.Cymru was searched specifically. Its AI stories concerned policy on the power of technology companies, an AI-generated political image and a legal clearance platform for journalism. These are AI stories in Wales or through Welsh media, but not evidence about the technology's Welsh-language capabilities.

It is fair to call this a quiet week for new Welsh products. The two genuine developments connect well. UNESCO says the language should be part of the service from the beginning. MultiBLiMP shows that the exact model needs testing at the end. MGhana-ST adds that the test must be repeated; the Iberian test says we must examine different speakers and domains; and RunyaNER says transfer sources should be chosen through evidence.

‘Supporting Welsh’ is therefore not a full specification. It should mean: accepting Welsh and code-switching, publishing data and rights, naming the checkpoint, testing different groups and tasks, repeating the run, and opening results to inspection. That is how international recognition becomes a trustworthy service.