More than speaking Welsh: data, law and real testing

A new programme for bilingual law and work on lower-resource languages show that AI readiness means authoritative data, governance and real testing.

No new Welsh model was released this week. But something potentially just as important in the long term was announced: the Welsh Government's second programme to improve the accessibility of Welsh law makes AI part of the work to make bilingual law easier to use.

The programme promises to explore how technology and AI can help people use the Welsh text of legislation, prepare explanatory material and assist legislative drafting. The Government will also continue expanding TermCymru and BydTermCymru as authoritative sources of legal terms and vocabularies.

This is not a live product. No model, supplier, budget or test result is named. The document does not say that legal text will be used to train a model. This is a commitment to explore, and that distinction must remain clear.

Two streams of legal pages passing through a transparent analytical machine under human control — Authoritative bilingual text is a useful resource, but the system must show its sources and uncertainty.

Why law is a particularly demanding test

Bilingual law is an attractive AI resource. It has corresponding Welsh and English text, controlled terminology, clear provenance and a traceable structure. But those same features raise the standard.

A fluent summary is not enough. A system would need testing on finding the right provision, matching Welsh and English versions, using the latest version, respecting legal terminology, citing sources and admitting uncertainty. In a high-risk field, users must be able to see the evidence and reach the authoritative text themselves.

This applies beyond law. A good Welsh service is more than a chat box that can compose Welsh sentences. It needs to give accurate answers, match the English service's standard, and not hide the route to a person when human judgement or care matters.

Three layers of readiness

new article by Cynog Prys of Bangor University, published by Eurac/MIDAS on 19 October, offers a useful way to organise the problem. His analysis identifies three layers of minority-language AI readiness.

The first is language readiness: are data, terminology, speech technology and other resources available? Welsh is comparatively well placed because of decades of education, broadcasting, legislation, standards and language technology work.

The second is organisational readiness: can a body use the technology without weakening its Welsh service, losing control over data, or replacing human contact where needed?

The third is user readiness: do people want to use the service in Welsh, do they trust it, and do they switch to English because the answer is better there?

The article is expert analysis, not a new study of user behaviour. But it poses a question every pilot should measure. ‘Can the model speak Welsh?’ is not the only test. The question is whether the technology makes it easier for people to live and work through Welsh.

A lesson from Twi: keeping control of terminology

GLOSS-MT, a new preprint on English-to-Asante-Twi translation, shows one practical way to retain control. The system looks up phrases in an offline dictionary, supplies the Twi terms to the model, then translates the mixed sentence.

On a benchmark of 84 documents in four domains, every fine-tuned model scored 2.85–5.48 chrF2 points above its base. In a direct control test, the models used 79%–97% of required terms. When every dictionary entry was changed, 70%–97% of the new terms appeared without retraining the model.

The method offers a clear idea for Wales: an organisation should be able to change an authorised term in TermCymru or an internal glossary without waiting for a new model version. That does not remove the need for a translator or editor. But it makes the rule visible and easy to change.

The paper and code are public, but the 8B model card uses CC BY-NC 4.0. That is another warning: ‘available’ does not always mean every commercial use is unrestricted.

Testing the real mess, not the clean sample

In new study of OCR for Lakota archives, fine-tuning small visual models on 35 transcribed pages reduced character error rate from 17.1% to 3.9%. Using only nine pages increased error by just 1.1 percentage points compared with the full set.

But replacing clean crops with realistic automatic page segmentation nearly doubled the best models' error. That is the most important result for Welsh archive projects. A perfectly cropped line is not the real test. The faded page, inconsistent column, historical spelling and detection of text boundaries are part of the system.

The authors place the work in the context of language reclamation and community conversations. That is not an ethical detail separate from the technology; it determines which documents, spellings and mistakes matter.

Streams of speech, dictionary cards and archives passing through different human checkpoints — Every mode—speech, text or image—brings its own errors, and the complete pipeline needs testing.

One speech model, 51 languages—but 51 different results

new SunflowerASR paper describes using supervised fine-tuning followed by GRPO to adapt Whisper-large-v3 for 51 African languages. The Apache 2.0 model is available on Hugging Face.

The authors report their best published average word error rate on two independent benchmarks: 0.227 on AfriVox-v2 and 0.372 on SimbaBench. But the model card also makes the variation clear: on Sunbird's internal benchmark, per-language error rates range from 0.08 to 0.67. The card also warns that the internal benchmark shares Sunflower's training domain, but not the comparison models'.

The lesson for Welsh is not to publish just one speech number. Results need breaking down by dialect, age, accent, microphone quality, noise, natural conversation and code-switching. At least one test set should be separate from training-data sources.

Nepali–English podcast benchmark adds a useful detail. On 13,044 conversational utterances, fine-tuning on less than an hour of synthetic speech reduced mixed error rate from 23.40% to 21.52%, and code-switch-point errors from 35.89% to 28.77%. GRPO added a much smaller improvement: 21.49% and 28.70%.

In this experiment, therefore, the simpler supervised stage did almost all the work. A Welsh–English project should measure errors specifically where the speaker changes language, and test whether a complex new method adds anything meaningful after conventional fine-tuning.

Do not let a model judge hide the difference

DeepSEA, an Apache 2.0 benchmark in 12 Southeast Asian languages, finds that smaller models deteriorate substantially on lower-resource languages. On reasoning and summarisation, models performed better when pivoting through English than working directly in the target language.

That suggests a simple Welsh test: ask the same question directly in Welsh, then ask the system to translate via English. If the second route is much better, Welsh capability is less independent than the interface suggests.

another study in 27 languages warns about using a language model to mark answers. Across nine prompting strategies and four open-weight judges, judge choice mattered more than prompt choice; some strategies' effects reversed with a different judge; and stability worsened in lower-resource languages.

Welsh is not included in the evidence summarised in these papers. That is why the results should not be assumed to transfer. But they offer concrete tests: direct Welsh versus an English pivot, multiple judges and prompts, and a sample marked by Welsh speakers.

Wrapping up the week

The week's story is not a model launch. It moves the question from ‘can AI produce Welsh?’ to ‘can Wales control, test and use the system responsibly?’

The law programme offers a concrete domain: authoritative bilingual text, controllable terminology and a measurable public service. The comparative work shows what that measurement should include: term control, data provenance and rights, the messiness of real documents or recordings, per-language results and human validation.

AI that speaks Welsh is a start. Readiness means knowing when it is right, who controls the terminology, what happens when it fails, and whether people choose to keep using Welsh.

Timing note: the final scan was completed at 09:07 BST on 23 October. Material published later on 23 October will be included in the next scan.