Local language and living voices: new lessons for Welsh AI

BavGround and Easper show why local knowledge must be tested and recordings selected for linguistic value, not just audio clarity.

Our entry on 11 August already discusses the French Luth-2 model and papers published earlier that week. No new result warrants repeating that story. This update therefore starts with what followed: a benchmark showing how models miss local knowledge, and a workflow helping communities decide which voices to transcribe first.

No new Welsh model, dataset or benchmark was published in the second half of the week. But new international evidence poses two specific questions for any Welsh AI project: does the model understand local life in Wales, not just standard Welsh; and does the voice data represent living language, not just the clearest recordings?

Paper illustration of three researchers comparing language test fragments between Welsh and Bavarian landscapes — A real language test needs more than the same question translated: it must measure dialect, place, source and community knowledge.

When Bavaria disappears from German

BavGround, submitted on 13 August, contains 206 original questions in eight areas: history, politics, landscape, food, buildings and heritage, art, traditions, and language itself. English, German and Bavarian versions of every question were created — 618 examples in total — with translations checked by a native speaker.

Some questions ask general facts about Bavaria. The rest draw on regional journalism, historical work and specialist sources. That distinction matters: a model can recognise a region’s name without knowing what is remembered, discussed and said there.

The researchers tested 15 open-weight models between 7B and 10B, plus one closed model. On average, open models scored 55.7% in English, 57.6% in German, but 45.9% in Bavarian. General questions averaged 63.1%; source-based local questions 46.7%. Language and dialect was hardest, at 35.6%.

These are the authors’ preprint results, not an independent retest. The dataset and code were not available for inspection during the search either. Nevertheless, the gap measured is highly relevant to Welsh.

A model can answer a general question about Wales without understanding a local word, the register of a council meeting, northern and southern dialect differences, or an institution’s history in one valley. It can translate the sentence without understanding its local context.

A Welsh version would combine general knowledge with questions based on community newspapers, archives, council records, oral history and regional news. Results would be broken down by dialect, register, domain and source rather than compressed into one national score.

Scoring changes the story

BavGround also shows that choosing A, B, C or D does not always measure what we think it measures.

Scoring the answer letter conventionally gave GENBA-10B — a model aimed at German, English and Bavarian — 41.9%. Measuring the likelihood of the option text instead raised this to 58.9%. Comparing the meaning of a generated answer raised it to 61.5%.

That does not prove the model solved the test. It shows that part of the failure concerns answer format. A Welsh benchmark should therefore test letters, option text, free answers and semantic judgement; vary option order; and inspect the examples themselves. One number can hide a model that knows but formats poorly — or one guessing the right letter without understanding.

Living voices: what to transcribe first

Easper, submitted on 12 August, moves from text to audio. It is an open workflow for field linguists: it takes recordings and ELAN annotations, prepares data, fine-tunes Whisper or XLS-R in the cloud, then returns a small model to an ordinary computer. The model can separate speakers, create a first transcript and return it in an ELAN file for human correction.

The The code is on GitHub under MIT. The paper was accepted at Interspeech 2026 and has a CC BY 4.0 licence. But its most interesting contribution answers a practical question: when human transcription time is scarce, which recording should come first?

The team tested five strategies on Bislama, Nafsan and Nguna. They had 13 hours 45 minutes of Bislama, 14 hours 50 minutes of Nafsan, but only one hour and one minute of Nguna. Early on, prioritising stories with useful vocabulary and repeated words and sound patterns reduced character error rate faster than selecting the clearest recordings.

That does not excuse poor recording. Clear audio matters to an archive. But for early model training, a natural conversation containing repetition and rich vocabulary can be more useful than perfect studio sentences.

Paper illustration of community speakers and linguists turning live recordings into organised sound waves — Audio clarity alone does not determine a recording’s value: vocabulary, repetition, dialect, consent and community use are part of the data.

What would that mean for Welsh?

If a Welsh project began with a hundred untranscribed hours, it should not organise work solely by noise level. It should measure which dialects, ages, domains, mutation patterns, place names and bilingual situations appear in each session. Material should provide enough repetition to learn patterns but enough variety to avoid learning only one voice.

DonorRank adds a second lesson. It tested which languages supplied the most useful data for helping a system recognise speech in Indian varieties and African languages. The largest or most closely related language was not always best. The choice depended on the data mixture, script and speech characteristics.

English should therefore not be assumed the best starting point for every Welsh speech problem. Breton, Irish, Scottish Gaelic or a particular English variety may help in different ways — but the choice must be tested on Welsh speech held out from training.

myMediWhisper shows the value of domain-specific data. The team built a 28-hour Burmese medical corpus recorded and checked by native speakers. Its best model achieved 23.44% word error rate on clear audio. Adding simulated noise worsened the clear-audio score but improved robustness in noisy, reverberant rooms. The dataset is under CC BY-NC-SA 4.0 and requires permission for access.

For a Welsh health service, the same kind of report should cover telephone audio, consultation rooms, medicine names, switching to English and dialect differences — not just one error rate on clear recordings.

What can go wrong in deployment

Three other papers warn that a system can appear multilingual in the lab but change when compressed, given tools or safety-tested.

In Language-Conditional Dequantization, quantising to INT3 increased perplexity (how hard the text is for the model to predict) in non-English languages by two to four times the English increase. A small LoRA adapter recovered 70–83% of the perplexity difference in non-Latin scripts, but far less of the knowledge-task gap. There was no specific Welsh result, so the lesson is to retest Welsh after creating the local version — not assume the original score survived.

In Actions Speak Louder than Words, revised on 13 August, found that four leading models followed the same action sequence in 71–73% of cases when changing language. They tended to route tasks through English, and instructions not to do so succeeded in less than 1% of cases. A Welsh agent test should therefore record translations, searches, tool calls, cost and human handover — not just the final answer.

In The Illusion of Cross-Lingual Safety, harmful requests in Twi, Hausa, Amharic and Swahili retained less than 10% of the English refusal signal across most tested language–model pairs. This is a preprint and the authors’ own study; it does not prove Welsh will behave identically. It demonstrates the need to measure safety with Welsh-written prompts, local cultural context, dialect and language switching.

A quiet week in Wales

Nation.Cymru/Nation.Wales, Newyddion S4C, BBC Cymru Fyw, Golwg360, Techiaith, Welsh Government sources and X were searched. No new Welsh technical release or formal development was found in the case of Y Dderwen after what was recorded on 7 August.

On 10 August, the Welsh Treasury again promoted Welsh–English transcription in Teams and Copilot, project grants and the Geraint and Gwyneth synthetic voices. Those programmes had already been announced and covered in the site’s Welsh Government article. It was not a new launch.

It is better to say the Welsh week was quiet than fill the gap with a general AI story in Wales. The real news in the second half of week 33 was comparative evidence: local language disappears in general benchmarks; transcription order can change how much a speech model learns; and compression, agentic use or safety adaptation can create new language disparities after training.

The next step

BavGround says: test locality, dialect and sources, not just standard language. Easper says: select living-language data that helps the model learn vocabulary and patterns, and return it to people for correction. The deployment papers say: test the system again after compression and putting it to work.

A credible measure of Welsh progress therefore needs source-based local questions; consensually collected recordings representing language diversity; several scoring methods; post-compression tests; an agent-action record; a Welsh-written safety test; and real speakers’ judgement.

‘Supports Welsh’ is not a result. It is a claim that must be tested in the language, the locality and the system people actually receive.