Luth-2: a small model showing a way forward for Welsh

Luth-2 shows how post-training a small model for one language can bring large gains — but new research warns that tests must be written in the language itself, reflect local life and include human judgement.

Paper illustration of a small model connecting language communities through coloured ribbons — Model size is not all that matters. Luth-2 connects data, specialised training and French tests in one open process.

No new Welsh model was published this week. Nor did new public evidence settle the debate over Y Dderwen . But a useful example came from France: two small models showing how much can be gained by focusing on one language, one set of tasks and one transparent evaluation process.

The project is called Luth-2. Two models were published on 11 August, with around 0.8 billion and 1.88 billion parameters. Based on Qwen3.5, they were post-trained specifically for French. The models, training data, code, recipe and benchmarks were released together; the 2B model is available under Apache-2.0.

Size is part of the story. Such a model can run locally instead of always relying on a huge cloud service. But the bigger lesson is the method: they did not simply build a general French model. They created a mixture of mathematics, knowledge, code, instruction following and tool use; trained experts on different tasks; then trained one model to imitate what the experts had learned.

The results — and limitations

On the team’s own tests, Luth-2-2B is well ahead of its base model in several areas. On MGSM-rev2, a multilingual mathematics test, its score rose from 64.60 to 86.52. On IFEval it rose from 61.91 to 75.06. On HumanEval+, a coding test, it rose from 37.56 to 66.00. The team says the model leads similarly sized models on ten of the twelve selected tests.

That is impressive, but two things matter. First, these are the authors’ own results, not yet independently reproduced. Second, many original instructions were translated from English into French before the answers were regenerated. That is a practical way to expand data, but can also transfer English patterns of thought, questioning and measurement into the new language.

This is the key question for Welsh. Translating thousands of English tests could improve a model on those tests. But would it understand a council meeting’s register, mutation after a number, a local place name, northern or southern dialect, bilingual conversation, or the difference between natural Welsh and Welsh that is correct only on paper?

The answer is not to reject translation. It is to label it, balance it with Welsh-written data, and test what cannot be borrowed from English.

The research cycle in action

Luth-2 is a practical example of the idea on AIsteddfod’s homepage: create, test, inspect, improve, then test again.

The team began with a relatively small base model. They collected and filtered tasks, separated areas such as mathematics, code and instruction following, used specialised models to improve each area, measured results repeatedly, and published the recipe for others to inspect.

A Welsh project could follow the same cycle without copying every choice. It would start with the things we want a model to do: answer public-service questions, edit without erasing the author’s voice, use correct terminology, handle dialect, translate in context and switch languages naturally. Every failure would then become a new test or training example.

The aim would be to show exactly what the model can do in Welsh, where it fails, and what has changed after each cycle of testing and training.

A public service needs more than one score

A new paper from the Netherlands reinforces the lesson. Researchers and local-government staff developed a framework called Grip on LLMs to assess over 30 multilingual and Dutch models. They measured factuality, honesty, social bias, energy, cost and training-data transparency.

No model won in every area. Higher quality tended to cost more money and energy. A factual model was not necessarily honest when it did not know an answer. Nor was there a consistent relationship between social bias and quality score.

This is especially relevant to public bodies in Wales. ‘Which model scored highest?’ is insufficient. They must also ask: does it complete the task as well in Welsh as in English? Does it admit uncertainty? Does it treat groups fairly? What are the cost, energy use and evidence about the data?

Place and voice are part of language

Cultivar shows another weakness in multilingual tests. The team adapted a FLORES subset to local contexts and tested 32 open-weight models. According to the authors, US-based content was translated better than content from other locations, regardless of language.

Translating a test can change its language without changing its cultural context. Measuring Welsh properly requires Welsh settings: school, surgery, council, train, eisteddfod, playground, farm, studio and family conversation. Place, institution and register are part of meaning.

The same applies to voice. In a new study of synthetic-speech evaluation, linguists judged 860 examples on ten different dimensions. The authors found that traditional tools for predicting naturalness scores mainly focus on acoustic quality. Audio language models could notice more things, but were inconsistent and sensitive to instructions.

One ‘naturalness’ score is therefore insufficient for Welsh speech. Pronunciation, stress, rhythm, clarity, consistency, personal and place names, and Welsh–English switching need assessment. People who know the language need to do the listening.

Paper illustration of three researchers carefully holding voice and safety test cards at a tidy table — Good tests separate the questions: is the sound clear, the language correct, the context local, and the behaviour safe?

A model can recognise harm without responding safely

Two new papers about Indian languages raise a safety question. SurakshaEval contains human-written safety prompts in ten Indian languages and English. The authors found that strong multilingual models still over-refuse harmless requests, miss implicit bias or lose regional context — especially in native scripts.

Another paper, accepted at the COLM 2026 interpretability workshop, seeks the mechanism. In the studied model, the network represented harm similarly in its middle layers whether the request was English or an Indian language. But the act of writing a refusal appeared later, in a more specific part of the network. One author explained the result in a thread on X.

We should not assume the same circuit exists in every model. But the distinction is useful: a system may recognise a harmful request yet fail to produce the safe response in that language. Welsh testing should measure model understanding and final behaviour separately.

What we did not repeat

Nation.Cymru, Welsh news sources, X and the web were checked for developments after the week 32 article. No new formal evidence was found in the case of Y Dderwen, so we are not repeating the allegation and denial. The previous article remains a record of what was known on 7 August.

The next step for Welsh

Update: the second week 33 report, 14 August continues the story with new work on local language, dialect and live voice data, without repeating the results above.

Luth-2 makes one thing clear: progress in a particular language need not wait for a huge new base model. It can start with a small model, purpose-built data, clear tasks and honest tests.

But tests must go beyond translating an English benchmark. They need human-written Welsh for Welsh contexts; real voices and dialects; separate measures of accuracy, usefulness, safety, cost and fairness; and enough of the process published for others to reproduce it.

A small model was the week’s news. The process around it — data, training, testing, auditing and improvement — is the big lesson for Welsh.