When the judges called a learner AI

An allegation, a denial and an unanswered question at the Eisteddfod — and lessons from Yiddish and Hindi research about data, evidence and testing AI in smaller languages.

Paper illustration of a writer holding a manuscript and facing a geometric machine shape — The debate over *Y Dderwen* now includes a clear allegation, a clear denial — and no public evidence settling the matter.

In three days, the week’s biggest Welsh AI story changed completely. It began with judges saying they suspected a machine had created a novel. Then the author came forward: a Welsh learner, a published author in English, and a woman who stated plainly that she had used no AI.

The author is Romy Wood. Y Dderwen is the novel. Lithodora was the pseudonym on the entry for the Daniel Owen Memorial Prize. ‘I just wrote a novel,’ Wood told Newyddion S4C on 6 August. ‘I haven’t used any AI.’

Wood had begun learning Welsh four years earlier. She said she expected her language to sound odd, fragmented or less fluent to native speakers. She had not expected to be accused of being AI on the Pavilion stage.

The judges did not change their view. Meinir Wyn Edwards, Lleucu Roberts and Llwyd Owen said they regretted causing distress, but were unanimous that ‘some form of artificial intelligence’ had created the work. The Eisteddfod said its judges were independent, but feedback should be respectful and constructive. The following day, the President of the Eisteddfod Court, Nic Parry, said the judges had been ‘treated unfairly’.

So this is the position: a serious allegation, a direct denial, the unanimous view of three judges and an institutional response. What we do not have is public proof capable of establishing who is right.

What is known — and what is not

Some facts are clear. Nine novels were submitted. Y Dderwen came last. Rebecca Thomas won with Ynys Gwales. Eisteddfod rules prohibit using AI to create an entry unless a competition states otherwise. The judges said the expression in Y Dderwen was garbled and error-ridden, and suspected AI was responsible, according to Newyddion S4C’s first report on 4 August.

But no model name, prompt record, document editing history, detector score, forensic method or admission was published. The manuscript was not released for others to examine. Wood’s denial alone is not technical proof that no AI tool was involved. The judges’ opinion, however firm, is not technical proof that AI was involved.

That distinction is central. An allegation should be called an allegation and a denial a denial. Neither should become an established fact because no more convenient answer is available.

When can a learner’s writing look machine-generated?

The case raises an uncomfortable question for Welsh culture. What happens when someone learning the language writes something that looks, to an experienced reader, like machine translation or generated text?

A learner may transfer word order, idiom or rhythm from another language. Mutations may be inconsistent. A sentence may be understandable but unnatural. Models also do these things — but they are not the only ones.

This was precisely the doubt Rhys Llwyd raised on X. He opposed AI use for creative prose, but argued that modern AI can produce correct, readable Welsh. If the text was so garbled, that alone did not prove AI — or AI alone — was responsible.

Poor text can be human-written. Fluent text can be machine-generated. Both can mix human writing, translation, editing, spellchecking, dictation and rewriting. Surface quality is evidence of literary merit; it is not a reliable digital signature of authorship.

Fluency is not mastery

New research this week shows why. The publication of M-GATE on arXiv on 4 Augustintroduced a benchmark of grammar, translation and tokeniser efficiency across 30 languages. It tested over 50 models in more than 80 configurations.

On the difficult grammar test, the best model’s MCC was only 0.36. Twenty-nine of the 53 models did no better than random guessing, with a tendency to accept ungrammatical sentences. Yet the same model families could translate much more convincingly.

That says nothing specific about the author of Y Dderwen. It says something generally important: a model can create fluent text without mastering a language’s structure. Conversely, a learner can have a real creative idea before mastering every grammatical pattern.

The closest comparison: Yiddish

A more direct comparison arrived on 6 August. A research team published MameLoshnLM, the first generative model with 8 billion parameters built specifically for Yiddish. Like Welsh, Yiddish has a rich literary tradition but a relatively limited digital presence.

The team did not assume everything labelled Yiddish in a web corpus was good Yiddish. They audited mC4’s Yiddish section. Only 42.2% of documents were classified as authentic Yiddish from known native sources. At least 29.8% appeared machine-translated, and 21.9% were misidentified Hebrew.

They then built Oytser: 915 million words, or 5.28 billion tokens, from books, forums, news, Wikipedia and other sources. The mixture matters. It includes not just standard web language but literature, contemporary discussion and several registers.

They also built Kashes, a nine-task benchmark testing translation, part-of-speech tagging, lemmatisation, parsing, transliteration, entity recognition and comprehension. Across 14 evaluation results, MameLoshnLM averaged 62.6, compared with 56.8 for its Llama 3.1 8B base model. On English-to-Yiddish translation it was over 11 COMET points ahead of its nearest competitor.

The The model weights are available under CC BY-NC 4.0. That does not mean all training data is freely available: it used a mixture of public-domain, licensed, non-commercial research material and specific agreements. The benchmark was still listed as forthcoming in the repository when checked.

The lesson for Welsh: before building a model, audit the data claiming to be Welsh. How much is native Welsh? How much is translation? How much is another language? Which registers, dialects and voices are missing? And when measuring the model, test what makes Welsh Welsh — not just a translation of an English test.

Paper illustration of community researchers listening to voice waves and arranging data fragments towards an open network — Dataset and model size are not the only questions: who is in the data, what is being tested, and how does the whole system work?

From data to service: the Hindi test

FormBharo, also published on 6 August, shows another side of the problem. It is a Hindi voice agent that fills in a form over a telephone call for maternal and child health services. It combines language models with explicit rules to verify information and manage conversation flow.

The researchers created 3,760 multi-turn tests across 960 simulated calls using human-recorded Hindi audio. Given accurate transcripts, some parts of the system worked extremely well: GPT-5.5 reached 99.8% accuracy on per-turn information extraction. But it was not the best model at completing the whole form.

When accurate transcripts were replaced with erroneous transcripts from real speech, form-completion success fell by up to around 41 percentage points. A hearing error can become an extraction error, then a wrong question, then incorrect data. Or a good rule can catch the error before it spreads.

This is directly relevant to Welsh voice services. A speech-recognition score alone does not show whether a system can complete a task. Tests need telephone noise, place names, accents and dialects, Welsh–English switching, clarification requests and handover to a person.

Millions of records, few voices

A third comparison concerned voice data. Hypa Intelligence published an analysis of Hypa-Voices on 3 August: 2,979,073 audio–text records in 22 languages, including Ibibio, Tiv, Efik, Yoruba, Hausa and Igbo.

The headline number is impressive. But the audit said 71.8% of the collection was translation. The top 10% of identified speakers accounted for 59.6% of audio hours, and the top 25% for 89.9%. Millions of rows do not necessarily mean millions of diverse voices.

The audit published no ASR or TTS score. Dataset size does not prove a model understands people underrepresented in it. A Welsh data card should show dialect, age, consent, provenance, repetition, each speaker’s contribution and how training and test data were separated to keep the same voice out of both.

What should the Eisteddfod do?

The debate over Y Dderwen shows that a one-line ban is not enough. A creative competition needs to say what ‘using AI’ means.

Is spellchecking allowed? What about grammar correction? Translation? Speech-to-text dictation? Asking for plot suggestions? Rewriting a paragraph? Generating the whole work? Accessibility tools?

A process is needed too. Where there is suspicion, it should be specified precisely, the entrant asked to respond, evidence of the writing process considered, reliance on one unknown detector avoided, and an appeal offered. The work’s quality should be judged honestly, without presenting its origin as fact without sufficient grounds.

This does not mean accepting every denial. Nor does it mean ignoring three judges’ professional opinion. It means authorship, literary quality and respect for an entrant are three different questions.

The end of the week

At the end of the week, the public evidence does not tell us whether AI was used to create Y Dderwen. We know Romy Wood denies it. We know the three judges maintain their view. We know the Eisteddfod defends judicial independence and asks for respectful feedback. And we know the President of the Court has defended the panel.

International research offers a better way to think about this. The Yiddish model says: audit data provenance and authenticity. The Hindi test says: test the whole system, not just individual components under ideal conditions. M-GATE says: do not mistake fluency for mastery. Hypa-Voices says: do not mistake size for diversity.

And the Eisteddfod story adds the most human lesson: do not mistake suspicion for proof — especially when a learner’s voice is at stake.