‘Artificial Intelligence (AI), Welsh and Higher Education’

The most detailed study yet of how the main AI tools handle real Welsh — from grammar and mutations to style and references.

In June 2026, Kara Lewis and the team at the University of Wales Trinity Saint David (UWTSD) Welsh Language Services Centre, sponsored by the Coleg Cymraeg Cenedlaethol, published a 70-page report that did something new: it gave real Welsh academic texts to ChatGPT, Copilot, Gemini, Claude and Perplexity, and assessed the results as a language tutor would.

For AIsteddfod, this is the most important qualitative benchmark published this year. Our quantitative evaluations measure how much models get right; this report shows how they fail.

The three questions

The research addresses three questions. Here is a summary of the report’s answers:

What was tested

The team inserted deliberate errors into sample essays in psychology, sport and Welsh literature: incorrect mutations, English syntax, confusion between and and and, between mewn and yn, and errors in negative forms. The tools were then asked to correct and explain the errors. These were the main findings:

• Correction is good; explanation is weak. ChatGPT and Copilot corrected ‘a substantial number of language and grammar errors’, but ‘the explanations offered… are not detailed, accurate or consistent enough to be meaningful’. For example, both tools corrected a mutation accurately, but only one explained why (‘soft mutation when an adjective follows predicative “yn”’).
• The tools themselves make Welsh errors when explaining. «"Apps" yw benthyciad o'r Saesneg» — emphatic syntax where no emphasis is needed — and coinages such as ‘hunanganolig’ (Claude), ‘theorïol’ (Copilot) and ‘ymddygiadoliaeth’ (Copilot) instead of the standard terms.
• Summarisation is good, but information can be lost. In trying to improve readability, tools may omit information when summarising, and may correct quotations ‘without any guarantee that the corrections are based on the original sources’.
• Fabricated references. When ChatGPT was asked to suggest sources for the report itself, two of the five sources were either fabricated or unconfirmed, and all were presented ‘in an entirely credible manner’.

The effect of prompt language

The most practically useful section is the series of experiments on Primo Research Assistant (PRA), an AI search interface using GPT-4.1 Mini, used by every university library in Wales through the WHELF consortium:

• The interface language had little effect on output.
• The prompt language does have an effect: giving instructions in Welsh leads to Welsh sources being found, but English sources may still take priority in fields where more English material is available, because of metadata, web traffic and titles carrying ‘greater weight’ in the catalogue.
• One sentence can change the result: after adding ‘Create the query using Welsh only’ to the end of the prompt, English disappeared from the Boolean query, and Welsh sources received much higher priority. The approach worked regardless of the original prompt’s language.

This captures one of the field’s major lessons: Welsh is not necessarily absent from the systems, but it needs to be requested explicitly.

Wales at the heart of the technology

Wales has helped create the technology reshaping the world, but our task now is to ensure that Welsh can take full advantage of its benefits.

— the report, noting that a Welshman, Llion Jones, was a co-author of the Transformer architecture (Vaswani et al., 2017)

The report also notes that the Welsh Government has a data-sharing partnership with OpenAI, and is working with Microsoft to enable Welsh prompts in Copilot — ‘ahead of many larger languages’, in Dr Jeremy Evas’s words.

Implications for AIsteddfod

At the end of the report, one sentence captures the reason for establishing AIsteddfod:

The question is not ‘whether higher education should engage with AI’ — that is already an inevitable reality. The question is how to do so in a way that… ensures Welsh does not lose out as education moves further into the digital age.

That is why our leaderboard treats idiom and naturalness separately from grammar, and why we publish details of our testing methods, including the prompts and harness. This is precisely the kind of transparency the report calls for.

Publication details