Correction and explanation: findings from the Coleg Cymraeg report

UWTSD’s study, sponsored by the Coleg Cymraeg Cenedlaethol, offers a detailed examination of the main AI tools’ Welsh. The results are mixed.

Four people sharing ideas in a colourful newsroom

A 70-page report by Kara Lewis and a working group at the University of Wales Trinity Saint David (UWTSD) Welsh Language Services Centre, sponsored by the Coleg Cymraeg Cenedlaethol, has attracted widespread attention — including an article on Golwg360. Its message is twofold: ChatGPT, Copilot and similar tools can provide real help to Welsh-speaking students, but should not be relied on without someone checking the work.

The report’s findings

Good correction. After the team inserted deliberate errors into sample essays — incorrect mutations, English syntax, a/ac, mewn/yn — ChatGPT and Copilot corrected a substantial number.
Weak explanation. When asked why, explanations were not ‘detailed, accurate or consistent enough to be meaningful’ — and the tools made new Welsh errors while explaining, including inventing terms such as ‘hunanganolig’ and ‘theorïol’ instead of standard ones.
Unnatural idiom. The output still sounds like translated Welsh rather than natural Welsh.
Invented references. Of five academic sources suggested by ChatGPT, two were false or unconfirmed — yet all looked entirely credible.

One line that improves the result

One practical finding is the experiment on Primo Research Assistant, the AI search system used by Welsh university libraries. It was prompt language, not interface language, that made the difference. After adding one sentence — ‘Create the query using Welsh only’ — the search queries used no English and gave Welsh sources much higher priority. This suggests that explicitly requesting Welsh can substantially change the system’s behaviour.

Our full analysis of the report is in the Language and translation section.

Discussing the results

There will be an opportunity to hear more at the Pembrokeshire National Eisteddfod. The Coleg Cymraeg Cenedlaethol will hold a panel on artificial intelligence in Welsh-medium and bilingual education in the Societies Pavilion at 1 pm on Monday 3 August. Dr Cynog Prys of Bangor University will chair, with Dewi Bryn Jones of Bangor’s Language Technologies Unit, Catrin Bradley of UWTSD and Richard Buckley of QAA discussing the findings. Dewi Bryn Jones won the 2025 Science and Technology Medal.

The report was published on 1 June 2026; public attention grew as the Eisteddfod approached in late July.