Getting better Welsh from AI tools
A practical guide summarising the Coleg Cymraeg Cenedlaethol report’s experiments on Primo Research Assistant and the main AI tools.
UWTSD’s report yn tests, rather than assumes, how users’ language choices affect the machine’s response. These are the findings and the practical steps that follow.
1. The interface language is not what makes the difference
The researchers repeated the same queries using different accounts and interface settings. The result was: ‘The interface language has no significant effect on output.’ Choosing a Welsh interface matters for the language’s status and the user experience, but does not change the behaviour of the model underlying the service.
2. Prompt language matters — but it is not the whole story
When Primo Research Assistant, which uses GPT-4.1 Mini, received Welsh instructions, its searches found Welsh sources. With English instructions, only English sources were returned. But even when asking in Welsh, English sources are prioritised in fields with less Welsh material, because of metadata, web traffic, and titles that are ‘more common and carry greater weight in the catalogue’. The same pattern appeared in a ChatGPT case study: a Welsh question about Dr Aneirin Karadog received a Welsh answer, but used mixed sources, including English versions of bilingual websites such as those of UWTSD and the Eisteddfod.
3. The instruction that makes a difference
Add one sentence at the end of your prompt:
‘Create the query using Welsh only.’
In the experiments, English disappeared from the Boolean query, and Welsh sources were given much higher priority. This worked regardless of the original prompt’s language: adding ‘Use only Welsh when making the query’ to an English prompt produced the same result.
The broader lesson, in the report’s words: if you want AI to restrict itself to Welsh sources, or prioritise Welsh versions of bilingual websites, ‘this should be stated directly in the instructions. Remember that AI does not read between the lines or make assumptions; it depends on the precise requirements it is given.’
4. Do not trust references without checking them
When asked to suggest academic sources, ChatGPT presented five. Two were either fabricated or unconfirmed, including a book Routledge had never published, and all were presented ‘in an entirely credible manner’. The latest versions of the tools try to express uncertainty, but have no awareness — ‘they do not know when they do not know’. Check every reference, every time. The report suggests the following prompt:
‘Can you find genuinely valid and reliable academic sources on [the topic]? Include links and publication details.’
5. The advice in brief
- Ask in Welsh if you want Welsh sources and a Welsh tone — but do not rely on that alone.
- State your language requirements clearly: ‘using Welsh only’, ‘prioritise Welsh sources’.
- Read every answer carefully, looking for English syntax and non-standard terms — these are the typical errors.
- Check every reference against the original source.
- You, not the machine, are responsible for the final content.