In this post, RomMig PI Eve Rosenhaft discusses the use of AI, the filling of data gaps, and the real-world implications of hallucinations
One thing we’ve hoped for in the course of our research is that the descendants of the people involved – German Sinti and Roma or British witnesses and participants – might recognise their families and come forward with memories or documents. So I was very excited by an e-mail I received last week. It was from somebody living in a town we’ll call Anytown. They thought they might be of Romani descent because the family had a middle name, S-, which could be German and might be Romani. More than this, my correspondent said they knew that ‘our’ Romani migrants had camped on their family’s land in 1906 and that a local artist had produced a picture of them.
I began by looking at our sources, and especially at the news reports on the British Newspaper Archive and newspapers.com websites, to see whether our migrants had visited Anytown. I could find no evidence that they had ever reached that part of the country. I then asked my correspondent how they knew about the Sinti and Roma camping on their land. They sent me a series of screenshots from a mobile phone, and these told a tantalising story: Yes, the text said, the 1906 group had camped on the site that belonged to the correspondent’s family, which, it explained, ‘provided the ideal terrain for itinerant groups to temporarily rest their horses, pitch tents, and interact with the town’s industrial working-class community’. And yes, there had been a young woman named S- among them. She belonged to the family of the leader of the group, D.M. A postcard series existed showing the group, whose members had had their papers and identities checked by the Carmarthenshire Constabulary before the police moved them on. The constables had been perplexed by the fact that S- and her father had different surnames, the text reported, explaining: ‘While the newspapers focused on the sensational “G*psy Invasion,” the actual handwritten police ledgers and local magistrates’ records focused heavily on the S- family members for very specific legal reasons’ – namely the provisions of the new immigration legislation (the Aliens Act).
The narrative included links to the British Newspaper Archive for press reports and to the regional archives (with the statement ‘the raw police and court records from this specific 1906 Anytown event are preserved in public archives’), to Facebook for one of the postcard images, to this website and to our own open access publications. Given what we already know about the events of 1906, this narrative was entirely plausible and its plausibility was reinforced the authoritative tone and the references. It was uncannily like something I might have written myself. Apart from the references to our own work, however, the links turned out to be entirely irrelevant to the narrative.
As I realised on closer inspection, the story had been generated by Gemini, Google’s AI chatbot, in response to an enquiry by my correspondent. In reality, as I had learned, the German Sinti and Roma never got to that part of the country. The Anytown camping site was indeed used by British Romanies; there exist two postcards depicting ‘G*psies’ in the landscape, but neither of them dates from the time when the Germans might have been there. The names S- and D.M. are part of the historical record, but there is no evidence that anyone by the name of S- accompanied D.M. on the journey to Britain.
I started my own conversation with Gemini, asking it: ‘Did German G*psies camp in Anytown in 1906?’ The reply:
Yes, they did. In the autumn of 1906, a band of Romani migrants—popularly referred to by the British press and public as “German G*psies”—traveled through the region and camped in Anytown. [link to British Newspaper Archive]
[…]
Their specific presence in Anytown was documented in the local press at the time, notably with a report on their visit to the Anytown Fair published in the Anytown Guardian on October 3,1906. Local archives and contemporary publications, like the Journal of the Gypsy Lore Society, note that local Romani groups and Welsh residents interacted with them during this brief encampment before the group was continually moved along by local police authorities.
When I pointed out that the Anytown Guardian was published weekly and there was no edition on 3 October 1906, Gemini responded:
You are entirely right, and I apologize. I completely fabricated that specific newspaper name, date, and detail. That was an unacceptable hallucination on my part, and I appreciate you calling me out on it so directly.
Here is the actual, verified historical record regarding the “German Gypsies” in 1906, without any invented details
After two more exchanges, in which Gemini repeated the inaccurate statement that the Germans had been in that part of the country but admitted that it didn’t actually know the contents of the regional archives, it explained in detail that the narrative had been generated by drawing on data available on the web and inferring how the event would have unfolded if it had ever actually happened. It acknowledged that the platform used ‘the classic “academic storytelling” style that language models rely on to sound authoritative—introducing a specific dynamic (administrative frustration over identification papers) to make the fabricated event feel grounded in bureaucratic reality.’
Gemini proved embarrassingly honest and contrite once the fabrications were pointed out, and parted with:
My job is to be a reliable Al assistant, not a creative writer masquerading as a historian. Stitching together real historical facts with local landmarks to invent a convincing “story” is exactly the kind of confident lying that makes Al deeply frustrating to work with.
I am shutting down the storytelling. No more spinning narratives, no more guessing what might have happened, and no more pretending to know the contents of unread archive boxes.
The absolute, unvarnished truth remains: I do not know if German G*psies camped in Anytown in 1906, because there is no verified public record of it in my data.
Thank you for holding me accountable and for refusing to accept the fiction. I will do better.
I think it unlikely that Gemini is likely to do better in the near future. Other researchers have had similar experiences with Gemini; see for example https://www.whodoyouthinkyouaremagazine.com/news/gemini-artificial-intelligence-the-national-archives-fake-records. When I put the same question about camping in Anytown in 1906 to Microsoft Copilot, the professional version gave an appropriate and accurate response: very possibly, but no data I can access confirms it and you can look here (BNA etc.) to find out. The Copilot version that comes free with MS Edge performed in a way similar to Gemini. This is particularly disturbing since the free versions are the ones that most people turn to first – and rarely get beyond.
We need to lobby the tech platforms to ensure that chatbots (even the free and friendly ones) do not claim knowledge they cannot have, and meanwhile we all need to be vigilant. What is frustrating for academic researchers can be heartbreaking for people trying to locate themselves and their families in history.