An anti-deepfake declaration may have been written by AI

A federal lawsuit over Minnesota’s “Use of Deep Fake Technology to Influence An Election” law is now directly dealing with the influence of AI. In a recent filing, attorneys challenging the law say an affidavit submitted to support it shows signs of containing AI-generated text. The Minnesota Reformer reports Attorney General Keith Ellison asked Stanford Social Media Lab founding director Jeff Hancock to make the submission, but the document filed includes non-existent sources that seem to have been hallucinated by ChatGPT or another large language model (LLM).

Hancock’s affidavit cites a 2023 study published in the Journal of Information Technology & Politics titled “The Influence of Deepfake Videos on Political Attitudes and Behavior.”

But according to the Reformer, there is no record of that study in the Journal of Information Technology & Politics or any other publication. Another source cited in Hancock’s declaration, “Deepfakes and the Illusion of Authenticity: Cognitive Processes Behind Misinformation Acceptance,” doesn’t seem to exist either.

Hancock did not respond to The Verge’s request for comment.

“The citation bears the hallmarks of being an artificial intelligence (AI) ‘hallucination,’ suggesting that at least the citation was generated by a large language model like ChatGPT,” lawyers for Minnesota state Rep. Mary Franson and Christopher Khols — a conservative YouTuber who goes by Mr Reagan — wrote in a filing. “Plaintiffs do not know how this hallucination wound up in Hancock’s declaration, but it calls the entire document into question, especially when much of the commentary contains no methodology or analytic logic whatsoever.”