TikTok’s AI tool accidentally let you put Hitler’s words in a paid actor’s mouth

TikTok mistakenly posted a link to an internal version of its new AI digital avatar tool without guardrails, letting users create videos that say just about anything. The hiccup was first spotted by CNN and allowed the outlet to generate videos containing quotes from Hitler and a message telling people to drink bleach, among other phrases. TikTok has since taken this version of the tool down, while the version TikTok intended to launch remains available.

Launched earlier this week, TikTok’s Symphony Digital Avatars let businesses generate ads using the likeness of paid actors. It also uses AI-powered dubbing that lets advertisers enter a script to make the avatars say what they want within TikTok’s guidelines. Even though only users with a TikTok Ads Manager account can access this tool, the version CNN found let anyone with a personal account try.

In a statement to The Verge, TikTok spokesperson Laura Perez says TikTok has resolved the “technical error” that “allowed an extremely small number of users to create content using an internal testing version of the tool for a few days.”

When CNN discovered the internal tool, it let the outlet generate videos reciting Osama bin Laden’s “Letter to America,” a white supremacy slogan, and a video telling people to vote on the wrong day. None of the videos CNN produced had a watermark disclosing that the video is AI-generated, which is something the proper version of TikTok’s Symphony Digital Avatars does.

CNN didn’t post the videos it created to TikTok, but Perez notes that if it had, the content “would have been rejected for violating our policies.” Even though TikTok has since taken this version of its tool down, it calls into question whether people will find other ways to abuse the digital avatar creator — and if TikTok is ready for it.