AI Tools Are Secretly Training on Real Images of Children

Over 170 images and personal details of children from Brazil have been repurposed by an open-source dataset without their knowledge or consent, and used to train AI, claims a new report from Human Rights Watch released Monday.

The images have been scraped from content posted as recently as 2023 and as far back as the mid-1990s, according to the report, long before any internet user might anticipate that their content might be used to train AI. Human Rights Watch claims that personal details and photos of these children were gathered by data repository Common Crawl and then URLs that linked to them were included in LAION-5B, a dataset that helps to train data for AI startups.

“Their privacy is violated in the first instance when their photo is scraped and swept into these datasets. And then these AI tools are trained on this data and therefore can create realistic imagery of children,” says Hye Jung Han, children’s rights and technology researcher at Human Rights Watch and the researcher who found these images. “The technology is developed in such a way that any child who has any photo or video of themselves online is now at risk because any malicious actor could take that photo, and then use these tools to manipulate them however they want.”

LAION-5B is based on Common Crawl—a repository of data that was created by scraping the web and made available to researchers—and has been used to train several AI models, including Stability AI’s Stable Diffusion image generation tool. Created by the German nonprofit organization LAION, the dataset is openly accessible and now includes links to more than 5.85 billion pairs of images and captions, according to its website. LAION says that it has taken down the links to the images flagged by Human Rights Watch.

The images of children that researchers found came from mommy blogs and other personal, maternity, or parenting blogs, as well as stills from YouTube videos with small view counts, seemingly uploaded to be shared with family and friends.

“Just looking at the context of where they were posted, they enjoyed an expectation and a measure of privacy,” Hye says. “Most of these images were not possible to find online through a reverse image search.”

LAION spokesperson Nathan Tyler says the organization has already taken action. “LAION-5B were taken down in response to a Stanford report that found links in the dataset pointing to illegal content on the public web,” he says, adding that the organization is currently working with “Internet Watch Foundation, the Canadian Centre for Child Protection, Stanford, and Human Rights Watch to remove all known references to illegal content.”

YouTube’s terms of service do not allow scraping except under certain circumstances; these instances seem to run afoul of those policies. “We’ve been clear that the unauthorized scraping of YouTube content is a violation of our Terms of Service,” says YouTube spokesperson Jack Maon, “and we continue to take action against this type of abuse.”