Close Menu
Technology Mag

    Subscribe to Updates

    Get the latest creative news from FooBar about art, design and business.

    What's Hot

    Epic just won its Google lawsuit again, and Android may never be the same

    July 31, 2025

    DJI won’t sell you an Osmo 360 in the US — but these retailers will

    July 31, 2025

    Reddit wants to be a search engine now

    July 31, 2025
    Facebook X (Twitter) Instagram
    Subscribe
    Technology Mag
    Facebook X (Twitter) Instagram YouTube
    • Home
    • News
    • Business
    • Games
    • Gear
    • Reviews
    • Science
    • Security
    • Trending
    • Press Release
    Technology Mag
    Home » Publishers Target Common Crawl In Fight Over AI Training Data
    Business

    Publishers Target Common Crawl In Fight Over AI Training Data

    News RoomBy News RoomJune 14, 20243 Mins Read
    Facebook Twitter Pinterest LinkedIn Reddit WhatsApp Email

    Danish media outlets have demanded that the nonprofit web archive Common Crawl remove copies of their articles from past data sets and stop crawling their websites immediately. This request was issued amid growing outrage over how artificial intelligence companies like OpenAI are using copyrighted materials.

    Common Crawl plans to comply with the request, first issued on Monday. Executive director Rich Skrenta says the organization is “not equipped” to fight media companies and publishers in court.

    The Danish Rights Alliance (DRA), an association representing copyright holders in Denmark, spearheaded the campaign. It made the request on behalf of four media outlets, including Berlingske Media and the daily newspaper Jyllands-Posten. The New York Times made a similar request of Common Crawl last year, prior to filing a lawsuit against OpenAI for using its work without permission. In its complaint, the New York Times highlighted how Common Crawl’s data was the most “highly weighted data set” in GPT-3.

    Thomas Heldrup, the DRA’s head of content protection and enforcement, says that this new effort was inspired by the Times. “Common Crawl is unique in the sense that we’re seeing so many big AI companies using their data,” Heldrup says. He sees its corpus as a threat to media companies attempting to negotiate with AI titans.

    Although Common Crawl has been essential to the development of many text-based generative AI tools, it was not designed with AI in mind. Founded in 2007, the San Francisco–based organization was best known prior to the AI boom for its value as a research tool. “Common Crawl is caught up in this conflict about copyright and generative AI,” says Stefan Baack, a data analyst at the Mozilla Foundation who recently published a report on Common Crawl’s role in AI training. “For many years it was a small niche project that almost nobody knew about.”

    Prior to 2023, Common Crawl did not receive a single request to redact data. Now, in addition to the requests from the New York Times and this group of Danish publishers, it’s also fielding an uptick of requests that have not been made public.

    In addition to this sharp rise in demands to redact data, Common Crawl’s web crawler, CCBot, is also increasingly thwarted from accumulating new data from publishers. According to the AI detection startup Originality AI, which often tracks the use of web crawlers, more than 44 percent of the top global news and media sites block CCBot. Apart from BuzzFeed, which began blocking it in 2018, most of the prominent outlets it analyzed—including Reuters, the Washington Post, and the CBC—spurned the crawler in only the last year. “They’re being blocked more and more,” Baack says.

    Common Crawl’s quick compliance with this kind of request is driven by the realities of keeping a small nonprofit afloat. Compliance does not equate to ideological agreement, though. Skrenta sees this push to remove archival materials from data repositories like Common Crawl as nothing short of an affront to the internet as we know it. “It’s an existential threat,” he says. “They’ll kill the open web.”

    Share. Facebook Twitter Pinterest LinkedIn WhatsApp Reddit Email
    Previous ArticleGuliKit’s anti-drift Hall effect sticks are coming for your PS5, PS4, Xbox, and Switch Pro gamepads
    Next Article Picsart teams up with Getty to take on Adobe’s ‘commercially-safe’ AI

    Related Posts

    Everything You Wanted to Know About China’s Auto Industry Takeover

    July 31, 2025

    Trump Ends Tariff Exemption for Small Packages

    July 31, 2025

    US Senator Urges DHS to Probe Whether Agents Were Moved From Criminal Cases to Deportations

    July 31, 2025

    Mark Zuckerberg Details Meta’s Plan for Self-Improving, Superintelligent AI

    July 31, 2025

    Join Us for WIRED’s AI Power Summit

    July 31, 2025

    Meta’s AI Recruiting Campaign Finds a New Target

    July 30, 2025
    Our Picks

    DJI won’t sell you an Osmo 360 in the US — but these retailers will

    July 31, 2025

    Reddit wants to be a search engine now

    July 31, 2025

    Tim Cook says Apple is ‘open to’ AI acquisitions

    July 31, 2025

    Everything You Wanted to Know About China’s Auto Industry Takeover

    July 31, 2025
    • Facebook
    • Twitter
    • Pinterest
    • Instagram
    • YouTube
    • Vimeo
    Don't Miss
    News

    Apple shipped its 3 billionth iPhone

    By News RoomJuly 31, 2025

    Apple has been the target of a fair amount of criticism over the past year,…

    Trump Ends Tariff Exemption for Small Packages

    July 31, 2025

    Apple says Trump’s tariffs are adding another $1 billion to its costs

    July 31, 2025

    US Senator Urges DHS to Probe Whether Agents Were Moved From Criminal Cases to Deportations

    July 31, 2025
    Facebook X (Twitter) Instagram Pinterest
    • Privacy Policy
    • Terms of use
    • Advertise
    • Contact
    © 2025 Technology Mag. All Rights Reserved.

    Type above and press Enter to search. Press Esc to cancel.