Close Menu
Technology Mag

    Subscribe to Updates

    Get the latest creative news from FooBar about art, design and business.

    What's Hot

    The Fujifilm X-E5 is a simple, familiar, and impressive travel camera

    July 27, 2025

    CookUnity Cracked the Code on Meal Delivery By Using … Gasp … Chefs

    July 27, 2025

    The Verge’s 2025 back-to-school shopping guide

    July 27, 2025
    Facebook X (Twitter) Instagram
    Subscribe
    Technology Mag
    Facebook X (Twitter) Instagram YouTube
    • Home
    • News
    • Business
    • Games
    • Gear
    • Reviews
    • Science
    • Security
    • Trending
    • Press Release
    Technology Mag
    Home » Cloudflare will now block AI crawlers by default
    News

    Cloudflare will now block AI crawlers by default

    News RoomBy News RoomJuly 1, 20253 Mins Read
    Facebook Twitter Pinterest LinkedIn Reddit WhatsApp Email

    The major internet architecture provider Cloudflare will now block known AI web crawlers by default to prevent them from “accessing content without permission or compensation,” according to an announcement on Tuesday. With the change, Cloudflare will start asking new domain owners whether they want to allow AI scrapers, and will even let some publishers implement a “Pay Per Crawl” fee.

    The Pay Per Crawl program will let publishers set a price for AI scrapers to access their content. AI companies can then view pricing and choose whether to register for the “Pay Per Crawl” fee or turn away. This is only available for “a group of some of the leading publishers and content creators” for now, but Cloudflare says it will ensure “AI companies can use quality content the right way — with permission and compensation.”

    Cloudflare has been helping domain owners fight AI crawlers for a while now. The company started letting websites block AI crawlers in 2023, but it only applied to ones that abide by a site’s robots.txt file, the unenforceable agreement that signals whether bots can scrape its content. Cloudflare began allowing websites to block “all” AI bots last year — whether they respect a site’s robots.txt file or not — and now this setting is enabled by default for new Cloudflare customers. (The company identifies scrapers to block by comparing them to its list of known AI bots.) Cloudflare also rolled out a feature in March that sends web-crawling bots into an “AI Labyrinth” to deter them from scraping sites without permission.

    Several major publishers and online platforms, including The Associated Press, The Atlantic, Fortune, Stack Overflow, and Quora, are on board with Cloudflare’s new AI crawler restrictions, as websites contend with a future where more people are finding information through AI chatbots, rather than search engines. “People trust the AI more over the last six months, which means they’re not reading original content,” Cloudflare CEO Matthew Prince said during the Axios Live event last week.

    Additionally, Cloudflare says it’s working with AI companies to help verify their crawlers and allow them to “clearly state their purpose,” such as whether they’re using the content for training, inference, or search. Website owners can then review this information and determine which crawlers to let in.

    “Original content is what makes the Internet one of the greatest inventions in the last century, and we have to come together to protect it,” Prince said in the press release. “AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate.”

    Share. Facebook Twitter Pinterest LinkedIn WhatsApp Reddit Email
    Previous ArticleMicrosoft Says Its New AI System Diagnosed Patients 4 Times More Accurately Than Human Doctors
    Next Article The Lenovo Chromebook Plus 14 Is the Best Chromebook Ever Made

    Related Posts

    The Fujifilm X-E5 is a simple, familiar, and impressive travel camera

    July 27, 2025

    The Verge’s 2025 back-to-school shopping guide

    July 27, 2025

    Apple beta season is here

    July 27, 2025

    Here are the laptops I’d tell any parent to consider for their back-to-school student

    July 26, 2025

    OnePlus Nord 5 review: selfie-centric midranger

    July 26, 2025

    Pebble is officially Pebble again

    July 25, 2025
    Our Picks

    CookUnity Cracked the Code on Meal Delivery By Using … Gasp … Chefs

    July 27, 2025

    The Verge’s 2025 back-to-school shopping guide

    July 27, 2025

    Razer’s Pro Click V2 Vertical Is the Ergonomic Gaming Mouse You’re Looking For

    July 27, 2025

    Apple beta season is here

    July 27, 2025
    • Facebook
    • Twitter
    • Pinterest
    • Instagram
    • YouTube
    • Vimeo
    Don't Miss
    Science

    The ICJ Rules That Failing to Combat Climate Change Could Violate International Law

    By News RoomJuly 27, 2025

    If a country fails to take decisive action to protect the planet from climate change,…

    Nemo’s Updated Dagger Osmo Tent Has Nicer Fabric and Better Design Details

    July 26, 2025

    Here are the laptops I’d tell any parent to consider for their back-to-school student

    July 26, 2025

    Do You Need a Barbecue Knife?

    July 26, 2025
    Facebook X (Twitter) Instagram Pinterest
    • Privacy Policy
    • Terms of use
    • Advertise
    • Contact
    © 2025 Technology Mag. All Rights Reserved.

    Type above and press Enter to search. Press Esc to cancel.