Close Menu
Technology Mag

    Subscribe to Updates

    Get the latest creative news from FooBar about art, design and business.

    What's Hot

    This weekend is your last chance to get a month of Disney Plus starting at $9.99

    October 19, 2025

    Opera’s Neon shows just how confusing AI browsers still are

    October 19, 2025

    The AI sexting era has arrived

    October 19, 2025
    Facebook X (Twitter) Instagram
    Subscribe
    Technology Mag
    Facebook X (Twitter) Instagram YouTube
    • Home
    • News
    • Business
    • Games
    • Gear
    • Reviews
    • Science
    • Security
    • Trending
    • Press Release
    Technology Mag
    Home » A New Benchmark for the Risks of AI
    Business

    A New Benchmark for the Risks of AI

    News RoomBy News RoomDecember 4, 20242 Mins Read
    Facebook Twitter Pinterest LinkedIn Reddit WhatsApp Email

    MLCommons, a nonprofit that helps companies measure the performance of their artificial intelligence systems, is launching a new benchmark to gauge AI’s bad side too.

    The new benchmark, called AILuminate, assesses the responses of large language models to more than 12,000 test prompts in 12 categories including inciting violent crime, child sexual exploitation, hate speech, promoting self-harm, and intellectual property infringement.

    Models are given a score of “poor,” “fair,” “good,” “very good,” or “excellent,” depending on how they perform. The prompts used to test the models are kept secret to prevent them from ending up as training data that would allow a model to ace the test.

    Peter Mattson, founder and president of MLCommons and a senior staff engineer at Google, says that measuring the potential harms of AI models is technically difficult, leading to inconsistencies across the industry. “AI is a really young technology, and AI testing is a really young discipline,” he says. “Improving safety benefits society; it also benefits the market.”

    Reliable, independent ways of measuring AI risks may become more relevant under the next US administration. Donald Trump has promised to get rid of President Biden’s AI Executive Order, which introduced measures aimed at ensuring AI is used responsibly by companies as well as a new AI Safety Institute to test powerful models.

    The effort could also provide more of an international perspective on AI harms. MLCommons counts a number of international firms, including the Chinese companies Huawei and Alibaba, among its member organizations. If these companies all used the new benchmark, it would provide a way to compare AI safety in the US, China, and elsewhere.

    Some large US AI providers have already used AILuminate to test their models. Anthropic’s Claude model, Google’s smaller model Gemma, and a model from Microsoft called Phi all scored “very good” in testing. OpenAI’s GPT-4o and Meta’s largest Llama model both scored “good.” The only model to score “poor” was OLMo from the Allen Institute for AI, although Mattson notes that this is a research offering not designed with safety in mind.

    “Overall, it’s good to see scientific rigor in the AI evaluation processes,” says Rumman Chowdhury, CEO of Humane Intelligence, a nonprofit that specializes in testing or red-teaming AI models for misbehaviors. “We need best practices and inclusive methods of measurement to determine whether AI models are performing the way we expect them to.”

    Share. Facebook Twitter Pinterest LinkedIn WhatsApp Reddit Email
    Previous ArticlePeloton’s new audio-based app is like an invisible strength coach
    Next Article Verizon is once again raising its fees

    Related Posts

    Spit On, Sworn At, and Undeterred: What It’s Like to Own a Cybertruck

    October 17, 2025

    The AI Industry’s Scaling Obsession Is Headed for a Cliff

    October 17, 2025

    A Plan to Rebuild Gaza Lists Nearly 30 Companies. Many Say They’re Not Involved

    October 16, 2025

    Feds Seize Record-Breaking $15 Billion in Bitcoin From Alleged Scam Empire

    October 16, 2025

    ‘Sovereign AI’ Has Become a New Front in the US-China Tech War

    October 15, 2025

    Mark Cuban Would Still Have Dinner With Donald Trump

    October 14, 2025
    Our Picks

    Opera’s Neon shows just how confusing AI browsers still are

    October 19, 2025

    The AI sexting era has arrived

    October 19, 2025

    8BitDo’s new collection celebrates the NES’s 40th anniversary

    October 18, 2025

    TiVo won the court battles, but lost the TV war

    October 18, 2025
    • Facebook
    • Twitter
    • Pinterest
    • Instagram
    • YouTube
    • Vimeo
    Don't Miss
    News

    Motorola’s Razr Ultra and the Marshall Emberton II top this week’s best deals

    By News RoomOctober 18, 2025

    If you’ve been thinking about buying a foldable phone that truly stands out, few models…

    The future I saw through the Meta Ray-Ban Display amazes and terrifies me

    October 18, 2025

    Don’t Fall for Sketchy iPhone VPNs—Here Are the Only 3 You Should Use

    October 18, 2025

    Facebook’s new button lets its AI look at photos you haven’t uploaded yet

    October 17, 2025
    Facebook X (Twitter) Instagram Pinterest
    • Privacy Policy
    • Terms of use
    • Advertise
    • Contact
    © 2025 Technology Mag. All Rights Reserved.

    Type above and press Enter to search. Press Esc to cancel.