Close Menu
Technology Mag

    Subscribe to Updates

    Get the latest creative news from FooBar about art, design and business.

    What's Hot

    NYT reports Elon Musk’s drug use went ‘well beyond occasional’ in 2024

    May 30, 2025

    Remove Body Hair From Everywhere With the Fur 2-in-1 Trimmer

    May 30, 2025

    Here’s where you can buy microSD Express cards for the Switch 2

    May 30, 2025
    Facebook X (Twitter) Instagram
    Subscribe
    Technology Mag
    Facebook X (Twitter) Instagram YouTube
    • Home
    • News
    • Business
    • Games
    • Gear
    • Reviews
    • Science
    • Security
    • Trending
    • Press Release
    Technology Mag
    Home » Meta got caught gaming AI benchmarks
    News

    Meta got caught gaming AI benchmarks

    News RoomBy News RoomApril 7, 20254 Mins Read
    Facebook Twitter Pinterest LinkedIn Reddit WhatsApp Email

    Over the weekend, Meta dropped two new Llama 4 models: a smaller model named Scout, and Maverick, a mid-size model that the company claims can beat GPT-4o and Gemini 2.0 Flash “across a broad range of widely reported benchmarks.”

    Maverick quickly secured the number-two spot on LMArena, the AI benchmark site where humans compare outputs from different systems and vote on the best one. In Meta’s press release, the company highlighted Maverick’s ELO score of 1417, which placed it above OpenAI’s 4o and just under Gemini 2.5 Pro. (A higher ELO score means the model wins more often in the arena when going head-to-head with competitors.)

    The achievement seemed to position Meta’s open-weight Llama 4 as a serious challenger to the state-of-the-art, closed models from OpenAI, Anthropic, and Google. Then, AI researchers digging through Meta’s documentation discovered something unusual.

    In fine print, Meta acknowledges that the version of Maverick tested on LMArena isn’t the same as what’s available to the public. According to Meta’s own materials, it deployed an “experimental chat version” of Maverick to LMArena that was specifically “optimized for conversationality,” TechCrunch first reported.

    “Meta’s interpretation of our policy did not match what we expect from model providers,” LMArena posted on X two days after the model’s release. “Meta should have made it clearer that ‘Llama-4-Maverick-03-26-Experimental’ was a customized model to optimize for human preference. As a result of that, we are updating our leaderboard policies to reinforce our commitment to fair, reproducible evaluations so this confusion doesn’t occur in the future.“

    A spokesperson for Meta didn’t have a response to LMArena’s statement in time for publication.

    While what Meta did with Maverick isn’t explicitly against LMArena’s rules, the site has shared concerns about gaming the system and taken steps to “prevent overfitting and benchmark leakage.” When companies can submit specially-tuned versions of their models for testing while releasing different versions to the public, benchmark rankings like LMArena become less meaningful as indicators of real-world performance.

    ”It’s the most widely respected general benchmark because all of the other ones suck,” independent AI researcher Simon Willison tells The Verge. “When Llama 4 came out, the fact that it came second in the arena, just after Gemini 2.5 Pro — that really impressed me, and I’m kicking myself for not reading the small print.”

    Shortly after Meta released Maverick and Scout, the AI community started talking about a rumor that Meta had also trained its Llama 4 models to perform better on benchmarks while hiding their real limitations. VP of generative AI at Meta, Ahmad Al-Dahle, addressed the accusations in a post on X: “We’ve also heard claims that we trained on test sets — that’s simply not true and we would never do that. Our best understanding is that the variable quality people are seeing is due to needing to stabilize implementations.”

    “It’s a very confusing release generally.”

    Some also noticed that Llama 4 was released at an odd time. Saturday doesn’t tend to be when big AI news drops. After someone on Threads asked why Llama 4 was released over the weekend, Meta CEO Mark Zuckerberg replied: “That’s when it was ready.”

    “It’s a very confusing release generally,” says Willison, who closely follows and documents AI models. “The model score that we got there is completely worthless to me. I can’t even use the model that they got a high score on.”

    Meta’s path to releasing Llama 4 wasn’t exactly smooth. According to a recent report from The Information, the company repeatedly pushed back the launch due to the model failing to meet internal expectations. Those expectations are especially high after DeepSeek, an open-source AI startup from China, released an open-weight model that generated a ton of buzz.

    Ultimately, using an optimized model in LMArena puts developers in a difficult position. When selecting models like Llama 4 for their applications, they naturally look to benchmarks for guidance. But as is the case for Maverick, those benchmarks can reflect capabilities that aren’t actually available in the models that the public can access.

    As AI development accelerates, this episode shows how benchmarks are becoming battlegrounds. It also shows how Meta is eager to be seen as an AI leader, even if that means gaming the system.

    Share. Facebook Twitter Pinterest LinkedIn WhatsApp Reddit Email
    Previous ArticleShopify CEO says no new hires without proof AI can’t do the job
    Next Article You can build these marble runs and connect them to your smart home over Wi-Fi

    Related Posts

    NYT reports Elon Musk’s drug use went ‘well beyond occasional’ in 2024

    May 30, 2025

    Here’s where you can buy microSD Express cards for the Switch 2

    May 30, 2025

    You can get 15 percent off Elden Ring Nightreign on Steam and Xbox

    May 30, 2025

    Nintendo will let you limit who your kid can GameChat with on the Switch 2

    May 30, 2025

    SEC drops Binance lawsuit in yet another gift to crypto

    May 30, 2025

    RFK Jr.‘s ‘Make America Healthy Again’ report seems riddled with AI slop

    May 30, 2025
    Our Picks

    Remove Body Hair From Everywhere With the Fur 2-in-1 Trimmer

    May 30, 2025

    Here’s where you can buy microSD Express cards for the Switch 2

    May 30, 2025

    A United Arab Emirates Lab Announces Frontier AI Projects—and a New Outpost in Silicon Valley

    May 30, 2025

    I Converted My Photos Into Short Videos With AI on Honor’s Latest Phones. It’s Weird

    May 30, 2025
    • Facebook
    • Twitter
    • Pinterest
    • Instagram
    • YouTube
    • Vimeo
    Don't Miss
    News

    You can get 15 percent off Elden Ring Nightreign on Steam and Xbox

    By News RoomMay 30, 2025

    Elden Ring Nightreign just launched, delivering a multiplayer-focused, roguelike-meets-Soulslike experience, and it’s already discounted. Grab…

    Why Anthropic’s New AI Model Sometimes Tries to ‘Snitch’

    May 30, 2025

    Nintendo will let you limit who your kid can GameChat with on the Switch 2

    May 30, 2025

    How to Win Followers and Scamfluence People

    May 30, 2025
    Facebook X (Twitter) Instagram Pinterest
    • Privacy Policy
    • Terms of use
    • Advertise
    • Contact
    © 2025 Technology Mag. All Rights Reserved.

    Type above and press Enter to search. Press Esc to cancel.