Close Menu
Technology Mag

    Subscribe to Updates

    Get the latest creative news from FooBar about art, design and business.

    What's Hot

    The Fujifilm X-E5 is a simple, familiar, and impressive travel camera

    July 27, 2025

    CookUnity Cracked the Code on Meal Delivery By Using … Gasp … Chefs

    July 27, 2025

    The Verge’s 2025 back-to-school shopping guide

    July 27, 2025
    Facebook X (Twitter) Instagram
    Subscribe
    Technology Mag
    Facebook X (Twitter) Instagram YouTube
    • Home
    • News
    • Business
    • Games
    • Gear
    • Reviews
    • Science
    • Security
    • Trending
    • Press Release
    Technology Mag
    Home » How Peter Thiel’s Relationship With Eliezer Yudkowsky Launched the AI Revolution
    Business

    How Peter Thiel’s Relationship With Eliezer Yudkowsky Launched the AI Revolution

    News RoomBy News RoomMay 21, 20254 Mins Read
    Facebook Twitter Pinterest LinkedIn Reddit WhatsApp Email

    Two members of the Extropian community, internet entrepreneurs Brian and Sabine Atkins—­who met on an Extropian mailing list in 1998 and were married soon after—­were so taken by this message that in 2000 they bankrolled a think tank for Yudkowsky, the Singularity Institute for Artificial Intelligence. At 21, Yudkowsky moved to Atlanta and began drawing a nonprofit salary of around $20,000 a year to preach his message of benevolent superintelligence. “I thought very smart things would automatically be good,” he said. Within eight months, however, he began to realize that he was wrong—­way wrong. AI, he decided, could be a catastrophe.

    “I was taking someone else’s money, and I’m a person who feels a pretty deep sense of obligation towards those who help me,” Yudkowsky explained. “At some point, instead of thinking, ‘If superintelligences don’t automatically determine what is the right thing and do that thing that means there is no real right or wrong, in which case, who cares?’ I was like, ‘Well, but Brian Atkins would probably prefer not to be killed by a superintelligence.’ ” He thought Atkins might like to have a “fallback plan,” but when he sat down and tried to work one out, he realized with horror that it was impossible. “That caused me to actually engage with the underlying issues, and then I realized that I had been completely mistaken about everything.”

    The Atkinses were understanding, and the institute’s mission pivoted from making artificial intelligence to making friendly artificial intelligence. “The part where we needed to solve the friendly AI problem did put an obstacle in the path of charging right out to hire AI researchers, but also we just surely didn’t have the funding to do that,” Yudkowsky said. Instead, he devised a new intellectual framework he dubbed “rationalism.” (While on its face, rationalism is the belief that humankind has the power to use reason to come to correct answers, over time it came to describe a movement that, in the words of writer Ozy Brennan, includes “reductionism, materialism, moral non-­realism, utilitarianism, anti-­deathism and transhumanism.” Scott Alexander, Yudkowsky’s intellectual heir, jokes that the movement’s true distinguishing trait is the belief that “Eliezer Yudkowsky is the rightful calif.”)

    In a 2004 paper, “Coherent Extrapolated Volition,” Yudkowsky argued that friendly AI should be developed based not just on what we think we want AI to do now, but what would actually be in our best interests. “The engineering goal is to ask what humankind ‘wants,’ or rather what we would decide if we knew more, thought faster, were more the people we wished we were, had grown up farther together, etc.,” he wrote. In the paper, he also used a memorable metaphor, originated by Bostrom, for how AI could go wrong: If your AI is programmed to produce paper clips, if you’re not careful, it might end up filling the solar system with paper clips.

    In 2005, Yudkowsky attended a private dinner at a San Francisco restaurant held by the Foresight Institute, a technology think tank founded in the 1980s to push forward nanotechnology. (Many of its original members came from the L5 Society, which was dedicated to pressing for the creation of a space colony hovering just behind the moon, and successfully lobbied to keep the United States from signing the United Nations Moon Agreement of 1979 due to its provision against terraforming celestial bodies.) Thiel was in attendance, regaling fellow guests about a friend who was a market bellwether, because every time he thought some potential investment was hot, it would tank soon after. Yudkowsky, having no idea who Thiel was, walked up to him after dinner. “If your friend was a reliable signal about when an asset was going to go down, they would need to be doing some sort of cognition that beat the efficient market in order for them to reliably correlate with the stock going downwards,” Yudkowsky said, essentially reminding Thiel about the efficient-market hypothesis, which posits that all risk factors are already priced into markets, leaving no room to make money from anything besides insider information. Thiel was charmed.

    Share. Facebook Twitter Pinterest LinkedIn WhatsApp Reddit Email
    Previous ArticleThe Best Bug Sprays to Keep Bites at Bay
    Next Article OpenAI is buying Jony Ive’s AI hardware company

    Related Posts

    Cursor’s New Bugbot Is Designed to Save Vibe Coders From Themselves

    July 26, 2025

    Americans Are Obsessed With Watching Short Video Dramas From China

    July 25, 2025

    Trump Says He’s ‘Getting Rid of Woke’ and Dismisses Copyright Concerns in AI Policy Speech

    July 25, 2025

    Trump’s AI Action Plan Is a Crusade Against ‘Bias’—and Regulation

    July 24, 2025

    A New Era for WIRED—That Starts With You

    July 24, 2025

    The Great Crypto Re-Banking Has Begun

    July 23, 2025
    Our Picks

    CookUnity Cracked the Code on Meal Delivery By Using … Gasp … Chefs

    July 27, 2025

    The Verge’s 2025 back-to-school shopping guide

    July 27, 2025

    Razer’s Pro Click V2 Vertical Is the Ergonomic Gaming Mouse You’re Looking For

    July 27, 2025

    Apple beta season is here

    July 27, 2025
    • Facebook
    • Twitter
    • Pinterest
    • Instagram
    • YouTube
    • Vimeo
    Don't Miss
    Science

    The ICJ Rules That Failing to Combat Climate Change Could Violate International Law

    By News RoomJuly 27, 2025

    If a country fails to take decisive action to protect the planet from climate change,…

    Nemo’s Updated Dagger Osmo Tent Has Nicer Fabric and Better Design Details

    July 26, 2025

    Here are the laptops I’d tell any parent to consider for their back-to-school student

    July 26, 2025

    Do You Need a Barbecue Knife?

    July 26, 2025
    Facebook X (Twitter) Instagram Pinterest
    • Privacy Policy
    • Terms of use
    • Advertise
    • Contact
    © 2025 Technology Mag. All Rights Reserved.

    Type above and press Enter to search. Press Esc to cancel.