Close Menu
Technology Mag

    Subscribe to Updates

    Get the latest creative news from FooBar about art, design and business.

    What's Hot

    Everything You Can Do in the Photoshop Mobile App

    July 5, 2025

    The Promise and Peril of Digital Security in the Age of Dictatorship

    July 5, 2025

    The Ploopy Knob is an open-source control dial for your PC

    July 4, 2025
    Facebook X (Twitter) Instagram
    Subscribe
    Technology Mag
    Facebook X (Twitter) Instagram YouTube
    • Home
    • News
    • Business
    • Games
    • Gear
    • Reviews
    • Science
    • Security
    • Trending
    • Press Release
    Technology Mag
    Home » How Peter Thiel’s Relationship With Eliezer Yudkowsky Launched the AI Revolution
    Business

    How Peter Thiel’s Relationship With Eliezer Yudkowsky Launched the AI Revolution

    News RoomBy News RoomMay 21, 20254 Mins Read
    Facebook Twitter Pinterest LinkedIn Reddit WhatsApp Email

    Two members of the Extropian community, internet entrepreneurs Brian and Sabine Atkins—­who met on an Extropian mailing list in 1998 and were married soon after—­were so taken by this message that in 2000 they bankrolled a think tank for Yudkowsky, the Singularity Institute for Artificial Intelligence. At 21, Yudkowsky moved to Atlanta and began drawing a nonprofit salary of around $20,000 a year to preach his message of benevolent superintelligence. “I thought very smart things would automatically be good,” he said. Within eight months, however, he began to realize that he was wrong—­way wrong. AI, he decided, could be a catastrophe.

    “I was taking someone else’s money, and I’m a person who feels a pretty deep sense of obligation towards those who help me,” Yudkowsky explained. “At some point, instead of thinking, ‘If superintelligences don’t automatically determine what is the right thing and do that thing that means there is no real right or wrong, in which case, who cares?’ I was like, ‘Well, but Brian Atkins would probably prefer not to be killed by a superintelligence.’ ” He thought Atkins might like to have a “fallback plan,” but when he sat down and tried to work one out, he realized with horror that it was impossible. “That caused me to actually engage with the underlying issues, and then I realized that I had been completely mistaken about everything.”

    The Atkinses were understanding, and the institute’s mission pivoted from making artificial intelligence to making friendly artificial intelligence. “The part where we needed to solve the friendly AI problem did put an obstacle in the path of charging right out to hire AI researchers, but also we just surely didn’t have the funding to do that,” Yudkowsky said. Instead, he devised a new intellectual framework he dubbed “rationalism.” (While on its face, rationalism is the belief that humankind has the power to use reason to come to correct answers, over time it came to describe a movement that, in the words of writer Ozy Brennan, includes “reductionism, materialism, moral non-­realism, utilitarianism, anti-­deathism and transhumanism.” Scott Alexander, Yudkowsky’s intellectual heir, jokes that the movement’s true distinguishing trait is the belief that “Eliezer Yudkowsky is the rightful calif.”)

    In a 2004 paper, “Coherent Extrapolated Volition,” Yudkowsky argued that friendly AI should be developed based not just on what we think we want AI to do now, but what would actually be in our best interests. “The engineering goal is to ask what humankind ‘wants,’ or rather what we would decide if we knew more, thought faster, were more the people we wished we were, had grown up farther together, etc.,” he wrote. In the paper, he also used a memorable metaphor, originated by Bostrom, for how AI could go wrong: If your AI is programmed to produce paper clips, if you’re not careful, it might end up filling the solar system with paper clips.

    In 2005, Yudkowsky attended a private dinner at a San Francisco restaurant held by the Foresight Institute, a technology think tank founded in the 1980s to push forward nanotechnology. (Many of its original members came from the L5 Society, which was dedicated to pressing for the creation of a space colony hovering just behind the moon, and successfully lobbied to keep the United States from signing the United Nations Moon Agreement of 1979 due to its provision against terraforming celestial bodies.) Thiel was in attendance, regaling fellow guests about a friend who was a market bellwether, because every time he thought some potential investment was hot, it would tank soon after. Yudkowsky, having no idea who Thiel was, walked up to him after dinner. “If your friend was a reliable signal about when an asset was going to go down, they would need to be doing some sort of cognition that beat the efficient market in order for them to reliably correlate with the stock going downwards,” Yudkowsky said, essentially reminding Thiel about the efficient-market hypothesis, which posits that all risk factors are already priced into markets, leaving no room to make money from anything besides insider information. Thiel was charmed.

    Share. Facebook Twitter Pinterest LinkedIn WhatsApp Reddit Email
    Previous ArticleThe Best Bug Sprays to Keep Bites at Bay
    Next Article OpenAI is buying Jony Ive’s AI hardware company

    Related Posts

    Despite Protests, Elon Musk Secures Air Permit for xAI

    July 4, 2025

    What Could a Healthy AI Companion Look Like?

    July 4, 2025

    For Today’s Business Traveler, It’s All About Work-Life Integration

    July 3, 2025

    Affluent Travelers Are Ditching Business Class for Business Jets

    July 2, 2025

    Airplane Wi-Fi Is Now … Good?

    July 2, 2025

    Business Travel Is Evolving Faster Than Ever. We’ll Help You Navigate It

    July 2, 2025
    Our Picks

    The Promise and Peril of Digital Security in the Age of Dictatorship

    July 5, 2025

    The Ploopy Knob is an open-source control dial for your PC

    July 4, 2025

    Laid-off workers should use AI to manage their emotions, says Xbox exec

    July 4, 2025

    Despite Protests, Elon Musk Secures Air Permit for xAI

    July 4, 2025
    • Facebook
    • Twitter
    • Pinterest
    • Instagram
    • YouTube
    • Vimeo
    Don't Miss
    Gear

    This Is Why Tesla’s Robotaxi Launch Needed Human Babysitters

    By News RoomJuly 4, 2025

    “This is a demo or test using safety drivers—it’s not an [autonomous vehicle] deployment,” says…

    Fairphone 6 gets a 10/10 on repairability

    July 4, 2025

    New Galaxy Z Fold 7 leaks may give first real look at Samsung’s slimmer foldable

    July 4, 2025

    This is not a tattoo robot

    July 4, 2025
    Facebook X (Twitter) Instagram Pinterest
    • Privacy Policy
    • Terms of use
    • Advertise
    • Contact
    © 2025 Technology Mag. All Rights Reserved.

    Type above and press Enter to search. Press Esc to cancel.