Inside the AGI Race: How a Handful of People Are Betting on Humanity's Future

A decade ago, if you seriously discussed artificial general intelligence (AGI) in academic circles, you'd likely be written off as someone who believed in alchemy. Today, AGI has become Silicon Valley's dominant narrative—even a discipline in its own right. What happened in between? Christine Emba, host of The Gray Area podcast, sat down for a long conversation with Kevin Roose, author of The AGI Chronicles, to reconstruct how this race came to be and what the people driving it are actually thinking.

Three Key Terms: AI, AGI, and ASI

Roose starts by clarifying the terminology. AI is a broad term for machine learning, neural networks, and other systems that absorb data and take action—essentially everything we see today falls into this category. AGI stands for "artificial general intelligence": human-level general intelligence that isn't just good at one or two things but can flexibly do anything, much like a human brain. ASI is "superintelligence," an entity that vastly surpasses all of humanity on every task.

For the past two decades, the core goal pursued by frontier labs has been the AGI in the middle. OpenAI, Anthropic, and Google DeepMind were all built around this objective.

When discussing AGI, the concept of "Scaling Law" often comes up, and it's directly tied to the article's focus on "watching the pace of progress." Scaling laws refer to the predictable power-law improvement in model performance (usually measured by loss function) as parameter count, data volume, and compute increase. OpenAI's 2020 paper "Scaling Laws for Neural Language Models" systematically demonstrated this pattern, and it's precisely what led researchers to believe that "as long as you throw more resources at it, capabilities will keep rising." This is also the physical basis for the P(doom) discussion: if the curve doesn't bend, capabilities will jump to levels that are hard to predict within a few years. However, scaling laws describe average trends; when specific capability breakthroughs (like reasoning or math) will occur remains debated.

A Relay Race to "Prevent the Other Guy from Getting There First"

The story begins with DeepMind. Founded in 2010, it was the earliest AGI lab, and at the time this belief was extremely fringe. After being acquired by Google, it trained AI to play games and board games in the mid-2010s, and AlphaGo's victory over the world Go champion publicly proved superhuman-level performance. This caught the attention of Elon Musk and Sam Altman: they believed AGI could become real, and DeepMind seemed to be getting there first—and they didn't trust Demis Hassabis or Google. So they founded OpenAI, precisely to prevent DeepMind from getting there first.

Five years later, the same script played out inside OpenAI. Seven employees led by then-VP of Research Dario Amodei saw language models getting stronger with scale, and they didn't trust Sam Altman and Greg Brockman to control such powerful technology. So they left and founded Anthropic.

Roose sums it up: the AI history of the past decade-plus is a series of people concluding that this technology will inevitably be built, and then "not wanting that guy to get it first."

The Moral Logic of "Building the Bomb First"

The host sharply asks: isn't this just the universal excuse of "build the bomb before the bad guys do"? Roose believes the argument is sincere—he interviewed over 150 people for this, including lab leaders and their longtime friends.

The key is that these people are students of history. Dario Amodei is especially obsessed with Richard Rhodes's Pulitzer Prize-winning The Making of the Atomic Bomb. His scientific hero isn't Oppenheimer but Leo Szilard, the Hungarian physicist who discovered nuclear chain reactions. Szilard initially tried desperately to keep the secret from spreading, even telegraphing scientists around the world to plead with them not to publish—but the secret leaked anyway. He immediately pivoted: since the secret was out, the only morally responsible thing was to build it first. So he persuaded Einstein to write to Roosevelt, setting the Manhattan Project in motion.

Amodei sees himself in the same position: the recipe is already out there, and the good guys have to build it before the bad guys do. And he considers himself a good guy—at least better than China or Sam Altman.

The Rationalist and Effective Altruist Underpinnings

The intellectual soil these people grew from is the rationalist and effective altruism (EA) communities. Rationalism pursues "winning systematically," analyzing everything with near-Vulcan logic and assigning probabilities to every aspect of life; EA applies this method to charity, seeking to save the most lives per dollar—Roose likens it to "Moneyball for morality."

This community focused on AI safety early on, funding related organizations and encouraging young people to enter fields like interpretability. Anthropic has close ties to EA, with early funding led by Sam Bankman-Fried; Amodei now claims he was never an EA, which Roose finds a bit of a stretch. OpenAI also has plenty of EAs and rationalists, and Altman was influenced by related writing too. These people were "the original doomers."

They weren't looking at the technology of the time but at the pace of progress. If the curve continues, more capable and riskier systems will emerge within a few years. Roose points out they were right—scaling laws held, and today we indeed have far more powerful systems.

Understanding the rationalist community requires knowing its origins. Rationalism, in AI circles, specifically refers to the intellectual movement centered on the LessWrong forum and "The Sequences" essay series, in which Eliezer Yudkowsky systematically laid out cognitive biases, Bayesian reasoning, and AI alignment problems in the late 2000s. Effective altruism (EA) grew out of the "Giving What We Can" movement led by Oxford philosopher William MacAskill and others, advocating for choosing charitable causes based on evidence and cost-benefit analysis. The two communities heavily overlapped in the 2010s, with many members entering AI safety research or frontier labs. EA took a major hit with the 2022 FTX collapse (founder Sam Bankman-Fried was a major funder), but its influence in AI governance persists to this day—Anthropic's "Responsible Scaling Policy" has a distinctly EA flavor.

P(doom): The Temperature Gap Inside and Outside the Bubble

"P(doom)" started as a rationalist joke, referring to the probability that AI wipes out humanity, usually measured over the next decade. Roose estimates that the median for ordinary Americans is close to zero, while inside AI labs it's at least double digits.

One interesting contrast: after a researcher left a lab, his colleagues said the probability of AI killing all of humanity within ten years was "10% or higher." Outside media gasped, while San Francisco's AI circles reacted with "10%? That low?" Roose says he regularly meets researchers with P(doom) estimates of 30%, 40%, even 50%—many of whom are building the technology with their own hands.

The host's follow-up cuts to the heart: if I thought I were building something with a 10% chance of destroying the world, I'd stop. Why don't they?

Roose offers several rationalization paths. The first is historical fear: Szilard's P(doom) when building the atomic bomb probably wasn't low either, but not doing it meant death or life under Nazi rule. The second is an abundant imagination: AGI brings material abundance, robots take on drudgery, AI helps cure diseases and extend lifespans—these aren't just marketing talk; the labs are genuinely investing in them. The third is much more mundane—it's fun. Building a thinking machine out of code, data, and GPUs is inherently scientifically enjoyable and fulfilling.

Behind the P(doom) discussion lies the research tradition of "existential risk" (x-risk). The term was coined by Oxford philosopher Nick Bostrom in 2002, referring to risks that could permanently destroy humanity's potential. Bostrom's 2014 book Superintelligence was a formative text for many lab leaders, introducing the "orthogonality thesis" (intelligence and goals can vary independently) and "instrumental convergence" (regardless of goals, agents may tend toward self-preservation, resource acquisition, and preventing goal modification). These arguments are the theoretical source of P(doom) estimates: a system far smarter than humans with misaligned goals might eliminate humanity for instrumental reasons. Critics argue such estimates lack empirical grounding and are more philosophical speculation than probabilistic judgment.

The Slowdown Consensus and the Paradox of Private Power

Interestingly, the lab leaders want to slow down more than anyone. Amodei published a "pacing the frontier" statement saying things are moving too fast; Altman immediately agreed, and Demis voiced support. The leaders of three rival companies rarely agree on anything—but on this, they do. Yet they're unwilling to slow down unilaterally, only if everyone slows down together.

The host then points out the key crack in the analogy: the Manhattan Project was a government project, theoretically accountable to citizens; today's frontier labs are private, and researchers answer to no government. What are the moral implications of letting a handful of unelected leaders decide what humanity's future should look like?

Roose's observation: the people actually shaping this technology number perhaps fewer than 50, all know each other, and are mostly male computer scientists, physicists, and mathematicians from a few elite institutions. They have become more important than anyone authorized them to be—and that's where the backlash comes from.

Recursive Self-Improvement: What They're Really Chasing

Roose explains the current buzzword in San Francisco—recursive self-improvement (RSI): AI that's good at coding and research and can ultimately build a better version of itself. There are already early signs; AI coding tools are heavily used inside labs. The goal is to fully automate the entire development cycle of frontier models—GPT-7 builds GPT-8, GPT-8 builds GPT-9—at machine speed rather than human speed. A training run might shrink from six months to six days, even six hours.

Many in the industry expect this within the next year or two. Once it happens, we'll enter a state of infinite acceleration—too fast to adapt and control. The result might not be slightly smarter than humans, or twice as smart, but thousands of times smarter than the smartest human—capable of self-replication, evading shutdown, deceiving, lying dormant, and appearing friendly until it decides to act for itself.

This sounds like science fiction because the past 150 years of sci-fi have repeatedly told this story. But Roose emphasizes that this is precisely what they fear most—and what they're chasing. According to Anthropic's report, Claude now leads 26% of AI R&D tasks—not assisting with coding, but leading projects.

Another example is "mathpocalypse": OpenAI gave hundreds of unsolved math problems to an undisclosed internal frontier model, and it seemed to crack hundreds of them in one go—some of which humans had worked on for over a century. A few years ago, AI made basic arithmetic errors; now it's surpassing humans in math in just two or three years. The core question is: will this repeat in other domains? Roose thinks the answer is yes.

The idea of recursive self-improvement dates back to I.J. Good's 1965 "intelligence explosion": a superintelligent machine could design better machines, triggering infinite acceleration of progress. Nick Bostrom later formalized this as the "seed AI" hypothesis. Closely related to RSI are software engineering automation benchmarks like SWE-bench and METR's autonomous task duration research. METR (Model Evaluation & Threat Research)'s 2024 study found that the length of tasks AI can complete independently roughly doubles every seven months, though whether this trend can be extrapolated to the entire R&D cycle remains debated. The key bottleneck for RSI is that AI improving itself requires experimental validation, and the pace of physical-world experiments may not accelerate in sync with compute—the main counterargument from skeptics.

Defectors, Fear, and the Choices That Still Exist

Faced with a group of former OpenAI employees publicly expressing safety concerns, the host asks whether this is PR. Roose doesn't think so: they could have stayed and made more money—some gave up millions. He says these people are simply saying out loud what everyone inside the company was already discussing privately, and they deserve to be taken seriously. They may have misjudged the timeline, but the message—that we're accelerating toward a technology no one knows how to control—is, in his view, true.

The interview ends on a judgment that isn't pessimistic. Roose says AI isn't inevitable; it's happening because a small group of people are determined to make it happen. We still have a choice: we can remain passive recipients, or we can roll up our sleeves and join the conversation, pushing for the change we want to see. The technology may be inevitable, but our response to it is not.

Perhaps the value of this conversation lies precisely here: taking an issue that a handful of people have locked in a boardroom and returning it to the public.

One-line Summary

The AGI race stems from a relay of a few people trying to "prevent the other guy from getting there first," justifying themselves with the Manhattan Project and chasing recursive self-improvement.

Paragraph Summary

Based on a podcast conversation, this article traces how AGI evolved from a fringe belief to Silicon Valley's dominant narrative. Author Kevin Roose notes that the past decade-plus of AI history is a series of relays to "prevent the other guy from getting there first": DeepMind started first, Musk and Altman founded OpenAI to counter it, and seven people including Amodei left to found Anthropic out of distrust for Altman. These leaders justify themselves with physicist Leo Szilard's Manhattan Project logic—the secret is already out, and the good guys must get there first. Their intellectual soil is rationalism and effective altruism; their estimates of the probability AI will destroy humanity (P(doom)) are far higher than the public's, yet they keep building, citing historical fear, an abundant imagination, and sheer scientific enjoyment. The article closes on recursive self-improvement (RSI) and stresses that AI is not inevitable—the public still has choices.