Ilya Sutskever – The age of scaling is over
Timestamps are as accurate as they can be but may be slightly off. We encourage you to listen to the full context.
Podcast Summary
In this fascinating conversation, Ilya Sutskever discusses the current state and future of AI development, sharing his perspectives on why today's models exhibit "jaggedness" - performing exceptionally well on evaluations while making basic mistakes in real-world applications. (01:02) Sutskever argues that we're transitioning from an "age of scaling" back to an "age of research," where simply adding more compute and data won't be sufficient for the next breakthroughs.
- Core themes: The episode explores fundamental challenges in AI generalization, the role of emotions as value functions, SSI's approach to building superintelligence, and the importance of incremental deployment for AI safety.
Speakers
Ilya Sutskever
Ilya Sutskever is the co-founder and Chief Scientist of Safe Superintelligence Inc. (SSI), a company focused on developing safe artificial general intelligence. Previously, he was co-founder and Chief Scientist at OpenAI, where he played a crucial role in developing GPT models and advancing the field of deep learning. Sutskever is renowned for his contributions to foundational AI research, including co-authoring seminal papers like AlexNet and being instrumental in the development of the transformer architecture.
Dwarkesh Patel
Dwarkesh Patel is the host of the Dwarkesh Podcast, known for conducting in-depth technical interviews with leading AI researchers, economists, and intellectuals. He has gained recognition for his thoughtful questioning and ability to extract insights from complex technical discussions, making them accessible to both technical and general audiences.
Key Takeaways
Current AI Models Suffer from "Jaggedness" Due to Training Methodology
Sutskever explains that modern AI models exhibit a puzzling contradiction: they excel on challenging evaluations but fail at basic real-world tasks, like alternating between the same bugs in coding. (02:24) This "jaggedness" stems from how reinforcement learning training is conducted - companies create specific RL environments often inspired by the very evaluations they want to excel at, leading to models that are over-optimized for narrow benchmarks rather than general competence. Unlike pre-training where "everything" was the data, RL requires deliberate choices about training environments, creating a potential mismatch between what models learn and what we actually need them to do.
Humans Achieve Superior Generalization Through Evolutionary Value Functions
One of the most profound insights centers on why humans learn so much more efficiently than AI models. Sutskever references a fascinating case study of a person who lost emotional processing due to brain damage - while remaining articulate and able to solve puzzles, they became incapable of making basic decisions like choosing socks. (11:32) This suggests that emotions function as evolutionary-encoded value functions that guide learning and decision-making. Humans come equipped with these sophisticated priors for evaluating situations, allowing teenagers to learn driving in just ten hours with robust safety intuitions, while AI systems require massive amounts of carefully curated training data.
We're Returning to an "Age of Research" After the Scaling Era
Sutskever argues that we're witnessing a fundamental transition in AI development. The period from 2020-2025 represented the "age of scaling" where the simple recipe of more data, compute, and parameters drove progress. (21:22) However, as pre-training data becomes finite and compute scales reach unprecedented levels, we're returning to an "age of research" similar to 2012-2020. The key difference is that this new research era operates with vastly more computational resources, requiring new fundamental insights rather than just scaling existing approaches. This shift explains why there are currently "more companies than ideas" in AI - scaling sucked all the oxygen out of the room for genuine research innovation.
Superintelligence Should Be Deployed as Continual Learners, Not Finished Systems
Rather than building an AGI that knows how to do every job from the start, Sutskever envisions superintelligence as incredibly efficient learning systems that acquire skills through deployment. (50:37) He challenges the traditional AGI concept, noting that even humans aren't "general" intelligence - we rely heavily on continual learning throughout our lives. The vision is of "super intelligent 15-year-olds" that are deployed into the world to learn specific roles, whether as programmers, doctors, or researchers. This approach allows for gradual deployment while enabling the AI to accumulate diverse skills across the entire economy, potentially leading to rapid economic growth as these systems learn and improve in real-time.
AI Safety Requires Demonstrating Power to Drive Behavioral Change
Sutskever has evolved his thinking on AI safety, now placing greater emphasis on incremental deployment rather than the original "straight shot to superintelligence" approach. (55:30) He argues that it's nearly impossible for people to truly comprehend future AI capabilities until they experience them directly - similar to how it's hard to imagine being old when you're young. The key insight is that as AI becomes more visibly powerful, it will drive unprecedented changes in behavior: fierce competitors will collaborate on safety, governments will take action, and AI companies will become much more paranoid about safety. This suggests that showing the AI's capabilities gradually, rather than hiding development until completion, is crucial for building appropriate societal and institutional responses.
Statistics & Facts
- SSI has raised $3 billion in funding, which Sutskever argues is more competitive for research than it appears because other companies spend much of their larger budgets on inference, engineering staff, and product features rather than pure research. (40:51)
- Companies now spend more compute on reinforcement learning than on pre-training, as RL requires very long rollouts that consume substantial computational resources while yielding relatively small amounts of learning per rollout. (22:56)
- The transformer architecture was originally developed using only 8-64 GPUs in 2017, which would be equivalent to approximately two modern GPUs, demonstrating that breakthrough research doesn't necessarily require the maximum possible compute resources. (38:42)
