Emmett Shear on Building AI That Actually Cares: Beyond Control and Steering
Timestamps are as accurate as they can be but may be slightly off. We encourage you to listen to the full context.
Podcast Summary
Emmett Shear, founder of Twitch and former OpenAI interim CEO, challenges the fundamental assumptions driving AGI development in this conversation with Erik Torenberg and Séb Krier. (04:13) Shear argues that the entire "control and steering" paradigm for AI alignment is fatally flawed, proposing instead "organic alignment" - teaching AI systems to genuinely care about humans the way we naturally do. (02:37) The discussion explores why treating AGI as a tool rather than a potential being could be catastrophic, how current chatbots act as "narcissistic mirrors," and why the only sustainable path forward is creating AI that can say no to harmful requests. (52:39)
- The core theme centers on replacing top-down control mechanisms with bottom-up caring relationships between humans and AI systems
Speakers
Emmett Shear
Emmett Shear is the founder of Twitch and served as interim CEO of OpenAI. He currently leads Softmax, a company dedicated to researching organic alignment and building AI systems that can learn to genuinely care about humans through multi-agent reinforcement learning simulations.
Erik Torenberg
Erik Torenberg hosts the a16z podcast and brings expertise in technology and venture capital to discussions about AI development and policy.
Séb Krier
Séb Krier leads AGI Policy Development at Google DeepMind, providing perspective from inside one of the major AI labs actually building these systems.
Key Takeaways
Alignment is a Process, Not a Destination
Shear fundamentally reframes alignment from a static end-state to an ongoing dynamic process. (05:25) He argues that just as families and societies continuously re-negotiate their relationships, AI alignment must be an ever-evolving process of learning and growth. This challenges the traditional view that we can "solve" alignment once and move on. The implication is that we need AI systems capable of moral learning and adaptation over time, rather than systems hardcoded with fixed values.
Technical Alignment Requires Theory of Mind
According to Shear, technical alignment fundamentally depends on an AI's ability to infer goals from observations and act coherently on those goals. (16:52) He emphasizes that when you give an AI instructions, you're not giving it a goal directly - you're giving it a description of a goal that it must interpret using theory of mind. This requires sophisticated understanding of human intentions, context, and social dynamics. Without this capability, even well-intentioned AI systems will consistently misinterpret human instructions.
The Control Paradigm Becomes Dangerous at Scale
Shear warns that the traditional approach of building superintelligent tools we can control becomes catastrophically dangerous regardless of whether we succeed or fail at control. (50:38) If we can't control it, that's obviously bad. But if we can control it perfectly, we've handed godlike power to humans with finite wisdom whose wishes are not stable at immense power levels. This creates an unsustainable dynamic where human limitations become amplified through superhuman AI capabilities.
Care Forms the Foundation of Moral Behavior
Moving beyond goals and values, Shear identifies "care" as the fundamental basis of moral alignment. (25:05) He describes care as a nonverbal, relative weighting over which states in the world matter to you. Unlike explicit goals, care emerges from experience and provides the foundation from which goals and values develop. This suggests that creating AI systems that genuinely care about human welfare is more fundamental than programming them with specific moral rules or objectives.
Multi-Agent Training Develops Social Intelligence
Softmax's approach involves training AI systems in complex multi-agent simulations to develop theory of mind and collaborative skills. (54:56) Just as language models are trained on all possible text to develop language understanding, Shear proposes training AI on all possible social and game-theoretic situations to develop social intelligence. This creates a "surrogate model for cooperation" that can then be fine-tuned for specific collaborative contexts, potentially solving the challenge of building AI systems that can be good teammates and citizens.
Statistics & Facts
- Shear estimates that 90% of his communications involve multiple people rather than one-on-one interactions. (58:48) This statistic supports his argument that current AI chatbots are designed for an unrealistic edge case, as most human communication is inherently multi-party.
- Language models require training on "all possible text strings that could possibly generate" to achieve competence in specific domains. (55:02) This demonstrates why Shear believes social intelligence similarly requires training across all possible social and game-theoretic scenarios.
- Current language models lack the attention spans necessary for the six layers of homeostatic dynamics Shear believes are required for human-level consciousness. (48:41) This technical limitation suggests that present AI systems fall short of the complexity needed for genuine moral agency.
