#490 – State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI
Timestamps are as accurate as they can be but may be slightly off. We encourage you to listen to the full context.
Podcast Summary
This podcast episode features a deep technical discussion about the state of artificial intelligence in 2025 with Sebastian Raschka, author of "Build a Large Language Model From Scratch," and Nathan Lambert, post-training lead at the Allen Institute for AI and author of the RLHF book. The conversation covers the rapid evolution of AI models, particularly after the "DeepSeek moment" in January 2025, examining everything from architecture innovations to the competitive landscape between Chinese and American AI companies. (16:29)
- Core themes include scaling laws, post-training techniques (especially RLVR), the rise of Chinese open-weight models, Silicon Valley work culture, and the future trajectory of AI development toward AGI.
Speakers
Sebastian Raschka
Sebastian is a machine learning researcher and educator, best known for his influential technical books including "Build a Large Language Model From Scratch" and "Build a Reasoning Model From Scratch." He focuses on making complex AI concepts accessible through hands-on implementation, believing that building systems from scratch is the most effective way to truly understand how they work.
Nathan Lambert
Nathan is the post-training lead at the Allen Institute for AI (Ai2) and author of "The RLHF Book," the definitive resource on reinforcement learning from human feedback. He's deeply involved in both research and policy work, including his Adam Project advocating for American open-weight AI models to compete with Chinese offerings.
Key Takeaways
The Era of Post-Training Innovation
The biggest breakthrough of 2025 has been reinforcement learning with verifiable rewards (RLVR), which allows models to learn through trial and error on problems with objectively correct answers like math and coding. (79:38) Unlike traditional RLHF which plateaus quickly, RLVR shows consistent scaling laws - you can keep investing compute and get better performance. This has enabled inference-time scaling where models can "think" for extended periods, dramatically improving their capabilities on complex tasks.
Chinese Open Models Are Reshaping the Landscape
Chinese companies like DeepSeek, Kimi, and MiniMax have released increasingly powerful open-weight models that match or exceed closed American models in many domains. (21:20) Their strategy leverages open licensing to gain global influence while American companies can't monetize Chinese APIs due to security concerns. This has prompted Nathan's "Adam Project" advocating for increased US investment in open-weight models to maintain technological leadership.
Programming Is Being Transformed by AI
Current AI coding tools like Claude Code and Cursor represent a fundamental shift in how software is created. The experience moves from micromanaging code details to thinking in design spaces and guiding systems at a macro level. (37:06) Professional developers are already shipping significant percentages of AI-generated code, with senior developers often using AI more extensively than juniors because they better understand how to leverage these tools effectively.
Architecture Innovation Continues Despite Stability
While transformer architectures remain dominant, meaningful innovations continue in attention mechanisms, mixture of experts (MoE) models, and efficiency improvements. (53:57) New approaches like text diffusion models and architectural tweaks for long context are being explored. The fundamental architecture hasn't changed dramatically since GPT-2, but optimizations in training, serving, and specialization continue to unlock new capabilities.
Silicon Valley's High-Intensity Culture Has Costs
The AI industry has adopted an intense "996" work culture (9am-9pm, 6 days a week) driven by fierce competition and belief in imminent breakthroughs. (155:36) While this drives rapid progress, it comes with significant human costs including burnout and work-life balance issues. The authors suggest this culture may be unsustainable long-term, though it's currently fueled by the perception that we're living through a transformative moment in history.
Statistics & Facts
- OpenAI's average compensation is over $1,000,000 in stock per employee per year, highlighting the premium placed on AI talent and the financial incentives driving the current talent concentration in frontier labs. (150:16)
- A survey of 791 professional developers (with 10+ years experience) found that 80% find programming with AI either somewhat or significantly more enjoyable, and 25% use AI for 50% or more of the code they ship to production. (106:48)
- DeepSeek's training costs were approximately $2-5 million at cloud market rates for their large models, demonstrating that frontier model training is becoming surprisingly affordable compared to the billions spent on serving them to users. (69:54)
