Synthetic Data and the Future of AI | Cohere CEO Aidan Gomez
Timestamps are as accurate as they can be but may be slightly off. We encourage you to listen to the full context.
Podcast Summary
This episode features Aidan Gomez, co-founder and CEO of Cohere, discussing his journey from co-authoring the transformative "Attention Is All You Need" paper at Google to building an enterprise-focused AI company. (00:00) Gomez shares insights on the current state of AI development, challenges facing the industry, and why he believes the focus on doomsday scenarios has been intellectually dishonest. (31:00) The conversation explores Cohere's approach to enterprise AI deployment, the talent wars in AI research, and predictions for the future of artificial intelligence in business and society. (34:06)
- Core themes include the evolution of AI from research breakthrough to enterprise implementation, the competitive landscape among AI labs, and the importance of building practical solutions over pursuing AGI hype
Speakers
Aidan Gomez
Aidan Gomez is the co-founder and CEO of Cohere, an enterprise-focused AI company valued at approximately $7 billion. He's a Google Brain alum and co-author of the seminal "Attention Is All You Need" paper, which introduced the transformer architecture that powers modern AI systems. He started as an undergraduate intern at Google at age 19 and later pursued his PhD at Oxford before founding Cohere.
Joubin Mirzadegan
Joubin Mirzadegan is a partner at Kleiner Perkins and host of the GRIT podcast. He focuses on exploring the personal and professional challenges of building history-making companies with leaders in technology and innovation.
Key Takeaways
Focus on Efficiency Over Scale
The core insight behind the transformer architecture wasn't just attention mechanisms, but designing for efficiency and scalability across multiple GPUs. (08:47) Gomez explains that they built the architecture to work well when scaling from one GPU to 32 GPUs, which turned out to be crucial as the industry moved toward training models on tens of thousands of GPUs. This focus on making training efficient rather than just effective became the foundation for all modern large language models and their ability to scale.
The AI Scaling Plateau is Here
Despite massive increases in compute spending, the rate of model improvement has significantly slowed down. (15:00) Gomez argues that we're entering "uneconomic territory" where doubling or 10x-ing training costs isn't yielding proportional improvements in model capability. This means the industry must shift focus from pure scaling to better data, improved training methods, and more efficient architectures. For businesses, this suggests the current generation of models will remain relevant longer than expected.
Memory and Learning from Experience is the Next Breakthrough
The most obvious missing capability in current AI systems is the ability to learn from experience and retain knowledge across sessions. (23:45) Gomez compares this to how humans improve over time through experience, while AI models reset to their original state with each new conversation. He predicts this will be the next major advancement in AI, allowing models to become more valuable to users over time by learning their specific needs and contexts, similar to how an intern becomes more productive after months on the job.
Enterprise AI Deployment is Still in Early Stages
Despite the hype, enterprise AI adoption remains focused on basic tasks like email summarization and meeting notes. (60:41) Gomez observes that companies are just beginning to move from proof-of-concept pilots to full-scale deployments across entire organizations. The real opportunity lies in augmenting white-collar knowledge workers who represent a supply-constrained part of the economy. This suggests massive untapped potential for AI tools in professional settings.
AI Doomsday Messaging was Strategic Fear-Mongering
The apocalyptic AI safety rhetoric from major labs was primarily a competitive strategy rather than genuine concern. (31:00) Gomez argues this messaging was designed to "pull the ladder up" by scaring competitors, investors, and regulators away from AI development. He calls this "intellectually dishonest" and believes it did a disservice to the world by slowing beneficial AI deployment. Instead of fear, he advocates for accelerating AI adoption to solve real-world problems and drive economic growth.
Statistics & Facts
- When Cohere started, there were only 15-30 people on the planet who had trained language models, according to Gomez. (46:33) This highlights how nascent the field was just a few years ago and explains the current talent shortage in AI research.
- The entire transformer project that revolutionized AI happened in just 12-16 weeks (about four months) in 2017. (10:16) This compressed timeline shows how quickly foundational breakthroughs can emerge in technology.
- Cohere has raised $1.7 billion and operates with a constraint of fitting models within two GPUs maximum, while competitors like Meta's models require tens of GPUs. (35:58) This demonstrates different strategic approaches to AI deployment and cost management.
