AI and the great developer speed-up, with Joel Becker of METR
Summary
Patrick McKenzie and Joel Becker discuss METR's research on AI coding assistants, revealing that expert open-source developers took 19% longer to complete tasks with AI tools, contrary to their own retrospective beliefs of being 20% faster. The conversation delves into the difficulty of accurately assessing AI productivity, the industrial organization of software development, the rapid evolution of AI tools, and the broader implications for AI's recursive self-improvement, security risks, and the future of the developer job market.Episode description
This week on Complex Systems, Patrick McKenzie (patio11) is joined by Joel Becker from METR. They discuss groundbreaking research on AI coding assistants.
Joel et al’s randomized controlled trial of 16 expert developers working on major open source projects revealed a counterintuitive finding: despite predictions of 24-40% speed improvements, developers actually took 19% longer to complete tasks when using AI tools, even though they retrospectively believed they were 20% faster. The conversation explores why even sophisticated professionals struggle to accurately assess their own productivity with AI tools, the industrial organization of software development, and the implications for AI's recursive self-improvement in research and development. It also touches on other perspectives from software developers using these tools professionally, and where we can expect them to improve rapidly.
–
Full transcript available here: www.complexsystemspodcast.com/the-great-developer-speed-up-with-joel-becker/
–
Sponsor:
This episode is brought to you by Mercury, the fintech trusted by 200K+ companies — from first milestones to running complex systems. Mercury offers banking that truly understands startups and scales with them. Start today at Mercury.com
Mercury is a financial technology company, not a bank. Banking services provided by Choice Financial Group, Column N.A., and Evolve Bank & Trust; Members FDIC.
–
Recommended in this episode:
- METR: https://metr.org/
- Joel Becker’s site: https://joel-becker.com/
–
Timestamps:
(00:00) Intro
(00:34) Understanding AI evaluation methods
(02:04) METR's unique approach to AI evaluation
(03:10) The evolution of AI capabilities
(06:44) AI as coding assistants
(09:15) Research on AI's impact on developer productivity
(13:55) Sponsor: Mercury
(15:07) Challenges in measuring developer productivity
(20:38) Insights from the research paper
(31:26) The formalities of software development
(32:07) Automated tools and human discussions
(32:47) AI and style transfer in software
(34:35) The role of comments in AI coding
(36:51) The future of AI in software engineering
(40:25) Economic implications of AI in software
(46:53) Challenges and risks of AI in software
(59:03) Security concerns with AI-generated code
(01:04:59) Wrap