AI Backfires: Study Finds Software Developers More Productive Without Assistance
A new study challenges the narrative of artificial intelligence as an automatic productivity booster, revealing that experienced software developers actually completed tasks faster when working without AI tools. Published this month, the research from the nonprofit Model Evaluation and Threat Research (METR) suggests a cautious approach to AI implementation in the workplace is warranted, particularly for skilled professionals.
The findings stem from an experiment involving 16 software developers with an average of five years of experience. Researchers tasked them with completing 246 work-related tasks, allowing the use of AI – primarily the code editor Cursor Pro and chatbots Claude 3.5/3.7 Sonnet – for half of them, and requiring a traditional, AI-free approach for the other half.
Developers initially predicted that AI would reduce their task completion time by an average of 24%. The reality proved strikingly different. Instead, tasks completed with AI assistance took 19% longer than those completed without, effectively reversing expectations. “While I like to believe that my productivity didn’t suffer while using AI for my tasks, it’s not unlikely that it might not have helped me as much as I anticipated or maybe even hampered my efforts,” one participant wrote in a blog post detailing their experience.
The “Tortoise and the Hare” of Software Development
The study’s results echo the classic fable of the tortoise and the hare, highlighting the potential for overconfidence in new technology. Researchers found that the developers’ existing expertise and contextual understanding often clashed with the outputs generated by AI. They spent significant time adapting AI-generated code to fit their projects and, crucially, debugging errors.
“The majority of developers who participated in the study noted that even when they get AI outputs that are generally useful to them—and speak to the fact that AI generally can often do bits of very impressive work—these developers have to spend a lot of time cleaning up the resulting code to make it actually fit for the project,” explained study author Nate Rush. The process wasn’t simply about accepting a finished product; it involved substantial refinement.
Furthermore, developers reported losing time crafting effective prompts for the AI chatbots and waiting for the technology to generate responses. This overhead, the study suggests, negated any potential time savings.
Challenging Economic Projections
The findings directly contradict optimistic forecasts about AI’s transformative economic impact. Predictions of a 15% boost to U.S. GDP by 2035 and a 25% increase in overall productivity now appear increasingly ambitious in light of this research.
However, Rush and his colleague Joel Becker cautioned against drawing broad conclusions. They emphasized the small sample size and the fact that the study focused on developers new to these specific AI tools. The authors acknowledged that future iterations of AI technology could potentially overcome these initial hurdles.
The primary goal of the study, they stated, was to encourage a more measured and data-driven approach to AI implementation. “Some of the decisions we’re making right now around development and deployment of these systems are potentially very high consequence,” Rush said. “If we’re going to do that, let’s not just take the obvious answer. Let’s make high-quality measurements.”
Diminishing Returns for Skilled Workers?
Economists are already taking note. Experts suggest that while AI may be effective at automating entry-level tasks, its benefits for highly skilled workers like experienced software developers may be limited. LinkedIn’s chief economic opportunity officer, Aneesh Raman, has observed that AI is beginning to impact entry-level positions.
Anders Humlum, an assistant professor of economics at the University of Chicago’s Booth School of Business, echoed this sentiment. “For those people who have already had five years of experience, maybe it’s not their main task that we should look for and force them to start using these tools if they’re already well functioning in the job with their existing work methods,” he stated. Humlum’s own research, conducted in May among 25,000 workers in Denmark, showed a modest 3% productivity improvement among those using AI tools.
This aligns with the views of MIT economist and Nobel laureate Daron Acemoglu, who argues that only 4.6% of tasks within the U.S. economy will actually be made more efficient by AI. Acemoglu warns that a rush to automate everything, even processes that shouldn’t be, will lead to wasted time and energy, failing to deliver the promised productivity gains. “The hard truth is that getting productivity gains from any technology requires organizational adjustment, a range of complementary investments, and improvements in worker skills, via training and on-the-job learning,” he previously wrote for Fortune.
A Call for Cautious Implementation
The METR study underscores the importance of carefully considering when and how AI tools are implemented. Previous research on AI productivity often relied on self-reported data or focused on isolated tasks. This new study, however, provides valuable insights from skilled workers navigating real-world challenges with the technology.
“In the real world, many tasks are not as easy as just typing into ChatGPT,” Humlum explained. “Many experts have a lot of experience [they’ve] accumulated that is highly beneficial, and we should not just ignore that and give up on that valuable expertise that has been accumulated.” He concluded, “I would just take this as a good reminder to be very cautious about when to use these tools.”
