Many people feel they are more productive with AI - and certainly bosses *expect* us to be. But a [study by Metr](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) in July 2025 advises some caution in that perception. It found that experienced developers using AI for a task took 19% longer than those without - with a significant underperformance compared to their expectations. ![[Pasted image 20250716212021.png]] What's interesting is that even after they had undertaken the experiment, developers still had an overly optimistic view of their performance Instead of purely coding, they spent significant time on composing prompts and waiting for AI responses, as well as addressing the issues in AI generated code, and this despite having experience in LLMs. There are a number of factors to consider for this experiment, however. Firstly, the developers were highly experienced, meaning there may be less of a % improvement that AI could add: less experienced developers may see greater gains of efficiency. Furthermore, they were aware they were taking part in a trial, which means they might have adopted different usage of AI than in the workplace, possibly augmented by them being paid by the hour for the experiment! The tasks selected for the experiment mirrored those in the real world - bug fixes, refactors, etc, - which may be too challenging for AI as it exists today. Smaller, self contained tasks may well have given different results: but it is, of course, real world tasks that we need to perform every day! So it's important, at least at this stage of AI's development, to be cautious about estimates and to be sure to undertake pilot studies with workflow measurements. Source: https://arxiv.org/abs/2507.09089