Coders spent more time prompting and reviewing AI generations than they saved on coding. On the surface, METR’s results seem to contradict other benchmarks and experiments that demonstrate increases in coding efficiency when AI tools are used. But those often also measure productivity in terms of total lines of code or the number of discrete tasks/code commits/pull requests completed, all of which can be poor proxies for actual coding efficiency. These factors lead the researchers to conclude that current AI coding tools may be particularly ill-suited to “settings with very high quality standards, or with many implicit requirements (e.g., relating to documentation, testing coverage, or linting/formatting) that take humans substantial time to learn.” While those factors may not apply in “many realistic, economically relevant settings” involving simpler code bases, they could limit the impact of AI tools in this study and similar real-world situations.
Ad a fairly senior developer, I’m not at all surprised. AI speeds me up in some circumstances like writing boilerplate; things like kubernetes manifests. It does not speed up my coding, but it does help me explore options, expand my knowledge, and point me down the right track on new methods and packages. It also lets me do things I wouldn’t normally bother with, but which are good practice like finding edge cases for unit tests, packaging for multiple architectures, writing scripts to profile my code, etc.
Essentially, I’m likely slower writing code with AI assistance but I think the code is higher quality because it let’s me quickly assess many options and implement best practices that are normally tedious to implement manually.
I almost never accept code AI has written without modification, but I think I gain a lot from its use.