Coders spent more time prompting and reviewing AI generations than they saved on coding.

Coders spent more time prompting and reviewing AI generations than they saved on coding. On the surface, METR’s results seem to contradict other benchmarks and experiments that demonstrate increases in coding efficiency when AI tools are used. But those often also measure productivity in terms of total lines of code or the number of discrete tasks/code commits/pull requests completed, all of which can be poor proxies for actual coding efficiency. These factors lead the researchers to conclude that current AI coding tools may be particularly ill-suited to “settings with very high quality standards, or with many implicit requirements (e.g., relating to documentation, testing coverage, or linting/formatting) that take humans substantial time to learn.” While those factors may not apply in “many realistic, economically relevant settings” involving simpler code bases, they could limit the impact of AI tools in this study and similar real-world situations.

  • HubertManne@piefed.socialEnglish
    352·
    2 days ago

    This does not seem surprising to me:

    “Overall, the developers in the study accepted less than 44 percent of the code generated by AI without modification. A majority of the developers reported needing to make changes to the code generated by their AI companion, and a total of 9 percent of the total task time in the “AI-assisted” portion of the study was taken up by this kind of review.”

    It sounds about right. The AI should be acting as an assistant. The big question to me is if the code that comes out 19% slower is at all of higher quality. Since the coder is doing more correction and review does it act a bit like a second set of eyes or a pho sort of collaboration. If so it could still be helpful. Granted my experience so far is that most of what it does can be done with plugins to an ide but like it is sorta handy to have it all set and going after an installation without having to find and start using the plugins. Im still worried about energy usage with these things but hoping that can be worked out and honestly im not sure if the energy usage for something integrated with an ide or such is as bad.

    • vortic@lemmy.worldEnglish
      27·
      2 days ago

      Ad a fairly senior developer, I’m not at all surprised. AI speeds me up in some circumstances like writing boilerplate; things like kubernetes manifests. It does not speed up my coding, but it does help me explore options, expand my knowledge, and point me down the right track on new methods and packages. It also lets me do things I wouldn’t normally bother with, but which are good practice like finding edge cases for unit tests, packaging for multiple architectures, writing scripts to profile my code, etc.

      Essentially, I’m likely slower writing code with AI assistance but I think the code is higher quality because it let’s me quickly assess many options and implement best practices that are normally tedious to implement manually.

      I almost never accept code AI has written without modification, but I think I gain a lot from its use.