Does AI actually help productivity?

Are we fooling ourselves?

Earlier this month, METR published research showing the counter-intuitive finding that software developers were LESS productive (by 19%) when using AI tools. Further, the developers own estimates of their productivity were wrong - they guessed that they were 20% MORE efficient. This was in a well-controlled and randomized experiment, testing with developers using AI tools in code that they had a lot of experience in.

Other formal studies have found productivity effects, so we shouldn't overreact to one study, but I think the real interesting finding was the self-deception - participants thinking they were more efficient with AI when they actually were not. Do we have a similar issue with AI in proposals?

In my experience with proposals and generative AI, if I can narrow down the guidance to a specific section and provide enough guidance, I can get good outputs that save me a notable amount of time. But I've also been the Color Team reviewer on sections that were obviously AI generated without much thought, and the rework and time lost on that made it clear that AI just made it worse. In the study, these developers were all working in an area they were very experienced - perhaps a similar dynamic happens in proposals that require detailed thinking and domain expertise?

I wonder how many of us in the proposal management community are fooling ourselves like the developers in this study?

At Proposability, we don't think proposal managers are just mindless prompters for an AI. We view proposal managers as conductors who guide a diverse orchestra to create something beautiful. Proposability helps you organize your thinking so that you can communicate well, to either your authors or an AI of your choosing. Contact us for a demo to learn more.

Ref: METR study: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity - METR