I will just point out the benefit is not as obvious as you think. Developers have consistently overestimated LLM productivity gains, which still seems true for agentic AI: https://metr.org/blog/2026-05-11-ai-usage-survey/ It is particularly striking how similar the results are to LLMs before agents.
Along with the total absence of long-term data, I think the benefit can be (weakly) denied. Maybe not in the employmemt marketplace, but certainly for myself.
> I will just point out the benefit is not as obvious as you think. Developers have consistently overestimated LLM
I think there are two different claims here:
- developers overestimate productivity gains, which is a solid finding in many of these studies. Skepticism of extremely large productivity gains is warranted and I flatly disbelieve "10x uplift" claims.
- LLMs give no productivity uplift at all, which is much harder to defend. A repeat of the famous METR RCT study did find evidence of improved productivity, and this seems to align with the experience of many experts I trust.
Specifically my claim is "the relatively minor productivity uplift I would personally get out of agentic development is offset by the high cost, along with unresolved questions about long-term code maintainability, so I am not convinced that it is actually beneficial."
IMO the bigger problem is that ~1.5x individual dev productivity uplift seems to translate into 1.05x uplift across the team. People have been waaaaayyyyy too overconfident about this stuff.
I am both a career developer and experienced team manager. from first hand experience the 1.5x im getting from AI is not flowing down to my team / org because why would i output 50% more when the pay environment and leadership are already underwhelming. That additional 50% productivity goes completely to side projects built on my second computer between 9-5 tasks
Actually, the METR report speculates that some of the overreported productivity uplift comes from grabbing unnecessary low-hanging fruit, things like "oh I'll make a web dashboard to keep track of this stuff // wow that would have taken all day without Claude!" But in the olden days they would have just used a notepad. Yet psychologically they built a real thing and saved a lot of time.
> But in the olden days they would have just used a notepad. Yet psychologically they built a real thing and saved a lot of time.
There was an old Onion article about the monetary savings from buying clothes on sale as computed by (a) the women buying the clothes; and (b) their male relatives.
That’s what I’ve heard from my dev team too. They’re using it to give themselves free time while still being on the clock, not to produce more output for the company. Roughly thinking about hours spent on projects I think have gone up per task, the opposite that should be happening.
I'm finding it almost impossible to fill that free time with work, unless it's just reading emails and chat messages.
I can context switch between two or three chats, but doing so speeds up my agent use at the cost of making reviews and discovery harder, so it might come out in the wash.
I'm finding this is a skill which I'm slowwwwwwly improving but, for now, my level means I tend to miss mistakes I'd notice if just working on one task. I also had to make accommodations to the way I work to make it achievable, including telling my employer that a 500 GiB disk just isn't enough any more when I need multiple builds in parallel.
Your comment makes total sense to me but generally, in regards to productivity gains from AI, I can never understand where these are realized for people. Maybe I'm just a laggard but never found myself 25%-50% behind on anything or that much more work/items/tickets available.
Great point. Devs have relatively no incentive to be more productive for their orgs, and all the incentive to be more productive on their own personal work. The AI benefit to devs is real, just not for large enterprises IMO outside of automating clerical/mundane work.
> Devs have relatively no incentive to be more productive for their orgs, and all the incentive to be more productive on their own personal work.
AI definitely supercharges the side project myth, but I think we’re going to see waves of developers quitting or losing their jobs because they’re chasing dreams of running a side code project that isn’t going to turn into a business.
The incentive to be more productive at work is that they get to keep their job and not be replaced by a cheaper junior. I think we’re in for a reckoning across the industry as companies realize that there’s little difference between a lazy senior armed with Claude and a halfway motivated junior who has aspirations of growing into something more. The latter costs less and might grow into a better dev.
> because why would i output 50% more when the pay environment and leadership are already underwhelming
The productivity uplift measured in some of these reports is from more tasks being done, not from developers giving themselves more idle time during the day.
If we do see job losses from AI, my feeling is that it’s going to be concentrated in people with an attitude like this one where workers try to remain anchored to their old delivery pace. It’s becoming easier than ever to replace a worker in that position with a lower paid junior equipped with Claude who has an interest in learning. There’s no reason to keep a higher paid worker around to copy and paste between Jira and Claude.
thats absolutely true regarding wysiwyg projects like a small website. aside from temporary cost savings, systematically ignoring institutional knowledge in favor of mass output is not gonna do humanity any favors ..
Yeah institutional knowledge is hard to capture, even more with agentic coding.
I just compressed what would take a team a few months to deliver in one or two months, the amount of gotcha I spot from the existing system we work on everyday is staggering.
How to deliver and keep these feedback is difficult, the other teams are overwhelmed by other tasks, so any tickets or communications will be forgotten in a few months.
And as an external ressources allowed to use frontier model, I have a hard time to keep up with my own pace, and all that institutional knowledge will be lost both by my employer and client.
I have seen no productivity uplift at all from LLMs. They are at least about neutral these days (they used to be a productivity drain), but no gains at all. By the time I get done reviewing the code to make sure it hasn't done anything crazy, I've spent the same amount of time I would've taken to write the code myself. The only people I personally know who claim productivity gains are getting those gains by completely disregarding quality, or understanding the code, and just YOLO letting the LLM do everything without checking. I'm not willing to do that.
I find Claude is shockingly slow. If it were genuinely instantaneous, it would be worth it, but I often find myself twiddling my thumbs while waiting for it to cook.
> LLMs give no productivity uplift at all, which is much harder to defend
It’s not really hard to defend. Because when people says that productivity is uplifted, they are talking about amount of work, not the ROI. That’s why you keep hearing about LOC, amount of PR and prototypes, and the time taken is actually “time to PR” and not “time to production + time spent on bugs”.
There's another even more abstract liability on the balance sheet here for "deprecated LOC" or something, where the thing they built gets scrapped because no one wants to untangle the rat nest, or the "documentation" was so incoherent that it all had to be rewritten and reviewed.
Coming from legacy enterprise I am using claude to do exactly this for some of my team's more bizarre and poorly documented "inventions" that started out as their pet project and then became part of our official workflow.
Maybe we can make an analogy with taking the plane vs driving to your destination.
A plane literally goes 10x faster than a car, so a 5 hour drive becomes a 30 minutes flight. But if you have to drive to the airport, arrive early, pass check-in, security, boarding, then pick up your luggage on arrival, rent a car and drive to your destination, you may realize that your 30 minute flight took you more than 5 hours in total.
You also have to consider the Amdahl's law: a 10x speedup on 10% of the project is just a 9% speedup overall.
> You also have to consider the Amdahl's law: a 10x speedup on 10% of the project is just a 9% speedup overall.
This is totally true for me. I'm working on more features/bugs at work and getting the coding done faster with claude but the coding part is at maximum half of the process and usually less. The rest of it is clarifying the requirements from our product team or given that we're a very legacy place, what the existing state of our infrastructure and other systems are for the new changes to fit in to, etc.
> You also have to consider the Amdahl's law: a 10x speedup on 10% of the project is just a 9% speedup overall.
It's a 10% speedup, unless you think that a 100% speedup would reduce completion time to zero rather than cutting it in half.
Assuming that "10% of the project" is measured by time requirements, when you get a 10x speedup on that part of the project, you'll end up completing 100 units of work in 91 (formerly 100) units of time. Your work rate is then 100 / 91 = 1.099 times what it was before; that's an improvement of 10%, not 9%.
(Sanity check: suppose you have a project that will take 100 days. That project is reassigned to someone 10% faster than you. They will take 100 / 1.1 = 90.91 days to finish.
If they were 9% faster, they'd take 100 / 1.09 = 91.74 days to finish.
Does a savings of 9 out of 100 days of work look more like the project that saves 9.09 days, or the one that saves 8.26 days?)
In the former case, which seems more likely, does it not fall to reason that like any tool there'll be some that will benefit from it and some that it just won't work as well for, but are equally as good at their job? Like that neovim, unless your an absolute zealot you wouldnt insist that everyone use it, or hire someone based off it, but those who do work well with it do find it to be a boost to productivity.
Also importantly, can it be a neovim? Neovim hasn't had literal trillions invested over it in the space of just 5 years, local llm's aside can those smaller productivity gains justify the huge investment put into them, and which will continue to be needed for further model development?
Developers are cashing in on the productivity gains. Meaning instead of using the increased productivity to do more work, they just become lazier or do fun irrelevant side projects instead, where as before there was just not much time for such things. I have definitely procrastinated on work simply because I know I can swoop in with an LLM and do it all in 15 minutes, whereas before I would have spent a few hours.
This is how LLMs can both result in greater productivity yet still not appear to yield much more benefit than the pre-LLM era.
And there is no benefit to the developer for doing the work first then just sitting idle. That’s how you get people putting more things on your plate.
Some unsolicited, probably obvious advice for you for the future: never do that again.
Managers will abuse you if you expand your workday to make sure deadlines don't slip. They will see that the work is getting done and see no urgency to hire more help.
You have to surface capacity issues to management, learn to say "no" diplomatically, and/or "I have capacity for 2 things out of these 5, prioritize which ones you want first."
The LLMs are better at enforcing their boundaries than developers. Run out of tokens? That's it. No more code. It doesn't care how much the company needs this for the customer call tomorrow.
Odd, it was the opposite for me. Last year, no burnout. But then agents got good and everyone wanted more code more features faster and faster. I got burnt to a crisp. Now? I don’t give a fuck. Ok people want to push tons of code straight to prod? Go ahead, looks good to me… no need to review it if it’s with the latest frontier models right?
Exactly, I finally migrated that old service to typescript and cleaned up some really annoying tooling, for example. I’m more productive and can get quick fixes out very quickly. We’re also seeing agents help identify incidents faster than human investigation can. But big structural things that are hard to integrate still take lots of time.
A few months ago, I heard a keynote speaker claim that LLMs did not give any measurable productivity gains, except in two fields: customer support and software development, where they resulted in about 30% productivity improvement. I forgot if he presented any sources, and if he did, I forgot those, but it sounds plausible to me.
But instead of productivity, I'm much more interested in using it to improve quality. You've got a tireless reviewer who is always ready to review your code and catch any gaps.
I don't know how they're measuring productivity for customer support, but it certainly isn't in terms of customer satisfaction. People have never been happy with chatbots when they're trying to reach a human.
We are getting ai responses from cloud providers. From both aws and google, my teams have received obviously ai generated and non-applicable advice.
The latest was that we need to write in object pooling in their client code, an option that doesn't exist yet in _their_ client code. Like, yo, that is a you-problem on performance bottlenecks; we expect you to provide solutions.
I think the real disaster is that once you let the LLM work on a project for a bit, you start to lose understanding of what exactly is even happening under the hood in the project. You can take steps to mitigate this, but agents don't exactly encourage the behavior required to maintain a good understanding of what's going on.
When a person becomes a manager, they do or do not have enough time and expertise to review all of the code that they trust the team to produce.
Managers usually get into automated testing; unit tests, integration tests, acceptance tests, and maybe also BDD syntax
Managers and developers are responsible for setting a test coverage threshold for merge approval.
If there is 100% branch coverage test coverage for a codebase, what would coverage-guided fuzzing or property testing find? If there is 100% branch coverage test coverage for a codebase, what is the value of spending resources on formal verification?
How does the value of LLM-produced 100% branch coverage compare to no-LLM 100% branch coverage?
I've never really understood this argument. If someone's a manager of an incompetent team, no amount of management skill will save the quality of the resulting software. I don't think "just treat LLMs like smart junior developers" fixes this, because well-functioning teams usually also have senior developers to keep things on track. Like, if we handed a team of genius-but-junior developers to the best "people person" manager in the world (i.e. who doesn't actually read the code), would we really expect decent results? Even if the manager tested the code by hand? I honestly don't think so, at least once the software gets past a certain (fairly low) threshold of size/complexity.
If you have a "this is an incompetent team" attitude, that's what quality software you'll get back; whether or not there's LLM use.
If you have a "I trust my competent team to write good enough tests and test coverage isn't my responsibility" attitude, that's what quality software you'll get back.
There are people producing good and excellent quality software with LLMs. Presumably you must discard low-quality code in order to maintain quality.
There's certainly a limit to code quality with current models. On number of lines of code per unit of time, LLM tools certainly already win.
Can costly automated code review for PRs catch most of the problems before they're under consideration for merge?
For example,
the vscode repo has extensive copilot integration. Every PR gets auto code reviewed. But with their tokens or the contributors'?
If I take poor quality code (AI-assisted or not) and spend a few hundred dollars on tokens for a next gen model and agent to get to 100% coverage and review for security bugs and CWE common weaknesses, what quality code will I have without refactoring with proven patterns and type annotations and polishing docstrings?
I don't think this argument holds. The OP made the very good point that most quality software teams are composed of both seniors and juniors, with the seniors (and even staff members sometimes) drawing the lines so that the system stays on rails.
If you're a manager now and your "team" is a bunch of coding agents, those agents are hardly junior engineers at best. It is equivalent or even worse than hiring a team of 3-5 junior engineers and letting them run rampant with your code.
> If you have a "I trust my competent team to write good enough tests and test coverage isn't my responsibility" attitude, that's what quality software you'll get back.
As much as this sentiment is nice, it is completely divorced from reality, unless the competence is verifiably there. If you take a bunch of juniors and say "yeah I trust them to do everything well enough," you're going to have a disaster on your hands.
> How does the value of LLM-produced 100% branch coverage compare to no-LLM 100% branch coverage?
This is such a salient question. Sometimes (definitely not always) the test suites produced by LLMs are so trivial it's scary. Coverage can be an illusion for sure.
I have been seeing such trivial shit getting submitted and deleting them from the test suites, but it's disheartening to see that the devs didn't have the better taste to see those LLM "tests" are so trivial they border on noop before submitting them...
What new or existing metric would better assess test coverage than line coverage and branch coverage?
I wrote a tool called tert - I guess it's called an agent harness now - to run various test runners and log test output and coverage output to disk. FWIU stripping spaces from JSON does save tokens. It seems like feeding coverage lines-missing maps into the prompt results in better output, better LLM-authored tests.
"Refactor these tests for maintainability and coverage. Use fixtures, mocks, and parametrization"
You can write tests that cover every line/branch without actually verifying the code does what it’s supposed to. Relying on (unreviewed?) LLM written tests that provide coverage may not be a great idea.
We try to use types as much as possible. And if we can't, then lint rules, etc. Basically, get everything as far left in the SDLC as we can.
And, if/where we need tests, we write the source so they are few, high value, and complementary. Like actual unit tests, not complex with stuff like mocks just to generate trivial coverage.
Formal verification is always more valuable than mere testing, but it's hitherto more expensive. The thing that ultimately matters is closing the loop: how well do the tests match the requirements, both as written and as unwritten in the mind of the customer?
Working with an LLM has given me a real eye opener on unwritten requirements. It's like outsourcing. "Yes, you've given me what I wrote down, but I never expected you do to it in that way"
> When a person becomes a manager, they do or do not have enough time and expertise to review all of the code that they trust the team to produce.
> Managers usually get into automated testing; unit tests, integration tests, acceptance tests, and maybe also BDD syntax
I can see managers getting involved into acceptance tests, but never in the other type of tests. And the verification mostly is involved into a quick manual testing/watching a demo. Code is not their concern. When there's a bug, they expect you to investigate and fix it.
Given that it's so easy to generate better than test stubs with a model and manually review what it thinks covers, I think it's fair to expect a manager to run and add to the tests.
If devops has done their job, it should be trivial for a manager to contribute to the tests and run the build on git push (or manually re-run the build with the web UI).
If a manager has deploy rights, they should be able to run the tests.
I think what this fails to cover is that managers rely on other people to do that understanding and keeping things in mind. When there's a problem, the manager isn't just saying to a random person "go fix this", most of the time they're relying on somebody who has a deeper understanding of the problem and is not likely to randomly break something else just to fix this one bug.
Fearless refactoring is possible given test coverage? True in my experience; both with higher-risk human-developed software and with agent-coded software.
I haven't yet made myself learn the new swarm of concurrent agents with different specializations/agent_instructions methods yet.
Are multiple worktrees worth the cognitive burden and merge overhead?
If slop is fine (and sometimes it is), the benefits are undeniable. If the dev was the kind that would have produced slop anyway - again, undeniable boost.
If the quality needs to be high I think it actually can slow you down, though.
Agreed, but also to expand, if the dev is mediocre and ai-assisted coding is a skill, the productivity gains are 10x more mediocre code. Since code is a liability, this is not desirable. Hence, mediocre devs being more productive is an underestimated problem of the age of ai-assisted coding. I see this every day.
The result is a whole bunch of dysfunctional systems unnecessarily dislodging perfectly acceptable processes.
"Code is a liability" is the part that usually gets hand-waved, and there's a number on it now. Veracode's 2025 GenAI report ran 80 tasks across 100+ models: when a task could be solved securely or insecurely, the models picked the insecure path 45% of the time, at roughly 2.7x the vulnerability density of human-written code (Java was worst, at a 72% failure rate).
So the mediocre-dev case may be worse than "10x more mediocre code." It's more code that also skews insecure by default, and that cost shows up downstream in review and incidents, not at the PR. Throughput goes up, and so does the liability per line.
Along with the total absence of long-term data, I think the benefit can be (weakly) denied. Maybe not in the employmemt marketplace, but certainly for myself.