All notes

AI code grades high at review. Production disagrees

Your AI code is not lying to you. But the review you run on it can't see what production will.

I've argued that evals are taste made measurable — if you can't state what good looks like, the agent can't hit it. New Relic just put a number on the failure mode behind that argument, and it's a big one.

94% of technology leaders say AI-generated code looks better than human code at review. 78% say it caused more production incidents once it shipped. Both numbers come from the same survey.

What did the 2026 State of AI Coding report find?

A grading gap so wide it almost reads like a joke.

Transparency first: I read New Relic's press release and skimmed the report PDF — I haven't rerun the survey. It sampled 200 US technology decision-makers with Hanover Research and dropped June 10, 2026. At review time, 61% of leaders rate AI code "somewhat higher quality" and 33% "much higher." Only 2% rate it lower.

Then the code ships. 78% report more incidents. 86% report senior engineers spending more time firefighting. 74% say at least a quarter of the AI code needs significant rework after deployment. 82% had at least one production failure tied to AI code in the past six months.

The report's term for this is "agent debt" — the bill arrives later, in production, in someone else's sleep.

Why does good-looking code still break?

Because your review measures how code reads, and production measures how it behaves.

A diff review checks naming, structure, style — qualities an LLM has learned to render beautifully. It can't simulate real load, weird user input, or two requests hitting the same state at once. That's exactly where AI code tends to fall apart: the happy path is polished, the edges are untested, and the failure doesn't show up in a pull request. It shows up at 2am.

New Relic found the highest-confidence teams were the most exposed: nearly two-thirds (62%) ship AI code without verifying it line by line. Confidence isn't a bug here. It's the mechanism. The code looks so good that the instinct to test real behavior goes quiet.

There's supporting history: a METR randomized trial found experienced open-source developers actually finished tasks 19% slower with AI tools — the perceived speed and the measured outcome point in opposite directions. Trust the dashboard, not the feeling.

What does agent debt cost a solo founder?

You don't have an on-call rotation. You are the on-call rotation.

A team pays agent debt in shifts. You pay it in focus — the exact thing your small product runs on. One broken deploy costs a day of debugging you'd planned to spend on the next feature, and you have no senior engineer to absorb the firefighting. Meaning the "86% more senior-engineer time" line, for us, is just "more of you."

The good news: debt you can name is debt you can limit. Keep every AI diff small enough that its blast radius is one feature. Ship behind a redirect or a feature flag so the rollback is minutes, not an archaeology dig. The discipline the big teams are rebuilding, you can afford to hold from day one.

What to check instead this week

Three swaps, all free, all before Friday.

Grade the run, not the read. After an agent finishes, don't ask "does this look right?" Ask "what did you run to prove it works?" If the answer is nothing, the diff is a hypothesis, not a feature.

Cover one unhappy path per PR. Feed it an empty form, a double-click, a network timeout. AI code rarely fails on the happy path — and review almost never feeds it an unhappy one.

Get told when it breaks. Error tracking with a five-minute alert is the cheapest engineer you'll ever hire. Production is the only review that counts, so at least it should be able to raise its hand.

The model didn't get worse since June. The audit did. Close that gap and agent debt is just a nice name for ordinary engineering — with better smells.

Frequently asked questions

  • New Relic coined the term to describe the operational burden that builds up when AI-generated code reaches production without enough oversight. The code reads clean at review, but the team pays later in incidents, rework, and senior-engineer firefighting — 82% of surveyed leaders had at least one production failure tied to AI code within six months.

About the author

mosh

mosh is a product designer for growth, working with design thinking and ever-improving design systems. What matters: fixing conversion, whether in B2B dashboards or direct-consumer apps.

Keep reading