Your AI coding rollout is working. You just can't prove it. Here's why that's the real problem.

Six months into the AI coding rollout, your board asks the question you've been dreading:

"What's the ROI?"

You have anecdotes. A staff engineer who shipped a migration in two days instead of two weeks. A team lead who says her PRs are cleaner. Cursor and Copilot usage dashboards showing 70%+ adoption. A McKinsey chart with the curve angled upward in all the places curves are supposed to angle upward.

What you don't have is a number you'd stake your credibility on.

So you hedge. You quote the adoption rate, because at least that's measurable. You mention "developer satisfaction is up." You promise a more rigorous measurement framework next quarter. Your board nods politely. They've heard this answer before, from the cloud migration, from the DevOps transformation, from the microservices rewrite. They are running out of patience for transformation programs that produce vibes instead of velocity.

This piece is about why the productivity gains are almost certainly real, why your current measurement approach can't see them, and what a CTO needs to put in place before the next board cycle to stop guessing.

Why the obvious metrics don't work

The instinct is to measure what's easy to measure. Lines of code per developer. PRs merged per sprint. Story points completed. Cycle time. Velocity.

Every one of these has a fatal flaw in an AI-assisted environment.

Lines of code go up, but most of them aren't written by humans anymore. A developer accepting a 40-line Cursor suggestion looks identical, in your metrics, to a developer who hand-wrote 40 lines. You're now measuring AI throughput, not human throughput, and you're calling it developer productivity.

PR counts and story points are gameable by exactly the people you're measuring. Sprint planning was always a negotiation. With AI assistance, that negotiation tilts further. Teams that finish early pull in more work. Teams that don't, don't. Your velocity chart goes up across the board and tells you nothing.

Cycle time captures the wrong window. AI's biggest contribution isn't faster coding. It's compressed time-to-first-commit on unfamiliar code. That happens before the PR opens. By the time cycle time starts counting, the gain has already happened and you've missed it.

The metrics aren't broken because measurement is hard. They're broken because they were designed for an era when the developer was the bottleneck. The developer is no longer the bottleneck. Which means the metric has to move upstream.

What's actually changed

Across the AI coding rollouts I've worked through over the last two quarters — multiple engineering teams, different domains, same delivery framework — one pattern keeps showing up.

Throughput doesn't change linearly. It changes bimodally. Some teams move close to 2x faster. Others move barely at all, sometimes slower. Adoption rates are nearly identical across both groups. Tool usage looks the same. The developers in the slow group aren't lazy or resistant. They're using the tools constantly.

The difference is upstream of the IDE.

Teams that get the productivity lift have specs that are AI-legible — clear acceptance criteria, named entities, explicit constraints, examples. Teams that don't, have specs written for humans who will fill in the gaps with tribal knowledge and a Slack thread. AI tools can't read tribal knowledge. They generate plausible-looking code that misses the actual intent, the developer spends an hour debugging the hallucination, and the productivity gain evaporates.

This means the gating factor on AI productivity isn't the developer or the tool. It's the spec.

And spec quality is something you can measure, before a single line of code gets written.

What to measure instead

Three signals, all leading indicators, all measurable from artifacts you already have:

1. Spec Readiness at sprint entry. Before a PBI enters a sprint, can it be scored green/yellow/red on whether it contains the structured context an AI tool needs — entities, constraints, acceptance criteria, examples? Green specs produce dramatically higher throughput. Red specs produce thrash. The percentage of green specs entering your sprints is the strongest leading indicator of whether your AI investment will pay back.

2. AI Code Modification Ratio. What percentage of AI-suggested code survives unmodified into production? High survival rates (>60%) indicate the upstream context is good. Low survival rates (<30%) mean your developers are spending their time debugging AI output instead of writing code. This is measurable from Git history and IDE telemetry your tools already capture.

3. Time from PBI ready to first commit. Not cycle time. The window between "PBI is workable" and "first line of code lands in the branch." This is where AI's biggest gain hides. If this number isn't dropping quarter over quarter, your AI rollout is not producing engineering speed regardless of what the adoption dashboard says.

None of these are perfect. All three are better than what you have now.

What this means for your next board cycle

If your AI productivity story for the next board meeting is still going to be "adoption is at X% and developer satisfaction is up," you have about 90 days to change it.

The honest version of the conversation is:

"We measured our rollout wrong for the first six months. The real bottleneck wasn't tool adoption — it was spec quality entering the sprint. We're now measuring three leading indicators that predict whether AI is producing throughput. Here's where we are on each, and here's where we'll be next quarter."

That conversation lands differently than the vibes version. It tells your board you understand what's actually happening in your engineering org, that you have a falsifiable hypothesis, and that you're not asking them to take productivity gains on faith.

Most engineering leaders running AI coding rollouts right now don't have this conversation ready. The ones who do — who can name three falsifiable signals and report on them quarter over quarter — will keep their programs through the next budget review. The rest will spend the meeting defending adoption rates against a board that's stopped accepting them as evidence.

Vinod Narayanswamy is an AI-Native Delivery Advisor and architect of the Cadence/AIDLC methodology for AI-native engineering teams. 17+ years in enterprise delivery; Scrum@Scale co-trainer with Jeff Sutherland. He works with engineering leadership on measurement frameworks for AI coding rollouts. Reach him on LinkedIn.