How Do You Know AI Actually Helped?

Now that tools are cheap, one conversation has become very common. Three months after rollout, the owner asks about results, and the person in charge says: "Everyone says it's a lot faster." The owner nods and renews. Three months later, same question, same answer, another renewal. The third time around, the owner finally follows up: how much faster is "a lot faster"?

In that moment, nobody can answer. Nobody wrote down at the start what it would be compared against, so three months later there's naturally nothing to compare.

Over the past few months, this series has been about what comes before you move: first see clearly which step you're on, then decide whether something is worth doing at all. This next stretch is about what comes after. The thing is already built. How do you know it's actually running?

The first thing is this sheet of paper.

01"It helped" says nothing on its own

Let's start with a premise that's easy to skip. Vague words let two people nod along in the same meeting while meaning different things.

Even ROI, a term that looks very hard-edged, works this way. One ROI cheat sheet breaks it into five kinds: return on ad spend (ROAS), project ROI, break-even ROAS, annualized return, and excess return. Five calculations, five conclusions. The same project can be profitable measured by ROAS, and a loss once you deduct costs and taxes and switch to project ROI.

That cheat sheet's conclusion is blunt: next time you hear ROI, first ask which kind it's actually calculating.

"It helped" is even looser than ROI. It can mean saving time, making fewer mistakes, no longer outsourcing something you used to outsource, or simply that everyone is happier using it. None of these four is unimportant, but each one is verified in a completely different way, and each leads to a completely different next step.

So the question isn't "which metric should we use." It's whether you've stated clearly, up front, what you're verifying. If you can't, every metric is just an answer to a question nobody defined.

02Acceptance criteria written after the fact always pass

This is the one thing this piece really wants to say, and it has nothing to do with metrics. It has to do with timing.

Say the rollout has been running for three months, and now the owner asks for a results assessment after the fact. Who writes it? Usually the person who pushed for the rollout, or the person who signed off on the budget.

You don't really need to guess what that assessment will look like.

Nobody writes a condition that the thing they bought can't pass. This isn't about integrity. When you only start writing after the fact, there's already a pile of ready-made results and ready-made data in front of you, and you pick whichever items come easily. So the acceptance criteria end up written to fit exactly what already happened. The metrics are the flattering ones; the baseline is the worst month.

What gets written can't be called false. Every number genuinely exists. It's just been selected.

Write it before rollout and the situation is completely different. At that point there are no numbers to pick from, only a promise that hasn't come due. You're forced to be specific: which action you're improving, how long it takes now, how far down you want to push it after the change, and who has the authority to say this one passed. Four sentences; written out, they're one paragraph long. And in that moment you'll find some fields you can't fill in. Not being able to fill them in is itself the most valuable checkup you'll get.

So this sheet of paper isn't a box to tick after the fact. It's for your earlier self to draw a line for your later self, one your later self can't quietly move.

One roadmap for enterprise AI transformation sets the pilot stage's task as verifying results first, and only then talking about scaling company-wide. Another, a six-step closed loop for companies with limited resources, spells the order out even more plainly: choose a scenario, try tools, verify results, turn it into an SOP, scale it up. Both put "verify" ahead of "scale." But both assume something they never say out loud: the standard for verifying has to exist before the act of verifying.

One set of notes on task orchestration boils this down to four words: accept first, delegate second. The order is spelled out clearly. "First" means first in time.

03One action, not four metrics

The fastest way to keep this from sliding into a list of metrics is to compress it into one sentence.

One sentence, four blanks:

Which action / How long it took before / How long it takes now / Who decides it passed

Note that these four blanks aren't four KPIs. They're four sides of the same action. When you fill it in, you may pick only one action, and catch-all phrases like "overall efficiency" or "customer service quality" aren't allowed. A phrase that covers everything turns out, on acceptance day, to cover nothing.

Filled in, it looks roughly like this: drafting the five client proposals each week used to take three hours per proposal; with AI, it has to come down to under an hour; and it counts as passing if the sales manager reads the draft and thinks it can go straight out.

Once this sentence is written, a lot of questions answer themselves. What do you compare in three months? Three hours against one hour. Who has the final say? The sales manager. What counts as failure? If the manager has to heavily rework every single draft, it didn't pass, no matter how smooth it feels to the team.

One set of notes on deciding whether to take on a client job uses almost exactly the same structure. Before taking the work, ask three things: who's paying, what result they're buying, and how it gets accepted. If you can't answer, you're just part of someone else's sales funnel. Those notes leave an even harsher line: between being able to do it and being able to make money from it lies an acceptance standard.

That was written from the contractor's side. Flip it around, and the company adopting AI is the client paying the bill. If the client can't answer "what result am I buying, and how will it be accepted" either, then what this money buys is, in practice, whatever the vendor says it is.

Another framework for deciding whether an AI feature is worth building has four dimensions, one of which is called "boundary assessment": it asks whether the scenario can actually tolerate errors. That's the "who decides it passed" blank in the sentence above. In low-tolerance scenarios, the person who decides has to be a human, and has to be named; only in high-tolerance scenarios can you hand that call over.

04What about exploratory work?

Here we have to face an objection honestly, because it's a real one.

Some rollouts were never meant to save time in the first place. Trying AI on reading contracts, organizing meeting notes, or comparing competitors: at the start you don't know what these will turn into, or what they're supposed to save. Demanding acceptance criteria up front sounds like strangling something before it's taken shape.

The objection's premise is right: exploration really shouldn't be judged by output efficiency. But the conclusion doesn't hold. Exploration has acceptance criteria too; it just verifies something different.

What exploration verifies is "what did we learn." Same four blanks, same single sentence: within three months, find out whether AI can reliably read the company's contract formats; the bar is twenty randomly sampled contracts with the key clauses caught correctly in at least sixteen; legal makes the call. This can be written down just the same, signed off just the same, and answered in three months just the same.

Exploration that can't produce criteria like this usually isn't too forward-looking. It's that nobody has thought clearly about what's being explored. The right move then is to go back and define the problem, not to wave it through first.

One piece on why enterprise AI fails to take hold traces the root cause to organizations never having written their judgment down. The data records the outcomes; the grounds for each judgment aren't recorded anywhere. Acceptance criteria are the smallest unit of exactly that kind of judgment. If you haven't even written down what "good" means, of course AI doesn't know which direction to go.

05What this sheet is really for: the nerve to call a stop

The most common misunderstanding about writing acceptance criteria is thinking its purpose is to prove success.

It isn't. Its purpose is to make "it didn't work" a conclusion you can say out loud too.

Without this sheet, a tool that's been in use for six months can't be touched. Want to expand it? You can't say on what grounds. Want to cut it? You can say even less, since everyone says it's useful and cutting it looks like dismissing your colleagues. So the easiest option is always to renew, year after year. This is exactly the pattern of applause for the demo, no sign of Beta 2: the first version gets cheers, and after that there's no mechanism to push it forward, and no mechanism to let it end with dignity. When the same source discusses a thirty-day restart, it puts "fill in the boundaries and acceptance criteria" one step ahead of designing the mechanisms. The same order, again.

With the sheet, it's different. Three months are up, so you lay it out: the target was under an hour, the actual is an hour and a half, and the manager has to heavily rework six drafts out of ten. That didn't pass. Not passing doesn't mean a full retreat. Maybe the scenario was the wrong pick, maybe the raw material wasn't ready. But at least now you can honestly say "this version didn't pass," and then decide whether to switch scenarios or pull out.

Before You rolled out a system and didn't know whether it worked until the year-end reports came in

Now One sentence on paper gives you, in three months, an answer that can be checked

And the most expensive thing on that sheet is the "who decides it passed" blank. It forces you, before you spend any money, to designate one person in the organization and give them the responsibility of saying "no" on acceptance day. Most rollouts didn't fail. It's just that nobody was ever authorized to declare that they had.

Start by writing that sentence

Tools are cheap enough to try any time, and that's a good thing. The price is that you have to supply the discipline of trial and error yourself: the cheaper the tools, the less anyone forces you to think through, first, what you're verifying.

Back to the scene at the start. The owner asks how much faster "a lot faster" is, and the reason nobody can answer isn't in that moment. It's in the sheet of paper that didn't get written three months earlier.

It's also the first document we ask for when we run an AI workflow diagnostic. Not the tool list, not the usage report. First we look at whether anyone wrote this one sentence down at rollout, and whether that sentence can still be found. If it can, the conversation that follows moves fast. If it can't, then we start by writing it. It's never too late for this; it just gets more expensive the later you write it.

Let AI move fast. Let people hold the anchor. Feelings are fine for discussion. They're not grounds for renewal.

Rooted in craft. Built for the new wild.