When we talk about the value of AI, we tend to measure it with speed. It’s the easiest thing to see and the easiest thing to put a number on. Work can be done faster. It makes sense.

​

But speed only counts if what's coming out is good. Which brings up a harder question, and the one no one really has a concrete handle on: how do you measure ‘good’?

​

Accuracy is a place to start. When there’s fewer errors, fewer things to redo and outputs you can check against the source, that’s a clear measurement. But accuracy really only tells you that AI didn't make things worse. It doesn't tell you it made anything better.

​

'Better' is the real question, and the first thing to realize is that ‘better’ isn't one thing. For some work, better is getting the thinking right. Not accuracy, but whether the system made the calls a smart person would make. Did it pull the right source material. Did it frame it for the client the right way. Did it know what mattered to put in and what to leave out. A system can be accurate to the data and get the thinking wrong.

 

For other work, better is AI coming up with new ideas that you never would have thought of. Not a remixed version of what you handed it, but a perspective that looks at things in a different, smart, creative way that solves the challenge at hand.

​

Two different ways to look at it, and there are more. Before you can measure whether AI is any good, you have to know what ‘better’ your work is aiming for.

​

Where I’ve landed is that better isn't pass or fail on a single output. It's a hit rate. And the right hit rate depends on the work. A creative idea system I built gives me twenty so I can use three. A proposal system needs to be right every time. They can’t be judged on the same scale because of the nature of the work.

Take the creative system and three-out-of-twenty ratio. To start, the twenty aren't twenty random thoughts. That wouldn’t need a system. But every one of the twenty cleared the mandatories: on brand, on brief and had originality to it. And still only 3 passed the bar of being ‘good.’ This sounds like failure but it’s the same standard we hold people to. Your best thinkers swing and miss most of the time on the hard stuff. 17 dead ideas and three good ones to work with is a great day. With people, we accept that bad ideas are part of how good ones get made. AI doesn't get the same leeway. One bad output and people decide the whole thing doesn't work. But a system built for creative work is only working if it has misses. The misses are the cost of the hits.

​

So the real move happens before any of this. The hit rate isn't something you give the system once it's running. It's the thing you have to know going in. Decide the work needs to be right almost every time, and you build one kind of system. It’s a system that’s tight and cross-checked and conservative. Decide it needs to create twenty ideas so three can land, and you build something completely different. You build a system that’s loose and generative and going to be wrong most of the time. Get that hit rate wrong at the start and you've built the wrong thing. The metric isn’t just a way to measure the system, it’s the blueprint.