Evaluation First: Building AI Systems That Keep Working

Why the companies that win with AI are the ones that decided what "correct" means before they decided which model to use.

The month six problem

Every company that has run an AI pilot knows how the story goes. Week one is a triumph. Someone demos an assistant that answers questions nobody thought a machine could answer, the room is impressed, and a budget appears. Month six is quieter. The system is still running, technically. People have quietly gone back to the spreadsheet. Nobody can say exactly when it got worse, or by how much, or why.

The instinct is to blame the model. It is almost never the model. A schema changed upstream. A vendor shipped a new version of an endpoint. The person who tuned the prompts moved to another team. A business rule that was true in March stopped being true in June. Each of these is a small event. Together they are the reason most AI pilots never become AI products.

What is missing is not intelligence. It is a mechanism that notices.

Evaluation is not a report card

In most organizations, evaluation shows up at the end. The system is built, then someone is asked to check whether it works, usually by clicking around for an afternoon. If the project is mature, there is a dashboard. The dashboard is real, the numbers on it are real, and it is still the wrong shape, because a dashboard is something you look at after a decision has already been made. By the time a chart shows a decline, the bad answers have already reached customers.

Evaluation first inverts the order. Before anything is built, you write down what a correct answer looks like for this business, in this context, right now. Not a vague aspiration like "accurate and helpful." A concrete set of real questions with answers your best person agrees with, and the rules that must never be broken. That artifact is the specification. The system is then built toward it, and more importantly, the system is not allowed to ship unless it meets it.

This is not a new idea. It is the same move the software industry has already made twice. Type systems took a class of correctness questions out of code review and made them a property of compilation. Continuous integration took "does the build still work" out of the tester's hands and made it a gate that blocks a merge. Both felt like bureaucracy when they arrived. Both are now invisible infrastructure that nobody would work without.

AI needs the same move, with one difference that makes it harder and more interesting. For ordinary software, correctness can often be proven. For an AI capability, correctness has to be measured. You cannot prove that a summary is faithful or that an answer is grounded. You can only define a standard, sample against it, and refuse to promote anything that falls short.

So the question is not whether your AI system is accurate. It is whether your organization has a standard that a system can fail, and the discipline to act when it does.

What this actually buys a business

Framed as engineering hygiene, evaluation first sounds like something to do later, when there is time. Framed correctly, it is the thing that decides whether AI becomes an asset or a recurring expense.

It makes change safe. This is the biggest one and the least obvious. Without a standard, every meaningful change is a leap of faith. Should you switch to the cheaper model that just launched? Nobody knows, so you either take the risk blind or you do not take it at all, and both answers cost money. With a standard, that question becomes an experiment that finishes by Tuesday and reports a number. Flexibility without evaluation is a liability. Flexibility with evaluation is leverage.

It survives people leaving. A capability validated by a conversation lives inside whoever had the conversation. When they change roles, the knowledge goes with them and the system starts to rot silently. A capability validated by an explicit, executable standard belongs to the company.

It makes AI governable. Every serious buyer, auditor, and regulator is converging on the same question: how do you know it still works? "Our team keeps an eye on it" is not an answer that survives contact with a procurement review. A standard that runs on every change, with a record of what passed and what did not, is.

It changes what you accumulate. This is the part I would put on a wall. When you build this way, the durable asset is not the model, and it is not the prompts. It is the evaluation set: the encoded, tested, hard won definition of what "correct" means inside your specific business. Models get better and cheaper every few months, and everything you built on top of the old one gets thrown away. Your definition of correct does not expire. It is the only part of an AI investment that compounds.

The flywheel

Most AI programs are structured linearly. Someone builds capability one. Then someone builds capability two, which costs about as much as capability one did. Meanwhile capability one starts to decay, because nothing is watching it. Add enough capabilities and the entire team is absorbed by maintenance. This is the model behind almost every failed enterprise AI initiative, and no amount of headcount fixes it, because headcount is exactly what it consumes.

A flywheel behaves differently. Each turn makes the next turn cheaper.

Ship narrow, with an explicit standard. One capability, one clear job, a modest set of real cases it must pass. Small enough that a domain expert can look at the whole test set in one sitting.

Let reality find the failures. It will find failures your test set never imagined, because real users ask questions in an order and a phrasing nobody designs for.

Turn every real failure into a permanent case. This is the hinge of the entire loop. A failure that gets patched and forgotten will come back. A failure that becomes a test case can never happen twice without someone being told.

Let the standard ratchet. The bar goes up as the system learns what it got wrong. Critically, it never quietly goes down. A change that would lower it gets blocked, not merged and apologized for later.

Reuse the machinery. By capability three, the test infrastructure exists, the review habit exists, the experts know what a good case looks like, and the definition of correct for shared concepts like "customer" or "late shipment" is already written down. The marginal cost of the next capability falls instead of staying flat.

That is the difference between an AI program that gets more expensive every quarter and one that gets cheaper. Not talent, and not tooling. The presence or absence of a loop that captures failures instead of discarding them.

Where the human actually belongs

"Human in the loop" has been diluted into meaning "a person checks the output." That version does not scale, and it is a tax that grows in direct proportion to how much you use the system. If every answer needs review, you have not automated anything. You have hired reviewers.

The useful version puts the human somewhere else. Not in the path of every output, but at the point where judgment is created rather than repeated.

Defining the standard. A domain expert is the only person who can say what a right answer looks like for their business. But that does not mean they should write test cases from a blank page for three weeks. It means their time should be spent on judgment, not transcription: show them candidate cases drawn from their real data and let them accept, edit, or reject. That is a task measured in minutes, and it is the highest value work a human does in the whole system.

Owning the vetoes. Some rules are not scores. Consider a system that answers ninety percent of pricing questions correctly and occasionally invents a contract clause that does not exist. It is tempting to call that ninety percent good. It is not. It is unsafe to ship, and any averaged score will hide exactly that, because a strong average is very good at burying a rare catastrophic failure. Someone has to name the failures that no aggregate is allowed to outvote, and that someone is a human who understands the consequences.

Approving what matters, and only what matters. Not every change needs sign off. A change to an internal research tool and a change to something that moves money do not deserve the same ceremony. Deciding which capabilities carry real risk is a human judgment, made once, that then determines how much friction the machinery applies for years.

Notice the pattern. In each case the human supplies judgment once and the system enforces it forever. That is the trade that makes AI worth building: not replacing expertise, but capturing it in a form that keeps working on a Tuesday in eighteen months when the expert is in a different job.

Build the gate before you build the builder

If there is one sequencing rule I would give any company starting this, it is this one.

It is tempting to first build the thing that lets people create AI capabilities quickly, and to add measurement later once there is something to measure. That order is backwards, and the failure it produces is worse than having no product at all. A fast way to ship capabilities without a way to measure them is a machine for producing unmeasured liabilities at speed.

The practical version, for a company with no AI infrastructure at all, is smaller than it sounds. Before you choose a model, before you choose a framework, before you write a line of orchestration: sit with the person who knows the work and write down thirty to fifty real questions, with the answers they consider correct, and the two or three things the system must never do. Run that set by hand against whatever you build. It is unglamorous and it takes a few days.

It is also the fastest way to find out whether you have a project. If nobody in the building can agree on what a right answer looks like, you do not have an AI problem yet. You have an unresolved business question, and no model is going to resolve it for you.

What I am building

This is the thesis behind Aulë, the system I am currently building. It starts from the position that evaluation is not a feature of an AI platform but the point of one, and that a capability should not be allowed to reach production until it has met a standard someone deliberately wrote down. It is very much in progress, and I will write more about it as it becomes something I can show rather than describe.

The broader argument stands on its own, though, and it does not require my system or anyone else's. The market has spent several years teaching companies that AI demos are easy and AI products are hard. The distance between those two things is not a model, a framework, or a budget line. It is a standard that a system can fail, a loop that turns every failure into a permanent lesson, and a human whose judgment is captured once instead of consumed forever.

Start with what correct means. Everything else is downstream of that.