There is an awkward moment in many technology programmes: the demonstration works, the room is pleased, and nobody wants to ask whether it belongs in the business. The model can produce an answer. That is a useful technical milestone. It is not yet a reason to fund an operating service.
When I consider sponsoring an AI pilot, I want the conditions for stopping it agreed before the first impressive demo. That may sound pessimistic. I see it as respect for the team's time and for the people who would have to live with the result.
Decide what a useful decision looks like
An AI pilot should begin with a person, a moment and a choice. Consider a support team that wants to prioritize cases likely to need specialist attention. The relevant question is not simply whether a model can classify past tickets. It is whether the right person receives a useful signal early enough to change the route of a case, with enough context to disagree when necessary.
Define the current practice as a baseline. Perhaps experienced agents already identify urgent cases accurately, but the queue gets delayed during peak periods. If the pilot cannot improve timeliness without creating more false alarms or extra handling work, it may not merit deployment. A strong benchmark includes operational measures as well as model scores.
I would ask four questions at the start: What decision changes? Who is accountable for it? What is the cost of a wrong recommendation? What would make the existing process a better choice? If a team cannot answer them, it is too early to evaluate a model, however interesting the data may be.
Give the pilot an exit door
A pilot can fail for several honest reasons. The records might not represent the cases that matter. Labels may capture past human decisions rather than the outcomes we hope to improve. The intervention might arrive after the useful moment has passed. A small accuracy gain could require more review work than the team can sustain.
These are findings, not embarrassments. I would set explicit review gates for data fitness, operational fit, safety and economics. At each gate, the options should be to continue, narrow the use case, redesign the process or stop. A “stop” decision should preserve what was learned: which fields were unreliable, which workflow assumptions proved wrong, and which simple rule performed surprisingly well.
Without an exit door, pilots become permanent experiments. Teams keep adjusting the model because money has already been spent, while the people who own the process wait for a service that never becomes dependable.
Let the people who handle exceptions judge it
It is easy to design a pilot around the average case. Operations feels the exceptions. A claims reviewer sees ambiguous evidence; a factory supervisor sees sensor readings during maintenance; a patient scheduling team sees a last-minute change that the historical data did not anticipate.
I would bring those people into evaluation before a rollout decision. Ask them to work through a small set of difficult, representative cases. Where would they trust the recommendation, seek more information or override it? What explanation would they need? What should happen when the model is uncertain or the system is unavailable?
Their answers may lead to a narrower, more useful product. A tool that routes obvious cases and escalates uncertain ones can be more valuable than a grand promise of full automation. Human review should have a real place in the workflow, with authority and time to act; it cannot be a disclaimer at the bottom of a policy.
What this means for leaders
I would fund AI pilots as learning investments with written decision gates, named process owners and a genuine alternative to deployment. Before approving scale, I would want evidence that the intervention improves the actual workflow, that exceptions can be handled, and that somebody owns performance after launch.
Stopping a pilot can be a successful leadership decision. It frees resources for a stronger opportunity and demonstrates that our standard for AI is usefulness in practice, not applause in a presentation.
Put this insight to work.
If you are weighing a similar decision, our team can help you clarify the problem, test your options and choose a practical next step.