Why most AI projects stall before launch
The failure is rarely the model. It's scope that never converges, evaluation that was never defined, and a demo mistaken for a product.
There is a particular shape to an AI project that dies. It starts with a strong demo, moves into six months of "almost ready", and ends quietly when the budget review comes round. The model was never the problem. The problem was that nobody agreed what finished looked like.
A demo is not a product
Demos are optimised for the happy path. Somebody types the question the system handles well, the output arrives, the room nods. Production is the opposite: the inputs are messy, the users are impatient, and the interesting cases are the ones nobody thought to try.
The distance between those two states is where most of the engineering actually lives — error handling, latency, cost per request, and the fallback behaviour when the model returns something unusable. Teams that budget for the demo and not the gap end up rebuilding.
Define the evaluation before the build
If you cannot say how you will know the system is good enough, you will never be able to say it is finished. This sounds obvious and is very widely skipped.
A workable evaluation needs three things:
- A fixed set of real inputs, drawn from the actual domain rather than invented
- A definition of a correct response that two people would agree on
- A threshold you are willing to ship at, agreed in advance
Write these down before the first prompt. They will change — but changing a written target is a decision, whereas changing an unwritten one is drift.
Constrain the surface area
The most common scoping error in AI products is building an assistant that can do anything. Open-ended surfaces are enormously harder to evaluate, harder to make reliable, and harder for users to understand.
A narrow tool that does one job dependably will beat a general assistant that does ten jobs unpredictably. It will also ship, which the general assistant usually does not.
Put a human where the cost of being wrong is high
Not every step needs automating to the same confidence. Drafting a reply is cheap to get wrong; sending it is not. Extracting a figure is cheap to get wrong; posting it to a ledger is not.
Design the checkpoint at the point of consequence, not uniformly across the whole flow. This buys you the ability to ship at a lower accuracy threshold without shipping risk — and it gives you a stream of labelled corrections that makes the next version better.
None of this is about model selection. Teams spend a lot of time on that question and comparatively little on the four above, which is roughly the inverse of how much each one determines whether the thing launches.