Your AI pilot has been a pilot for 18 months. Here's the way out.

There's a Slack channel at your company called something like #ai-pilot, and it's been quiet for a while. The prototype impressed everyone in Q1 of last year. It summarized support tickets, or drafted proposals, or answered policy questions from your internal docs, and the room went quiet in the good way. Leadership blessed a pilot. Eighteen months later, the pilot is still a pilot: a notebook, a monthly status update, a demo environment three people can access, and zero production traffic.

Nobody kills it, because the demo still works. Nobody ships it, because shipping means answering questions the pilot was never built to answer. We call this pilot purgatory, and after scoring the AI dimension on enough Readiness Index assessments, we can tell you it has a consistent anatomy and a concrete exit.

Why pilots stall (it's almost never the model)

Four failure modes, in the order we usually find them.

Nobody owns the path to production. The pilot was born in an innovation team, a growth team, or one enthusiastic engineer's spare Friday. Production lives somewhere else. Engineering owns deploys, uptime, and on-call. Data owns the pipelines and the warehouse. Security owns the review that decides whether customer data can touch a model API at all. The pilot sits outside all three, which means each function can veto it and none of them can ship it. A project with three part-time stakeholders and zero full-time owners doesn't move. It orbits.

The pilot runs on demo data. Fifty hand-picked documents. A clean CSV someone exported in March. The real corpus is different: it lives in six systems, half of it carries customer PII, a third is in formats nobody documented, and access is permissioned in ways the pilot cheerfully ignored. Closing the gap between the demo corpus and the production corpus is weeks of unglamorous pipeline and permissions work, and nobody scoped it because the demo already "worked."

There's no eval harness. "It looks good" is not a metric. Without a labeled test set and automated scoring, every prompt change is a matter of opinion, every model upgrade is a leap of faith, and every regression gets discovered by a user. Teams respond rationally: they stop changing things. The system freezes at demo quality forever. We'll defend this on any call: if you can't run an eval suite against a change, you don't have an AI product. You have a demo with a budget line.

Nobody built a cost model. The pilot costs $40 a month in tokens, so nobody asked. Then someone in finance asks what it costs at 50,000 documents a month, with retries, long-context calls, and a human reviewing 20% of outputs, and the room goes quiet in the bad way. Unit math that nobody calculated gets parked, and parked means purgatory.

The 90-day exit

This is the roadmap we run. It's deliberately boring, which is why it works.

Days 1 to 15: shrink the scope, name the owner. One workflow, not five. Pick the one with a measurable outcome and a tolerant failure mode: first-draft responses for support, contract clause extraction, meeting notes into the CRM. Write down the baseline number you're trying to move: average handle time is 11 minutes, reviewing a vendor contract takes 4 hours, whatever it is. Then name a single owner with budget and the authority to pull engineering, data, and security into the same weekly meeting. A committee is not an owner. A center of excellence is definitely not an owner.

Days 16 to 30: build the eval harness before touching the model again. Collect 200 to 300 real examples from the actual workflow, label them with what "good" looks like, and wire automated evals into CI with Braintrust, Langfuse, promptfoo, or plain pytest if that's your culture. Every prompt change, retrieval tweak, and model swap now runs against the same set before it merges. This is the most valuable artifact the whole program will produce, and it outlives every model you'll ever use.

Days 31 to 60: rebuild on production data. Real pipelines from the real systems, with permissions carried through to retrieval so the system can't answer questions using documents the asker isn't allowed to open. If retrieval sits at the core of your use case, this phase is where most of the effort goes; we wrote up the details in our RAG systems production guide. Put a gateway like LiteLLM in front of every model call for routing, logging, and spend caps. And run the security and legal review in parallel, not as a gate at the end; our post on AI governance covers what that review needs to include, and none of it requires a committee.

Days 61 to 90: limited production with kill criteria. Route 5 to 10% of real traffic, or one full team, through the system. Keep a human in the loop wherever an error is expensive. Stand up one dashboard with three numbers: quality (eval pass rate on sampled production traffic), cost per unit of work, and p95 latency. Review it weekly. And write the kill criteria before launch day: "if it isn't resolving 15% of tickets on its own by day 90, we shut it down." At day 90 you make one of three calls: scale it, fix the one thing blocking scale, or kill it. All three beat month 19 of a pilot.

The decisions that matter more than the model

The model is the most replaceable component in the entire system. If you have the eval harness and the gateway, moving from one frontier model to another is a config change plus an afternoon of eval runs. The durable assets are the eval set, the data pipelines, and the owner. Companies get this backwards constantly: they'll spend six weeks debating model choice and zero days building the harness that would make the debate empirical.

The same logic applies to agents. If your roadmap says "agents" anywhere, the ownership and eval questions get harder, not easier; we covered why in our post on AI agents for B2B companies.

Budget the run, not just the build. The number that kills AI projects isn't the development sprint. It's the monthly token invoice plus the human review hours nobody counted. Model both at target volume before day 60 and the finance review becomes a formality instead of the meeting where the project dies.

And put a kill date on everything. An AI pilot without a kill date isn't caution. It's a subscription to feeling modern, renewed monthly, paid in the credibility of the next AI project you propose.

This is exactly what the AI dimension of the DGTL Readiness Index scores: ownership, data readiness, evaluation discipline, and cost visibility, not model choice. If your pilot has celebrated a birthday, the fastest way out is usually a small senior team that has shipped this loop before. That's the work our AI practice does, alongside the engineering and security practices it ships with.

Related: RAG systems in production → · AI governance → · AI agents for B2B →