RAG systems that survive production: why demos work and pipelines break

The demo took two weeks. Someone indexed 40 product docs, put a retrieval step in front of an LLM, and it answered ten questions in a row without missing. Budget approved. Ninety days later, the same system told a prospect that the enterprise plan includes a feature deprecated in March, and nobody could explain why.

We've rescued enough of these systems that the pattern stopped being interesting. The demo wasn't a lie. It was a different system than the one production needs. Retrieval-augmented generation is the right architecture for most B2B knowledge problems: support answers, sales enablement, policy lookup, contract intelligence. But the distance between "answered questions in a meeting" and "answers 40,000 questions a month without inventing pricing" is where the real engineering lives. Almost none of it involves the model.

Why the demo works

A demo runs under conditions production never sees again.

The corpus is small and hand-picked, usually 20 to 100 documents someone curated the night before. The questions come from the people who wrote those documents, so they phrase queries the way the docs phrase answers. The index is a day old, so nothing is stale. One user, zero concurrency, and nobody watching the meter.

Production inverts all of it. The corpus is thousands of documents nobody curated. Users phrase questions in ways the docs never anticipated, in two languages, with typos. The index starts aging the moment you build it. And finance asks what each query costs by month two.

None of those failures are visible in a demo, which is exactly why the demo convinces people.

The six places production breaks

Data freshness. Your docs change weekly. Your index doesn't know that. Demo pipelines index once and never again; production needs a sync strategy with a freshness SLA. For most B2B corpora that means event-driven updates when a source document changes, or at minimum a nightly rebuild with change detection so you're not recomputing embeddings for 10,000 unchanged pages. The failure above, recommending a deprecated feature, wasn't a hallucination. Retrieval worked perfectly against an index that was four months old. And if your source data is already scattered across six tools with no pipeline, fix that first. We wrote about that sequencing in the data infrastructure guide.

Chunking. Splitting documents every 500 tokens is the default in every tutorial, and it's wrong for most business content. Naive splits separate a pricing table from the paragraph that says which customers it applies to, or a policy from its exceptions. Production systems chunk by document structure: headings, sections, tables kept whole, 10 to 20 percent overlap, and metadata (source, section, date, product) attached to every chunk. Sizes between 300 and 800 tokens cover most content, but the right answer varies by document type. That's why chunking is a design decision, not a config value.

Retrieval quality. Vector similarity isn't relevance. Pure vector search misses exact identifiers, product codes, and rare terms. Pure keyword search misses paraphrase. Production pipelines run hybrid retrieval, dense vectors plus BM25, with a reranker (a cross-encoder like Cohere Rerank) over the top 50 candidates. In our engagements, that one change moves retrieval hit rate more than any model upgrade. When an answer comes out wrong, teams blame the generator. About 70 percent of the time, the retriever handed it the wrong context.

Evals. The demo's eval was "the room nodded." Production needs a golden set: 150 to 300 real questions with reference answers and the documents that should be retrieved, scored on retrieval hit rate, faithfulness, and answer relevance. Ragas, LangSmith, and Braintrust all turn this into a CI step, so every prompt tweak, chunking change, or model swap runs against the same set before it ships. Stated plainly: if you can't name your retrieval hit rate as a number, you don't have a RAG system. You have a demo with users.

Hallucination guardrails. Grounded generation needs enforcement, not intention. That means citations required in the output and validated against retrieved chunks. A faithfulness check before high-stakes answers go out. Confidence thresholds below which the system says "I don't know" and routes to a human. Logging every answer with its retrieved context so failures can be audited. A system that honestly declines 5 percent of queries beats one that answers 100 percent and is confidently wrong on 4 percent. If you sell into regulated industries, this section is the difference between a tool and a liability; the policy side is covered in our AI governance post.

Cost per query. A query costs embeddings plus retrieval plus reranking plus generation. With frontier models and long contexts, a careless pipeline lands between $0.05 and $0.15 per query, which at 100,000 monthly queries is real money. Production systems route: a cache for repeated questions (support traffic repeats heavily, 30 to 40 percent cache hit rates are normal), a small model for easy queries, the frontier model only when the router decides it's needed. Cost per query belongs on the dashboard next to hit rate, not as a surprise on the invoice.

What a production-grade pipeline looks like

Seven stages, each with an owner and a metric:

1. Ingestion. Connectors to the real sources (wiki, CRM, ticketing, contracts) with change detection. Manual exports are how indexes go stale.
2. Processing. Structure-aware chunking with metadata on every chunk.
3. Indexing. A vector store that matches your scale. pgvector handles millions of chunks inside the Postgres you already run; Qdrant or Pinecone earn their place at heavier filtering and scale.
4. Retrieval. Hybrid search plus reranking, measured as recall against the golden set.
5. Generation. Prompts that require grounding and verifiable citations.
6. Evaluation. The golden set running in CI on every change.
7. Observability. Per-query traces showing what was retrieved, what was generated, cost, latency, and user feedback, on a dashboard someone actually reads.

For a B2B company with a real corpus, that's an 8 to 12 week build for a first production system. The demo is week one. Everything after week one is what makes the system survive contact with users.

When RAG is the wrong tool

About half the RAG projects we're asked to scope shouldn't be RAG.

If the answer lives in a database, you want tool calling and SQL, not semantic search over exports. "What was our Q2 churn" is a query, not a retrieval problem.

If the corpus is under 50 stable documents, put it in the context window and move on. Long-context models made mini-RAG obsolete, and an index is maintenance you own forever.

If the job is taking action instead of answering questions, you need an agent with tools, where retrieval might be one tool among several. That's a different architecture, and we covered it in AI agents for B2B companies.

If the source content is wrong or contradicts itself, RAG will retrieve the contradiction with perfect fidelity. No pipeline fixes a corpus that disagrees with itself.

RAG earns its complexity when the corpus is large, changes often, and every answer has to trace back to a source. That covers a lot of B2B knowledge work. It doesn't cover all of it, and teams that skip this question ship the wrong system with great confidence.

Where this lands

The gap between demo and production isn't a model problem. It's an engineering problem with six known failure points and a known fix for each one. That's the work our AI practice exists for: production-grade AI systems that ship with evals, guardrails, and a cost model attached, because measurable ROI requires the measuring part. And if you're still deciding whether your data layer can feed a RAG system at all, that's exactly the kind of question the DGTL Readiness Index was built to surface.

Related: AI Agents for B2B Companies → · The Data Infrastructure Guide → · AI Governance →