Retrieval before generation: why RAG is an architecture decision, not a feature

Retrieval-augmented generation has a reputation problem. Half the industry treats it as a plug-and-play feature: embed your documents, call the API, ship. The other half dismisses it because their first attempt returned confident nonsense. Both experiences come from the same mistake - treating RAG as something you add rather than something you architect.

The corpus is the product

A retrieval system can only be as good as the collection it searches. Before any embedding model is chosen, three questions about the corpus need answers.

First, what counts as a document? A forty-page policy manual and a two-hundred-word FAQ entry carry information at completely different grain. Chunking the manual into fixed five-hundred-token slices guarantees some chunks will split an argument in half. Structure-aware chunking - splitting on headings, keeping sections intact when they are short enough - consistently outperforms naive slicing in practice.

Second, what is stale? Content platforms accumulate material like sediment. A retrieval index that silently ranks a 2019 pricing page alongside this quarter's will produce confident, outdated answers. Freshness belongs in the index metadata, and decay or filtering belongs in the ranking step.

Third, what should never be retrieved? Drafts, internal notes, legal holds and personal data need exclusion rules enforced at ingestion, not filtered out after a leak.

The contract between retriever and generator

Once retrieval returns candidates, the generation step receives a constructed context. Treat that construction as a formal contract:

• Context is untrusted input. Retrieved text is data, never instructions. A document that says "ignore previous directions" is attempting manipulation, and the system prompt plus output validation should make such attempts inert.
• Sources are first-class output. The response schema includes the references actually supplied, so the interface can show exactly what the answer drew from.
• Insufficiency is a valid outcome. When no candidate clears a relevance threshold, the correct behaviour is a refusal: "I could not find enough in the available content to answer that confidently." Users forgive missing answers far faster than fabricated ones.

Evaluation closes the loop

A RAG system without evaluation is a demo forever. The minimal viable harness scores three things: does the answer follow only from the retrieved context (groundedness); does it address the question (relevance); and did retrieval surface the right material at all (source coverage). Run these on a fixed question bank after every corpus change, prompt change and model change.

Teams that wire this into CI catch regressions the day they happen. Teams that evaluate by vibes discover them from customers.

Start boring

The production-grade path is unglamorous: keyword search over clean, structured, fresh content; then hybrid ranking; then embeddings where they measurably help; then reranking if the logs demand it. Organisations that skip straight to vectors on top of messy content buy themselves an expensive mystery. Organisations that build the boring foundation first find that the fancy layers have less to fix.