RAG is not dead. Bad retrieval is.

Long context solves one problem: fitting more raw material into a single call. That's genuinely useful for tasks like reviewing one long contract, summarizing one long transcript, or holding a full codebase in view for a single refactor. It's a different problem from search: finding the right handful of facts out of a corpus that's too large, too permission-gated, or too fast-changing to fit in any context window, no matter how large.

A million-token context window doesn't make a 50-million-token knowledge base fit in one call. It doesn't add per-user row-level security to a shared document store. It doesn't reduce the marginal cost of every single query re-processing the entire corpus. And it doesn't give you a citation trail pointing at the exact paragraph an answer came from, which matters enormously the moment a customer or auditor asks "where did that number come from."

• Cost. Sending your entire knowledge base as context on every query means paying for those tokens on every query. Retrieval sends only the relevant slice, which at real query volume is the difference between a manageable bill and a runaway one.
• Freshness. A document updated five minutes ago should be reflected in the next query. Re-indexing a retrieval store for a changed document is fast and cheap. Re-processing it into a giant static context blob for every future call is neither.
• Permissions. Different users see different subsets of your data. Retrieval lets you filter at query time by the requesting user's access level. A single shared context blob has no such boundary unless you rebuild it per user, which erases the cost advantage of long context entirely.
• Auditability. Regulated industries and any serious enterprise buyer will ask for a citation: which document, which section, produced this answer. Retrieval gives you that pointer for free. A model reasoning over an undifferentiated context blob gives you a paraphrase with no traceable source.

When a client tells us "we tried RAG and it didn't work," the pipeline almost always has the same handful of gaps. In order of impact:

1. Structure-aware chunking. Replace fixed 512-token windows with chunks that respect document structure: headings, sections, table boundaries. A chunk that splits a table in half or cuts a definition off mid-sentence will never retrieve well, no matter how good the embedding model is.
2. Hybrid search, not pure vector similarity. Combine keyword/BM25 search with vector search and merge the results. Vector similarity alone misses exact matches on product codes, error codes, and proper nouns, which is exactly the kind of query real users ask most often.
3. A reranking pass. Retrieve a wider candidate set (30 to 50 chunks) with a cheap first-pass search, then rerank with a cross-encoder or a small model call to pick the true top five. This single step recovers more accuracy than almost any embedding model swap.
4. Metadata filtering before similarity search. Filter by document type, date, department, or permission level first, then run similarity search inside that filtered set. This fixes both accuracy (less noise to search) and security (no cross-tenant leakage) in one change.
5. Query rewriting for the actual question. Real user questions are often underspecified or conversational. Rewrite the query into a clean search query before hitting the retrieval layer, using the conversation history for context.

The popular claim this year is that with million-token and beyond context windows becoming standard, retrieval is a transitional technology that serious teams are migrating away from, and that "just put it all in context" is the simpler, more reliable architecture going forward.

It's wrong for the same reason "just add more RAM" was never a full substitute for database indexing. It conflates "can technically fit" with "is the right architecture." Even in a world of infinite context windows, you still don't want to pay to reprocess your entire knowledge base on every query, you still need per-user access control, and you still need a citation trail. Those requirements are architectural, not a limitation of window size, and they don't disappear as the window grows. What actually changes with longer context is the retrieval unit: instead of returning tiny 300-token chunks, you can now return full documents or full sections to the retrieved set, which reduces some chunking pain. That's a real improvement inside a RAG pipeline. It's not an argument for deleting the pipeline.

We still see teams get this backwards in the field: reaching for a bigger context window to paper over a retrieval pipeline they never fixed, and getting slower, more expensive, less auditable answers as a result. Long context is a tool inside the pipeline, not a replacement for it.

Before touching the generation prompt, isolate retrieval and measure it on its own:

1. Build 30 to 50 real questions with a known correct source chunk for each, pulled from actual usage or support tickets, not invented examples.
2. Run retrieval alone against your current pipeline and record whether the correct chunk appears in the top 5 results.
3. Calculate recall@5. Below roughly 80%, the retrieval layer is the bottleneck and the prompt is not worth touching yet.
4. Change one variable at a time (chunking, hybrid search, reranking) and re-run the same eval set to isolate what actually moved the number.
5. Once recall@5 clears 90%, move to evaluating the generated answer quality on top of retrieval, using the same question set.

This is the same discipline we use in the eval work behind our agent eval suite playbook, applied specifically to the retrieval layer rather than the full agent loop.

Retrieval isn't the answer to every problem either. If your corpus is a single document (one contract, one long transcript) that fits comfortably in context, doesn't change per query, and doesn't need per-user filtering, just put it in context and skip the pipeline entirely. Building a retrieval system for a 40-page PDF is unnecessary complexity that adds latency and cost for no benefit.

The decision rule is simple: if the corpus is large, shared across users with different permissions, changes frequently, or needs a citation trail, build retrieval. If it's a single, small, static document read by one user in one session, don't. Most enterprise knowledge bases, support systems, and internal wikis land firmly in the first category, which is why retrieval keeps earning its place in production stacks despite the yearly obituary.

Retrieval is one layer in a larger context strategy that also includes tool results, conversation history, and system instructions, all competing for the same token budget. Our context engineering field notes cover how we allocate that budget across sources. And if retrieval is feeding a customer-facing agent, the accuracy of what comes back directly determines whether that agent is production-ready in the first place; see our framework in what production readiness actually means.

If you've been told RAG is dead and you're considering ripping out a working pipeline in favor of a bigger context window, get a second read on the actual numbers first. That's a fast, focused review, exactly the kind we run inside the $1,000 AI Audit.