Table of Contents

Your RAG can’t find a document you know exists? If your pipeline searches AI-generated summaries instead of the original text, this is often why — and it’s not a retriever problem. When you do RAG over summaries, any keyword the summary dropped is simply not in the index, so no query can match it.

If your architecture keeps the original document in an external system and stores only an AI-generated summary, then your search runs against the summary — not the original. That single fact has a consequence most teams discover too late: the summary is your search index, and any identifier the summary drops becomes permanently unsearchable. The bottleneck in this kind of system is almost never the retriever. It is upstream, at the moment the summary was written.

The setup: searching a derivative, not the source

There are good reasons to store only a summary. The original may live in a system of record you don’t own — an email server, a document store, a compliance archive. Retaining full copies may be expensive, or restricted. So the pipeline extracts a compact summary, writes that into your own store, and the original stays where it was.

This works well until someone wants to search. Because now the search — whether keyword, filter, or vector — runs against whatever text you actually stored. And what you stored is not the document. It is a lossy compression of it.

The core claim

The moment you store a summary instead of the original, the summary becomes your search index. A word the summary omitted is a word no query can ever match, because the underlying text simply does not contain it. No retriever, however good, can find a document by a term that isn’t in the index.

This is easy to miss because summaries are usually judged as reading material — is it accurate, is it concise, does it capture the gist? Those are the wrong criteria if the summary is also going to be searched. A summary can be an excellent read and a terrible index at the same time. It captures the meaning a human wants and drops the exact part number, the vendor name, the dollar figure, the attachment filename — precisely the tokens someone will later search for.

Why this outranks retriever tuning

Most search-quality discussion is about how you query the index: chunking strategy, embedding model, reranking, hybrid fusion. All of that assumes the index contains the thing you’re looking for. If the identifier was dropped at summarization time, every one of those techniques is operating on text that never had the answer in it.

So the causal order is: index quality first, retrieval technique second. A team that spends weeks tuning retrieval while the summarization prompt is quietly discarding identifiers is optimizing the wrong layer. The failure looks like “search can’t find this document,” and the instinct is to blame the search. The actual cause is that the document, as stored, doesn’t mention the thing being searched for.

Design implications

Treating the summary as an index rather than only as reading material changes how you build the pipeline.

Make searchability an explicit requirement of the summary. The summarization prompt usually optimizes for a readable gist. If the summary is also the index, the prompt has a second, competing job: preserve what people will search by. State it directly — the summary serves a dual purpose, reading and retrieval — so identifiers aren’t sacrificed for brevity.

Extract key entities into their own structured fields, not prose. Names, organizations, product or part numbers, dates, amounts — these are the search axes. Burying them inside summary prose makes them fragile (a reworded summary can drop them) and hard to filter on. Pull each identifier type into its own field. One search axis, one field. The prose summary carries meaning; the structured fields carry the handles people grab by.

Pin down what “sensitive” means, or the model will over-redact. A vague instruction like “exclude sensitive information” is harmless when the summary is just for reading. It becomes dangerous the moment the summary is an index. Faced with an undefined rule, a cautious model strips out organization names, figures, and identifiers — and those were your search axes. If redaction is required, define exactly what counts as sensitive (say, personal contact details) and what must be preserved (say, organization and product names). Leaving it to model discretion means silently losing the ability to search.

The general principle

This generalizes beyond summaries. Whenever an AI pipeline stores a derivative of the source and then operates on that derivative — a summary, an embedding, an extracted record — the moment of derivation silently sets the ceiling on everything downstream. What the derivation drops, the downstream can never recover. The design leverage is therefore upstream: get the derivation right, because no amount of downstream cleverness can restore information that was thrown away before it was stored.

Frequently asked questions

Why can’t my RAG find a document I know is there? The most common structural reason: your RAG is searching a summary, not the original document. If the summarization step dropped the exact term you’re searching for — a part number, a name, a phrase — that term isn’t in the index at all. No retriever can match a word that isn’t in the text it searches. Before tuning the retriever, check whether the searchable text still contains what you’re looking for.

Does RAG over summaries lose information? Yes, by definition. A summary is a lossy compression of the original. It preserves meaning and drops specifics. That’s fine when the summary is only read by a human, but when the summary is also your search index, every dropped specific becomes an unsearchable document. Searching summaries trades completeness for compactness.

Should I search the summary or the full text? If you can afford to index the full original, that gives you complete keyword coverage. If you must store only a summary (for cost, privacy, or because the original lives elsewhere), then treat the summary as an index, not just as reading material: make searchability an explicit requirement of the summarization prompt, and extract key identifiers into their own structured fields so they survive.

How do I stop summarization from dropping searchable keywords? Three things. State in the summarization prompt that the summary serves retrieval, not just reading. Pull names, IDs, numbers, and dates into dedicated fields instead of leaving them in prose. And define exactly what “sensitive” means if you redact — a vague redaction rule makes a cautious model strip the very identifiers people search by.

Takeaway

If you store summaries and search over them, you are not searching your documents — you are searching your summaries. Judge those summaries as indexes, not just as reading: make searchability an explicit requirement, lift identifiers into their own fields, and never leave redaction to undefined discretion. The bottleneck was never the retriever. It was the summary, written before anyone thought about search.

Related: for why you summarize enterprise email before feeding it to an LLM in the first place, see the companion piece on normalizing email before the LLM. This piece is its flip side — what that summarization costs you.