Table of Contents

Should you summarize before extracting with an LLM? Often no — and if you already built your pipeline that way, this is probably why it keeps losing the one detail you needed. When an LLM analyzes an AI summary instead of the raw text, whatever the summary dropped is gone for good, no matter how good the extractor is. The details that vanish first are negations and reversals — the “we decided not to” lines that turn out to be decisive later.

If you build an LLM pipeline as raw text → summary → extract-from-summary, the weak signals that look trivial today but turn out to be decisive later are silently destroyed at the summary step. Negations and reversals—statements about what was decided not to do—are the first to go. So when your goal is decision tracing or signal detection, you must either force the summary step to preserve that class of information, or analyze the source text directly. This post maps that decision boundary with a real case.

What is the summarize-then-extract pitfall?

The summarize-then-extract pitfall is the information-loss structure that appears when you feed an AI summary—rather than the source text—into a downstream extraction step: whatever the summary discarded during compression can never be recovered later, no matter how good the extractor is.

The root cause is a mismatch in objective functions. A summarizer is trained and tuned to compress the gist of a document. But what you often need to extract is not the gist—it is a single line that looks unimportant now and becomes decisive later. Chain two steps whose objectives diverge, and the first step throws away exactly what the second step needs.

A real case: a one-line reversal buried in a thread

Consider an email thread about a sample shipment. Production schedule, inventory status, owner changes, status updates—a dozen messages of a long, dominant narrative. Somewhere in the middle, one person wrote a single line:

“We’ve decided not to ship this particular batch this time.”

The AI summary dropped that line. It preserved only the dominant narrative: “sample production and shipment proceeding on schedule.” Months later, when someone asked “why did that batch never ship?”, a system that retained only the summary had no answer. The trace of the decision had already been destroyed at the storage step.

The lesson is blunt: the cleaner your summary gets, the more reliably the one line you were looking for disappears.

Why summarizers drop exactly that line

Two mechanisms compound here.

1. Low salience. Against a “proceeding on schedule” narrative repeated a dozen times, “not shipping” is a sliver of text. Summarizers infer importance from frequency and repetition, so a short line that appears once reads as noise.
2. Negation and reversal bias. Summarizers are good at keeping “what we did” (affirmative actions) and quick to discard “what we decided not to do” (negations, exceptions, reversals). Yet in decision tracing, those negations and reversals carry the highest information. What deviated from the plan matters more than what went according to it.

Put simply:

The objective function of summarization (compress the gist) ≠ the objective function of decision tracing (preserve exceptions and reversals).

That mismatch is the essence of the pitfall.

When to analyze raw text vs. a summary

This is not an argument against summarization. Summaries earn their place: they ease token limits, improve retrieval (embedding) efficiency, amortize cost and latency, collapse per-recipient duplicates into one narrative, and—under security policy—may be the only derived form you are allowed to put in the cloud. The deciding question is simple: is the answer in the gist, or in the exception?

Goal	Right input	Why
Topic classification, overall trends	Summary	The gist is the answer; loss is harmless
Decision tracing, reversals & exceptions	Raw text	The answer lives in what the summary discards
Weak leading-signal detection (demand, inventory, capex)	Raw text	Signals hide in low-frequency, fine-grained phrasing
Quick skim, UI preview	Summary	For human reading; accuracy demands are low

The rule is short: if the answer is in the main storyline, analyze the summary; if the answer is in an exception, reversal, or weak signal, analyze the source text directly.

The fix: two-track storage and decision-aware summarization

In production, two measures used together contain this pitfall.

First, two-track storage. Keep the summary as the “skim” layer and the raw text as the “trace” layer—retain both. If the source stays in a controlled private store (for example, an on-prem database), then even a decision the summary dropped can be recovered when a tracing query descends to the source. The key is to never treat the summary as the single source of truth.

Second, decision-aware summarization. Pin the preservation rules into the summarization prompt as a contract:

Preserve the following verbatim from the source, even when brief:
- Decisions / confirmations (e.g., "decided to do X")
- Reversals / cancellations (e.g., "decided NOT to do X", "prior decision withdrawn")
- State changes (e.g., schedule, owner, or scope changes)
- Negations / exceptions (e.g., "except for Y")
Never omit these items, however small their share of the overall narrative.

By correcting the objective function explicitly, you force the summarizer to hold on to the low-salience signals it would otherwise discard by default.

Frequently asked questions

Should I summarize text before extracting with an LLM, or extract from the raw text? It depends on where the answer lives. If you need the gist — topic, overall trend, general classification — summarizing first is fine and even helpful. If you need an exception, a reversal, or a weak signal, extract from the raw text, because the summary likely dropped exactly that. The short rule: gist in the main storyline → summary is fine; answer in an exception → go to the source.

Why does my LLM summary miss important details? Because a summarizer optimizes for the gist, and it infers importance from frequency. A detail that appears once against a dozen repeated messages reads as noise and gets dropped. Negations and reversals (“decided not to…”) are dropped fastest of all, even though they often carry the most information for decision tracing.

Does extracting from a summary lose information compared to the source text? Yes. Anything the summary discarded during compression is unrecoverable downstream — the extractor can only work with what the summary kept. If the class of information you need is prone to being compressed away (exceptions, negations, fine-grained signals), extracting from a summary structurally loses it.

How do I keep the summary from dropping negations and reversals? Two measures together. Keep both the summary and the raw text (two-track storage) so a tracing query can descend to the source. And write a decision-aware summarization prompt that explicitly contracts the summarizer to preserve decisions, reversals, state changes, and exceptions verbatim, however brief. This corrects the summarizer’s default objective, though it reduces loss rather than eliminating it.

The one-line takeaway

A summary keeps “what was done” and drops “what was decided not to do.” If tracing is the goal, never discard the source text.

A reality note: the measures above reduce loss; they do not eliminate it. A decision-aware prompt still misses things depending on the model and the document, and two-track storage takes on the cost and latency of source lookups. But the direction holds—in any system that deals in weak signals, a summary is an optimization, not a source of truth, and the source of truth must remain the raw text all the way down.