Table of Contents

You ship AI search. It’s good. It answers questions the old keyword box never could. And then, almost on cue, someone asks the obvious question: so can we turn off the old search now?

It feels like a fair question. Why keep the clunky keyword box when the smart one does more? But the answer is no — and not because anyone’s attached to the old thing. It’s because the two tools aren’t competing for the same job in the first place. One is not a better version of the other. They’re looking at different things.

The instinct: newer is a superset of older

The assumption hiding inside “can we retire it” is that AI search is a superset — everything the old search did, plus more. Retire the subset, keep the superset. Clean.

Except it isn’t a superset. And to see why, you have to look at what each tool is actually searching. That’s the whole game, and it’s easy to miss because both of them present as “a search box.”

What each tool is actually looking at

Here’s the thing most people skip past. In a setup where you store an AI-generated summary and search over that, your AI search is reading the summary — the derivative. Your old keyword search is reading the original.

That’s not a small distinction. It’s the whole distinction.

The summary is a lossy compression of the document. It keeps the meaning and quietly drops specifics — the full part number, the attachment filename, that one phrase someone typed once in the third paragraph. Those live in the original. So the old keyword search can find them, and the AI search structurally cannot. Not “finds them less well.” Cannot. The words aren’t in the text it’s searching.

So when someone says “the new one does more,” that’s true for a whole class of questions — and flatly false for another class. The AI search wins at “find me the emails where a customer sounded unhappy about pricing.” The old search wins at “find me the email with this exact part number in it.” Neither can do the other’s job. That’s not a maturity gap you’ll close next quarter. It’s a difference in what they’re pointed at.

This is a difference in domain, not in quality

Worth saying plainly, because it’s the load-bearing idea: this is not a performance difference. It’s a domain difference — as in, the set of things each tool can possibly reach.

A performance gap closes over time. A better model, a bigger index, a smarter reranker — you tune it and the weaker tool catches up. A domain gap doesn’t close, because it’s not about how well the tool works. It’s about what’s in front of it. No amount of making the AI search smarter puts the dropped part number back into the summary. The information isn’t there to be found.

Once you frame it as domain rather than quality, “retire the old one” stops even being a coherent proposal. You’d be deleting reach, not deleting redundancy.

Two more reasons the old box earns its keep

The domain argument is the big one, and it stands on its own. But there are two smaller, practical reasons that pull the same direction.

Sometimes the user already knows the exact string. They don’t want interpretation. They have a part number, or an error code, or a filename, and they want the document that contains it — now. Routing that through an LLM adds a beat of latency and, worse, a chance the model over-thinks a query that needed no thinking. You typed something precise and it decided to be helpful about it. For those users, the plain keyword box isn’t the fallback — it’s the better tool.

(One honest caveat: how much of the original the keyword search actually covers depends on your setup — full text or just some fields. Worth confirming for your own system before you lean on this. The domain argument above doesn’t depend on it; this one does.)

And the old box is your safety net. Your AI search is only as complete as your indexing pipeline. Anything that pipeline missed — a document that never got summarized, an extraction that silently failed — vanishes from AI search entirely. The keyword search still sees it, because it reads the original. Which means the old box quietly doubles as the thing that catches your AI search’s blind spots. Someone finds a document through keyword search that AI search swore didn’t exist? Congratulations, you just found a gap in your index. That’s an operational feedback loop you don’t want to throw away.

The bigger pattern: ask about domain, not IQ

Step back from search for a second, because this shows up all over the place.

AI features are almost always built on a derivative. Not the raw thing — a summary of it, an embedding of it, a set of fields extracted from it. That’s usually a fine, even necessary, design choice. But it means the AI feature inherits a smaller domain than the source it was built from. It can only reach what made it into the derivative.

So the next time “let’s just replace the old thing with AI” comes up — and it will — the useful question isn’t which one is smarter. It’s do these two things even cover the same ground? If the AI feature stands on a derivative and the old thing stands on the source, the honest answer is: they don’t. Keep both, and stop framing it as replacement. The framing that actually holds up is division of labor — the AI feature makes newly-answerable questions answerable, without pretending it swallowed the old ones.

Frequently asked questions

Does AI search replace keyword search? No — not for a whole class of queries, and not because of quality. If your AI search reads summaries (a derivative) while keyword search reads the original, anything the summary dropped is outside AI search’s reach permanently. That’s a domain gap, not a performance gap, so a better model never closes it. Keep both and split the work.

Should we keep traditional keyword search after adding AI search? Yes. Keyword search covers exact-string queries (part numbers, error codes, filenames) where an LLM only adds latency and a chance of over-interpreting a precise query. It also catches documents your indexing pipeline missed — it reads the original, so it doubles as a safety net that surfaces gaps in your AI index. Retiring it deletes reach, not redundancy.

Is AI search better than keyword search? Neither is “better” — they cover different ground. AI search wins at meaning-based, fuzzy questions (“emails where a customer sounded unhappy”). Keyword search wins at exact-string retrieval (“the email with this part number”). Framing it as which-is-smarter is the wrong question; the right one is whether the two tools even cover the same set of documents.

When should I use AI search vs keyword search? Use AI search when the query is about meaning or situation and some approximation is acceptable. Use keyword search when you already know the exact string and need every document that contains it, with no interpretation. The deciding question isn’t IQ, it’s domain: is the answer something the derivative (summary) still contains, or only the original?

Takeaway

AI search isn’t a better keyword search. It’s a different search, pointed at a different thing — a summary instead of the source. Anything the summary dropped is outside its reach forever, no matter how good the model gets, because that’s a domain limit, not a quality one. So keep the old box. It answers the exact-string questions, it catches your indexing gaps, and it covers ground the AI search was never standing on. “Replace it” was the wrong verb. “Divide the work” is the right one.

Related: this is the direct sequel to why the summary is your search index — that piece explains why the derivative drops things; this one is what that costs you when you try to consolidate tools.