Replacing Legacy Enterprise Search With a Retrieval-Grounded AI Agent
Keyword search was returning documents; people needed answers. The engineering was less about the model than about making sure every sentence it produced could be traced to a source.
- Client
- Enterprise knowledge team
- Industry
- Enterprise software
1000s
Documents indexed
Hybrid
Dense + keyword retrieval
Source-linked
Every response
Sub-second
Retrieval latency
The problem
The existing search worked exactly as designed and was useless in practice. It matched keywords across thousands of internal documents and returned a ranked list. Employees then opened five of them, skimmed, and either found the answer or gave up. Most gave up and asked a colleague, which is how institutional knowledge quietly becomes a tax on the few people who remember things.
The ask was an assistant that answers, not a search box that lists. The constraint was that a wrong answer delivered confidently is worse than no answer at all, because people act on it.
The architecture
We built a Retrieval-Augmented Generation system on OpenAI models with Pinecone as the vector store and LangChain orchestrating retrieval and composition. The generation step is the small part; almost all of the engineering effort went into what gets retrieved and what the model is permitted to do with it.
Documents are chunked with overlap tuned to the document types rather than a global default, embedded, and stored with metadata — source, department, document type, recency — that the retriever can filter on. A hybrid strategy runs dense vector similarity alongside keyword matching, because dense retrieval alone reliably misses exact identifiers: part numbers, policy codes, contract references, the things people actually search for.
Hallucination control
The system is not allowed to answer from parametric memory. Responses are composed strictly from retrieved passages, each answer carries citations back to the source documents, and when retrieval returns nothing above the relevance threshold the correct output is an explicit "not found in the knowledge base" rather than a plausible paragraph.
That last behaviour is unglamorous and it is the whole product. An assistant that admits ignorance ten percent of the time gets trusted for the other ninety. An assistant that always answers gets audited once, caught once, and abandoned.
Metadata filtering does more work here than the model does. Scoping a query to the right document class before retrieval removes most of the confident-but-irrelevant failure mode at source, and it costs nothing at inference time.
Latency and cost
Token consumption is the running cost of a RAG system and it is easy to let it drift. We capped retrieved context, ranked and truncated aggressively before composition, and cached embeddings for unchanged documents so re-indexing only touches what actually moved.
Retrieval sits under a second, and the perceived response time is dominated by generation rather than search — which is the correct place for the time to go, and the place where streaming hides it from the user.
Results
The team replaced legacy search with an assistant that returns grounded, source-linked answers across the full internal corpus, at a latency that keeps it in the flow of work rather than as a thing you go and do.
The client's summary after delivery: "He quickly understood the scope, tackled the technical challenges with precision, and ensured the system performed as expected. Communication was smooth throughout the project, and he remained responsive to feedback."
The caveat worth carrying
RAG quality is bounded by corpus quality, and no amount of retrieval engineering fixes contradictory source documents. Where two policies disagreed, the honest system surfaced both and said so — which turned out to be genuinely useful, because it exposed conflicts nobody had noticed.
If your documents have never been audited for currency, budget for that before you budget for the model. It is the least interesting part of the project and the one that determines whether it works.
