250,000 Documents, 94% Retrieval Accuracy, 75% Less Research Time
At a quarter of a million documents, retrieval strategy stops being a detail and becomes the entire product. Here is what we measured, and what we refused to promise.
- Client
- Enterprise knowledge management team
- Industry
- Enterprise operations
250,000+
Documents indexed
~94%
Retrieval accuracy
~75%
Less research time
Citation-backed
Every answer
The problem
A quarter of a million internal documents had accumulated across a decade — policies, specifications, reports, correspondence, procedures — in several systems, with several conventions, and no reliable way to find the current version of anything.
People were spending a substantial fraction of their week on retrieval rather than work. Worse, they were sometimes finding the wrong version and acting on it, which is a category of error that does not show up in any dashboard.
Why scale changed the design
Approaches that work at a thousand documents fail at a quarter of a million. Naive top-k retrieval over a corpus that size returns passages that are semantically near and contextually useless, because at that scale there is always something that looks similar.
So the architecture front-loads narrowing. Metadata filtering — document class, department, effective date, status — reduces the candidate space before semantic search runs. Recency weighting means a superseded policy does not outrank its replacement on wording alone. Chunking is tuned per document type rather than globally, because a procedure and a contract fragment differently.
What we built
OpenAI embeddings over a vector database with the filtering layer above, semantic search combined with keyword matching for identifiers, and a composition step constrained to retrieved content with citations attached to every claim.
Structured outputs, so downstream systems can consume answers programmatically rather than parsing prose, and role-based access enforced at retrieval time — a user cannot receive a grounded answer built from a document they are not cleared to read, which is a subtler leak than it first appears.
API integrations so the assistant is reachable from where people already work rather than being another destination they have to remember.
Measuring accuracy honestly
The ~94% figure is retrieval accuracy against a labelled evaluation set — did the system surface the correct source document for the question — not an assertion that 94% of answers are perfect. Those are different claims and conflating them is how vendors get caught.
We built the evaluation set from real questions the team had actually asked, including the awkward ones, because an evaluation set written by the implementer tends to test what the implementer built.
Results
Retrieval accuracy sits around 94% against that set, and internal research time fell roughly 75%. Answers arrive with citations, so the remaining 6% is visible and checkable rather than silently wrong.
The secondary effect the team valued: because every answer links its sources, people started noticing and retiring superseded documents. The assistant improved the corpus it was reading.
What we'd flag
A corpus this size will contain contradictions, and no retrieval system resolves them. Ours surfaces both documents and says they disagree. That is the honest behaviour and it does mean the assistant occasionally hands back a question instead of an answer.
Access control has to be enforced at retrieval, not at display. Filtering the final response is not sufficient — the model has already read the passage by then, and the leak is in the summary.
