Why Your AI Knowledge Base Cannot Find Answers (2026)
Most RAG deployments break at retrieval, before the model writes a single word. Three failure modes come back often: semantic search alone misses exact tokens, missing reranking pushes the right passage out of context, and stale documents poison the answers. Here is what to demand from a vendor.
When a knowledge assistant answers "I do not have that in the documents" and the document is two folders away, the model is rarely the culprit. Retrieval failed to hand it the right passage.
11 min read
When the document exists and the assistant still answers "I don't know", start with retrieval. The right passage may never have reached the context, because of the search method, the ranking, or the state of the sources. Here are three common failure modes and the acceptance questions to ask a vendor. Our knowledge base builds start from €6,000 net.
Quick answer
A language model sees only what reaches its context. If retrieval did not find the right paragraph, that paragraph does not exist as far as the model is concerned, even if it sits in the same folder. So "our knowledge base does not work" usually means "our retrieval does not work".
Three causes account for most cases:
- One retrieval method. Semantics alone misses exact tokens; lexical search alone misses paraphrase.
- No reranking. The system does find the right passage, at rank fourteen, while the first five reach the context.
- Dirt in the sources. Stale versions, duplicates and contradictory documents produce confident falsehoods with a citation attached.
Diagnosis starts with a free process scan (€0): 30 minutes with an engineer and a written takeaway in two business days. A dedicated knowledge base build starts from €6,000 net, typical full projects fall in the €6,000–35,000 net range, and an implementation specification with architecture and a fixed quote is €1,200 net. If you commission the system build from us, we credit the specification fee toward the build. Rates are on the pricing page.
Why is semantic search alone not enough?
Vector search compares meanings. That is excellent when an employee asks "how do we settle weekend overtime" and the procedure talks about "work on non-working days". Not a single word matches, and the vectors do.
The same property becomes a defect when the question contains an exact string. Invoice number INV/2026/08/117, error code E-4471, a part symbol or a counterparty name carry no semantic meaning. To an embedding model, one invoice number looks much like every other invoice number. Full-text search finds them instantly, because it looks for that exact string.
This is why the market converged on hybrid search: both paths run in parallel and their results are fused into one list. The standard fusion method is Reciprocal Rank Fusion (Cormack, Clarke, Büttcher, SIGIR 2009), which merges rankings without having to compare incomparable relevance scores.
The size of the improvement shows up in measurements. In its "Contextual Retrieval" write-up (September 2024) Anthropic reported that contextual embeddings cut top-20 retrieval failures by 35%, combining them with contextual BM25 by 49%, and adding a reranker by 67%. Those are Anthropic's numbers on Anthropic's datasets, so treat them as a direction and measure your own corpus on your own question set.
Production confirmation came from Cerebras, which published a description of its internal knowledge base in 2026, serving more than 15,000 questions a day (a company account of its own system, with no independent verification). Their architecture is exactly this pattern: full-text and vector search with IDF weighting and result ageing, RRF fusion, reranking from top-20 to top-10, LLM distillation of Slack threads into structured records, and MCP tools through which agents query the base.
What does reranking do, and what breaks without it?
Retrieval returns a list of candidates. Only the first few enter the model's context, because context costs money and bigger is not better. If the right passage landed at rank fourteen, the system technically found it and the answer will still be wrong.
A reranker is a second, more accurate model that scores each candidate against the specific question and reorders the list. It is slower than retrieval, so it runs on a short list only, for example over the top-20 to select the top-10. It is one of the cheapest ways to raise the quality of a knowledge base, which is why its absence from a vendor's offer is a warning sign.
It is also worth discarding the popular shortcut that you can simply feed the model more text. The "Context Rot" study (Chroma, July 2025, 18 models) showed that long-context quality degrades unevenly and starts falling well before the advertised window limit. NoLiMa (ICML 2025) adds a sharper result: when the question and the target information share no vocabulary, 11 of the models tested drop below 50% of their short-context accuracy already at 32k tokens. Padding the context window makes results worse.
How do stale and contradictory documents ruin answers?
This is the cause that is hardest to discuss with a management board, because no model fixes it.
If a 2024 price list sits on the drive beside a 2026 one, retrieval will find both. The model gets contradictory sources and does one of three things: it picks at random, averages them into a smooth untruth, or cites the older one because it matched the question better. Each outcome is worse than having no tool, because it looks credible and carries a citation.
Good architecture has an answer for this. Result ageing lowers the rank of old documents when newer versions of the same material exist (Cerebras described this as part of its ranking, 2026). An explicit conflict status shows the user that sources disagree and leaves the resolution to the document's owner. An owner assigned to every document set means there is somebody to ask. What that looks like in a running system is described in the piece on permission-aware RAG enforced before the model, where a source conflict is a separate answer status.
Cleaning up documents is often a substantial project line of its own. Price the removal of duplicates, the resolution of contradictions, OCR for scans with no text layer, and the assignment of owners before you sign. That work sits on the company side, and an offer that does not price it will move the cost onto you after signature.
Symptom, cause, what to check
A table to run against your own system. The third column matters most, because each of these symptoms can be diagnosed within hours on a set of a few dozen real questions.
| Symptom | Likely cause | What to check |
|---|---|---|
| "I don't know" although the document exists | retrieval never handed the passage to the context | dump the raw retrieval results for that question and see where the right document ranks |
| Misses invoice numbers, error codes, part symbols | vector search only, no full-text layer | ask the vendor whether search is hybrid and how results are fused |
| Right document in the results, absent from the answer | no reranking or too small a context budget | check how many passages reach the model and whether a reranker runs on the short list |
| Answers correct but out of date | no ageing and no document versioning | count the duplicates and versions of the same material in the corpus |
| Two different answers to the same question | contradictory sources with no conflict status | ask what the system does when two documents disagree |
| Answers contain data the asker should not see | permissions enforced after generation | ask what happens to a forbidden document before the model starts writing |
| Great demo, weak production | demo run on hand-picked files | request a test on 30 of your own questions and your own documents |
How we test this on our own bench: the Czytelnia Pamięci demonstrator
We built a Polish knowledge reading room called Czytelnia Pamięci. It runs locally over a synthetic corpus of 600 documents belonging to a fictional company. We call it a demonstrator, because it is no client deployment and it measures nobody's production. It does contain one number that explains this article better than any other.
The scope test comes out perfect: 150 golden questions across 5 identities, 750 checks in total, and a single out-of-scope leak fails the entire run. Alongside it: byte-identical refusal text (so the shape of a refusal cannot reveal that a closed document exists), a prompt-injection isolation test, and a check that the database role the API connects with was created nosuperuser nobypassrls, so the PostgreSQL row policy actually binds it. Answer accuracy, citation correctness and refusal correctness: 1.0 each. Zero scope violations.
Except every one of those 150 questions carries the document's reference signature, so that 1.0 scores permission enforcement and says nothing at all about retrieval quality. So we added a second sample: 24 paraphrases with no signature, phrased the way a user would phrase them. Top-1 accuracy on that sample came out at 0.511 over 45 scored pairs. Four paraphrases asking about facts absent from the catalogue ended in refusals across all five identities, and they did so by relevance threshold, with no keyword rule involved.
That half is exactly the symptom in the first row of our table. The dense layer in the demonstrator is 256-dimensional feature hashing: a deliberately simple pipe that verifies the construction end of the system. Choosing an embedding model for Polish is simply not part of it. A question built from different words than the document falls below threshold and ends in an honest refusal. Picking the embedding model is deployment work, measured on the client's real corpus.
The lesson to carry into your own demo: scope and accuracy are two separate measurements. A system can score 1.0 on permissions and 0.5 on retrieval, and neither number tells you anything about the other. A vendor showing one figure is showing you half the picture. The scope architecture described here is unpacked further in the piece on permission-aware RAG enforced before the model.
What to demand from a vendor at a demo
A knowledge assistant demo always looks good, because it runs on documents chosen to match the questions. These acceptance questions cut through the script:
- Show the raw retrieval results before the model writes an answer. You want the candidate list and its order. The polished paragraph alone proves nothing.
- Is search hybrid, and how do you fuse the results? An answer of "we use a vector database" means one path and predictable gaps on exact tokens.
- Is there reranking, and over how many candidates does it run?
- What does the system do with contradictory versions of the same document?
- What happens to a document the asker has no right to, before the model starts generating?
- How do you measure quality? There should be a golden question set with expected answers and a regular measurement. Team impressions are insufficient.
- Who cleans up the documents, and is that work inside the quote?
The seventh question usually changes the price of an offer the most, and it is the one most often left out of it.
How to start
- Collect 30 real employee questions your current base answers badly. That is the cheapest diagnostic set that exists.
- Book a free process scan: 30 minutes with an engineer and a written takeaway in two business days. We will walk through those questions and tell you whether the problem is retrieval, the corpus, or both.
- If it turns out your corpus is small and stable, you will hear that plainly. When building RAG is unnecessary is the subject of the piece on an AI knowledge base without RAG.
Book a free process scan | See pricing
Related articles
Frequently asked questions
- Why does an AI knowledge base say "I don't know" when the document exists?
- Usually because retrieval never handed the right passage to the model. The model sees only what reaches its context, so a document that missed the result list does not exist as far as the model is concerned. The usual causes are semantic search with no lexical layer, no reranking, and chunks that are too large or badly split.
- Why does semantic search miss invoice numbers and error codes?
- Vector search compares meanings, and an invoice number, an error code or a part symbol carries no semantic meaning. For those queries full-text search works better, because it looks for the exact string. That is why hybrid search, which runs both paths and fuses their results, is the market standard.
- Do hybrid search and reranking really improve accuracy?
- In its "Contextual Retrieval" write-up (September 2024) Anthropic reported that contextual embeddings cut top-20 retrieval failures by 35%, combining them with contextual BM25 by 49%, and adding a reranker by 67%. Those are Anthropic's measurements on their own datasets, so treat them as a direction and measure your own numbers on your own question set.
- How much of a knowledge base project is document cleanup?
- It is often a substantial project line of its own, and its size depends on the state of the sources. It covers removing duplicates and stale versions, resolving contradictions, OCR for scans with no text layer, and assigning an owner to every document set. Price that work before you sign; we name it in the quote explicitly.
Free process scan
Start with a free process scan.
- A 30-minute call with the engineer who would lead the work.
- A review of the processes that cost you the most time and money.
- A written summary of what to automate first and the likely cost range.
The scan is free and creates no obligation. If automation is unlikely to pay off, the written recommendation will say so.
€0
30 minutes · written takeaway within 2 business days
Times are shown in your own time zone. We work with clients across time zones.
Describe the process in the form