Why an AI knowledge base misses answers

When an internal assistant says that it cannot find a document that clearly exists, inspect retrieval before replacing the model. The model can answer only from the passages placed in its context. A source that never reaches that context is invisible to generation.

Decide whether the failure calls for better retrieval, cleaner sources, a permission fix, or a smaller tool. The right inspection prevents a rebuild that leaves the underlying problem untouched.

Decide whether retrieval is the fault

Capture the complete path for a failed question:

1. the normalized query sent to search;
2. each candidate returned by every retrieval path;
3. the score and rank for each candidate;
4. the passages passed to the model; and
5. the source and locator shown in the final answer.

If the correct passage is absent from the candidates, inspect indexing, parsing, chunking, query rewriting, and filters. If it is present but absent from context, inspect ranking and the context budget. If it reaches the model but the answer is still wrong, inspect the prompt, source conflicts, and the answer validator. These are different faults with different fixes.

What reaches the model

RAG has two distinct stages. Retrieval finds candidate passages from an indexed source set. Generation receives a selected subset and writes an answer from that context. A polished final paragraph hides the handoff between the stages, so ask to see it during review.

Chunking affects that handoff. A chunk that is too short can lose the definition or table heading that gives a sentence its meaning. A chunk that is too long can push the useful passage out of the context or mix versions of a procedure. Preserve headings, section identifiers, document dates, and links as metadata that can travel with each chunk.

Contextual Retrieval is one published design for this problem. Anthropic describes adding document context to chunks before embedding and combining semantic retrieval with a lexical search path and optional reranking. The article documents a method; quality on your documents still requires measurement at each stage with a question set from your company.

Exact terms need lexical search

Vector search is useful when a question paraphrases the source. An employee can ask how weekend work is settled while the policy uses another phrase. Semantic similarity can bridge that wording difference.

Exact identifiers need a different signal. Invoice numbers, error codes, product symbols, policy IDs, and names can have little semantic meaning. A lexical search such as BM25 looks for the token itself. A hybrid system runs both paths and combines their candidate lists. The fusion method matters less than the acceptance test: an identifier query should return the record that contains that identifier, while a paraphrased question should still find the relevant policy.

If the corpus is small and stable, full-text search may be enough. Do not add vector infrastructure simply because a vendor uses the word RAG. Choose the least complex search that meets the question set and the permission model.

Reranking changes the order

Initial retrieval is optimized for speed. It returns more candidates than the model will receive. A reranker evaluates that short list against the complete question and changes the order using a more expensive relevance calculation. It can recover a useful passage that initial search placed too low, but it cannot recover a source that indexing or permissions excluded.

Treat reranking as a separate component in the test plan. Record candidate recall before reranking, the selected context after reranking, citation correctness, and the effect on latency. A vendor that only shows final answers cannot tell you which stage needs work.

Source state can invalidate a good match

Retrieval can be technically correct and still return the wrong business answer. Common causes include:

• an old and current version with the same title;
• duplicate copies with different access rules;
• a scan whose text layer was never checked;
• an owner who has not confirmed that a procedure is current; or
• two sources that state different dates, limits, or responsibilities.

Give documents an owner, status, effective date, and supersession relationship. If two current sources conflict, return a conflict state and route it to that owner. Do not ask the model to average the statements. Source maintenance belongs inside the knowledge-base project and must be planned before launch.

Permissions come before generation

A model should never receive a passage that the requesting person cannot see. Apply access filters while selecting candidates or before they enter the model context. Filtering a generated answer is too late: the model has already seen the restricted content.

Test a permitted query, a forbidden query, a mixed-permission query, and a change in the user's role. Check both the answer and the retrieval trace. The permission-aware RAG design describes why this boundary belongs before generation.

Test with your own questions

Build a reviewed set from real failed searches. Include:

• exact identifiers and names;
• paraphrases and alternate terminology;
• questions whose answer is absent;
• current versus superseded documents;
• conflicting sources;
• tables and scanned files;
• permission-restricted records; and
• prompt-injection text embedded in a source.

For each question, record the expected answer, expected source, acceptable refusal, and access scope. Measure retrieval recall, rank of the expected passage, citation correctness, refusal correctness, and latency. Repeat the set after a parser, chunking, embedding, reranking, or source-permission change. NIST's AI Risk Management Framework recommends representative testing, defined measures, and monitoring after deployment.

Questions for a vendor

Ask the vendor to answer these with your own corpus:

1. Can we inspect raw results before generation?
2. Which queries use lexical, vector, or hybrid retrieval?
3. Where does reranking run, and what candidates does it receive?
4. How are document versions and conflicts represented?
5. When are permissions applied?
6. Which source locator appears in an answer, and can a user open it?
7. What happens when no source meets the threshold?
8. Who owns parser, source-cleanup, and question-set maintenance?

The answers reveal whether the product is a searchable system with controls or only a chat interface over a prompt.

When search is enough

Use a full-text search or a structured FAQ when the source set is small, stable, and easy to navigate. Choose a RAG assistant when questions cross several maintained sources, users need citations, or access rules must follow the employee. Choose a custom workflow only when the answer must also create or update a record with an approval rule.

FAQ

Should we increase the context window when answers are weak? First inspect retrieval and ranking. More context can add noise, stale versions, and permission risk. A larger window does not repair a missing or ambiguous source.

Does hybrid search guarantee correct answers? No. It broadens the candidate set. Source quality, ranking, permissions, generation, and validation still need separate tests.

What is the first useful check? Take a handful of real failed questions and inspect the raw candidate list, selected context, source version, and access decision for each one.

Sources

• Anthropic: Contextual Retrieval
• NIST AI Risk Management Framework 1.0
• GDPR, Regulation (EU) 2016/679