A reliable enterprise knowledge system is not primarily a prompt or model-selection problem. Prioritise permission-aware retrieval, well-maintained source content, and an evaluation set based on real questions before tuning generation. Treat citations as evidence to review; a cited answer can still be wrong.
Start with the knowledge boundary
Retrieval-augmented generation, or RAG, retrieves relevant content and supplies it to a language model when answering. It is useful when answers must reflect changing organisational material without retraining the model. It does not make source material accurate, current, complete, or authorised.
Define the system’s knowledge boundary before choosing a retrieval design:
- Which repositories are included?
- Which document types and versions are authoritative?
- Which users may retrieve each item?
- How quickly must changes appear?
- What must the system do when evidence is missing or contradictory?
- Which questions require a human decision rather than an answer?
Enterprise content is heterogeneous: policies, records, intranet pages, spreadsheets, scanned PDFs and databases all have different structures and freshness requirements. Microsoft’s current Azure AI Search guidance identifies query understanding, multi-source access, token limits, response time and security as separate RAG challenges, rather than treating retrieval as a single search setting. Its documented security approaches include document-level trimming, permission metadata and query-time filters. These describe what the platform can do; whether a particular deployment is correctly secured is separate work. Microsoft’s Azure AI Search RAG overview
Make authorisation part of retrieval
A permission check after generation is too late. Unauthorised text may already have entered the prompt, logs, caches or model context. The retrieval layer should enforce the user’s effective permissions before content is passed to the model.
Recommended controls include:
- Preserve source-system identifiers, ownership, classification and access-control metadata during ingestion.
- Apply security trimming at query time, not only when documents are indexed.
- Test direct access, inherited access, revoked access, group membership changes and confidential-content edge cases.
- Include negative tests designed to make the system reveal documents the user should not see.
- Treat permission changes and document deletion as freshness events requiring measurable propagation times.
A search index is a derived copy. It can become more permissive than the source if permissions are flattened incorrectly, stale, or applied only at the application interface, so an organisation still needs its own access-control testing.
For New Zealand organisations handling personal information, the Office of the Privacy Commissioner’s guidance, published in September 2023, says the Privacy Act applies across the stages of using AI, including user input, responses and actions based on those responses. It recommends a Privacy Impact Assessment before use and regular updates as risks change. Office of the Privacy Commissioner guidance on AI and the Information Privacy Principles
Improve retrieval before changing the model
Retrieval quality depends heavily on document preparation. Preserve headings, tables, page references, dates, document status and links to the canonical source. Remove duplicate versions where possible, or make version precedence explicit. Chunk documents around meaningful sections rather than using only fixed character lengths.
Use more than one retrieval signal for enterprise content:
- Keyword search handles exact policy names, product codes and legal phrases.
- Vector search handles related concepts and terminology variation.
- Hybrid retrieval combines both.
- Reranking can improve ordering after a broader candidate search.
- Metadata filters can constrain results by business unit, jurisdiction, document status or effective date.
The right design depends on the question set. A 2025 benchmark called EKRAG evaluated enterprise-style questions over corporate documents and considered correctness, relevance and faithfulness. It shows that enterprise question answering needs its own evaluation; how well any system sustains that performance in production is a different question. EKRAG in the Association for Computational Linguistics Anthology
Chunking can remove context. Anthropic’s contextual retrieval article describes a provider experiment in which contextual information was added to chunks before indexing, and reports improvements on its tested datasets. Those results are provider-reported experiments, with task, corpus and model choices controlled by the authors. Treat the gains as a hypothesis to test on your own corpus. Anthropic’s contextual retrieval article
Design for uncertainty and source conflict
A grounded answer can still be wrong. The retriever may miss the decisive passage, return an obsolete policy, combine incompatible documents, or retrieve a plausible but irrelevant section. The model may then produce a fluent answer that overstates what the evidence supports.
Set explicit response behaviours:
- Answer only when the retrieved evidence meets a defined sufficiency threshold.
- Say when no adequate source was found.
- Show source title, date and location, not just a generic citation.
- Distinguish requirements, recommendations, examples and historical material.
- Surface conflicts instead of silently selecting one document.
- Route high-impact questions to a named owner or review process.
Grounding features differ by provider. Google’s documentation describes grounding with sources such as search, enterprise data and third-party systems, while Microsoft documents retrieval pipelines that return citations and query metadata. These pages describe product behaviour and supported configurations; independent validation of answer quality, security and fit is separate work. Google Cloud’s grounding overview
Evaluate retrieval and answers separately
A useful evaluation set should contain real or carefully redacted questions, including ambiguous wording, terminology differences, multi-document questions, outdated-content traps, permission boundaries and questions with no answer in the corpus.
Measure at least four layers:
- Retrieval recall: Did the system retrieve the evidence needed to answer?
- Retrieval precision: Were the returned passages relevant and sufficiently focused?
- Answer faithfulness: Are claims supported by the retrieved evidence?
- Answer usefulness: Is the answer complete, understandable and appropriate for the user’s task?
Also measure abstention quality, citation correctness, permission leakage, latency and freshness. Do not collapse these into one score; a system can improve answer relevance while worsening security or freshness.
Measure adoption through task evidence rather than usage volume alone: completed tasks, repeat questions, correction and escalation rates, and whether citations are opened. High query volume may mean the system is useful, or that users cannot find a reliable answer.
ARES, published at NAACL in June 2024, evaluated context relevance, answer faithfulness and answer relevance using automated judges calibrated with a small set of human annotations. This is a useful evaluation pattern, but its reported results came from benchmark tasks and domain-shift experiments, not an enterprise production deployment. Automated judges should therefore be sampled against human review, especially for specialised terminology and high-consequence content. ARES evaluation framework
Manage freshness as an operating capability
Freshness is more than re-indexing on a schedule. Define service expectations for:
- Time from source change to searchable index update.
- Time from permission change to retrieval enforcement.
- Handling of deleted, superseded and withdrawn documents.
- Detection of failed ingestion jobs.
- Identification of stale answers and citations.
Store document effective dates and version status as retrievable metadata, and prefer canonical links and source-system timestamps over copied publication dates. Fast-changing material may need a shorter indexing interval or retrieval directly from the source.
Apply a staged decision test
A sensible pilot should pass these gates before broader release:
- Content readiness: authoritative sources, owners, versions and retention rules are known.
- Permission readiness: access-control tests cover both allowed and denied cases.
- Retrieval readiness: the system retrieves evidence for representative question types.
- Answer readiness: responses distinguish evidence from inference and abstain when necessary.
- Operational readiness: freshness, logging, incident response and evaluation ownership are defined.
- Privacy readiness: personal-information handling has been assessed for the proposed sources and workflow.
If the corpus is poorly governed, RAG will surface the confusion faster than it resolves it. If questions require reasoning over structured records, calculations or undocumented business context, a document-only design may be insufficient. If the cost of a wrong answer is high, retrieval should support human review rather than replace it.
What we are watching
Research is moving towards more realistic enterprise benchmarks, claim-level faithfulness checks, multimodal documents and comparisons between retrieval and long-context approaches. These results remain sensitive to corpus design, query type, model, retriever and evaluation method.
We are watching whether independent evaluations establish durable relationships between benchmark scores and production outcomes. We are also watching permission-aware retrieval under changing group membership, document deletion and cross-repository inheritance. Finally, organisations should watch for evidence that adoption measures reflect better decisions and completed work, rather than novelty, confidence or answer volume.