- Evaluate retrieval, answer quality and access controls separately.
- Measure latency across the entire journey using an explicit protocol.
- An assistant should be able to flag missing or insufficient information.
What the KDDI case reports
On 8 September 2026, Google Cloud and KDDI describe work on Buffmee, a consumer RAG application. The case reports a 38% reduction in total latency and an improvement of nearly 18% in time to first token. These results belong to that project and were published by the vendor; they are not a forecast for another company.
The practical interest of this publication is the connection between evaluation and optimisation. For an enterprise assistant, we suggest making three dimensions visible: what was retrieved, what was answered and what the user was permitted to access. A favourable overall impression does not distinguish them.
Build a corpus you can govern
The first task is identifying sources, owners and update rules. An approved procedure, a draft and an old document should not be treated as equivalent references. Useful metadata includes date, status, business scope and access permissions.
Define how documents enter, change and leave the collection. If a procedure is replaced, the assistant should retrieve the relevant version rather than continuing to cite the old one as current. Ingestion is an ongoing process, not a one-time import at the project’s start.
Create a representative evaluation set
Collect real user questions and expected answers approved by the business team. Add ambiguous wording, absent information and contradictory documents. For each question, identify relevant sources and what makes an answer acceptable.
Evaluation should locate the problem. Was the right document retrieved? Does the answer respect its content? Does the citation point to the correct passage? Did the assistant add an unsupported claim? These observations lead to different corrections: retrieval, corpus preparation or generation.
Measure the complete journey
Time to the first response and time to the completed result describe different experiences. An answer that starts quickly may still take a long time to finish. Measure authentication, retrieval, any reranking, generation and display.
Compare variants on the same questions under stable test conditions. Document repetitions and unusual values. A speed optimisation should not be accepted if quality falls beyond the agreed threshold. Costs and operational effort are also part of the decision.
Check permissions before answering
The assistant must respect user permissions in sources and indexes. Control cannot depend solely on a textual instruction asking the model not to reveal certain content. Authorised documents should be selected before they are passed to generation.
Explicitly test two people with different permissions, a revoked permission and a moved document. Check logs, caches and possible exports as well. A factually correct answer is still wrong if it exposes information that the person should not receive.
Plan for uncertainty and maintenance
Define when the assistant should ask for clarification, state that it lacks sufficient sources or refer the user to a person. Users should be able to open cited material and flag problematic answers. That feedback needs an identified owner.
A useful pilot combines a bounded corpus, representative users and reference cases. After launch, rerun evaluations when models, indexes or documents change materially. The objective is quality monitored over time, with explicit limits and a correction path.
| Dimension | Control question | Case to include |
|---|---|---|
| Retrieval | Is the right source found? | Synonym, recent document, ambiguity |
| Answer | Is each claim supported? | Missing or contradictory source |
| Access | May this person see the source? | Revoked permission, different users |
| Performance | What duration does the user experience? | Repeated measurements, first token and complete answer |
Sources & methodology
This insight combines cited publications with editorial analysis. Illustrative examples are not client results. Vendor features and terms may change.