A Three-Layer Benchmarking Framework for AI Knowledge Assistants - Part 1: Methodology
Most AI benchmarks test whether a system can answer questions. Ours tests whether it can be trusted to answer them in environments where a wrong answer has real consequences.
We built a three-layer benchmarking framework for our AI knowledge assistant, powered by a proprietary retrieval architecture designed specifically for regulated industries.
Step 1 : Curate benchmark question set that comprehensively tests information throughout the knowledge base
Generic QA benchmarks don't reflect the complexity of real-world regulated documents. We curated questions that mirror the types of lookups users actually perform in production, including the ones most systems struggle with:
- Answers buried inside multi-column tables
- Codes and identifiers that appear exactly once in dense policy manuals spanning hundreds of pages
- Questions requiring synthesis across multiple sections
- Acronym-heavy queries where general-purpose AI models have no context
- Policy rules hidden inside forms, nested tables, and unstructured layouts
- General summarization questions
Step 2 : Measure retrieval separately from generation
This is where most benchmarks fall short. They only measure the final answer. But in regulated environments, correctness alone isn't sufficient. A right answer pulled from the wrong source can't be audited, can't be trusted to hold up as content changes, and masks the retrieval failures that will eventually surface as visible errors. Measuring retrieval and generation as separate layers is how you tell the difference between a system that works and one that appears to.
We split the evaluation into two layers:
- Retrieval: Did the system find the correct page? Did it rank it first? What happens as you retrieve more or fewer pages? Key performance metrics include:
- Multi-labelPrecision: What proportion of the predicted pages were properly sourced
- Multi-label Recall: What proportion of the ground truth pages appeared in the prediction
- Multi-label F1: The harmonic mean of multi-label precision and recall
These metrics are computed for each individual inference and aggregated across examples for evaluating over all generalizability.
- Generation: Given what was retrieved, how accurate, complete, and well-formatted was the answer? We use an independent AI model in an LLM-as-a-judge scheme, scoring each response against verified ground truth answers. The key metrics are:
- Accuracy: Does the response contain the correct factual answer, including exact codes, modifiers, and identifiers where applicable
- Completeness: Does it fully address all parts of the question, or only part of it
- Hallucination Safety: Are all claims traceable to the source document, or does the response introduce information not supported by the source
- Conciseness: Is the response length appropriate for the complexity of the question
This separation matters because a system can retrieve the wrong page and still generate a plausible-sounding answer. Worse, with a large enough context window, the correct answer might appear somewhere in the retrieved material even when the retrieval itself was inaccurate. Both scenarios mask retrieval failures. The first produces confident, well-formatted, completely wrong responses. The second appears to work but breaks unpredictably as content changes or scales. In regulated environments, that’s the dangerous failure mode, and it’s invisible without layered evaluation.
Step 3 : Evaluate user experience metrics
Retrieval and generation quality tell you if the system works. These metrics tell you if it works in production.
- Latency: How long does the end-to-end response take? We measure time-to-first-token separately from total generation time to distinguish retrieval latency from model latency.
- Formatting Quality: Does the response structure match the complexity of the question? Clean one-line answers for simple lookups. Structured tables and bullet points for multi-part policy questions.
- Cost: Good retrieval pays compounding dividends here. When the retrieval layer consistently surfaces the right source material, the LLM doesn't need to compensate with size. That means smaller become viable, which in turn opens up self-hosting on reasonable GPU instances, or API-based deployments that consume significantly fewer tokens per query.Better retrieval directly expands your deployment options and reduces cost at scale.
Takeaways
- Modular troubleshooting: Knowing where a system breaks is as important as knowing that it breaks. This modular evaluation approach gives you that clarity.
- Transparent evaluation: Every score is traceable to a specific question, retrieved source, and judge rationale. Stakeholders can audit the results, not just trust them.
- Swap-and-tune flexibility: Evaluate retrieval and generation independently; swap models or tune parameters and measure the impact in isolation.
- Informed deployment decisions: If a smaller model matches frontier model accuracy on your domain and your benchmark proves it, you have the evidence to justify self-hosting or right-size your API spend.
In Part 2, we share the full results - what the numbers actually looked like, where each system broke down, and what the benchmark revealed about the tradeoffs between retrieval breadth, answer quality, and cost.
.webp)
