When a major financial services firm tested three enterprise search vendors last year, all claimed over 90% accuracy on their demo queries. Six months into production, the firm’s compliance team flagged dozens of incorrect citations in customer-facing documents — answers that looked authoritative but pointed to the wrong source or invented facts entirely.
The problem wasn’t the technology. It was how accuracy got measured in the first place.
The Overfitting Trap in RAG Evaluation
Retrieval-augmented generation, or RAG — where an AI pulls information from your documents before generating answers — has become the default architecture for enterprise knowledge assistants. Microsoft’s Copilot, Google’s Vertex AI Search, Elastic’s Search AI, and countless startups built on OpenAI’s APIs all use some version of this approach.
The evaluation problem is subtle but serious. Most vendors test their systems on carefully selected query sets that match their training conditions. Hit them with the exact questions they optimised for, and performance looks stellar. Deploy them against the messy, unpredictable queries your employees actually ask, and hallucination rates spike.
Researchers call this “query overfitting” — a system that aces the test but fails in the real world. For regulated industries like banking, healthcare, or legal services, this gap between demo and production isn’t just embarrassing. It’s a liability.
Two Metrics That Actually Matter
A growing push among AI evaluation researchers points to two metrics that procurement teams should demand: query coverage and claim verifiability.
Query coverage measures whether a RAG system can handle the full range of questions users might ask — not just the golden set vendors prepare for demos. A system with high coverage performs consistently across diverse, unexpected queries. One with low coverage might nail common questions but produce confident nonsense on anything unusual.
Claim verifiability goes further. It asks: can every factual statement in the AI’s response be traced back to a specific passage in your documents? Systems that score well on verifiability make it easy to audit answers. Those that don’t leave you guessing whether the AI retrieved real information or generated plausible-sounding fiction.
Together, these metrics give you query-agnostic evaluation — a benchmark that doesn’t depend on knowing which questions users will ask in advance.
Why Vendors Resist (And How to Push Back)
Most enterprise AI vendors won’t volunteer these metrics. Query-agnostic evaluation is harder, requires more compute, and often produces less flattering numbers than cherry-picked benchmarks. Some vendors genuinely haven’t built the infrastructure to measure claim verifiability at scale.
That’s not your problem. During procurement, ask vendors to demonstrate performance on a query set you provide — one they haven’t seen before. Request a breakdown of how many claims in sample responses can be traced to source documents. If a vendor balks, that tells you something important about how their system will perform once the sales team leaves.
For pilots, build claim-checking into your success criteria. Have subject matter experts sample 50 to 100 responses and verify each factual statement against source documents. Track the percentage that check out. If you’re seeing verification rates below 85%, you have a hallucination problem that will scale with adoption.
The Compliance Angle Gets Sharper
India’s Digital Personal Data Protection Act and sector-specific regulations from RBI and SEBI are raising the stakes on AI accountability. When an AI assistant gives an employee wrong information about customer data handling or regulatory requirements, “the AI made it up” won’t satisfy auditors.
Enterprises deploying RAG for anything touching compliance, customer communications, or financial decisions need an audit trail. That means systems designed for verifiability from the start — not accuracy scores calculated on friendly test sets.
Microsoft and Google have begun adding citation features to their enterprise AI products, a nod toward verifiability. But features and metrics aren’t the same thing. A citation button means nothing if 20% of citations lead nowhere useful.
What This Means for You
If you’re evaluating RAG-based search or knowledge assistants, stop accepting vendor accuracy claims at face value. Demand testing on your queries, not theirs. Build claim verification into every pilot. And treat any vendor who can’t explain their query coverage methodology as a red flag.
The enterprises that get RAG evaluation right will deploy AI assistants their employees actually trust. The ones that don’t will spend the next two years explaining hallucinations to compliance teams and wondering why adoption stalled.