Benchmark methodology
A de-risked approach, backed by multiple reproducible benchmarks — each run head-to-head against standard RAG on the same model, the same questions, and the same retrieval.
Claim by claim
Two measurement families back our data: an accuracy-and-efficiency study at 1,000-document scale on an 8B model, and a fleet-throughput ramp on a 30B model. We keep their attributions separate.
| Claim | How it was measured |
|---|---|
| 91× less text per question | In a 1,000-document test on an 8B model, Smart CAG re-read only the ~15-token question — where a standard RAG setup re-read ~1,352 tokens of document text, both given the same retrieved documents. |
| Same answer quality | Across 500 questions, an independent — and more capable — model graded the answers for meaning and scored the two even: a statistical tie with RAG (details below). |
| 10× faster first token | In the 30B fleet test with one request at a time, the first token came back in 28 ms versus 276 ms for RAG. |
| 3.1× throughput / GPU at fleet scale | In the same 30B test with 24 requests running at once, each GPU finished 9.2 answers per second versus 3.0 for RAG. |
| $0.14 vs $0.43 per 1k queries | At that same 24-at-once point: take the box's hourly price and divide by the answers it sustained per hour (formula below). |
Fleet-scale throughput
We ramp in-flight concurrency up to 24 on one serving GPU tier and measure both arms on the same model and retrieval. The RAG arm runs in the condition a busy fleet actually creates: a per-request nonce defeats the engine's prefix cache — exactly what other tenants' traffic does under churn — so RAG re-reads its context every request while the resident-KV arm does not.
Qwen3-30B-A3B (FP8), tensor-parallel across a 4-GPU tier (4× L4, $4.602/hr). Same model and same retrieved documents feed both arms.
Time-to-first-token is the real time to the first streamed token. Throughput = completed requests ÷ wall-clock at each concurrency level — queueing included.
GPU $/1k queries = instance $/hr ÷ (sustained queries/sec × 3600) × 1000. The real running price of the box, not a list rate.
| Concurrency | Smart CAG q/s | RAG q/s | CAG TTFT | RAG TTFT |
|---|---|---|---|---|
| 4 | 3.92 | 2.04 | 47 ms | 296 ms |
| 8 | 5.25 | 2.41 | 61 ms | 388 ms |
| 16 | 7.55 | 2.73 | 80 ms | 399 ms |
| 24 | 9.22 | 2.98 | 91 ms | 648 ms |
RAG throughput barely moves as concurrency climbs — it is compute-bound re-reading context on every request — so the resident-KV lead widens with load. Full ramp and the raw run outputs are in the source-of-truth results log.
Answer Quality
Query answer accuracy is a statistical tie with a standard RAG architecture. On a 1,000-document haystack, resident-KV serving matches an equally-retrieved RAG within noise, while re-processing far less text per question.
A 1,169-document QASPER haystack, 500 questions, on an 8B model. One shared hybrid retriever (keyword + dense + reranker) feeds both arms — so the comparison isolates the serving path, not retrieval quality.
A paired significance test put the gap between the two well inside statistical noise — on answer quality, they're not distinguishable.
The result is corroborated by a second, independent frontier judge — a different, more capable model than the one under test, so the verdict doesn't depend on a lenient self-grade.
We'll share the connector, the conformance suite, and the prefix-cache benchmark so your own engineers can run the head-to-head on your traffic shape.
Get in touch