How it works
Read each document into the model's memory once — one forward pass, no training — then serve every question about it from that memory, for every workload on your fleet.
The problem
Every question pushes thousands of context tokens back through the model — fleet GPU time spent re-reading documents that were already read an hour ago, and yesterday, and the day before.
Every query re-prefills thousands of document tokens. The re-reading burns fleet capacity in proportion to traffic.
Time-to-first-token climbs with every token stuffed into the prompt — and it's your SLO that absorbs it.
The engine's built-in cache is keyed to exact token prefixes and evicted whenever the pool churns — so clients re-ship whole documents to rebuild state that existed an hour ago.
The approach
Engram Smart CAG reads each document through the model once and saves the model's internal working memory of it; durable, resident on your GPUs, and addressable by document id. A flexible, composable document base cache, made permanent. Same model, same answers; the reading work is done once instead of thousands of times.
Onboarding is sub-second to single-digit seconds per document, once. After that each answer prefills dozens of tokens instead of thousands — measured 91× less re-processed text per question.
The saved memory covers the full document, not a chunk. Matches the quality of re-reading the entire document, where chunk-RAG loses context.
Attend the whole document's memory (CAG) — or, when storage is tight, a training-free router loads only the slice that answers the question (QRC). Same measured accuracy.
The pipeline
pip install the plug-in — a stock vLLM connector. No fork, no new engine; wire it to your tenant API or document store.
One frozen forward pass per document saves its KV memory. No training, no fine-tuning, ever.
Bring your own retrieval or use the built-in hybrid stack — either way it just picks document ids; a training-free router can narrow to the answering slice.
The model answers from the resident memory — only the question itself is prefilled, dozens of tokens. Grounded, sourced, first token in ~30 ms.
Engram runs inside your VPC with no public endpoint. Your customers' documents never train a shared model, never leave your perimeter, and deleting a memory removes the document from serving — data-residency guarantees you can put in a contract.
Get in touch