What Is CAG (Cache-Augmented Generation)? Comparing CAG vs RAG with Testing Up to 620,000 Tokens

CAG (Cache-Augmented Generation) is a method that, instead of narrowing down documents through search, puts the full text into the prompt as a preamble, caches that part, and reuses it for each question. While it eliminates the search misses that occur with RAG, the larger the document, the heavier each single process becomes. We had Claude on Amazon Bedrock answer 24 questions based on a fictional internal company policy created for testing, at five scales ranging from about 10,000 to 620,000 tokens, and compared this against three search-based approaches and Self-Route (a method that answers via search and falls back to the full text when search is insufficient). At 36,000 tokens, CAG matched or exceeded search in accuracy rate, cost, and latency across the board, and at 108,000 tokens, cost and latency were roughly on par. Beyond 300,000 tokens, while accuracy was maintained, the time to the first character and the cost per question ballooned to 2 to 7 times that of search.

For information systems and DX personnel considering an AI chat that answers questions about internal company policies or FAQs, we summarize how to choose between these approaches using numbers.

CAG is the opposite of RAG, which searches before passing content—it passes everything, and caches the part that was passed.

First, we will organize how the mechanism works and how to read the figures from the frequently cited original paper.

RAG splits documents into fragments (chunks) of a few hundred characters each, builds an index using embeddings or BM25, and for each question passes only the few most relevant pieces to the model. CAG does not build an index. It places the full text of the document in the same order at the beginning of the prompt every time, adds a cache boundary marker at the end, and appends only the question after that. This approach is made possible by the expansion of the context window; the Claude Sonnet and Haiku models used this time accepted documents of up to 620,000 tokens as-is.

The first time, the full text is read in and written to the cache; from the second time onward, it is read from the cache, and only the question part is newly processed. What corresponds to storing the KV cache (the intermediate result of attention computation) in the original paper is, in the API, prompt caching.

The cache only takes effect when the preamble up to the boundary marker does not differ by even a single byte. The default lifetime is 5 minutes, extended each time it is read, with a 1-hour setting also available. The unit price for Claude Sonnet in Bedrock's Bangkok region was 2.00 USD per million tokens for normal input, 2.50 USD for cache writes (5-minute lifetime), and 0.10 USD for cache reads (these are reference values as of the time of writing; please check the AWS pricing page for the latest rates). Since reading is one-twentieth the cost of normal input, using the same preamble twice recovers the premium paid for writing. Cache design is also covered in the article on prompt caching in multi-tenant setups.

The name CAG spread from the paper "Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks" (arXiv:2412.15605, a WWW '25 short paper). Using an 8B-scale open model (Llama), the paper compiled SQuAD and HotpotQA documents into 20,000 to 85,000 tokens and compared the results against RAG using BM25 and vector search.

The frequently cited figure of "about 40 times faster" is the ratio, in HotpotQA's largest setting (about 85,000 tokens), between 92.08 seconds when the full text was passed without caching and 2.26 seconds with CAG. It is not a comparison against RAG. The accuracy (BERTScore) for the same setting showed CAG at 0.7407, slightly below RAG's 0.7535 when passing the top 5 results via BM25. In SQuAD's largest setting (about 50,000 tokens), CAG's 0.7734 exceeded RAG's highest value of 0.7658.

What the original paper showed was that when documents are small, passing the full text is about as accurate as RAG, and caching avoids the slowness of processing the full text. This time, we verified just how far this "when documents are small" condition extends.

Rather than using documents from an actual company, we measured using a fictional internal company policy written for testing purposes, along with the text of laws and regulations to increase the scale.

The documents, questions, and methods compared are as follows.

The core of the materials is a fictional set of internal regulations for a Japanese-affiliated manufacturing company in Thailand, written for testing purposes. It consists of 7 documents and 41 sections: Employment Regulations, the old and revised versions of the Business Travel Expense Regulations, an FAQ on expense reimbursement, Information Security Regulations, leave application procedures, and organization/contact information. We deliberately included places where amounts and deadlines differ between the old and revised versions, as well as an FAQ that was left unupdated after the revision.

There are 24 questions, divided into 4 categories: 8 questions that can be answered by reading a single location, 6 questions that require combining two documents, 5 questions where the old version/FAQ conflicts with the revised version, and 5 questions with no answer in the materials (such as the length of childcare leave or the retirement age). An example of a combination question is: "Who approves an expense reimbursement of 60,000 baht, and what is that person's name?" The rule for who approves is written only in the procedure manual, and the name is written only in the organization chart.

To increase the scale, we added the full text of laws unrelated to internal regulations—such as the Patent Act, Copyright Act, and Building Standards Act—from the e-Gov Law Search API, and interspersed the 7 internal regulation documents at even intervals among the laws. The volume of materials was set at 5 levels, from about 11,000 tokens (internal regulations only) up to 36,000, 108,000, 310,000, and 620,000 tokens, with the answers always residing on the internal regulations side.

Scoring was done by having a separate Claude (Sonnet) grade the answers, given the correct answers and the full text of the internal regulations: 1 point for a correct answer, 0.5 points for an answer missing part of the key point, and 0 points for an incorrect answer. Instances of citing outdated values and fabricating rules not present in the materials were recorded separately, and for every judgment other than "correct," we checked the reasoning one by one.

For generating answers, we called Claude Sonnet and Claude Haiku via a global inference profile from the Amazon Bedrock Bangkok region (ap-southeast-7). Thinking was enabled only between tool calls for Sonnet (no tools were used this time) and disabled for Haiku. The materials were passed in the form of Claude's search_result so that citations could be attached, and the instruction text (prioritize the newer version in case of conflict; state clearly if the answer is not in the materials) was the same across all methods.

We compared 6 methods. CAG passed the full text, with a cache boundary marker appended at the end. All three retrieval-based methods had Sonnet generate the answer. The first used vector search alone (Cohere's embedding model, top 6 results), and the second used hybrid search, which blends vector search and BM25 rankings (top 6 results). The third layered Cohere's reranking model on top of Anthropic's published Contextual Retrieval (a method where each chunk has 50–100 characters added via Claude Haiku describing its position within the document, before building the index), reranking the top 30 results and passing either 6 or 12 of them. In Anthropic's own measurements, this combination reduced the top-20 retrieval miss rate from 5.7% to 1.9%. The last method, Self-Route, has the third retrieval method answer first, and only falls back to CAG when the model judges the materials to be insufficient. Retrieval and Self-Route were measured at two scales: 36,000 and 620,000 tokens.

Latency was measured from sending the question from a local terminal to receiving the first character of the response, and for retrieval methods, this includes the round trip for embedding (Singapore) and reranking (Tokyo). For CAG, the second and subsequent questions were cache-read runs; separately, we also measured, for each scale, 2–3 runs where a different single line was appended to the front each time to prevent the cache from taking effect. Cost is a calculated value obtained by multiplying the number of returned tokens by the unit price from the pricing API.

CAG scored between 23.5 and 24 out of 24 at every one of the 5 scales, while retrieval scored between 21.0 and 23.5.

Most of the difference comes from questions that span multiple documents and from the mixing-in of outdated values.

At 36,000 tokens, CAG was equal to or better than retrieval in accuracy, latency, and cost across the board. At 620,000 tokens, CAG still had the highest accuracy, but the median time to first character was 15 seconds.

The cost for CAG is the average for cache-read runs (the cost of the initial write is discussed below).

The differences among the three retrieval methods showed up mainly on cross-document questions. Adding contextual descriptions and layering on reranking increased the number of questions where all the necessary sections made it into the top 6 results, from 16 out of the 19 answerable questions (vector search alone) to 18, and accuracy improved accordingly. Even after adding laws to reach 620,000 tokens, retrieval accuracy barely changed.

The question that every retrieval method got wrong was: "Who approves an expense reimbursement of 60,000 baht, and what is that person's name?" The fact that the approver is the General Manager of the Administration Division is stated in the procedure manual, and every retrieval method answered this part correctly. However, the name is found only in the organization chart, and that section contains neither the word "expense" nor "approval." Because the question's wording doesn't overlap with that section, it never made it into the top 12 results, even with contextual descriptions added and reranking layered on. The retrieval methods' answers were along the lines of "We cannot provide the approver's personal name, as it is not stated in the materials provided"—answering "not present" even though the answer does exist in the materials.

CAG, even at the 620,000-token scale, answered: "According to Organization and Contact Information (as of April 1, 2026), the name of the General Manager of the Administration Division is Sato." Because the full text is visible, it can use the first answer (General Manager of the Administration Division) as a clue to find the second (the name). The Self-Route paper (Li et al., EMNLP 2024) also lists multi-step questions—where the previous answer determines what to search for next—as one of the main reasons RAG fails.

On the question of adding together the daily allowance and lodging expenses (a regular employee traveling to Chiang Mai for 2 nights, 3 days), both vector search and hybrid search missed the revised version's section. Vector search's answer calculated using the old version's daily allowance and lodging expenses, arriving at 4,200 baht, while hybrid search's answer stated, "Article 5 (Daily Allowance) of the 2026 revised version is not included in the materials," and failed to produce the post-revision total. The correct answer, using the revised version, is 5,900 baht. When retrieval misses a necessary section, the model ends up constructing its answer from whatever outdated material it has on hand.

The 5 questions seeded with discrepancies were, surprisingly, answered almost correctly by search as well. The old and revised versions have the same headings and very similar wording, so both were retrieved together, allowing the model to choose the newer one.

What proved problematic for search was a question unrelated to the discrepancies. For "When are reimbursed expenses transferred?", the context-added + reranking (top 6) answer correctly stated the transfer date (closing on the 20th of each month, transfer on the 25th of the same month), but then added, "Note that travel expense settlements must be filed through the expense settlement system within 14 days of returning." The 14 days was an old value left over in the FAQ; the revised version specifies within 10 business days. Because only the FAQ section was retrieved and the model never saw the revised version, it presented the old value as the current rule.

Such contamination by old values occurred in 0–3 cases per condition with the search method, but in zero cases for both Sonnet and Haiku with CAG. The fundamental fix is to delete old FAQs or correct them with every revision, but CAG, which lets the model see the entire document set, is less susceptible to the effects of such missed corrections.

Accuracy did not drop even at 620,000 tokens, but time to first token and cost per question both increased with the volume of documents.

What the cache mainly reduces is cost; at large scale, it does not shorten latency very much.

For Sonnet, the median time for cache-read turns stayed under 2.5 seconds up to 108,000 tokens, but rose to 5.2 seconds at 310,000 tokens and 15.0 seconds at 620,000 tokens.

"With cache" covers 23 runs from the second question onward; "without cache" covers 2–3 runs (for the 11K Sonnet case, one outlier of 52.9 seconds is excluded).

For Sonnet at 310,000 tokens, the 13.5–16.2 seconds without cache shrank to a median of 5.2 seconds, but at 620,000 tokens, the 19.3–22.5 seconds only dropped to 15.0 seconds. This is presumably (unconfirmed) because even when loading is skipped, the process of composing an answer while referencing a long preamble still remains.

Haiku took 3–4 seconds median even at small scale, and was never faster than Sonnet at any scale. At 11,000 tokens, the answer length was not very different from Sonnet's, yet the output token count was roughly double; the cause has not been determined.

For cache-read turns, the cost for Sonnet was 0.007 USD per question at 36,000 tokens and 0.066 USD at 620,000 tokens.

Units are USD (average per turn, including output). Search (context-added + reranking, top 6) stayed around 0.012 USD per question whether at 36K or 620K, unaffected by document volume. CAG's Sonnet was roughly on par at 0.014 USD at 108,000 tokens, but rose to about 3x at 310K and about 5.5x at 620K. At 1,000 queries per day, CAG at 620,000 tokens would cost 66 USD per day, versus 12 USD for search.

Since the cache lifetime is 5 minutes, if queries are more than 5 minutes apart, the next question becomes a write turn. At 620,000 tokens, a single write costs 1.55 USD, so the cost per question balloons during periods when queries are sparse. If the lifetime is extended to 1 hour, the write unit price rises from 2.50 USD to 4.00 USD per million tokens (Sonnet).

Haiku, at 100,000 tokens or fewer, is dramatically cheaper—under 0.001 USD per question—and its accuracy, at 23–24 points, was not far behind Sonnet's. However, for turns where input exceeds 100,000 tokens, input, cache, and output unit prices all shift to a 5x tier; the figures in the table from 108,000 tokens onward are calculated at this rate.

Sending the same preamble again with the full 108,000-token document set resulted in 107,510 tokens being read from cache, with 1.20 seconds to first token and a cost of 0.013 USD. Next, sending it with just one comma added to the first document caused the cache read to drop to 0; 107,511 tokens had to be rewritten, taking 3.33 seconds and costing 0.270 USD. The cache is only used when the entire preamble up to a breakpoint matches exactly, so if there is only one breakpoint at the very end, editing any part of the documents forces a full rewrite of the entire text.

In practice, this assumes the document order stays fixed and that things which change every time—such as timestamps or user names—are not placed before the breakpoint. By putting rarely-changing regulations first and frequently-updated FAQs or notices later, with a breakpoint placed at that boundary as well (up to 4 breakpoints are allowed per call), editing the documents only requires rebuilding the latter portion.

Also, the search_result format used to attach citations adds about 110 tokens per block; for the 41 sections of the internal regulations, this increased the count from 5,721 tokens to 10,386 tokens. With CAG, it is cheaper not to split sections too finely.

Self-Route becomes slower and more expensive the more a question lacks an answer in the source material, and it could not rescue questions that search answered only halfway.

Self-Route is a method in which search is used to answer first, and only when the model judges that the source material is insufficient does it fall back to full-text CAG. The paper (Li et al.) reports that this approach answers 57–82% of questions on the search side, lowering cost by 39–65% compared to passing the full text, while keeping accuracy nearly intact.

This time, 5 out of 24 questions fell back to the full text, and these were exactly the 5 questions that had no answer in the source material. At 620,000 tokens, confirming that no answer exists requires reading the full text, so these 5 questions took 9.2–16.0 seconds to the first character, and the average cost per question was also about twice that of search alone (0.026 USD). On the other hand, the question about the approver whose name could not be answered ended at the search stage with "not stated in the source material," and never fell back to the full text. When the provided material allowed for a provisional answer to be written, the model did not judge the material to be insufficient.

The higher the proportion of questions asking about things not covered in the regulations, the thinner the benefit of Self-Route becomes. Rather than leaving the fallback decision entirely to the model, mechanisms are needed to compensate—for example, by rule, routing questions that likely require other documents, such as "approver" or "person in charge," to the full text automatically.

From this round of numbers, a rough guideline is: use CAG when the source material is around 100,000 tokens or less, and beyond that, use hybrid search with context plus reranking.

In this round's source material, 100,000 tokens corresponds to roughly 100,000 characters.

Based on this round's measurements, the line to draw is: choose CAG when the source material is around 100,000 tokens or less, and choose search once it exceeds 300,000 tokens.

These thresholds are values for the case of using Sonnet via Bedrock. Anthropic, in its article on Contextual Retrieval, states that if the source material is under 200,000 tokens, the full text can be placed in the prompt. However, for inquiry use cases where the median time to first character needs to stay within 3 seconds, this round's measurements put the ceiling at around 100,000 tokens.

The frequency of updates and the scope of what may be shown also factor into the decision. Material that is updated daily requires rewriting the cache each time, and if different departments are shown different material, the cache is split by combination. Both of these are conditions that search handles more easily. How to build search is summarized in the article on hybrid search.

There were 24 questions, and missing even one question shifts the score by 0.5–1 point. Superiority cannot be claimed between conditions with small score differences. The added laws and regulations were text from a field different from internal company rules. In environments with many documents of similar content—such as slightly different regulations lined up department by department—search is expected to miss more, and CAG's accuracy rate could also drop.

Errors occurred with CAG as well. With Haiku, in the 36,000-token round, it added a routing notice and a nonexistent item number not found in the source material, and in the 620,000-token round, it returned one answer in which the opening sentence contradicted the conclusion in the body text.

What was measured were two models, Claude's Sonnet and Haiku, with Japanese-language questions only. The times are values from calling Bedrock's Bangkok region one question at a time from a local machine; wait times when many inquiries arrive simultaneously were not measured. GraphRAG and Agentic RAG, where the model repeats searches, were also not compared.

Here are answers to 3 frequently asked questions about choosing between CAG and RAG.

If the documents are around 100,000 tokens or less (about 100,000 characters in this study) and updates are infrequent, starting with CAG is the simplest approach. It requires no indexing or search mechanism, and in our measurements the accuracy was equal to or better than search. If the volume exceeds that, or if the documents shown differ by user, choose hybrid search with context and reranking instead.

This still happens. What CAG reduces is errors caused by necessary documents being missed during retrieval. Errors from misreading documents, and hallucinations that fabricate content not present in the documents, still remain. Checks such as blocking answers without citations are necessary with either approach.

If there is only one chunk boundary at the end, then no matter where in the document you make a fix, the next single query will require rewriting the entire text. With a 108,000-token document, a difference of a single comma raised the cost of one query from 0.013 USD to 0.270 USD. By grouping frequently updated documents toward the end and placing a boundary just before them, and by applying updates in a batch at a set time, you can reduce both the amount and frequency of rebuilding.

CAG is a method that caches the entire text of a document and reuses it, so there is no omission from retrieval. On 24 questions about a fictional internal company policy, it scored 23.5 to 24 points regardless of scale, equal to or better than search (21.0 to 23.5 points). The differences arose on questions that could only be answered by referencing a different document, and on questions where outdated FAQ values got mixed in.

However, once the document exceeds 300,000 tokens, even with caching, the median time to the first character reached 5 to 15 seconds, and the cost became about 3 to 5.5 times that of search. As a rule of thumb, use CAG up to around 100,000 tokens, and beyond that, layer hybrid search with context and reranking. In either case, the order of the documents, the update method, and citation checks need to be decided at the design stage.

Our company supports AI chat systems that answer questions based on internal company policies and FAQs at a Thailand location, from measuring document volume and update frequency through to selecting the method and building the system. For details, see AI & DX Support Services.

References (accessed November 8, 2026)

• Brian J. Chan et al., "Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks" https://arxiv.org/abs/2412.15605
• Anthropic, "Introducing Contextual Retrieval" https://www.anthropic.com/engineering/contextual-retrieval
• Zhuowan Li et al., "Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach" https://arxiv.org/abs/2407.16833
• e-Gov Law Search https://laws.e-gov.go.jp/