Prompt & Tool-Result Caching

Definition

Agent caching reuses previously computed information. Prompt-prefix caching reuses model input processing; application caches reuse retrieved data or results. These mechanisms have different correctness, privacy and invalidation requirements.

Prompt caching does not mean reusing the final answer. It can reduce repeated input processing while the model still generates a fresh response. An application answer cache skips more work but also assumes the stored result remains appropriate for the current request and user.

What it covers

Reuse unit

Distinguish prompt processing, retrieved records and generated answers.

Validity

Bind reuse to data, model, tool and policy versions as appropriate.

Isolation

Include tenant and authorization context in application-cache boundaries.

Economics

Account for writes, misses, storage and invalidation overhead.

Design the cache key around correctness. A document identifier without its version may return outdated content; a query without tenant context may leak information. Cached authorization decisions need particular care because permissions and approval state can change independently of the document.

In practice

Practical design patterns by Min.

• Start with repeated stable content whose reuse can be measured.
• Recheck access before returning cached sensitive material.
• Track stale-result failures alongside hit rate and estimated savings.

Measure net benefit over a relevant period. Include cache population, read charges where applicable, misses and the work needed to maintain validity. Frequently changing prefixes or low reuse can produce poor economics even when caching is technically enabled.

Research notes

Source-grounded details and editorial applications. Follow each link for the original document.

Working example

Why the controls matter

The catalog example separates different freshness needs. A description may remain useful while stock, price and shipping restrictions change. The purchase boundary should validate current conditions; a convenient cached answer is not sufficient evidence for a consequential transaction.

Common pitfalls

• Treating a semantically similar question as proof that an old answer is still valid.
• Serving cached data after a user’s access has been revoked.
• Assuming a high cache hit rate guarantees lower overall cost.

Document retention and deletion for application caches as part of the data lifecycle. Expiration alone may not satisfy an urgent correction or deletion requirement. Verify the actual provider feature terms rather than transferring assumptions from another product’s caching implementation.

Questions for operational review

Use these questions to decide what evidence your team needs. They are editorial prompts, not a compliance checklist.

• What exactly is being reused?
• Which changes invalidate it?
• Can reuse expose information across access boundaries?