Definition
A memory layer that stores an AI's recent output so that similar future requests are answered from the cache instead of re-running the expensive model. It makes chatbots faster and cheaper — but if the cache mixes up who owns what, one customer's private conversation can be served to another. A breach in the widely used LiteLLM AI gateway showed cached tool results can even be used to hijack a user's agent actions.
Why it matters
Caching delivers the speed and cost savings leaders want, yet it sits directly between customers and the model; a cache bug can quietly hand one tenant's data to another.