KV cache
The model's working memory of the conversation so far, kept on the GPU so it does not recompute the whole prompt for every new token.
how it works · the vocabulary
key-value cache
It is the main consumer of GPU memory during serving, and it grows with the context length and the number of concurrent users. Prompt caching sells you access to a KV cache that outlives your request; provisioned throughput sells you the memory to hold one.