A long-context assistant regularly sends the same large reference prefix followed by a short changing question. What optimization should be evaluated on a supported model?

Published
  1. Assume caching guarantees a hit for every request and every model.
  2. Insert a unique timestamp before all shared context on every request.
  3. Keep reusable content at the prefix and use supported prompt caching, measuring actual cache usage.
  4. Cache the final answer globally without considering user authorization.

Comments — 0