This website uses cookies to ensure you get the best experience.
Learn more.
A team uses a model priced separately by input and output tokens. It repeatedly sends a very long prompt and requests lengthy answers. Without changing the number of requests, which change can directly reduce token-based inference usage while preserving required content?
Published
- Keep sending the same oversized prompts and expect a lower temperature to eliminate input-token charges.
- Count only the number of characters and assume every model uses one character per token.
- Increase the requested answer length because output tokens are always free.
- Remove unnecessary prompt material and set an appropriate response-length limit, then evaluate answer quality.
Comments — 0
Sign in to comment