This website uses cookies to ensure you get the best experience.
Learn more.
The first token arrives quickly, but long responses finish slowly. Which measurements best distinguish startup from generation throughput?
Published
- Treat total latency divided by request count as a per-token generation rate.
- Use only the number of retrieved documents.
- Use only the first-token time as proof of complete-response performance.
- Measure time to first token, output token rate, and full request latency separately.
Comments — 0
Sign in to comment