The first token arrives quickly, but long responses finish slowly. Which measurements best distinguish startup from generation throughput?

Published
  1. Treat total latency divided by request count as a per-token generation rate.
  2. Use only the number of retrieved documents.
  3. Use only the first-token time as proof of complete-response performance.
  4. Measure time to first token, output token rate, and full request latency separately.

Comments — 0