A customer assistant should begin showing text quickly while a model generates a long answer. Which design best improves perceived responsiveness without claiming faster total generation?

Published
  1. Increase maximum output tokens and assume first-token latency becomes zero.
  2. Cache every private answer globally and return it to all users.
  3. Wait for the full response and display a fabricated answer during generation.
  4. Use supported streaming inference and handle partial output, errors, and cancellation in the client.

Comments — 0