This website uses cookies to ensure you get the best experience.
Learn more.
A customer assistant should begin showing text quickly while a model generates a long answer. Which design best improves perceived responsiveness without claiming faster total generation?
Published
- Increase maximum output tokens and assume first-token latency becomes zero.
- Cache every private answer globally and return it to all users.
- Wait for the full response and display a fabricated answer during generation.
- Use supported streaming inference and handle partial output, errors, and cancellation in the client.
Comments — 0
Sign in to comment