A serving team increases batch size and throughput rises, but p95 latency worsens. What explains this?

1. The evaluation set must be invalid
2. Queueing and larger batch execution can delay individual requests
3. More throughput always guarantees lower tail latency
4. The model's output vocabulary must have changed