What tradeoff should be considered when enabling inference batching?

Published
  1. Batching guarantees unchanged model accuracy for all deployments
  2. Batching removes all GPU memory constraints
  3. Throughput may improve while queueing increases latency
  4. Batching always lowers every request's tail latency

Comments — 0