A Triton deployment receives many small independent inference requests. Which feature can improve GPU utilization by combining compatible requests?

Published
  1. Dynamic batching configured for the model.
  2. Returning a cached HTTP success code without inference.
  3. Forcing the server to idle between all requests.
  4. Renaming every request to batch.

Comments — 0