This website uses cookies to ensure you get the best experience.
Learn more.
A Triton deployment receives many small independent inference requests. Which feature can improve GPU utilization by combining compatible requests?
Published
- Dynamic batching configured for the model.
- Returning a cached HTTP success code without inference.
- Forcing the server to idle between all requests.
- Renaming every request to batch.
Comments — 0
Sign in to comment