A GPU has enough arithmetic throughput but insufficient memory for the chosen model and batch. What should be considered?

Published
  1. Add network bandwidth while ignoring model placement
  2. Increase only host swap and assume GPU capacity is unlimited
  3. Choose the largest batch without measuring device memory use
  4. Memory capacity, model partitioning or a smaller validated workload configuration

Comments — 0