A distributed training job spends substantial time exchanging gradients between nodes. Which infrastructure matters most directly?

1. Only the storage capacity used for final model archives
2. Only the visual resolution of training dashboards
3. Inter-node bandwidth, latency and collective communication efficiency
4. Only the number of CPU cores in an unrelated management node