Training runs are dominated by repeated manual hyperparameter guesses. Which approach best fits?

1. Evaluate data-parallel distributed training.
2. Use a bounded, tracked hyperparameter tuning strategy.
3. Profile and optimize the input pipeline with suitable caching and parallelism.
4. Evaluate a supported model-parallel or sharded training strategy.