A model's state exceeds one device's memory, requiring its computation or parameters to be divided. Which approach best fits?

1. Evaluate a supported model-parallel or sharded training strategy.
2. Evaluate data-parallel distributed training.
3. Profile and optimize the input pipeline with suitable caching and parallelism.
4. Use a bounded, tracked hyperparameter tuning strategy.