This website uses cookies to ensure you get the best experience.
Learn more.
Most tasks in a Spark join finish quickly, but one reducer runs much longer and spills heavily. A single customer key accounts for a large fraction of input rows. Which diagnosis and mitigation best fit the evidence?
Published
- Data skew on the join key; evaluate skew-aware execution or an appropriate salting strategy while preserving join correctness.
- Uniformly distributed work; reduce the job to one executor to remove the outlier.
- Increase the number of output files after the join without changing how the hot join key is processed.
- Increase the number of workers but leave the single dominant join key assigned to one reducer.
Comments — 0
Sign in to comment