Most tasks in a Spark join finish quickly, but one reducer runs much longer and spills heavily. A single customer key accounts for a large fraction of input rows. Which diagnosis and mitigation best fit the evidence?

Published
  1. Data skew on the join key; evaluate skew-aware execution or an appropriate salting strategy while preserving join correctness.
  2. Uniformly distributed work; reduce the job to one executor to remove the outlier.
  3. Increase the number of output files after the join without changing how the hot join key is processed.
  4. Increase the number of workers but leave the single dominant join key assigned to one reducer.

Comments — 0