A Glue Spark job joins a multi-terabyte fact dataset with a small dimension table that comfortably fits in executor memory. The current shuffle join moves large amounts of data across the cluster. Which optimization should the engineer evaluate first?

Published
  1. Coalesce the complete fact dataset to one partition before joining.
  2. Broadcast the entire multi-terabyte fact dataset to every executor.
  3. Broadcast the small dimension table so workers can join locally against the large dataset.
  4. Convert every join into a Cartesian product and filter after materialization.

Comments — 0