This website uses cookies to ensure you get the best experience.
Learn more.
A Glue Spark job joins a multi-terabyte fact dataset with a small dimension table that comfortably fits in executor memory. The current shuffle join moves large amounts of data across the cluster. Which optimization should the engineer evaluate first?
Published
- Coalesce the complete fact dataset to one partition before joining.
- Broadcast the entire multi-terabyte fact dataset to every executor.
- Broadcast the small dimension table so workers can join locally against the large dataset.
- Convert every join into a Cartesian product and filter after materialization.
Comments — 0
Sign in to comment