This website uses cookies to ensure you get the best experience.
Learn more.
A Glue Spark job reads a large dataset successfully, then the driver runs out of memory when the code collects all rows locally to calculate an aggregate. Executor tasks are otherwise balanced. Which change should the engineer make first?
Published
- Add more executors but keep collecting the entire dataset into the same driver process.
- Combine all input into one larger partition before collecting it.
- Perform the aggregation with distributed operations and collect only the small final result if necessary.
- Increase source partition count but retain the final collection of every row into the driver.
Comments — 0
Sign in to comment