This website uses cookies to ensure you get the best experience.
Learn more.
A Spark notebook repeatedly collects a large DataFrame to the driver and fails with memory errors. Which design change best addresses the cause?
Published
- Repartition the source but still collect every row to the driver
- Keep transformations distributed and collect only small necessary results
- Convert every row to a longer JSON string before collecting
- Raise the result-display row limit and repeat the collection
Comments — 0
Sign in to comment