A Spark notebook repeatedly collects a large DataFrame to the driver and fails with memory errors. Which design change best addresses the cause?

Published
  1. Repartition the source but still collect every row to the driver
  2. Keep transformations distributed and collect only small necessary results
  3. Convert every row to a longer JSON string before collecting
  4. Raise the result-display row limit and repeat the collection

Comments — 0