A feature engineering job repeatedly scans a large text dataset to select a few columns for one date. Which data organization should be evaluated?

Published
  1. Duplicate all columns into every file regardless of query needs.
  2. Partition on useful predicates and use a suitable columnar format such as Parquet.
  3. Combine every date into one uncompressed text object.
  4. Use random partition labels unrelated to the queries.

Comments — 0