A source directory has millions of historical files and ongoing arrivals. What ingestion design is appropriate?

1. A driver-side full download of every file on each run
2. A fresh destination table for every discovered file
3. A checkpoint stored only in driver-local temporary storage
4. Auto Loader with durable progress and a chosen discovery mode