A team repeatedly tunes against the same holdout until the score looks excellent. What is the main evaluation concern?

Published
  1. A holdout becomes larger every time it is scored
  2. The holdout has become part of model selection and is no longer an untouched final test
  3. The model registry automatically prevents evaluation leakage
  4. Every improved score proves better future performance

Comments — 0