A team reports an improved AI assistant using a test set repeatedly inspected during prompt tuning. What is needed for a credible final quality estimate?

Published
  1. Use only the training examples because their expected answers are known.
  2. Choose the best single run and omit all other outcomes.
  3. Use a separately held-out representative evaluation set and document the acceptance criteria.
  4. Remove difficult cases after seeing their results.

Comments — 0