A team compares two retrieval configurations in production. What makes the comparison more trustworthy?

Published
  1. Comparable traffic assignment, defined metrics, and controls for relevant differences
  2. Route easy users to one variant and difficult users to the other without adjustment
  3. Judge only whichever variant had more total requests
  4. Change the model and prompt unpredictably in only one variant

Comments — 0