A labeling project has low agreement between annotators on an important class. What should happen before scaling annotation?

1. Remove all disagreement records without understanding the affected population.
2. Clarify labeling instructions, review ambiguous examples, and measure agreement on a pilot.
3. Train a larger model to make conflicting labels consistent automatically.
4. Choose the fastest annotator labels as unquestioned ground truth.