Human review and calibration

• Aggregate accuracy (for example 97%) can mask poor performance on specific document types or fields.
• Stratified random sampling of high-confidence extractions measures the ongoing error rate and detects novel error patterns.
• Field-level confidence scores route reviewer attention, but only after calibration against a labeled validation set.
• Validate by document type and field segment before automating anything.
• Route low-confidence extractions and ambiguous or contradictory source documents to review, prioritizing limited reviewer capacity.
• Calibration means the stated confidence matches the measured accuracy.

• Decide which segments can be automated and which cannot.
• Design a review-routing rule from calibrated field-level confidence.
• Keep a sampling program running after automation to catch drift.
• Handle a document whose own values contradict each other.

• "High overall accuracy means every segment is fine."
• "The model's confidence is accurate out of the box." It must be calibrated.
• "Once automation works, sampling can stop." Formats change; drift is silent.

Options that cut review uniformly, review by arrival order or length, or trust an uncalibrated threshold. The best answer measures by segment first and spends reviewer time where errors are most likely.

• 4.4Validation, retry and feedback loops
• 4.6Multi-instance and multi-pass review
• 5.2Escalation and ambiguity resolution