BERT · Classification · MLflow
Mortgage Document Classification
A model-first document workflow replacing brittle rules while preserving human oversight of low-confidence predictions.
THE PROBLEM
Why this system existed
Rules-based triage consumed analyst time and struggled with document variation, while ambiguous cases still required review.
OUTCOME
What changed
Reached 93.4% macro F1 and reduced analyst time per batch from 8.2 hours to 1.4 hours.
REFERENCE ARCHITECTURE
Controls around the model
01Classify
02Route
03Retrieve
04Rerank
05Generate
06Validate
07Observe
DECISIONS
Trade-offs considered
Confidence-gated automation
Human review tail
Tracked model variants
Compliance-approved REST deployment
FAILURE CASE
What did not work
Automation without confidence routing would hide uncertainty; below-threshold documents remained in the analyst queue.