Fine-tuning · MLOps · Cost Control
Domain-Adapted Llama 3
QLoRA fine-tuning, MLflow experiment tracking, vLLM serving, and staged promotion with quality checks at each gate.
THE PROBLEM
Why this system existed
High-volume domain tasks need strong quality and lower unit cost without sacrificing reproducibility or rollback.
OUTCOME
What changed
Improved task F1 from 0.74 to 0.91 and reduced per-query inference cost by 38%.
REFERENCE ARCHITECTURE
Controls around the model
01Classify
02Route
03Retrieve
04Rerank
05Generate
06Validate
07Observe
DECISIONS
Trade-offs considered
Parameter-efficient QLoRA adaptation
Dataset and hyperparameter versioning
Canary promotion
Evaluation-gated rollback
FAILURE CASE
What did not work
Fine-tuning gains are task-specific; model promotion stayed gated by domain evaluation rather than benchmark performance alone.