FIELD NOTE / DECISION NOTE
Release gates for LLM systems
An AI feature should not be promoted because a handful of examples look impressive. It should be promoted when the evidence package explains what it does well, where it fails, and how the team can recover.
Start with the task, not the model
A release gate begins with a task contract. What is the system expected to answer, classify, retrieve, or execute? Which cases are unacceptable even if the average score looks healthy? This framing keeps evaluation tied to user and operational outcomes rather than whichever benchmark is easiest to report.
Separate the gates
I prefer separate checks for retrieval, generation, security, latency, and operational behavior. Retrieval can fail while the model remains fluent. A response can be relevant while leaking an unauthorized field. A system can be accurate but too slow or expensive for its intended workflow. One composite score hides those differences and makes ownership unclear.
Carry evidence into deployment
The release artifact should identify the model and prompt versions, evaluation set, retrieval configuration, policy checks, latency and token observations, and rollback target. This is not bureaucracy around an AI feature; it is the minimum needed to explain a decision later. For high-impact workflows, a human approval boundary belongs in the release path as well.
Treat the gate as a control loop
A failed gate should produce a useful next action: expand the test slice, change retrieval, tighten authorization, add a fallback, or defer release. The goal is not to make a system appear safe. The goal is to make behavior measurable enough that improvement is repeatable.