FIELD NOTE / SYSTEMS NOTE
Cascade Failures in Multi-Agent LLM Systems: How One Bad Tool Call Corrupts the Whole Pipeline
A legal review pipeline returned confident summaries of clauses that did not exist. The prompt was innocent; the state moving between agents was not. This investigation follows that incident through compounding error, detection, architecture, and the reliability program that made the pipeline fail loudly.
The clause that was never there
At 02:17 an on-call engineer found a legal-document review queue citing a termination clause absent from every source page. The classifier identified the contract family, the retriever returned a nearby indemnity section, and the summarizer sounded certain. Rewriting the prompt changed wording but not the result. A trace showed an OCR tool had returned a valid envelope with an empty clause list; three downstream agents treated that object as evidence. The pipeline returned HTTP 200 for 11,842 documents before a reviewer noticed. A small tool error had become shared state.
Why a pipeline fails differently
A single model call has one observable boundary. A multi-agent workflow creates several state boundaries, and each can convert uncertainty into an assumption. If five agents are each 95% accurate, every handoff is correct only 0.95⁵ = 77.38% of the time. In clinical triage, a 95% extraction component became 78% end-to-end agreement after five steps. Loud failures stop a run; silent failures continue with a plausible object, which is why corrupted state is more dangerous than a wrong sentence.
Five ways cascades begin
Tool output poisoning occurs when a payments API returned null settlement amount and reconciliation treated null as zero, misclassifying 3.8% of same-day transfers. Context accumulation drift appeared in insurance claims: each reviewer appended a guessed incident date until 14% of long traces carried the wrong date as fact. Confidence laundering hit retail banking when a fraud classifier's 0.61 confidence was stripped before approval, raising false approvals by 9 points. State mutation cascades hit logistics when an unvalidated depot code entered shared state and added 22 minutes per shipment. Retry amplification appeared in SaaS support: four retries appended partial answers, prompt size grew 6.2x, and hallucination rate doubled.
02:17:14 state mismatch
02:17:15 retry appended
02:17:16 downstream acceptedDetection: schema validation02:17:14 state mismatch
02:17:15 retry appended
02:17:16 downstream acceptedDetection: checkpoint comparison02:17:14 state mismatch
02:17:15 retry appended
02:17:16 downstream acceptedDetection: confidence propagation02:17:14 state mismatch
02:17:15 retry appended
02:17:16 downstream acceptedDetection: state diffing02:17:14 state mismatch
02:17:15 retry appended
02:17:16 downstream acceptedDetection: retry budgetsConfidence is data, not decoration
In a pharma safety workflow, Agent A returned a suspected interaction with confidence 0.61. Agent B serialized only the result string, and Agent C interpreted it as a verified contraindication. The defect was not the threshold; it was the vanished uncertainty. Preserve confidence, evidence IDs, and abstention reasons as typed fields. A downstream agent must handle them, not merely be invited to read them.
The arithmetic of compounding error
End-to-end accuracy is pⁿ when steps are independent. At 99%, pipelines of 3, 5, 7, and 10 agents are 97.03%, 95.10%, 93.21%, and 90.44%. At 97% they are 91.27%, 85.87%, 80.78%, and 73.74%. At 95% they are 85.74%, 77.38%, 69.83%, and 59.87%. At 90% they are 72.90%, 59.05%, 47.83%, and 34.87%. At 85% they are 61.41%, 44.37%, 32.08%, and 19.69%. A financial approval workflow cannot inherit the best single-agent benchmark; it must pass a complete state transition.
That means 1 in 1 pipeline runs produces a fully correct result.
The pipeline is usable only with checkpoints and abstention.
Why the pipeline oracle is missing
Model-level evaluation has a labeled answer; intermediate states rarely do. A legal extractor may produce an equivalent span, a planner may choose two valid routes, and a verifier may refuse for a novel reason. Reviewers see the polished final answer, not the state that caused it. In e-commerce fulfillment, sampling final order messages missed a 0.4% inventory-state corruption rate for three weeks.
Five detection strategies with tradeoffs
Schema validation catches missing fields and impossible enums; a payments deployment added 3 ms p95 and cut malformed-state incidents 81%, but missed semantic errors. Confidence propagation added 8 ms in insurance and reduced false certainty 31% after class calibration. Checkpoint comparison cost 14 ms and 2.1% storage in logistics while detecting 73% of mutation cascades. Adversarial canaries added 0.4% traffic in clinical search and found injection regressions 26 minutes earlier. Cross-agent consistency checking added 180 ms and 11% compute in legal review, with 4.7% false positives, but caught unsupported summaries schemas could not.
Resolving the legal incident
The incident was tool poisoning followed by context drift. A stricter prompt moved unsupported-clause rate only from 7.4% to 7.1%. A schema gate rejected OCR payloads whose spans were empty while page count was nonzero; checkpoint comparison exposed retries appending rejected objects instead of replacing them. Replacing retries, carrying evidence_status, and adding canaries dropped unsupported summaries to 0.6%, cut detection from 19 minutes to 42 seconds, and added 0.3% queue latency.
Architecture that contains damage
A firewall agent should sit between untrusted tools and shared state when a workflow handles money or regulated text. A clinical deployment added 24 ms and reduced unsafe writes 88%. Uncertainty-tagged outputs suit fraud and underwriting; carrying ranges cost 6% more tokens and reduced false certainty 31%. Human checkpoints belong before irreversible side effects; an insurance payout workflow added 42 seconds median review and prevented 97% of severe attempts. Circuit breakers block calls after a threshold, probe recovery, and reopen only on success; logistics reduced repeated carrier failures from 12 minutes to 90 seconds.
Building a cascade-failure eval set
Sample traces by transition, not only final outcome. Build 300–500 production-shaped runs across normal, absent-evidence, malformed-tool, stale-data, retry, and adversarial cases. Label every handoff with expected schema, provenance, uncertainty, and permitted mutation. Keep a low-base-rate holdout untouched. Synthesize failures by dropping one field, swapping a version, delaying a tool, or duplicating a retry before composing multi-fault cases. In pharma document processing, 62% of synthetic cascades began with parser defects rather than model defects.
Hover a cell to inspect expected failures.
Monitoring the first crack
Track per-step accuracy, rejection rate, confidence deltas, retry depth, context growth, and downstream decisions made from quarantined data. Cascade depth counts how many agents consume tainted state; mean time to cascade detection measures distance between origin and alert. Leading indicators include null-but-successful fields, scores compressing toward 1.0, duplicate keys, and retries that grow prompts. A retail banking monitor caught a 14% context-size increase and stopped a cascade 11 minutes before false approvals crossed its impact threshold.
The reliability questions teams should answer
Can the team name the state contract at every boundary and show which agent last changed a field? Can it distinguish tool failure, retrieval miss, model error, and mutation? Where is uncertainty preserved? Which transitions replay, which side effects are idempotent, and which require a human? What is the breaker threshold and probe? How are rare cascades represented in the holdout, and what is the maximum tolerated depth? In the legal pipeline, answering these questions turned a mysterious bad summary into a rejected payload, named owner, and measurable recovery path.
Graceful failure is the product
A reliable multi-agent system does not promise every agent is right. It promises one wrong agent cannot silently authorize the next ten decisions. Validation, uncertainty, checkpoints, canaries, bounded retries, and circuit breakers make failure inspectable. In regulated financial review, these controls increased median latency 17% while reducing severe cascade incidents 84%. That is the trade: a visible stop is cheaper than discovering corruption in a customer record.
Share a thought
Comments appear immediately. Email is optional and never shown.
No comments yet. Be the first to share a thought.