Why AI Agents Fail in Production: 6 Failure Modes and How to Fix Them

The agent passed every demo run. Then you shipped it. A week later you are looking at a token bill you did not expect, a database with rows you cannot explain, or a support complaint about an action you never authorized.

Demo environments share one property: you wrote the prompt and you wrote the test inputs. In production, inputs arrive from users and upstream APIs you did not anticipate. The failure modes that matter are the ones triggered by inputs you never thought to test.

This guide covers the six failure categories I diagnose most often in agents that passed their demo phase and broke in production. For each one: the observable signal that tells you which failure type you are facing, and the containment fix. This is not a pre-build checklist -- if you want the checks to run before the agent ships, that is the pre-build companion. This guide is for agents that are already running and already wrong.

Of the production failures I have had to diagnose, nearly all were preceded by a demo phase where every single test passed. The problem was not the demo -- it was that the demo inputs were a subset of one.

The 6 Failure Categories: A Triage Table

The fastest way to waste time on a broken production agent is to start with the wrong failure category. A loop explosion and a silent drift failure can look nearly identical in output quality. A permission scope failure and a tool timeout both produce incomplete results with no obvious cause. Getting the category right in the first 15 minutes determines whether this takes 90 minutes or a full day.

The sixth category -- state and memory corruption -- is almost always traceable to one decision: whether stateful context is validated before use. The five categories that need a full diagnostic sequence each get their own section below.

Permission Scope Failures

What you see in production: the agent wrote to a table it had no business touching. Or it sent a message to a recipient outside the defined scope. Or -- worst -- you find out from an external audit, not from your own monitoring.

Permission scope failures in production are forensic, not preventive. The action has already happened. The work is attribution and containment.

Diagnostic sequence:

1. Pull the full audit log for the agent's service account for the last 7 days. Sort by action type, not timestamp -- patterns in what the agent did matter more than when it did them.
2. Identify the first out-of-scope action and trace it to the prompt input that triggered it.
3. Check whether the permission was intentionally granted or inherited from a development credential that was never narrowed before go-live. This is the root cause in most cases.
4. Scope down to the minimum permission required for the defined task. Redeploy. Pull the audit log again 24 hours later and verify.

One concrete benchmark: an agent that only needs to read from one database table should have a credential scoped to exactly that table and nothing else. If an audit of your service account shows 4 tables in the permission set and only 1 is ever accessed, that is a 3-table overreach. Every unused permission is potential blast radius in the event of prompt injection or an unexpected input.

This is the only failure mode in this guide that warrants stopping writes immediately -- before finishing the diagnosis. Scope down the credential first, then complete the audit.

Observability Failures: No Traces, No Evals

What you see in production: the agent completes successfully. Output looks reasonable. Three weeks later you discover it has been consistently wrong in a way that compounded downstream -- incorrect data in reports, wrong summaries delivered to users, bad inputs passed to agents running further down the pipeline.

You cannot diagnose what you cannot observe. An agent with no trace logging and no output evals is an opaque process. When it breaks, you have the final output and nothing else.

Diagnostic signals:

• No structured log for individual tool calls -- only the final agent response is captured, not the intermediate steps
• No eval set: no collection of input/output pairs where you know what correct looks like
• No human spot-check loop: nobody reviewing a sample of agent outputs per week

What trace logging means in practice: every tool call emits a structured log entry with (a) tool name, (b) input sent, (c) output received, (d) timestamp. This is trivial to add to most agent frameworks and makes every other failure mode in this guide diagnosable in under 10 minutes rather than hours.

What an eval set means in practice: 15 to 30 input/output pairs where you know what a correct output looks like. Run the agent against them once a week and track pass rate over time. A pass rate declining from 94% to 81% over four weeks is a recoverable signal you can act on. Finding out when a user complains is not.

Remediation sequence for a running agent with no observability:

1. Add a logging wrapper to every tool call. Deploy it without changing any other agent logic.
2. Run the agent manually against 10 recent inputs. Record the outputs. This is your starting baseline.
3. Set a weekly calendar reminder to run those same 10 inputs again and compare. Imperfect, but it catches regressions immediately while you build the automated version.
4. Build the automated eval pipeline in parallel. The manual loop covers the gap.

Loop and Retry Explosions

Take an agent that classifies incoming support tickets using an LLM. The upstream ticket API starts returning 503s intermittently during a 40-minute window. The agent's retry logic is coded as "retry until success." It fires on every 503. When the API recovers, the agent has made 1,800 classification calls -- 50 tokens each, 90,000 tokens total. At typical GPT-4o pricing that is roughly $0.45 for one queue. But the agent runs 12 concurrent ticket queues. Actual cost for a 40-minute outage: $5.40. At a longer context window and a pricier model, that math reaches $400 without anyone noticing until the monthly invoice. Use the AI Agent ROI Calculator to work out what a loop explosion costs for your specific agent configuration.

Diagnostic signals -- check the bill first, then the logs:

• Token usage spike on a single day that does not correspond to a traffic spike
• Retry count in logs exceeds the limit you believed you had configured
• Duplicate side effects: multiple database rows created, multiple emails sent, multiple payment events triggered for the same input

The "retry limit you thought you set" problem deserves a direct statement: prompt-level retry instructions do not enforce a ceiling. Writing "if the tool fails, try up to 3 times" in a system prompt is guidance, not a constraint. The model interprets it probabilistically. The ceiling must be enforced in code.

Remediation:

1. Audit every tool call for its retry configuration. If any is prompt-level, move it to code today -- not as a follow-up task.
2. Add a hard per-session token budget with a kill switch. A warning is not sufficient; the session must terminate when the budget is hit.
3. Add dead-letter queuing: items that exceed the retry ceiling go to a queue for human review, not back into the retry loop.
4. Verify downstream idempotency: if a tool call does get retried despite the ceiling, does the downstream system handle the duplicate gracefully? Fix the retry loop first, but the idempotency check matters because it does not undo duplicate side effects already in production.

Silent Model Drift

This is the failure mode that takes longest to notice. The agent keeps returning correctly-formatted outputs. The structure is valid. Automated schema validation passes. Monitoring shows no errors. Users notice something is off; dashboards do not.

Take an agent that summarizes customer feedback and tags it by sentiment. For three months it works correctly. Then the base model receives a provider-side update. Summarization quality degrades slightly -- outputs become more generic and lose specific detail. Sentiment tagging accuracy drops from 91% to 78%. No error is thrown. No alarm fires. The sentiment dashboard shows a flatter distribution than reality, and the product team draws conclusions from it for six weeks before anyone questions the data source.

Where drift comes from in production:

• The API provider updates model weights behind a non-versioned endpoint -- calling "gpt-4o" instead of a dated pinned version means you are accepting every provider model update as an untested production change
• Context window handling or tokenization behavior shifts at the provider level without a public announcement
• The prompt references terminology or categories that have shifted in the updated model's behavior

Diagnostic sequence:

1. Check the model version string in every API call the agent makes. If any is a non-dated alias, note every endpoint where this is true before doing anything else.
2. Run your eval set and compare pass rate against the last recorded baseline. If no baseline exists, that finding is equally important -- you cannot detect drift you have no reference for.
3. Take 5 recent agent inputs that you have also run 90 days ago. Run them now and compare outputs side by side. Qualitative shifts surface in the first few pairs.

The fix for confirmed drift: pin to the last known-good dated model version, re-run your eval set on that version, and confirm the regression disappears. Then either stay pinned or rewrite prompts to be robust to the new model behavior before migrating forward.

Tool Timeout and External-Dependency Failures

Production agents depend on external services: databases, APIs, file storage, search indexes. Those services have their own reliability profiles. When they degrade, the agent's behavior depends entirely on whether the tool call has a timeout configured and what the agent does when it hits one.

Take an agent that fetches data from three external APIs and synthesizes a report. One API starts responding in 12 seconds instead of its usual 1.5 seconds during a period of elevated load. The agent has no timeout on that tool call. Every run blocks for 12 seconds at that step. This does not surface as an error -- it surfaces as an SLA miss. Report delivery goes from 3 seconds to 14 seconds. Without per-step timing in the logs, the degradation is nearly impossible to attribute to the right dependency.

Diagnostic sequence:

1. List every external call in the agent: database queries, API calls, file system reads, search index lookups. For each, find the timeout value configured. Any call with no configured timeout is a P1 finding regardless of whether it has caused a failure yet.
2. Pull p95 response latency for each dependency over the last 30 days. If p95 exceeds 60% of the configured timeout, you have no margin for a slow day.
3. Determine what the agent does when a tool call hits its timeout. If the behavior is unclear or undefined, add explicit timeout handling before any other fix.

Remediation:

• Set explicit timeouts on every external call. The specific value matters less than having one at all -- any ceiling contains the failure.
• On timeout, return a structured partial-result object with a flag listing which tools succeeded and which did not complete.
• Let the caller decide whether a partial result is acceptable for the use case. The agent should surface that decision, not make it silently.

The Recovery Protocol: A Prioritized 90-Minute Diagnostic Sequence

When a production agent is broken and you need to move fast, the order of checks matters as much as the checks themselves. Running model evals before confirming you have observability wastes 30 minutes. Auditing permissions before ruling out a loop explosion misses the P0. Here is the sequence that matches failure frequency against diagnostic speed.

Minutes 0-15: Rule out the P0 failures first

• Pull token usage for the last 24 hours. A spike inconsistent with traffic volume is a loop explosion signal. Go directly to the Loop section above.
• Pull the audit log for the agent's service account. Any out-of-scope writes: stop agent writes immediately and work through the permission sequence before continuing. Do not skip this even if you are certain it is not a permission issue.

Minutes 15-30: Confirm you have observability before diagnosing further

• If your logs do not include per-tool-call records, you cannot confirm the root cause of anything else in this sequence. Add the logging wrapper, run the agent on a safe test input, and verify the logs appear before continuing.
• Review the last 20 agent runs: completing normally, partially completing, or hanging? Hanging runs with no timeout error point to the tool timeout category.

Minutes 30-50: Check external dependencies

• Pull response latency for every external tool call in the last 24 hours. Any p95 above 5 seconds warrants investigation.
• Verify each external dependency is reachable from the agent's execution environment right now -- not from your local machine. Network reachability from inside the execution context is the variable that matters.

Minutes 50-70: Run your eval set

• If you have an eval set, run it now and compare pass rate against baseline. This tells you whether the problem is systematic or specific to a subset of inputs.
• If you do not have an eval set, run 10 recent inputs manually and document the outputs against expected results. This becomes your starting baseline for the next incident.

Minutes 70-90: Check model version and drift signals

• Confirm which model version string the agent is calling. If any is a non-dated alias, note it as a finding regardless of whether drift is the current root cause.
• Run one input from 90 days ago through the current agent. If the output has qualitatively changed, you have a drift signal to investigate after stabilizing the immediate issue.

Document every finding in this sequence with a timestamp as you go. The sequence produces a triage report, not just a fix. When the problem recurs -- and production failures tend to recur -- the next person running this protocol has a comparative baseline. For a structured version with pass/fail checkboxes and a notes column, the agent preflight checklist includes a production-audit variant alongside the pre-build checks.