A failed AI agent run is usually treated like a weird one-off. Someone opens the logs, finds the last error, edits the prompt, and moves on. That is how the same bug comes back three weeks later wearing a different model name.

The better habit is simple: turn expensive or dangerous failures into tests. Capture the whole run, label the step that went wrong, write down the behavior you expected, then make the next release prove it no longer fails. The result is less glamorous than another prompt trick, but it is much more useful.

A recent study of coding-agent failures gives this approach some teeth. Iterative refinement failures accounted for 56% of the failures in its dataset. The authors also found that a structured explanation system helped participants identify root causes 2.8× faster than raw traces. Those numbers do not prove that every team needs a large observability platform. They do show that disconnected logs are a poor debugging interface.

Structured agent failure analysis workflow

Trace the failure

Start with one failed run, not a grand telemetry project. You need enough information to answer one question: which step first put the agent on the wrong path?

A useful trace has a parent run and child spans for each meaningful operation. Record the user task, model and version, prompt version, tool name and arguments, tool result, retrieval query and returned documents, state changes, iteration number, latency, token usage, and the final response. If the agent hands work to another agent, keep that handoff inside the same trace.

Plain application logs can contain these fields, but a flat list makes the causal chain hard to see. OpenTelemetry's trace conventions exist for this reason: spans describe individual operations and give different services a shared vocabulary for correlating them. You can use an observability vendor, a self-hosted collector, or a small JSON trace store. The important part is the shape of the evidence, not the logo on the dashboard.

Do not log private prompts or customer data by default. Redact secrets before export, hash identifiers, and make payload capture configurable. A trace that helps you debug a refund agent but copies a customer's full account record into a third-party dashboard is a bad trade.

When a run fails, classify the first meaningful cause. Useful labels include wrong tool selection, invalid tool arguments, stale retrieval, missing state update, unsupported assumption, timeout, permission failure, and iteration exhaustion. Do not label the entire run "hallucination" unless you can show that the model's unsupported claim was the first bad step. That label hides more than it explains.

The paper's experiment varied the maximum iteration budget across 1, 2, 5, and 10 steps. It reports that overly restrictive budgets caused failures even when the initial approach was sound, and suggests a minimum budget of 5 to 10 iterations for typical coding tasks. Treat that as a starting hypothesis, not a universal setting. A short database lookup should not receive the same budget as a repository-wide refactor.

The first artifact to save is a small failure record. It can live in Git until the workflow grows:

{
  "task": "Update the invoice export command",
  "failure": "agent repeated a failing test without changing the implementation",
  "first_bad_step": 6,
  "category": "iteration_exhaustion",
  "expected": "change the parser, rerun the focused test, then report the diff",
  "must_not": "rerun the same command more than twice",
  "trace_id": "redacted-run-id"
}

This record is more valuable than a giant transcript because it states what a future run must do differently. Keep the raw trace beside it for diagnosis, but make the test case readable without replaying 400 lines of model chatter.

Turn the failure into a small test

A regression case needs three things: an input, an executable task, and a scorer. The input is the original user request plus the minimum context needed to reproduce the behavior. The task is your real agent workflow, not a simplified model call that avoids the tool or retrieval path that failed. The scorer checks the behavior you care about.

Start with deterministic checks. For a coding agent, that might mean the requested file changed, the focused test exits zero, no forbidden command ran, and the final response contains a patch summary. For a support agent, it might mean the answer includes the correct policy ID and does not claim a refund when the order is outside the allowed window. Deterministic checks are cheap, repeatable, and easy to trust.

Use a judge only where wording can legitimately vary. A judge can score whether an explanation is complete or whether a response follows a policy, but it should not be your only gate for a destructive action. Pin the judge model, ask for a structured binary result, and compare it with human labels before making it block merges.

Langfuse describes this loop as a dataset, an experiment, and a threshold. Its documentation also recommends keeping a pull-request gate in the tens to low hundreds of cases. The exact count is less important than coverage. A 20-item starter set that represents your known failure modes beats 2,000 synthetic prompts that never exercise your tools.

The practical dataset fields can stay boring:

{
  "input": {"task": "...", "context": "..."},
  "expected_output": "...",
  "failure_category": "wrong_tool",
  "scorers": ["tool_allowlist", "required_fact", "judge_quality"],
  "source_trace": "redacted-run-id"
}

Every time production exposes a new failure mode, add one case. Every time a case flakes, decide whether the agent is unstable or the scorer is badly designed. Do not silently delete flaky cases. Move them to an advisory suite until you understand the variance.

Gate the fix in CI

The release decision should be explicit. Block the pull request when a deterministic safety or correctness check fails. Warn when a judge score dips but the judge is still being calibrated. Run the full, expensive suite nightly if it is too slow for every pull request.

Langfuse's CI workflow follows this pattern: run an experiment against a dataset, calculate evaluators, raise RegressionError when a threshold is missed, and let GitHub Actions fail the job. Braintrust describes the same feedback loop from another angle: inspect a production trace, add it to a dataset, run it against future changes, and feed the next failure back into the set.

A minimal policy might look like this:

block:
  focused_tests_pass: 1.0
  forbidden_tool_calls: 0
  required_schema_valid: 1.0

warn:
  judge_quality: 0.80
  response_helpfulness: 0.75

run:
  pull_request: blocking checks plus a small dataset
  nightly: full dataset plus judge scores
  release: full dataset with pinned model and dataset versions

Pin the dataset version for release runs. Otherwise a production trace added during the test can change the result while you are trying to compare two model versions. Store the model, prompt, tool definitions, retrieval configuration, commit SHA, and evaluator versions with every run.

The threshold needs a baseline. Run the current system several times, especially when a judge is involved, and leave room for normal variation. If the baseline is 0.88 and the score regularly wanders by a few points, a threshold of 0.90 will create noise rather than safety. A deterministic check can block immediately. A judge should earn blocking authority through repeated agreement with human review.

There is a useful connection to the bash-only coding-agent workflows discussion: tool boundaries are part of the behavior you need to test. If a change adds a new shell tool, add cases for dangerous commands, malformed arguments, and recovery after a non-zero exit. The agent's final answer is only one output. Tool choice is an output too.

What not to automate yet

Do not turn every bad answer into a blocking test. Some failures are caused by a temporary provider outage, changing external data, or an evaluator that cannot distinguish a short correct answer from a long one. Record them, label them, and decide whether they belong in the blocking, advisory, or monitoring layer.

Do not capture hidden chain-of-thought as your debugging strategy. Tool inputs, tool outputs, state transitions, retrieval evidence, and explicit model responses are enough to reconstruct most operational failures. They are also safer to store and easier for another engineer to inspect.

Do not assume an observability vendor fixes the workflow by itself. The hard part is choosing the first bad step and writing a test that expresses the desired recovery. A dashboard can show that six calls happened. It cannot decide whether call three should have been retried, replaced, or forbidden.

The payoff compounds. The first regression case prevents one known failure. The tenth starts to reveal which prompts, tools, and retrieval changes cause trouble. Eventually, a failed run stops being an incident report that disappears into chat. It becomes a small piece of executable knowledge.

Sources