Commission
Research
All research areas →

Agent Evaluation and Coordination

Generated-Code and GPU-Kernel Correctness

Federated Learning and Privacy

Decentralised Systems and Protocol

Methods
Papers
Collaborate
About Contact Bring a technical claim

Method · Failure coding

Classify agent failures so two reviewers would agree

A single success rate tells you an agent failed, not where. Incident coding turns traces into counts of planning, retrieval, tool, coordination and escalation failures — but only if the codebook is specific enough that independent reviewers apply it the same way.

The situation

How do we classify failures consistently rather than collapsing planning, retrieval and tool errors into one score?

What you leave with

A versioned codebook with annotated examples, ambiguity and precedence rules, and a reviewer agreement plan with measured agreement per code.

For: Agent researcher; evaluation team

Why one score is not enough

An agent that fails 30% of tasks might be planning badly, retrieving the wrong evidence, calling tools with malformed arguments, or colliding with another agent. Each points to a different fix. Collapsing them into one number makes regressions invisible when one failure kind falls and another rises.

Two of my preprints show what becomes visible when failures are recorded at the right level:

  • In Before the Pull Request, concurrent coding agents coordinated through grite, an append-only log stored in git. The share of work that merely re-did a teammate’s task fell from 78% to 0%, and concrete failure modes — conflicting edits, lock starvation, redundant rediscovery and race-to-close — were recoverable from the log with provenance, several of them invisible in pull-request history. grite is a Neul Labs project, and I founded Neul Labs.
  • The bounded-autonomy harness tracks six indicators — citation validity, source grounding, schema compliance, escalation correctness, constitutional alignment and unsafe-action rate — and distinguishes them by source of supervision: programmatic verifiers, task-level escalation labels or AI-judge scores. Its small pilots also showed configuration variance large enough to mask tuning effects, so coded failure rates need the same uncertainty treatment as any other metric (see uncertainty and repeated trials).

Neither paper is a codebook study. They supply the layers and the warning; the protocol supplies the consistency.

Assumptions

Traces must contain enough intermediate state to locate a failure; reviewers must code independently before seeing each other’s labels; and the codebook must be frozen before the main set is coded, with later changes logged as amendments (see pre-specified evaluation plans).

Illustrative example — not a client engagement. A research team has 600 traces from a tool-using support agent with a 64% task success rate. The pilot codebook has eleven codes; after double coding 60 traces, two retrieval codes show poor agreement because reviewers disagree on whether a stale document counts as a retrieval or a grounding failure. They are merged, and a precedence rule is added. On the main set, tool-argument errors account for the largest single share of primary codes, and the team’s planned planner rewrite is deprioritised in favour of argument validation.

What this can and cannot establish

It can establish consistent, auditable counts of defined failure kinds, how reliably each kind can be identified, and how the mix changes between versions. It cannot prove why an agent failed beyond what its traces show, guarantee the codebook is complete, or make counts from one agent comparable with another without re-piloting.

Useful if

  • You have agent traces or logs and a pass rate, but cannot say which stage of the agent is responsible for most failures.
  • Two teams describe the same incidents differently, so failure trends cannot be compared across releases.
  • You plan to use an LLM or a log-mining script to label failures at scale and need to know whether its labels can be trusted.

Not the right fit if

  • There are no traces — only final outputs. Without intermediate steps, failures can be scored but not located.
  • You need real-time alerting on a production agent. That is an operations build, not a coding study.
  • You want a universal agent taxonomy. Codebooks are built for a system and a question; reuse elsewhere is tested, not assumed.

What this produces

  • Versioned codebook. Each code with a definition, inclusion and exclusion criteria, a positive example and a near-miss that should not receive the code.
  • Ambiguity and precedence rules. How to code cascades (root cause versus downstream symptom), multi-label cases, and traces with too little information to decide.
  • Annotated example set. Traces coded by two reviewers, with disagreements and the adjudicated label and reason kept, not overwritten.
  • Agreement report. Chance-corrected agreement overall and per code, so weak codes are visible rather than averaged away.
  • Coded frequencies. Failure counts per code with intervals, linked back to the trace IDs that support them.

What you need before starting

  • Traces with intermediate steps. Plans, retrieval calls, tool invocations and their results, inter-agent messages or coordination events, and final outputs, with stable IDs.
  • A question the coding should answer. For example, which stage to invest in next, or whether a release changed the failure mix.
  • Two reviewers. People who understand the domain and can code independently. One may be me; the second should come from your side or be agreed in advance.

Protocol

  1. 1
    Fix the unit, then draft codes from traces. Decide what is coded — episode, step, tool call or coordination event — and separate outcome failure (the task was wrong) from process anomaly (the task succeeded despite a fault). Then read a stratified sample and group failures by layer: planning and decomposition, retrieval and grounding, tool selection and arguments, output and schema, coordination between agents, escalation and governance. Write each code with criteria and a near-miss.
  2. 2
    Write ambiguity rules before coding. A single primary code per unit for the earliest failure that explains the outcome, optional secondary codes for symptoms, and an explicit 'insufficient evidence' code instead of a forced guess.
  3. 3
    Pilot double coding and revise. Two reviewers code the same sample independently. Compute Cohen's kappa for two coders (or Krippendorff's alpha for more), read every disagreement, revise definitions, and freeze the codebook with a version number.
  4. 4
    Code the main set with a double-coded fraction. Double-code a pre-agreed share of units, adjudicate disagreements with a logged reason, and report agreement on that share. Codebook changes after this point are logged amendments.
  5. 5
    Validate any automated coder. If an LLM judge or log-mining script assigns codes at scale, compare it with the adjudicated human labels per code before using its counts, and report where it diverges.
  6. 6
    Report frequencies with provenance. Counts and rates per code with intervals, each linked to trace IDs, alongside the agreement table and the codebook version.

Limits and unfavourable results

  • A codebook measures the failures it defines. Failure kinds no one anticipated appear as 'other' or are missed; the pilot reduces this but cannot rule it out.
  • Traces show what an agent did, not always why. Root-cause codes are judgements under stated rules, not causal proof.
  • Agreement statistics depend on how often each code occurs; a rare code can show low kappa even when reviewers mostly agree. Per-code counts are reported alongside.
  • A codebook built for one agent architecture may not transfer to another without re-piloting.

If reviewers cannot reach acceptable agreement on a code after revision, the code is merged or dropped and the report says so. A taxonomy with fewer, reliable codes is more useful than a detailed one nobody applies consistently.

Evidence behind this page

  • grite (GitHub) — Git-native append-only coordination log used as the source of mined coordination failures in Before the Pull Request. MIT. A Neul Labs project; I founded Neul Labs.

Questions

How should AI agent errors be classified?

By the layer where the earliest explanatory failure occurred — planning, retrieval, tool use, output, coordination or escalation — using written inclusion and exclusion criteria, a precedence rule for cascades, and an 'insufficient evidence' code. The categories matter less than whether two reviewers apply them the same way.

Is there a standard agent failure taxonomy?

Not one that transfers reliably across architectures and tasks. Published taxonomies are useful starting points, but a codebook should be piloted on your own traces and its agreement measured before its counts are trusted.

Can an LLM code agent incidents instead of people?

Possibly, for codes where it agrees with adjudicated human labels — but that has to be measured per code. In my Principle-Bench work, two LLM judges from different model families agreed only at Cohen's kappa of 0.16 on an adversarial split, which is a reminder that automated labels can diverge sharply where it matters most.

What makes agent incident coding research reproducible?

A frozen, versioned codebook, the double-coded sample with disagreements and adjudications kept, per-code agreement, and trace IDs behind every count, so another team can recode a sample and compare.

See also

Prepare an evaluation brief

Describe the agent, what its traces contain and the decision the failure breakdown should inform. A redacted sample trace is useful but not required to start.

Last reviewed 2026-10-07.