The Nervous System: You Cannot Debug an Agent by Reading Its Answer

Dana Okafor's offboarding agent reports success. Fourteen accounts reassigned, six systems updated, final paycheck cleared. The summary reads clean. Nine months later, an auditor finds a production database Dana could still reach on day forty. The agent never lied. It just never looked.

If you are picking this up mid-series, the premise fits in a paragraph. An agentic system is eleven layers grouped into seven organ systems, and the model is one of them. The series walks one ordinary event through all eleven: Dana, a senior engineer, resigns. Part 0 drew the map, Part 1 gave the agent a name of its own, Part 2 gave it a plan, Part 3 gave it hands that reach real systems, Part 4 gave it a full picture of what she owns, and Part 5 gave it the rules for what it may do alone.

Fourteen days later the agent says it is done. This layer, Observability and Evaluation, is the only reason anyone can check.

A summary is a claim, not evidence

Ask the agent whether it finished, and it will tell you yes or no. Either answer is a verdict, not a record.

Dana's agent said yes: "offboarding complete." That sentence does not show which of her twelve steps ran, in what order, or against which account. It also does not explain why her production database access was still open forty days after her last day, because a pass does not carry evidence, only a verdict.

Now flip it. Say the equity forfeiture step failed instead, because the 2003 vendor app changed a field name and the agent's browser automation could not find it. "Equity computation failed" is the whole message. It does not say which field, what the agent tried, or what was on screen right before it gave up. Somebody now has to redo the entire run by hand just to find the one step that needs fixing.

A second AI reading that same message has no more to work with than you do. It is grading a verdict with no evidence attached, whichever way the verdict went.

A pass gives you nothing to verify. A failure gives you nothing to fix. Both are verdicts, and neither is evidence. The fix is not a better summary, from this model or any other. It is a different kind of record entirely: a log of what the systems actually did, captured at the moment they did it, with no model's judgment sitting between you and it.

What a trace actually is

A trace records the full sequence, end to end, in order:

• which agent acted, the specific instance running Dana's offboarding, not a shared service account
• which tool it called, the CRM's reassignment endpoint, not a vague "updated the CRM"
• which system it touched, and which record inside it, account 00214, not just "the CRM"
• what it read, the actual input, like the forfeiture number it pulled off the 2003 equity app
• what it wrote, the value it actually set, like the new owner on that account
• when, down to the second, not the day

That is a very different thing from a log line that says "offboarding completed."

Part 3 described the 3:40 am beat: the model provider rate-limited mid-task, the gateway failed over to a backup, and the run picked up from a checkpoint two minutes old. Nobody noticed, because nothing broke. But nobody noticing only proves nobody was watching. A trace proves the run actually recovered: the rate limit at 3:40 am, the fallback firing, the checkpoint loading, and the next tool call landing on the right account four seconds later, in the right order. That is the gap between believing it worked and being able to show it.

The same trace is what tells you where step nine actually failed, rather than leaving you to re-run the whole fourteen-day process and hope the failure repeats. Tracing does not prevent the miss. It is the only thing that makes the miss visible at all.

Four metrics, and the one nobody watches

Dashboards tend to collect whatever is easy to collect. Four numbers carry the weight for Dana's fourteen-day run, and for every run like it.

Latency. How long the run took, and where the time actually went. Dana's fourteen days were not fourteen days of the model thinking. Most of that clock was her final paycheck sitting in her manager's inbox for six hours before he approved it. A latency number that only reports the total hides that the bottleneck was a person, not the model.

Cost. Token spend per run, tied back to the routing decisions Part 3 covers. Revoking Dana's Okta and Slack access is routine, and a cheap model handles it fine. Computing her equity forfeiture off a 2003 vendor app with no API is not routine, and it justifiably costs more. A run that costs three times the median because it routed the routine 90% to the expensive model is a routing bug, not a hard task.

Success rate, broken down by autonomy rung. Part 5 sorted Dana's twelve verbs onto four rungs: hand off, ask first, act and tell, act and log. Revoking her production database access sits on act-and-log, and if it silently fails, that is one kind of problem. Her final paycheck sitting on ask-first and getting rejected by her manager is a different kind, because a rejection there might mean the approval step is doing exactly what it is for. A flat success-rate number marks both with the same red X and hides which one you actually have.

Human override rate. How often a person changes or rejects what the agent proposed. This is the one most platforms do not track, and it is the leading indicator. Say her manager keeps correcting the agent's proposed reassignment of one of Dana's six direct reports, routing them to someone already at capacity. That rising correction rate tells you the reassignment logic is wrong before it produces a real staffing problem. A correction rate that drops on a verb that used to need fixing every time is the opposite signal: the fix worked.

None of these four require reading a single transcript. That is what makes them cheap to run continuously instead of occasionally.

An LLM judge triages. It does not verify

You cannot read every trace from every run. Across a whole offboarding program, at any real volume, nobody can. The common answer is to have a second model read the transcript and grade it: did the agent do what it was supposed to do.

This works, inside a specific limit. Point an LLM judge at Dana's twelve steps and it catches the obvious failures fast and cheap: the wrong account reassigned, a missing revocation, a malformed payroll record. Used to triage a large volume of offboardings down to the small fraction that actually needs a person, it is the only economical option.

Used as the only check, it has a specific failure mode. Say the agent computed Dana's equity forfeiture off that 2003 vendor app and got her vesting-cliff date wrong by a day. A judge model trained on data similar to the model it is grading tends to miss the same thing, the way two students who studied from the same wrong textbook would grade each other's exam identically wrong. Both are confident in the same incorrect date, so the judge approves it, because "confident and wrong" reads identically to "confident and right" from the outside. Feed the judge Dana's summary and it is grading the same verdict-without-evidence the summary already gave you, just with a second layer of confidence stamped on top. Feed it the trace instead, the actual vesting table the agent read and the date it computed from it, and the judge finally has a record to check the number against, not just another claim.

Human review: sample the disagreements

Nobody has time to read every trace, and nobody needs to. What is worth reading is the small set where the LLM judge and a human reviewer land on different verdicts. Say the judge approves the reassignment of Dana's fourteen CRM accounts because all fourteen now show a new named owner. A human reviewer who actually knows the team notices one of those new owners is out on parental leave and cannot pick up the account for another month. The judge checked a field. The human checked reality.

That disagreement rate is a metric in its own right. A low, stable rate means the judge is doing its job and the sampling can stay light. A rising rate means something changed, in the agent's behavior, in the judge's calibration, or in what counts as correct, and it is worth finding out which before the gap widens further. The disagreement is the signal. The account reassigned to someone on leave is just where you go to read it.

Drift is a silent failure

Offboardings like Dana's used to take three days. Now they take six. Nobody touched the code, nobody changed a config, and if you only watch pass/fail metrics, nothing here trips an alarm, because the runs are still succeeding.

This is drift, and it is almost always upstream:

• a model provider silently updates the model behind an API you call by name rather than version
• a vendor changes a form layout the Computer Use step depended on
• a rate limit tightens

Something the system depends on moved, quietly, and most metrics are built to catch failure. A slow six-week creep does not trip them.

Catching it requires a baseline and a comparison, not a threshold. Track the distribution of latency, cost, and step count for a given task type over time, and watch for the whole distribution shifting, not just for individual runs crossing a line. A gradual six-week creep from three days to six will not trip a single-run alert. It will show up immediately on a week-over-week chart, which is the entire argument for building one before you need it.

The learnings log: the component almost nobody has

Six months after Dana's offboarding, a different engineer resigns. Her agent hits the same 2003 equity app, the same missing API, and times out the same way Part 3 described. Somebody already solved this. Nobody remembers, because nothing recorded that it happened, why it happened, or what fixed it.

That is the learnings log, and it answers a different question than Part 4's procedural memory. Procedural memory is what the agent reads to act correctly next time; it is built for the machine. The learnings log is what the team reads to stop rebuilding the same fix; it is built for the humans running the platform. Confuse them and you get one of two failures: the agent's operating memory turns into an ungoverned pile of incident postmortems, or the team's incident history lives nowhere, and every failure gets rediscovered at full price.

Without a learnings log, your platform has no memory of its own mistakes, even while the agents running on it have memory of everything else. That asymmetry is the entire argument for treating it as a real component, not a wiki page somebody meant to keep updated.

What the nervous system decides

Each layer before this one gave Dana's agent something:

• the front door gave it a name
• the brain stem gave it a plan
• the bloodstream gave it reach
• the memory gave it the full picture
• the immune system decided what it could do alone

This layer is the only reason any of the previous five are checkable at all.

• A trace turns "it worked" into a record you can defend.
• Four metrics tell you where to look before something breaks.
• A judge model triages the volume no human could read.
• A disagreement rate tells you when the judge itself needs a second look.
• Drift detection catches the failure that never trips an alert, because nothing failed, something just got slower.
• A learnings log is the only thing standing between your team and paying for the same lesson every quarter.

Get this layer wrong and every other layer becomes a claim instead of a fact. Dana's offboarding either revoked everything on schedule, or it didn't, and "the summary said so" and "here is the trace" are two very different answers to that question.

Next: The Skeleton. Dana's agent can revoke her access and compute her paycheck. It cannot, on its own, be the thing HR, payroll, and the CRM agree actually happened, and that is a different and harder problem.

Matthew Kruczek is Managing Director at EY, leading Microsoft domain initiatives within Digital Engineering. Connect with Matthew on LinkedIn to discuss the observability layer underneath your agent program.

References

1. Gartner. "Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027." June 25, 2025.
2. Anthropic. "Building Effective Agents." December 2024.
3. OpenTelemetry. "Semantic Conventions for Generative AI Systems." 2026.
4. Matthew Kruczek. "The Bloodstream: Your Model Is a Commodity, Your Tools Are the Risk." matthewkruczek.ai, 2026.
5. Matthew Kruczek. "The Immune System: How to Decide What Your Agent Does Without Asking." matthewkruczek.ai, 2026.
6. Matthew Kruczek. "Measuring Agentic AI Impact: The Three-Layer Framework." matthewkruczek.ai, 2026.