“Agent” has become the industry’s most elastic noun - stretched over everything from a chatbot with a plugin to software that runs a workflow for hours unattended. For coverage to be checkable, the word needs edges.
Here is the working definition this desk applies: an agent is a system that pursues a goal over multiple steps, choosing its own actions - calling tools, reading results, deciding what to do next - in a loop, without a human approving each move. Three ingredients matter: a model that decides, tools that act on the world, and a loop that feeds outcomes back in. Remove the loop and you have a feature. Remove the tools and you have a conversation.
Why the loop changes everything
The definition matters because the loop transforms the reliability question. A chatbot’s error is one bad answer; an agent’s error compounds, because step twelve builds on step eleven. The arithmetic is unforgiving:
| Steps in the task | At 95% per step | At 99% per step |
|---|---|---|
| 1 | 95% | 99% |
| 5 | 77% | 95% |
| 10 | 60% | 90% |
| 20 | 36% | 82% |
| 50 | 8% | 61% |
A system that feels impressively reliable in a demo - nineteen times out of twenty per action - completes a twenty-step task barely a third of the time. This single table explains most of the gap between agent demos and agent deployments, and why serious evaluations report task-level success over many trials rather than per-response quality.
How to read an agent claim
The checkable questions follow directly. How many steps does the benchmark task actually involve? What counts as success - full completion, or partial credit? How many attempts does the agent get, and is a human allowed to intervene? And what is the cost per completed task, since loops that retry and reflect burn tokens in multiples? An agent claim with those four answers attached is evidence. Without them, it is a demo with a definition problem.
Guardrails that actually move the number
The compounding table also explains which safety measures matter, because anything that catches an error mid-loop changes the exponent rather than the base. Verification layers - checking a tool’s result against an independent source before proceeding - convert silent failures into retries. Constrained tools (a sandboxed browser, an allow-listed API, a database the agent can read but not drop) cap the blast radius of the errors that slip through. Human gates at irreversible steps - payments, sends, deletes - concentrate scarce attention where mistakes are unrecoverable. And honest task decomposition keeps chains short, since two ten-step tasks with a checkpoint between them succeed far more often than one twenty-step run. None of this is exotic; it is the same discipline distributed-systems engineers apply to unreliable components, arriving in a field that briefly forgot software fails.
The other number: cost per completed task
Reliability’s twin is economics. Loops that reflect, retry and verify burn tokens in multiples of a single response, and a failed twenty-step run costs everything and delivers nothing - so the honest unit is cost per completed task, failures amortised in. It is routinely several times the naive per-call estimate, occasionally ruinous, and almost never in the launch post. An agent product quoted in price-per-request is being priced like a chatbot; ask for price-per-outcome and watch the conversation change.
The benchmark landscape, honestly
Agent evaluation is younger and rougher than model evaluation, and knowing its failure modes is part of reading it. Public agent benchmarks tend to reward environments that are easy to score - code that compiles, web forms that submit - which over-weights tidy digital tasks and under-weights the ambiguous, judgement-laden work agents are sold for. Success criteria vary wildly: some suites grant partial credit that flatters incomplete runs, others count a task solved if any of several attempts lands, quietly importing a best-of-N settings game. And because agents act on live tools and websites, results decay as the world changes, so an eight-month-old score describes a vanished environment. None of this makes the numbers useless - it makes them evidence with a short half-life and generous error bars, exactly the reading posture the rest of this piece recommends.
“Agent” is doing the same work “smart” did a decade ago: converting a spectrum into a sticker.
THE FOUR-QUESTION AUDIT
When a product says agent, ask: what actions can it take without a human click; over how many steps does it keep state; what happens when a step fails; and what is the blast radius of its worst credentialed mistake. The answers place it on the spectrum. The marketing never does.
THE DESK’S BOTTOM LINE
Autonomy is a dial, not a badge. This desk describes systems by what they are permitted to do unattended - and we’ll keep translating “agentic” into that sentence until the industry does it unprompted.
