An agent that only knows how to continue is not autonomous. It is a running task looking for someone to pull the plug.

TL;DR

Teaching an agent to execute without defining its stopping condition delegates risk, not work. A useful system needs to know what result completes the task, how much time, cost, and scope it may consume, which actions need human authority, and what to do when evidence is insufficient or the same error repeats. Stopping is not failure: it can mean completing, requesting a decision, refusing an action outside its authority, or returning to a safe state. Responsible autonomy is not measured by how many steps a machine took alone. It is measured by how much it can do without crossing a boundary that should remain human.

There is a slightly childish habit of judging an agent by how willing it looks.

It researches. Opens tabs. Calls tools. Creates files. Tries again. Calls another tool. When attempts fail without producing new evidence or changing the conditions of the problem, it does what many systems do when nobody taught them otherwise: it keeps executing.

In a demo, that looks like autonomy.

In operations, it can be an expensive way of not knowing how to finish.

An agent needs more than a list of things it can do. It needs a stopping contract: the conditions under which it must complete, pause, request a decision, refuse an action, or step back.

Without that, “be proactive” becomes the technical equivalent of handing someone a power drill and leaving the room before explaining where the pipes run.

Execution is only half the verb

A deterministic automation usually has a well-behaved ending: it receives an input, applies a rule, and produces an output. If a condition does not hold, it fails and reports the error — or it should.

An agent has more room to choose its next step. It can research, compare sources, query a system, prepare a draft, or coordinate other tasks. That latitude is exactly what makes it useful in less linear work.

It is also what makes “solve this” inadequate.

Solve what, exactly? How far? Using which sources? Within what time and cost? What counts as an acceptable answer? Who decides when an exception exceeds the system’s authority?

The NIST AI Risk Management Framework treats risk as dependent on the context of use and notes that risk tolerance is not prescribed by the document: it must be defined by those accountable for the system. It is a voluntary framework, not a ready-made recipe for agents. Still, it puts the question in the right place: the boundary does not come from the model. It comes from consequences an operation does or does not accept.

A good stopping condition turns those consequences into observable behavior.

Four correct ways to stop

Stopping does not mean every task ends with a red error and someone losing an afternoon. A well-designed agent can reach four legitimate endings:

1. Complete: it produced a result that satisfies the pre-defined acceptance criteria. Acceptance criteria are verifiable conditions that separate “looks ready” from “is ready for this use.”
2. Escalate: it encountered a decision that requires someone with authority, business context, or accountability the system does not have.
3. Abstain: there is not enough evidence for a reliable recommendation, sources conflict, or the task asks for a claim it cannot support.
4. Step back: a failure, unproductive repetition, or excessive risk occurred; the system stops trying and preserves or restores a safe state when possible.

The problem is not when an agent reaches the second, third, or fourth ending.

The problem is hiding them because a sales presentation prefers to pretend every route ends in “done.”

A limit is not one giant red button

When people talk about limiting an agent, they often picture a single emergency brake. It is necessary in some contexts, but it cannot solve the whole problem alone.

Healthy autonomy usually has layered limits:

Limit	Question it answers	Practical example
Time	How long may a task run before it loses value?	Stop research after 20 minutes without a sufficient source and return the gap.
Cost	How much resource may this benefit consume?	End a query chain before turning a simple answer into a small financial audit.
Scope	Which systems, data, and actions belong to this task?	Query the customer relationship management system (CRM) to draft a reply, but do not alter a record or send a message.
Authority	What may the system do alone, and what may it only recommend?	Prepare a payment; never approve or execute it without the accountable person.
Repetition	When do identical attempts stop being persistence and become a repetition cycle with no progress (loop)?	Stop after two failures from the same integration and open an incident with context.
Irreversibility	Which action cannot be safely undone?	Require approval before deleting data, revoking access, publishing, or communicating externally.

This is not bureaucracy placed on top of AI to ruin the fun.

It reduces the risk of sending the wrong email to five thousand people when an ambiguous instruction such as “clean the database” is executed without review.

Escalation is not handing every decision to a tired human

There is an opposite extreme too: an “agent” that asks for approval on every click and calls that safety. If someone must confirm every query, classification, and sentence in a draft, they did not gain capacity. They turned every low-impact action into an approval bottleneck.

The point is to draw the right boundary.

A low-impact task can finish automatically when inputs are complete, the source is defined, and criteria are clear. A task with financial, legal, reputational, or human impact needs additional review steps. Between those poles is a wide field of recommendations, drafts, simulations, and reversible actions that can remove work without pretending accountability evaporated.

The OpenAI safety guidance for agents recommends combining layered controls and retaining human oversight for higher-impact actions. That is a vendor’s guidance for its own tools, not a certification that any workflow built with them is safe. But the principle is broader than one platform: human review should sit where it changes the consequence, not serve as decorative stamping at the end.

I prefer this rule:

An agent may advance while consequences remain within the authority it received and evidence remains sufficient. When either stops being true, it returns the decision with context.

Returning it with context matters. “I could not do it” is a notice. “I could not do it because two official sources conflict; I tried these queries, rejected these alternatives, and need you to choose between A and B” is a handoff.

Terminal evidence: the receipt that the work ended

In more complex systems, the final answer should not be a pleasant sentence saying everything was solved. It should leave a terminal record: the minimum evidence that the task ended, how it ended, and what remains unresolved.

For every relevant execution, I want to be able to answer:

• what objective it received;
• which sources, tools, and permissions it used;
• which acceptance criterion was satisfied — or which one was not;
• which action was taken, by whom, and under which authorization;
• which uncertainty, exception, or residual risk remains;
• how to go back when a reversible change was made;
• who needs to decide the next step when the result was escalated.

That does not require recording every intermediate thought, much less dumping sensitive data into an event record (log). A useful record is not machine voyeurism. It is enough traceability to check a decision, correct the workflow, and avoid reconstructing an incident through digital archaeology.

That is the difference between an agent that looks busy and a system that can be operated after the person who configured it goes on vacation.

Irreversible actions deserve an explicit boundary

Some actions can be undone without drama: saving a draft, creating an internal task, producing a simulation, flagging a divergence for review.

Others spread into the world before anyone can apologize: publishing a statement, deleting data, changing permissions, sending a charge, closing an account, or making a commitment on someone else’s behalf.

It is not enough to write “be careful” in the instruction that guides the system (prompt). Care is an admirable human quality; as an operational control, it is conceptual jelly.

The boundary needs to appear in the architecture: separate tools, least-privilege access, a test environment, human confirmation for material impact, and a return path when one exists. In my article on generic agents, I argued that a demo usually omits exceptions, the source of truth, and the decision owner. The stopping condition is where those three omissions finally send the bill.

The agent does not need to become “less intelligent” to accept a boundary.

It only needs to be less irresponsibly loose.

Repeating an error is not perseverance

Another classic sign of a bad contract is a loop that changes clothes with every attempt.

An integration returns an error. The agent tries again. The error returns. It changes a random parameter, tries again, calls another tool, repeats the same operation, then produces a two-hundred-line report explaining that it worked very hard.

Persistence is useful when the next attempt carries new information: credentials were renewed, a dependency returned, an additional source appeared, or someone resolved an ambiguity.

Without a changed condition, repetition is simply automated stubbornness.

That is why the contract needs to declare what counts as a different attempt, which error ends the operation immediately, and when a pattern requires escalation. This connects to what I wrote in “You wrote it, did not read it: AI does not sign for you”: fluent output does not end responsibility. In the same way, a long sequence of actions does not prove progress.

Rollback begins before the error

Rollback is the technical term for returning to a known earlier state when a change fails or produces an unwanted consequence. But it does not appear by magic after the problem.

To reverse something, you need to have preserved state, identifiers, permissions, and a tested path before you need them. That is why a recovery copy (backup), versioning, approval, and action limits belong in the same conversation.

In the article on operational cleanup, the question was when an automation may delete. Here, the question comes earlier: when may an agent change something knowing it might need to undo it?

Both questions have the same uncomfortable answer: not when the model looks confident, but when the operation knows what it protects and how it recovers.

Autonomy that fits a real company

I do not want agents reduced to decorative assistants that ask permission to breathe. I also do not want a loose machine, with too much access and an optimistic instruction to “do whatever it takes.” Between those two characters there is engineering.

An agent can be extremely useful in researching options, consolidating information, finding inconsistencies, preparing decisions, executing reversible actions, and organizing work. In many situations, that frees people for what requires negotiation, human context, accountability, consequence-reading, and direction.

But autonomy is not the number of tasks you abandoned in the hands of a system.

It is the quality of the boundary you managed to design.

At i-9.ai, that starts before choosing a model or a tool: understanding the task, sources, impact, authority, acceptance criteria, and what must happen when reality leaves the script. Architecture comes after that. And only then comes the autonomy a company can actually sustain.

If your company is discussing what an agent could do, start with a less pleasant and much more useful question: in which situation should it stop?

Further reading

• NIST AI Risk Management Framework 1.0: a voluntary framework to map, measure, govern, and manage AI risk; it supports the importance of context, risk tolerance, and responsibility, but does not prescribe one agent design.
• OpenAI — Safety in building agents: technical guidance on guardrails, tool limits, and human oversight in agent workflows; it is vendor documentation, not an independent validation of an implementation.
• NIST — AI RMF Playbook: operational examples associated with the framework; use it as a source of questions, not as a substitute for defining organizational authority and risk.
• Common Weakness Enumeration — CWE-636: reference for the principle of failing securely; applying it to agents in this article is an operational synthesis, not an AI-only rule.

References and limits of use

• NIST supports that risk, impact, and tolerance depend on context and that governance crosses the lifecycle. It does not define the correct cost, time, or approval limit for a specific company.
• OpenAI’s documentation supports layered controls and human intervention for higher-impact actions. It does not guarantee that guardrails eliminate error, misuse, or unexpected consequences.
• CWE-636 supports the general notion that a failure should not move a system into a less safe state. Translating that into “stop and escalate when evidence is missing” is the operational interpretation defended here.

Keep reading

• The best answer is not the one I like most
• Generic agents may be your worst first contact with AI
• If AI will do the work, do not fire the person who knows when it is wrong