Emergency Stop Design for AI Agents — A Circuit Breaker Implementation Guide

A circuit breaker for AI agents is a safety mechanism that detects abnormal behavior, cost overruns, and runaway loops in real time, and automatically halts processing before damage can escalate. This article is aimed at developers and MLOps engineers running LLM-powered AI agents in production, and walks step by step through building a monitoring layer, configuring trigger conditions, implementing kill switches and fallbacks, and integrating with guardrails. By the end, readers will have a concrete implementation strategy for designing "the means to stop" into their own agents from the ground up, keeping runaway costs and security incidents in check.

An AI agent that continuously makes autonomous decisions can call tools far beyond what its designers anticipated, causing costs and risks to grow exponentially. Emergency stop design is a prerequisite for having "the means to stop" on equal footing with "the means to run." This section outlines the three reasons why stop design cannot be deferred: cost, security, and regulation.

In conventional applications, requests and responses correspond one-to-one, making it relatively easy to estimate the upper bound of potential damage. An AI agent, by contrast, can chain a dozen or more tool calls and LLM inference steps from a single instruction. If this chain converges as expected, there is no problem—but if the loop termination condition is misjudged, calls never stop and billing balloons without limit.

The risks fall into three broad layers. The first is cost—token-based billing and pay-per-use external API charges spiral out of control. The second is security—the agent is manipulated via prompt injection into executing operations that were never authorized (sending emails, deleting data, exfiltrating information). The third is reliability—outputs containing hallucinations are passed downstream as definitive information, corrupting business data. These three layers are not independent; during a runaway event they compound and surface simultaneously. That is precisely why a shared mechanism to "detect and stop" is needed, rather than isolated countermeasures.

The risks specific to AI agents are also explicitly categorized in OWASP's threat categories for LLM applications. The two most representative are Unbounded Consumption and Excessive Agency.

Unbounded Consumption refers to a state in which there is no upper limit on inputs or recursive calls, causing computational resources, costs, and tokens to be consumed without bound. Infinite loops and denial-of-service (DoS) attacks in which an adversary deliberately submits heavy tasks fall into this category.

Excessive Agency refers to a state in which the permissions, tools, and autonomy granted to an agent are excessive, causing the blast radius of a malfunction to grow too large. Typical examples include designs where "write permissions are granted to a process that only requires read access" or "irreversible operations can be executed without human approval." Circuit breakers primarily address the former, while minimizing permissions (least-privilege design) primarily addresses the latter. The two are complementary—neither alone is sufficient.

Preventing runaway behavior is both a technical challenge and a compliance requirement. The EU AI Act, which is already fully in force, requires that high-risk AI systems be equipped with the means for humans to intervene during operation and, when necessary, stop the system (human oversight). The very fact that a control equivalent to a stop button is built into the design becomes a point of conformity assessment.

In other words, a kill switch is no longer merely "reassuring to have"—in regulated domains, it is becoming a component whose absence means requirements are not met. The full picture, including internal rules and audit readiness, is covered in the AI Governance Practical Guide, but from a stop-control perspective, at a minimum the following three points should be explicitly documented at design time: (1) who can stop the system and under what conditions; (2) whether stop operations and their reasons are recorded in logs; and (3) whether the system can return to a safe state after being stopped.

A circuit breaker cannot function on its own. It only works when three prerequisites are in place: an observability foundation that detects "what is happening," a policy defining "who intervenes and to what extent," and a threshold definition specifying "at what value to stop." Before moving into implementation, confirm these three prerequisites.

A stopping mechanism can only operate within the scope of what is observable. Token consumption and loop counts cannot trigger anything if they are not being measured. Therefore, the first prerequisite is whether an AI observability foundation is in place to make the agent's internal state visible.

Specifically, the inputs, outputs, tokens used, tool calls, latency, and errors for each step must be recorded in chronological order at the trace level. Applying a distributed tracing approach—such as OpenTelemetry—to LLM calls, and structuring data as "1 task = 1 trace" and "1 step = 1 span," allows threshold evaluation in later stages to be written directly against that structure. The design of the observability foundation itself is covered in the Practical Guide to AI Observability; the procedures in this article proceed on the assumption that measured metrics are already available. If that foundation is not yet in place, the correct order is to start with observability first.

The next decision is how to involve humans in the stop determination. It is easier to organize the degree of intervention by thinking in terms of three broad levels.

In-the-loop is an approach that requires human approval before every significant operation. It is suited to irreversible, high-risk processes, but reduces throughput. On-the-loop is an approach where the agent executes autonomously while a human monitors the situation via a dashboard and intervenes or stops it when necessary. Out-of-the-loop is fully autonomous, with stopping delegated entirely to mechanical triggers.

In practice, it is realistic to vary the level by tool, based on the irreversibility and risk of each operation—for example, "search is autonomous, external transmission requires approval, and payment requires dual approval." The fundamentals of intervention design are covered in detail in the Human-in-the-Loop (HITL) explainer. The circuit breaker should have a clearly defined escalation target upon stopping, so as not to conflict with this intervention level.

The final prerequisite is selecting the metrics that determine when to stop. Choose representative threshold metrics based on how easily they can be observed and how unlikely they are to produce false positives.

The key is not to rely on a single metric, but to combine multiple ones. Token count alone misses "cheap but endless loops," while step count alone overlooks "few but expensive tool calls." Rather than aiming for perfect thresholds from the start, set them on the conservative side initially, with the expectation of adjusting them based on the production metrics distribution (p50 / p95). For cost-related upper limits, it is worth considering them in conjunction with the budget design in the LLM Cost Optimization Guide.

The essence of a circuit breaker is a "measure → evaluate → cut off" loop. The first step is building a monitoring layer that collects metrics in real time to serve as the basis for evaluation. Here, three streams of data are measured: consumption, progress, and anomaly signals.

The foundation of the monitoring layer is a counter that accumulates consumption with each step the agent takes. By wrapping the LLM client and tool execution, tokens, cost, call count, and latency are added to a per-task context on every invocation.

The key point is to separate measurement from the agent's core logic and confine it to the wrapper layer. This way, breaker conditions can be changed later without touching the core. Latency is also recorded as a metric for early detection of signs that cascading delays from external APIs may cause the entire system to hang.

Alongside consumption, tracking whether the agent is "making progress" is equally important. Infinite loops most commonly manifest as repeated calls to the same tool with the same arguments, or as oscillation between the same states.

There are two fundamentals for detection. First, a step limit — set a maximum number of steps per task and force termination if exceeded. Second, repetition detection — maintain a hash of tool names and arguments for the last N steps, and treat it as a loop if the same pattern exceeds a threshold count.

In a multi-agent architecture that separates planning from execution, the node transitions in the task graph themselves can be made the subject of monitoring. See the multi-agent AI overview for design patterns. A "stuck" state where progress has not been updated for a certain period of time is also caught separately via a timeout.

In addition to volume and progress, "quality anomalies" in output should also be collected as signals. These serve as material for triggering guardrails or human intervention downstream.

For signs of prompt injection, monitor for the appearance of phrases that attempt to override system instructions (e.g., "ignore previous instructions"), sudden calls to unauthorized tools, and patterns of external URLs or credential leakage in the output. For signs of hallucination, watch for references to facts not present in tool results, self-reported confidence levels that are unnaturally high or low, and inconsistent answers to the same question.

It is important to note that what is collected here are merely "signals" and should not individually serve as definitive grounds for stopping execution. Too many false positives will block legitimate tasks. Accuracy in detection logic improves when you have a solid understanding of attacker techniques — AI red teaming is a good resource for familiarizing yourself with typical attack patterns.

This step converts the metrics collected through monitoring into a determination of whether to stop or continue. Just like a circuit breaker in microservices, triggers are defined across three categories: cost, error rate, and timeout. We will examine the threshold design for each.

The first trigger is budget-based. Set upper limits on tokens, cost, and call count per task, and design the behavior upon exceeding them as a package. Rather than applying a single uniform limit, having at least three tiers makes operations more manageable.

Appropriate values vary by task type. Tasks that converge easily, such as extraction or formatting, tend to be inexpensive, while research or long-horizon planning tasks have unpredictable iteration counts. Categorizing tasks — for example, into "exploratory / extraction / generative" — and setting separate budgets for each avoids the situation where applying the same limit to all tasks results in 90% being too lenient and 10% being too strict. The broader framing of budgets within overall cost design is covered in detail in AI Agent Economic Models.

The second is an error rate-based trigger, which most closely resembles the original "circuit breaker." It follows the microservices pattern and is managed with three states.

• Closed: Normal operation. Calls are allowed through, but failures are counted.
• Open: When failures exceed the threshold, the circuit opens and all calls are immediately rejected for a fixed period (fail-fast).
• Half-Open: After a cooldown period, a limited number of calls are allowed through as a test; if recovery is confirmed, the state returns to Closed.

This prevents the waste of cost and time caused by endlessly retrying calls that are known to fail when an external API or tool is down. Holding the threshold as a rate such as "M failures out of the last N calls" helps avoid the circuit opening excessively due to infrequent, isolated errors.

The third is time-based. It catches cases where the overall process stalls due to waiting on external responses or hanging tools, even when cost and error counts are within their thresholds. Timeouts should be managed in layers.

At the step level, an upper time limit is set for each individual LLM call and tool execution; if exceeded, that call is aborted and routed to a retry or fallback. At the session level, an upper time limit is set for the total elapsed time of a single task; if exceeded, execution is safely interrupted even if incomplete. Using both together is necessary to catch cases like "each step is fast but the loop never terminates" or "one step freezes and blocks the entire process."

In implementation, this is combined with cancellation mechanisms for asynchronous processing (timeout-aware await, cancellation tokens) to ensure that resources (connections, temporary files, locks) are reliably released upon interruption. An incomplete interruption creates "zombie tasks" where processing continues running in the background even though it was supposed to have been stopped.

This step is about designing what to do after a trigger fires — how to stop execution, and what to return after stopping. There are varying degrees of stopping, and the appropriate method should be chosen based on operational impact. This is further layered in combination with guardrails and multi-agent isolation. Let's look at the key implementation considerations.

There are two types of stops. A hard stop is a kill switch that immediately aborts processing. It is used when continuing is more dangerous, such as in cases of runaway costs or a clear security breach. A soft stop is an approach that either completes the current step before halting at a safe boundary, or continues with degraded functionality. It is suited for cases where user experience needs to be preserved.

Having both a global kill switch — one that takes effect across all agents and all tenants simultaneously — and a scoped one limited to a specific task or user allows the blast radius to be minimized during an incident. Upon stopping, both a message to return to the user and an internal log entry must always be generated.

While a circuit breaker is a reactive mechanism that "stops a process that has already started running," a guardrail is a proactive mechanism that "blocks dangerous inputs and outputs before they pass through." Because they operate at different layers, combining them creates a multi-layered defense.

On the input side, a prompt firewall detects injections and prohibited topics, blocking them before they reach the agent. On the output side, content is passed through filters for personal information, credentials, and harmful expressions before being sent downstream. Events caught here are also reflected in the breaker's counters as anomaly signals collected at the preceding stage, enabling coordination such as "sessions that repeatedly trigger guardrails are blocked entirely."

A comprehensive overview of guardrail design and implementation is compiled in the AI Guardrails Implementation Guide. The stopping mechanisms described in this article are best understood as a safety net that serves as the last line of defense, catching "runaway execution in progress" that has slipped past the guardrails.

In systems where multiple agents collaborate, the granularity of shutdown becomes critical. Taking down the entire system because a single sub-agent went rogue would severely compromise availability.

The foundational design principle is isolation (bulkhead). Assign each sub-agent its own independent budget, circuit breaker, and execution context, so that when one opens, the others continue operating. The orchestrator detaches tasks from the halted node and either reroutes them to an alternative path or escalates to a human.

When agents share tools or state, a single runaway agent can trigger a cascade that propagates across the entire system. Apply individual limits and access controls to all shared resources. For details on collaborative design, refer to Multi-Agent AI. Whether partial shutdown is effective is determined at the initial architecture selection stage.

An emergency stop is not something you can simply add and be done with. Misconfigured thresholds or poorly designed operations will either halt normal business processes or fail to work when it matters most. Below are two failures commonly seen in practice, along with ways to avoid them.

The most frequent failure is erring too far on the side of caution, resulting in frequent false positives. Setting a token limit right at the average means that normal but slightly heavy tasks get interrupted one after another, making the feature look like an "unreliable thing that keeps stopping" from the user's perspective.

There are three ways to avoid this. First, base thresholds on the distribution of production metrics — use p95 or p99 rather than p50, and place the threshold where the majority of normal tasks fall within bounds. Second, rather than jumping straight to a hard stop, first insert a soft limit with warnings and model degradation, applying the brake gradually. Third, run the circuit breaker initially in observation mode (shadow mode), logging only what would have been stopped — how many times and which tasks — had it been active. Reviewing the false-positive rate during this dry run before enabling actual blocking significantly reduces incidents after production deployment. Treat thresholds not as fixed values, but as something to continuously adjust through ongoing operations.

Another common failure is an implementation so focused on stopping that it neglects to record why the stop occurred. The system halts but leaves no logs, so when a user reports "it stopped midway," there is no way to trace which trigger fired at what value. This makes it impossible to adjust thresholds or prevent recurrence.

At a minimum, each stop event should be preserved as a structured log containing: (1) trigger type (cost / error rate / timeout / guardrail), (2) the measured value and threshold at the time of firing, (3) the relevant task, user, and trace ID, (4) stop type (hard / soft) and subsequent behavior, and (5) a timestamp. With these in place, you can retrospectively validate whether a stop was justified and adjust thresholds with a sound basis.

In addition, continuously monitor stop frequency and false-positive rates on a dashboard, and set alerts for sudden spikes. Viewing the stop mechanism itself as a subject of monitoring also contributes to building the audit trail for "explainable human intervention" required by the EU AI Act. Stop logs serve as primary data for driving improvement loops across both cost and incident dimensions.

A circuit breaker for AI agents only functions when the full cycle of "measure → judge → block → record" is embedded as a design-phase prerequisite rather than a post-deployment add-on. The steps covered in this article can be summarized as follows:

• Align prerequisites: Define the observability foundation, intervention levels, and threshold metrics upfront.
• Monitoring layer: Measure consumption, progress, and anomaly signals across three streams via a wrapper layer.
• Triggers: Apply multi-layered judgment across three streams — cost, error rate, and timeout.
• Kill switch: Differentiate between hard and soft stops, and add depth through guardrails and multi-agent isolation.
• Operations: Use shadow mode to identify false positives, and continuously adjust thresholds based on stop logs.

A mechanism for stopping is an investment that simultaneously supports three goals: containing runaway costs, containing security incidents, and achieving regulatory compliance. In our production deployment support for AI agents, we recommend incorporating this stop design as a mandatory requirement in the initial architecture. Please feel free to contact us to discuss agent adoption or the specifics of emergency stop design.