AI Agent Guardrail Operations Checklist: Permissions, Review, CI, and Kill Switch Design

Guardrails for AI agents refer to the mechanisms of permissions, review, mechanical constraints, and stopping that prevent agents from executing incorrect operations in production environments. It is not uncommon for AI agents that worked normally in a PoC to perform unexpected operations after being deployed to production, leading to incidents. Most causes can be attributed to lax permission design, inadequate review structures, a lack of mechanical verification through CI, and stopping mechanisms for abnormal situations being deprioritized.

This article organizes these issues into a checklist divided into four areas: permission design, human review and approval, CI and testing, and stop design. In particular, permission design and stop design tend to be the starting points of incidents, so we go into detail with specific configuration examples and failure patterns. On the other hand, since CI and testing items are relatively easy to incorporate into existing development processes, we present them in a form that allows checking key points concisely. This content allows operations personnel and engineers considering AI agent adoption to check what is missing from their own designs in this order. We also present this in a form that can be directly incorporated into the implementation of Harness Engineering.

Whether guardrails are designed as a "structure" rather than as "instructions" changes how prone the system is to incidents. Simply writing cautionary notes in a prompt cannot prevent problems if the agent ignores or misreads the instructions. The true role of guardrails is to constrain the agent through mechanisms such as permission restrictions, approval flows, and mechanical detection.

Cautionary notes in prompts and guardrails are often confused, but their roles are fundamentally different. While minor operational mistakes can sometimes be suppressed with cautionary notes in prompts, operations involving permissions or monetary amounts require mechanical constraints. Even if you write "Do not delete this file" in a system prompt, that caution note becomes ineffective if the agent forgets the instruction amid a long conversation history, or prioritizes instructions mixed in through indirect input. In cases like prompt injection, where unintended instructions are introduced via external data, relying on cautionary notes can actually become a vulnerability.

Guardrails, on the other hand, are mechanisms that function regardless of the agent's intent or interpretation. Examples include not issuing an API key with deletion permissions in the first place, excluding writable tools from the whitelist, and having the system branch to prevent operations from being executed without approval. It is important to first distinguish that cautionary notes are "instructions we hope will be followed," while guardrails are "constraints that cannot be broken."

This distinction overlaps with discussions of Excessive Agency in AI agents. An agent with overly broad permissions retains room to deviate in the face of unexpected inputs or complex tasks, no matter how carefully the cautionary notes are written. In production operations, it is essential to distinguish between suppression through instructions and constraints through structure, and designing with priority given to the latter is the starting point for preventing incidents.

Guardrails function not as standalone rules, but as part of the design philosophy known as Harness Engineering. Harness Engineering is a design approach that structurally constrains the environment in which an agent operates, localizing damage even if mistakes occur. Minimizing permissions, inserting review steps, verification through CI, and emergency stop mechanisms are all components that make up this harness.

A design that treats guardrails as something that can be substituted by the agent's "performance" or "intelligence" will not hold up in production operations for long. In reality, as model accuracy improves, the surrounding structural constraints become even more effective at preventing incidents. Even excellent models cannot avoid unexpected inputs or chains of misjudgment, so designs that rely on improving the model alone remain fragile.

The four areas of this checklist are not independent countermeasures; they become easier to implement when read with the premise of incorporating them into the overall Harness design. Permission design serves as the entry point of the harness, review as an intermediate checkpoint, CI as pre-verification, and emergency stop as the final line of defense. The overall design picture is explained in What Is Harness Engineering? A Design Approach to Structurally Prevent AI Agent Mistakes.

How far should the permissions granted to an AI agent be restricted? The basic principle is that of least privilege, allowing only the operations necessary for the task at hand. By clearly separating read-only and write permissions, and limiting the target tools and connection destinations in advance, the scope of damage from unexpected operations can be narrowed.

The tools an agent can call should be managed using an allowlist approach. A default-deny policy, in which only the tools required for the task are individually registered, results in fewer gaps than a denylist approach. It is safer to route the addition of any new tool through a review process each time.

API keys and DB connection information should also have permission granularity separated at the tool level. Tools responsible for search or aggregation should be assigned read-only keys, while operations requiring write or delete access should be given keys with separate permissions. A configuration in which a single all-purpose key operates all tools is a pattern to avoid, since if part of the instructions is hijacked via prompt injection, the damage can easily spread across the entire system.

For internal verification environments, prioritizing development speed with a looser whitelist, while in production environments defaulting to read-only keys and temporarily enabling write-capable tools only through an approval flow, is a realistic form of staged control. Defining the granularity of whitelists and keys together with the structural constraint design discussed in What Is Harness Engineering? A Design Approach for Preventing AI Agent Mistakes Through Structure makes management easier.

During development-stage verification, it is unlikely to cause problems even if an AI agent touches the production DB directly, but in production operation, that same configuration can easily become an entry point for incidents—this is the key difference. Since the system will function even without separating connection targets between development and production, this separation tends to get deprioritized, but fully separating connection information by environment significantly reduces the cost of later reworking permission design. It is safer to give the agent a replica of the production DB or a read-only view, and to verify any processing that requires writes in a staging environment before reflecting it in production.

Secrets management should also avoid writing credentials directly into code or system prompts. Storing them in a secrets management service such as SSM (AWS Systems Manager) and retrieving them as temporary credentials at runtime makes it easier to revoke access and audit permissions in the event of a leak. Using a standard method such as AES-256 for encryption at rest is a common configuration.

Combining IP address restrictions and Zero Trust Network Access (ZTNA) mechanisms with DB connection information, so that connections from unexpected networks are blocked outright, makes it easier to minimize damage even if a key is leaked.

Permission design alone cannot prevent the risk of an agent making "an incorrect judgment within the scope it is permitted to act." What matters here is a design decision about where to place human involvement. In the Loop (a human approves before execution), On the Loop (a human can monitor and intervene after execution), and Outside the Loop (the agent operates fully autonomously without human involvement)—whether these three levels of involvement can be assigned according to the severity of the task determines whether the approval process becomes a mere formality. Making everything require human approval is safe but makes operations unworkable; leaving everything to full autonomy is fast but fails to nip incidents in the bud. Below, we look at how to design this line-drawing in each area.

Inserting HITL into every operation leads to approval fatigue. Where to place In the Loop is determined by the "cost of error."

There are mainly two axes of judgment. The first is irreversibility. Operations that cannot be undone afterward—such as deleting a database, executing a payment, or sending an external notification—carry a high cost of error and should be subject to mandatory human approval before execution. Conversely, drafting internal documents or generating read-only analysis results can be easily corrected even if mistaken, so operating under On the Loop with after-the-fact confirmation, without inserting approval, causes little issue.

The other axis is the scope of impact. Processing for a single customer's data and processing that affects thousands of records in a batch job differ exponentially in the cost of error for the latter. It is practical to set up conditional branching such that, even for the same operation, HITL is triggered once the number of affected records exceeds a threshold.

For processes involving monetary amounts, a design that switches whether to insert HITL based on an upper limit amount is also effective. A staged design—where Agentic AI is entrusted with small automated transactions, but switches to In the Loop once a certain amount is exceeded—tends to work well in practice. Combining the three axes of irreversibility, scope of impact, and monetary amount, and preparing in advance a table of which combinations of conditions require human involvement, can prevent inconsistent judgment once operations begin.

How to balance the frequency and quality of approvals is the core of this design.

If the scope of approval is expanded too broadly, reviewers become preoccupied with checking notifications and end up pressing the approval button without scrutinizing the content. This is a state in which the approval mechanism remains only as a formality and has lost its substantive checking function—a typical cause of incidents occurring even when guardrails are in place.

Inserting HITL into every operation does not necessarily mean it is safe. Narrowing the scope of approval to operations with high irreversibility and wide impact, and switching the rest to On the Loop after-the-fact confirmation, allows reviewers' attention to be concentrated where it is needed. Reducing the number of cases also has the effect of securing more confirmation time per case.

As concrete design measures, it is effective to provide a screen that allows batch approval of similar requests together, to reduce the material needed for judgment by displaying summaries of impact scope and differences, and to automatically escalate cases where approval has been pending beyond a certain time. Rotating approvers among multiple people so that the load does not concentrate on a specific person is also effective in preventing oversights caused by approval fatigue.

Combined with mechanical constraints from CI and testing, it becomes possible to reduce the number of situations that require human approval in the first place.

If you rely solely on human review, the risk of decision fatigue remains, but if you incorporate mechanical constraints, you can eliminate the majority of issues before review. Pre-commit hooks and regression detection via evaluation sets are mechanisms that stop dangerous changes before a reviewer even checks them. Combined with a least privilege design, you can further narrow down what requires approval.

There are clear criteria for which rules should be moved to pre-commit hooks. Priority should be given to rules that can be judged mechanically as pass/fail, without requiring human judgment.

Specifically, this includes detection of embedded secrets, direct hardcoding of production DB connection strings, calls to dangerous system commands, and imports of unapproved tools. Since these can be judged as "present/absent," it is more reliable to leave them to detection tools than to have reviewers search for them visually.

On the other hand, context-dependent judgments—such as the validity of business logic or impact on user experience—should not be moved to pre-commit hooks. If too many rules are added to hooks, developers will start looking for workarounds, and the guardrails will ultimately become hollow. Since tests run across the entire CI and rules stopped at pre-commit serve different roles, it is practical to keep only lightweight checks that require fast execution in the hooks, while placing time-consuming evaluation set runs on the CI pipeline side.

After an agent makes a code change, how much do you rely on manual verification to confirm it works correctly?

Pre-commit hooks prevent mechanical rule violations, but they cannot detect whether an agent's behavior has "degraded compared to before." This is where evaluation sets become necessary. Just as human interviews use the same list of questions each time to compare candidates, you should prepare a fixed set of input-and-expected-output pairs for the agent as well, and score it against the same criteria with every change.

Evaluation sets should include not only frequently occurring tasks but also input patterns that caused problems in the past. This allows continuous confirmation that fixed bugs have not recurred, improving the accuracy of regression detection. For the scoring method, rather than requiring an exact match of the output, a practical approach is to score whether the expected elements are included.

By automatically running this evaluation set in CI and designing the system to block merges when the score falls below a threshold, you can mechanically stop performance degradation that is difficult to notice through visual review alone. The management of evaluation sets also connects to the design philosophy covered in What is Harness Engineering? A Design Method to Structurally Prevent AI Agent Mistakes.

Decision axis: Where do you stop unexpected runaway behavior?

Even if you reduce the probability of incidents through permission design and review, you cannot bring it to zero. As a last line of defense, you need a mechanism that halts processing when abnormal behavior or a sudden cost spike is detected. We will examine the design of trigger conditions and the setting of upper limits on token/API costs separately.

Design the conditions separately: stop based on a call count limit when the same tool call occurs repeatedly in a short period, and stop based on an error rate when unexpected error responses continue.

Specifically, it becomes easier to operate if you maintain circuit breaker trigger conditions across three categories: "frequency," "anomaly detection," and "boundary violation." The frequency category is designed as "the same tool called consecutively beyond a certain number of times," the anomaly detection category as "failures or errors exceeding a certain rate among the most recent N responses," and the boundary violation category is designed to immediately stop upon detecting "an attempt to access a resource outside the whitelist" or "an unexpected write/delete operation."

The boundary violation category is particularly high priority. If a write-capable API is called during a task that was supposed to be read-only, or if a connection to a production DB is detected, it is safer to uniformly stop regardless of cost or frequency. Conversely, for the frequency and anomaly detection categories, setting thresholds too strictly will stop even normal retries, so room must be left to adjust thresholds according to the nature of the task.

How to handle the situation after a stop is also something that should be decided in advance. Whether to resume automatically or resume only after human approval is a judgment where, the higher the cost of an error for a given task, the more appropriate it is to choose the latter.

Setting upper limits on tokens and API costs plays the role of stopping financial damage before anomaly detection even occurs. Even when everything appears to be running normally without errors, there are cases where token consumption alone accumulates without limit due to near-infinite-loop retries or bloating context windows. Have you ever experienced learning only after the fact, from a billing statement, that "before I knew it, monthly API usage costs had ballooned to several times the expected amount"?

Limits become easier to operate when maintained across three tiers: "task level," "session level," and "daily/monthly." At the task level, you set an upper limit on the number of tokens usable in a single execution and force termination when exceeded. At the session level, you set upper limits on the number of consecutive calls or API calls by the same agent. At the daily/monthly level, the design monitors the organization-wide cost ceiling and issues alerts.

A task-level limit alone cannot prevent cases where an agent generates numerous tasks in a short period. Conversely, a daily limit alone allows a single runaway task to keep consuming wasteful costs until it hits that ceiling. Combining both allows single-incident accidents and cumulative accidents to be detected separately. For detailed implementation of token consumption visualization and limit design, see What Is a Token Trap? Practical Consumption Management to Prevent Hidden Cost Explosions in AI Agents.

Why do accidents still occur even after permission design, review, CI, and stop mechanisms have each been put in place individually?

In many cases, the cause lies not in the mechanisms themselves but in the "seams" between them. Even if a read-only key is configured for the production DB, if the tool called with that key permits shell command execution, the permission design becomes merely a formality. Least privilege must be reconsidered not by judging individual keys or tools in isolation, but by examining the entire combination of operations the agent can reach.

Accidents occur even with an approval flow in place when approval fatigue has progressed to the point where reviewers routinely press the approve button without scrutinizing the content. In the early stages of HITL adoption, checks tend to be conducted carefully, but as the volume increases, checks tend to become perfunctory.

Even if Lint and unit tests pass in CI, if the evaluation set remains outdated, behavioral changes following a model update cannot be detected. Whether tests "exist" and whether they can "detect current behavior" are separate issues.

Even with circuit breakers or cost ceilings, if thresholds are set only at the task level, the combined cost of multiple tasks executed in parallel falls outside the scope of monitoring. Even when individual guardrails are functioning, accidents occurring outside their boundaries cannot be prevented. During design, it is essential to explicitly define the scope of application for each mechanism and check the combination for gaps.

Q1. To what extent should logs be retained? In addition to the tool names called by the agent, input parameters, output results, and execution times, the approver's judgment content should also be logged when an approver is involved. Even if you retain only diffs of prompts or system prompts, you cannot trace the cause of an accident without knowing which tool was actually executed under which permissions.

Q2. How often should permission audits be conducted? Immediately after operations begin, changes are frequent, so monthly reviews are advisable; even after operations stabilize, conducting an audit once a quarter is a realistic practice. It is not uncommon for tools added to an agent or temporarily granted permissions to remain in place past their intended expiration.

Q3. Should audit logs and application logs be managed separately? Audit logs, such as permission changes and approval history, should ideally be retained in a tamper-resistant format separate from ordinary application logs used for debugging. Mixing the two in the same location tends to complicate the task of extracting only the information needed during an audit, reducing review accuracy.

Q4. What are the typical problems commonly found during audits? Typical cases include test tokens that should have been decommissioned still remaining active, or multiple agents sharing the same administrator-privilege key. These accumulate as a result of permissions that were appropriate at the time of initial design being gradually loosened with each feature addition. Combining log recording and permission management implementation with the visualization mechanism covered in Converged Management of LLM Inference Logs can help reduce the burden of audit work.

Q5. Who should be responsible for conducting audits? If left solely to the development team, permission reviews tend to be deprioritized in favor of feature additions, so a structure involving multiple perspectives—including operations staff and security staff—is preferable. Clearly designating a person responsible for permission design is a prerequisite for sustaining audits over time.

Q1. What is the difference between guardrails and Harness Engineering? Guardrails refer to individual constraint rules such as permissions, review, CI, and stop mechanisms, while Harness Engineering is a design approach that embeds those rules into the agent's entire execution environment to structurally prevent accidents. It is easier to organize your thinking by considering guardrails as components and Harness as the design philosophy that combines those components. Detailed design methods are explained in What Is Harness Engineering? A Design Approach to Structurally Prevent AI Agent Mistakes.

Q2. Should all guardrails be introduced from the small-scale PoC stage? At the PoC (proof of concept) stage, a realistic approach is a phased introduction in which only permission design and emergency stop mechanisms are put in place first, with rigorous CI/test operations and approval flows strengthened at the time of production migration. Attempting to perfect everything from the outset tends to slow down validation speed and delay the evaluation of AI ROI (AI return on investment) itself.

Q3. Can accidents be prevented by increasing human review? Reports indicate that increasing review volume leads to approval fatigue, which conversely increases erroneous approvals. A design that raises review quality is more effective—for example, automatically executing operations with low error costs while routing only high-cost operations through HITL (Human-in-the-Loop).

Q4. How should circuit breaker activation conditions be determined? It is common to set conditions by combining multiple indicators, such as cost ceilings, sudden spikes in error rates, and abnormal repetition of the same operation. Relying on a single threshold alone can cause erroneous stoppages even during normal high-load periods, so AND/OR designs combining multiple conditions should be considered.

Q5. Once this checklist is put in place, can it be operated indefinitely without change? Because permission and tool configurations change during operation, the checklist requires periodic review. In particular, neglecting log, audit, and permission reviews risks having the original guardrail design become a mere formality, making it impossible to trace the cause of accidents.

The four domains—permission design, human review, CI, and emergency stop—only exert real accident-prevention power once they are embedded into an execution environment called a Harness, rather than functioning independently. The key point is not to create a checklist once and leave it there, but to translate it into pre-commit hooks, CI pipelines, and permission management systems, creating a state in which it is automatically applied every time the agent runs.

The least-privilege whitelist should be placed at the entry point of tool calls, the approval flow immediately before irreversible operations, the evaluation set in CI before deployment, and the circuit breaker in the runtime monitoring layer. These are constraints that operate at different timings, and strengthening any single one still leaves other loopholes remaining.

Once operations begin, logs and permissions must be audited periodically to confirm that checklist items are actually functioning. It is also reassuring to design, as part of the Harness itself, a mechanism for reviewing whether rules have become mere formalities. The overall design picture can be reviewed in What Is Harness Engineering? A Design Approach to Structurally Prevent AI Agent Mistakes.