An AI agent is a system in which an LLM (Large Language Model) autonomously executes tool calls and multi-step reasoning to automate business processes. In recent years, while success stories at the PoC (Proof of Concept) and pilot stages have been increasing, organizations that stumble when scaling to production continue to emerge.

This article is aimed at engineers, project managers, and IT decision-makers considering the transition of AI agents to production, and systematically explains the practical steps from pilot to full-scale deployment. From five perspectives—quality management, system integration, AI governance, organizational structure, and AI ROI (Return on Investment) measurement—it presents concrete causes of scaling failures and ways to overcome them. By the end, you should have a clearer outline of a roadmap toward production deployment.

Many companies feel promising results at the PoC (Proof of Concept) and pilot stages of AI agents, yet cases of stumbling during the transition to production environments continue to be reported. This "pilot-to-production" gap arises not only from technical issues, but from a complex interplay of quality management, system integration, governance, and organizational readiness.

This section examines why this gap occurs, drawing on data and structural factors. Understanding the barriers to scaling is the first step toward full-scale deployment.

Despite the fact that many companies have embarked on AI agent PoC (Proof of Concept) and pilot initiatives, only a handful of projects ever reach production operation. Industry research repeatedly reports a trend in which the vast majority of AI projects stall at the pilot stage, with only around 10–20% successfully transitioning to production.

This "pilot-to-production" gap is not simply a matter of technical capability. It arises from a complex combination of factors such as the following:

• Lax evaluation criteria: Because pilots verify operation using small volumes of well-prepared data, they cannot withstand the diverse inputs encountered in real-world operations.
• Ambiguous definition of success: KPIs (Key Performance Indicators) stop at "it worked," leaving the connection to business value unclear.
• Scope creep: After a successful pilot, requirements are added one after another, causing development costs and timelines to balloon.
• Organizational unpreparedness: The structures built by the technical team are not met with a readiness on the operational side to adopt them.

What is particularly easy to overlook is the "complexity inherent to production environments." Challenges such as integration with legacy systems, security requirements, and stability under high load—issues that could be sidestepped during the pilot—frequently surface all at once just before the production transition.

Because AI agents have an LLM (Large Language Model) at their core, the non-deterministic nature of their outputs makes quality management across the entire system difficult. Hallucinations that were tolerable at the PoC stage translate directly into operational risks in production.

To prevent a pilot from ending as a mere "experiment," a design philosophy that anticipates the production transition from the very outset is indispensable.

Even when a pilot succeeds, there are common patterns in the cases that stumble during the production transition. Below are five factors that are repeatedly reported in the field.

① Lack of a quality management framework While manual checks are feasible at the PoC stage, as processing volume increases, hallucinations and erroneous outputs tend to go undetected. Without guardrails and a monitoring infrastructure, quality tends to degrade rapidly.

② Underestimation of legacy system integration costs API integration with ERPs and core databases often requires several times more effort than anticipated. Mismatches in authentication methods and data formats are discovered after the fact, causing projects to stall.

③ Absence of governance and approval workflows Proceeding to production without defining who gives final approval for agent outputs and at what risk level HITL (Human-in-the-Loop) should be introduced leads to confusion when incidents occur.

④ Insufficient AI literacy within the organization When frontline staff do not understand the limitations of agents, it results in judgment errors caused by over-reliance, or conversely, avoidance of use due to excessive distrust. Investment in AI literacy education tends to be deprioritized.

⑤ Vague metrics for measuring AI ROI When KPIs remain loosely defined as "some kind of efficiency gain," it becomes impossible to verify return on investment, making it difficult to justify continued funding. Designing quantitative success metrics from the pilot stage onward is a prerequisite for scaling.

These five factors are not independent problems—they compound one another, deepening the severity of failure. The next chapter takes a closer look at concrete measures for addressing the first barrier: quality management.

Quality issues that tend to be overlooked at the pilot stage surface all at once in production environments. When increased user volume, greater diversity in input patterns, and instability in external APIs converge, cases of defects that could not be reproduced in development environments occurring in rapid succession have been reported.

Quality management for AI agents requires a different approach from conventional software testing. Given that outputs are probabilistic, the essence of the challenge is not to aim for "zero bugs," but to design a framework that continuously maintains quality within an acceptable range.

The following sections explain concrete implementation approaches from two perspectives: output monitoring and guardrail design, and reliability assurance through HITL (Human-in-the-Loop).

When an AI agent repeatedly produces incorrect outputs in a production environment, trust in the system can collapse rapidly. This is why the design must incorporate monitoring and guardrails at both stages — before and after output is generated — rather than simply having a human review the output after the fact.

Key Monitoring Metrics

• Hallucination rate: Track the frequency with which responses deviate from facts through periodic sampling evaluations
• Tool call success rate: Monitor the ratio of failures and timeouts in tool calls (Function Calling)
• Latency and token consumption: Detect early signs of context window bloat
• User feedback rate: Set the proportion of negative reactions (rejections and correction requests) as a KPI

The baseline configuration involves aggregating these metrics into an AI observability platform and visualizing them in real time on a dashboard.

Two-Layer Guardrail Design

Guardrails should be implemented at two layers: the system prompt level and the output level.

1. Input guardrails: Embed rules into the system prompt to detect prompt injection attacks, and restrict references to prohibited topics and confidential information
2. Output guardrails: Inspect generated text using regular expressions and classification models, and automatically flag policy violations and personal information leakage patterns

The critical point is not to leave guardrails as "static rules" and forget about them. New edge cases will continuously emerge after the system goes live, so an operational review cycle — weekly or monthly — must be built into the design plan from the outset. Combined with the HITL approach covered in the next section, this creates a dual defense of automated detection and human judgment.

HITL (Human-in-the-Loop) is a design methodology that incorporates human review and approval steps into an AI agent's processing flow. When combined with output quality monitoring, it allows human judgment to compensate for risks that guardrails alone cannot prevent.

Designing Intervention Levels Across Three Tiers

HITL has three modes depending on the depth of intervention.

• In the Loop: A human reviews all outputs before execution. Suitable for the initial phase where accuracy is the top priority
• On the Loop: The AI operates autonomously, with escalation to a human only when anomalies are detected
• Outside the Loop: Fully automated by default, with quality assured through post-hoc auditing

A practical approach is to start with "In the Loop" immediately after going live in production, then gradually transition to "On the Loop" as reliability data accumulates.

Explicitly Defining Escalation Conditions

Operating without a clear definition of "which cases to route to a human" tends to result in one of two outcomes: an excessive burden on staff, or high-risk cases slipping through unreviewed. The following should be defined in advance.

• Outputs where the confidence score falls below a threshold
• Inputs that resemble past misclassification patterns
• Processes involving high-risk data such as monetary amounts or personal information

Feeding Review Results Back into the Loop

Human review results should not end at a simple approve/reject decision. By leveraging them for model retraining and prompt improvement, the agentic flywheel keeps turning. Storing review logs as structured data enables smooth integration into downstream MLOps processes.

For AI agents to deliver real value in business operations, integration with existing systems such as ERP and core databases is unavoidable. However, in many organizations, differences in API specifications and inconsistent data formats cause integration work to drag on.

This section covers two approaches: architectural patterns for practically advancing connectivity with legacy systems, and a standardization approach leveraging the MCP and A2A protocols.

When running AI agents in a production environment, the first obstacle encountered is connectivity with legacy systems. Core ERP systems, on-premises databases, and traditional batch processing infrastructure — these often lack direct APIs and cannot accept requests from agents.

The representative integration patterns can be organized into the following three categories.

• API gateway / wrapper approach: An adapter layer is placed on the legacy side, enabling agent access via REST/GraphQL. While this minimizes changes to existing code, it tends to increase latency
• Event-driven (message queue) approach: A message broker such as Kafka or RabbitMQ is interposed to exchange data asynchronously. This configuration is commonly adopted in smart factory environments in manufacturing and logistics where batch processing is prevalent
• Data virtualization / feature store approach: A virtual layer integrating data from multiple systems is presented to the agent. This enables centralized management of the features required for real-time inference in AI workflow automation

The criteria for choosing between them are "update frequency" and "consistency requirements." For revenue management or dynamic pricing where real-time responsiveness is required, the API gateway approach is a better fit; for inventory management where nightly batch processing is sufficient, the event-driven approach tends to be more suitable.

One issue to watch out for is the N+1 query problem that arises when an agent, operating as a composite AI system, calls multiple tools. When queries to legacy systems cascade, response latency and system load tend to spike sharply. In practice, it is considered effective to use process mining to visualize actual data flows before connecting, in order to identify bottlenecks in advance.

After resolving the challenges of legacy integration, the next problem that awaits is the proliferation of competing communication standards between agents. When each agent operates on its own proprietary API schema, the cost of connectivity grows exponentially. The two protocols that serve as the key to standardization here are MCP (Model Context Protocol) and A2A (Agent-to-Agent Protocol).

The Role of MCP and When to Apply It

MCP is a specification that standardizes the interface used when an LLM calls tools and data sources. Its main advantages are as follows.

• Abstraction of tool calls: Access to databases, APIs, and file systems can be described using a common schema
• Ease of replacement: When backend systems change, there is generally no need to modify the agent side as long as the MCP layer is maintained
• Context window efficiency: Because only the necessary information is passed in a structured form, wasteful token consumption is reduced

The Role of A2A and When to Apply It

A2A is a communication standard for agents to directly delegate and coordinate tasks with one another. It is particularly effective when coordinating multiple specialized agents in a multi-agent system, and the following benefits can be expected.

• Standardization of task delegation: Chains such as "research agent → summarization agent → approval agent" can be explicitly defined
• Visibility into agent orchestration: It becomes easier to track which agent is processing what using AI observability tools

Points to Note During Implementation

Since both protocols are still evolving specifications, it is recommended to always refer to the official documentation and to pin versions during operation. A phased approach — first stabilizing tool integration with MCP, then expanding agent-to-agent coordination with A2A — is a realistic way to minimize risk in a production environment.

The moment an AI agent moves into production, the question of "who is responsible?" pierces through the entire organization. Authority and approval flows that could remain ambiguous during the pilot stage simply cannot function in production without a clear governance design.

Building the Framework for AI Governance

The first priority is establishing a decision-making authority matrix.

• AI Owner: The business-side accountable party. Manages KPIs and usage policies.
• AI Engineer: Ensures the quality and safety of models and infrastructure.
• AI Risk Officer: Monitors regulatory compliance from the perspectives of the EU AI Act and AI TRiSM.

Without a clear separation of these three roles, accountability tends to become ambiguous when incidents occur, leading to delayed responses.

Policy Design to Prevent Shadow AI

Shadow AI—where employees informally begin using AI tools on their own—creates data leakage risks and gaps in oversight. It is necessary to standardize the usage request flow and change management for system prompts, and to establish rules that prevent unauthorized agent deployments.

AI Literacy and Organizational Transformation

Even with governance documentation in place, it becomes a mere formality if frontline AI Literacy is low. Building in regular training and Knowledge Transfer mechanisms so that frontline staff can understand an agent's scope of judgment and escalation conditions is a practical countermeasure.

Organizational readiness should be viewed not as a "cost," but as the foundation that accelerates scaling. In the next section, we will look at how to measure whether this investment actually pays off.

To gain executive approval for moving an AI agent into production, it is necessary to quantitatively demonstrate AI ROI (Return on AI Investment). However, a vague sense that "things have become more convenient" is not sufficient justification for continued budget allocation. The golden rule is to build measurement design into the process before the pilot begins.

A Three-Layer Structure of Metrics to Measure

• Efficiency Layer: Reduction rate in processing time, changes in error rates, hours saved per staff member.
• Quality Layer: Output accuracy scores, Hallucination frequency, escalation rates.
• Business Layer: Revenue contribution, customer satisfaction scores, reduction in opportunity loss through lead time compression.

The critical point is to record a baseline before the pilot begins. Without a point of comparison, any claim of "improvement" cannot be substantiated with numbers.

Common Pitfalls

Focusing solely on cost reduction tends to cause quality degradation and personnel reallocation costs to be overlooked. Monitoring costs, infrastructure expenses, and retraining costs incurred after AI agent deployment must also be accounted for as Total Cost of Ownership (TCO). Limiting KPIs (Key Performance Indicators) to five to seven and running a quarterly review cycle makes them more manageable.

Designing a Phased ROI Reporting Structure

Agreeing in advance on a roadmap that reports "Efficiency Layer" metrics on a weekly basis immediately after going live, then transitions to "Business Layer" metrics after three to six months, reduces the risk of a premature decision to abandon the project during the early period when short-term numbers are difficult to produce. Automating the measurement framework itself with AI Observability tools is also an effective option for reducing operational overhead.

Q1. What is so different between a pilot and production?

Because a pilot runs with a limited set of users and data, the requirements for throughput, error rates, and security are all significantly more lenient than in production. In production, the number of concurrent connections and the volume of data increase by orders of magnitude, and integration with legacy systems, governance, and SLA compliance are all demanded at once. Failing to understand this "environment gap" in advance tends to result in frequent quality degradation and system failures after the transition.

---

Q2. Where should I measure the quality of an AI agent?

It is common practice to set KPIs across four axes: output quality, latency, Hallucination frequency, and tool call success rate. Setting up a system in advance that uses AI Observability tools to visualize these in real time and trigger alerts when thresholds are exceeded leads to early detection of problems.

---

Q3. To what extent should HITL (Human-in-the-Loop) be retained?

A practical approach is to design this in stages based on risk level. A three-layer structure is commonly adopted: high-risk decisions (contracts, medical, legal, etc.) use In the Loop, where a human must always approve; medium-risk uses On the Loop, where humans intervene only in exceptional cases; and low-risk uses Outside the Loop, with full automation.

---

Q4. Is AI governance necessary even for small teams?

Regardless of scale, the risks of Shadow AI and regulatory compliance requirements such as the EU AI Act cannot be avoided. Starting with at least these three points—documenting a usage policy, establishing an incident reporting flow, and conducting regular model audits—tends to significantly reduce the cost of building out governance retroactively.

Moving an AI agent into production requires simultaneously addressing three layers: technology, quality, and organization. The root cause of low production adoption rates, even after a successful pilot, is that a "gap" remains somewhere in one of these three layers.

Looking back at the key points covered in this article, they can be summarized in the following five areas:

• Quality Management: Use guardrails and AI Observability to detect output deviations early.
• HITL Design: Leverage both In the Loop and On the Loop approaches to balance the cost of human involvement with accuracy.
• System Integration: Standardize connectivity with existing ERP and legacy assets using MCP and A2A protocols.
• Governance: Institutionalize risk assessment, audit logs, and access management within the AI TRiSM framework.
• ROI Measurement: Define KPIs and continuously evaluate return on investment from both cost reduction and revenue contribution perspectives.

The aspect most often overlooked is "organizational structure." No matter how excellent an architecture is designed, if the people and processes responsible for operations are not in place, the production environment will sooner or later become dysfunctional. Simultaneously advancing AI Literacy improvement and Knowledge Transfer forms the foundation for sustainable scaling.

Going into production is not a one-time event, but a continuous improvement cycle. By steadily accumulating small successes at the MVP stage and continuing to turn the agentic flywheel, AI agents grow into a source of competitive advantage for the organization. Achieving production operation for a single process first, then expanding horizontally from there—that steady accumulation of incremental progress is the shortest route to scaled deployment.