What is AI Observability? A Practical Guide to Monitoring LLMs in Production

AI Observability refers to a framework for visualizing the input, inference, and output processes of LLM applications, and for continuously monitoring and improving quality, cost, and latency.

Once you begin running LLMs in production, problems emerge one after another that traditional APM (application performance monitoring) cannot capture. Hallucination frequency, cost spikes from token consumption, cascading agent failures — these create a state of "running but broken."

This article is aimed at LLM application operators, MLOps engineers, and product managers, and provides a systematic explanation covering the fundamental concepts of AI Observability, implementation steps, and key points for tool selection. By the end, you will have a clear picture of what to measure in your own LLM operations.

AI Observability refers to a framework for continuously visualizing and measuring the internal state of applications built with LLMs (Large Language Models). Its defining characteristic is that it goes beyond simple uptime monitoring to provide a multifaceted view encompassing prompt and response exchanges, token consumption, and the occurrence trends of hallucinations.

It differs from traditional APM and MLOps in both what it monitors and what it aims to achieve, requiring a new approach designed to handle the non-deterministic nature of LLMs and the quality evaluation of natural language output. The following sections will systematically outline these differences and the specific challenges they present.

Traditional APM (Application Performance Management) monitors around numerical metrics such as latency, error rate, and throughput. MLOps tracks feature drift and model accuracy degradation. Both are designed on the premise that "a correct answer can be defined."

Applications built with LLMs (Large Language Models) break this premise.

Key differences from APM and MLOps

• Probabilistic output: Even with identical input, a different piece of text is generated each time. Automatically determining whether something is an error or normal is difficult.
• Subjective quality: Whether "an answer is accurate" or "the tone is appropriate" cannot be measured by numbers alone.
• Unstructured input: Free-form prompts, unlike traditional typed API requests, contain an infinite variety of unexpected patterns.
• Dynamic cost: Inference costs fluctuate based on token count and cannot be managed by simple request volume alone.

Data drift detection used in MLOps is a technique for capturing statistical changes in feature vectors. With LLMs, however, it is necessary to track semantic-level degradation, such as meaningful shifts in prompt intent or an increase in hallucinations.

AI Observability is a concept that emerged to bridge these gaps. It integrates tracing, evaluation, and cost management to address the unique complexity of LLMs. It is best understood not as an extension of existing observability infrastructure, but as a monitoring framework redesigned to fit the characteristics of LLMs.

Applications that incorporate LLMs (Large Language Models) face fundamentally different monitoring challenges compared to traditional software. While code behavior is deterministic, LLM output is probabilistic — the same input can return a different response every time. This is the single greatest factor that complicates monitoring design.

The main challenges can be summarized in the following four points:

• Difficulty quantifying output quality: Even when an API returns a 200 response code, the answer may be a hallucination that differs from the facts. "Running" and "answering correctly" are two separate issues.
• Context Window management: When the total token count of a prompt — including conversation history and RAG retrieval results — grows large, it leads to cost spikes and response degradation. A mechanism to track these changes in real time becomes necessary.
• Complexity of chains and agents: In Compound AI Systems and AI agents, multiple LLM calls and tool executions are chained together. Identifying which step caused a quality drop requires end-to-end tracing.
• Security risks such as Prompt Injection: In RAG and agent systems that handle external data, malicious input can alter behavior. This cannot be detected by conventional code vulnerability scanning.

What these challenges have in common is that "simply collecting logs is not enough." This is the background against which a dedicated framework for cross-cutting monitoring of semantic output quality, cost efficiency, and security — namely, AI Observability — is required.

Cases where LLMs (Large Language Models) move beyond experimental PoC stages and are integrated into production systems are rapidly increasing. Along with this, operational issues such as "quality is degrading even though it seems to be running" and "costs have ballooned unexpectedly" are becoming more apparent.

The growing attention on AI Observability is driven by these risks that are unique to production operations. Identifying the root cause of failures is particularly difficult in Compound AI Systems, where multiple AI agents work in concert.

The following H3 sections will take a closer look at the necessity of AI Observability from two perspectives: the operational risks of the agent era, and market trends.

The nature of operational risk has changed significantly in an era where AI agents autonomously complete tasks by calling multiple tools. Unlike traditional systems where monitoring a single API call was sufficient, agents perform chained processing spanning dozens of steps. Without visibility into what is happening along the way, identifying the root cause of failures becomes nearly impossible.

The main risks can be organized into three categories:

• Cascading failure amplification: A "snowball effect of errors" tends to occur, where a hallucination in one step propagates to subsequent steps, causing the final output to deviate significantly
• Unpredictable costs: In multi-step reasoning, token consumption per request can spike dramatically, and cases of unexpected costs have been reported
• Latent prompt injection risks: Agents that retrieve and process external data have a broad attack surface, making them vulnerable to malicious inputs that trigger unintended behavior

Furthermore, as agent orchestration grows more complex, it becomes increasingly difficult to identify which LLM call is the bottleneck from logs alone. Even when latency suddenly increases, fine-grained tracing is essential to determine whether the cause lies on the model side or the tool-call side.

With fully autonomous agents that lack a HITL (Human-in-the-Loop) mechanism, the risk also rises that no one notices a problem until it becomes visible. AI observability functions as the foundation for detecting such risks early and supporting safe production operations.

As the production deployment of LLM applications accelerates, interest in AI observability is also growing rapidly. What was once a domain addressed only by a handful of forward-thinking companies has become widely recognized as a risk — "operations without monitoring" — alongside the proliferation of generative AI.

Several factors are driving this shift.

• Increasing production incidents: As more companies integrate LLMs into chatbots and internal search, cases of hallucinations and degraded response quality causing business impact have been reported
• The need for cost management: With LLMs billed per token, API costs tend to balloon without monitoring
• Tightening compliance requirements: AI governance legislation is advancing across countries, including the enforcement of the EU AI Act, and organizations are increasingly required to maintain and explain logs of model behavior

The range of available tools is also expanding. Dedicated platforms such as LangSmith, Langfuse, and Arize have been released in quick succession, and products that integrate with existing MLOps stacks have also emerged. Cloud vendors have begun offering LLM monitoring capabilities as managed services, and the barrier to adoption is gradually lowering.

On the other hand, organizations that have yet to adopt these practices commonly cite a shared challenge: "We don't know what to measure." Continuing operations without defined metrics and alert thresholds tends to result in significant time spent identifying root causes after an incident occurs. The next chapter explains the "Four Pillars of AI Observability" — a systematic framework for the metrics that should be measured.

Making AI observability work requires breaking down the scope of monitoring into appropriate components. In the operation of LLM (Large Language Model) applications, four pillars — tracing, evaluation, cost management, and latency optimization — complement one another.

If any one of these is missing, identifying the root cause of failures and maintaining quality becomes difficult. Each of the following H3 sections explains the role of each pillar and key points for implementation in detail.

Tracing is a mechanism for recording and visualizing the entire processing flow from user input to LLM output. It is analogous to distributed tracing in web applications, but differs significantly in that LLM-specific elements are added.

Key elements to record in an LLM trace

• The system prompt and the actual user input
• Token count, latency, and model name for each LLM call
• When using RAG, the search query and the content of retrieved chunks
• Arguments and return values of tool calls
• The timing of errors and retries

In configurations where multiple LLM calls are chained — such as multi-agent systems or Agentic RAG — it is particularly difficult to identify at which step quality degraded. By assigning a trace ID to each span and preserving parent-child relationships, it becomes possible to later reconstruct "which sub-agent received which input."

A typical Before/After example

Without tracing: When a user reports that "the answer seems wrong," there is a tendency for root cause investigation to take a long time because it is impossible to confirm which prompt was sent.

With tracing: The problematic call ID can be identified, and the input, output, and context window contents can be reproduced within minutes.

On the implementation side, many tools adopt OpenTelemetry-based SDKs, making integration with existing APM infrastructure straightforward. However, in cases where prompts contain personal information, masking of trace data must be designed in advance. "Evaluation," covered in the next section, operates using this trace data as input, so the granularity of tracing directly affects the accuracy of evaluation.

The output quality of an LLM (Large Language Model) cannot be assessed by "error codes" the way traditional software can. Even when a response is returned successfully, the content may be inaccurate, inappropriate, or out of context. This is the primary factor that makes evaluation difficult.

For automated measurement of quality metrics, the following indicators are primarily used:

• Correctness: Verifies whether the answer is factually accurate by cross-referencing it against the source documents in RAG (Retrieval-Augmented Generation)
• Faithfulness: Measures whether information not contained in the retrieved results has been generated — i.e., the presence or absence of hallucination
• Relevance: Calculates the degree of semantic alignment between the user's question and the answer using a similarity score from semantic search
• Toxicity/Safety: Detects whether inappropriate expressions or discriminatory content are present using AI Guardrails

Manually reviewing all of these would require enormous effort. For this reason, an approach known as "LLM-as-a-Judge" is widely used, in which a separate LLM acts as an evaluator and functions as a substitute for human grounding checks.

However, automated evaluation has its limitations. Since the evaluation model itself may carry biases, it is recommended to periodically cross-check results against human reviews to verify accuracy.

An evaluation pipeline is easiest to operate when structured as follows: "sampling production traffic → automated scoring → alerting when thresholds are exceeded." By visualizing score trends on a dashboard, quality degradation caused by model updates or prompt changes can be detected early.

In production LLM deployments, managing cost and latency alongside quality is an ongoing challenge. Costs can vary significantly depending on the number of tokens included in a single API call and the choice of model, making optimization without visibility extremely difficult.

AI observability measures and records the following metrics in real time:

• Token consumption: Input/output token counts for both prompts and completions
• Latency breakdown: Time to First Token (TTFT) and overall response time
• Per-model cost: Comparative costs when using multiple models
• Cache hit rate: Reuse of identical or similar prompts

By continuously recording these metrics, it becomes possible to identify "which prompts are high-cost" and "which steps are bottlenecks." For example, in cases where unnecessary information is being packed into the Context Window, prompt compression tends to reduce token counts.

Regarding latency, in multi-step reasoning using agent orchestration, delays accumulate easily across each step. Visualizing the time spent at each step using trace data has been reported to reveal processes that can be parallelized and steps that can be switched to a lighter-weight SLM (Small Language Model).

The basic approach to optimization is as follows:

• Route lower-priority tasks to cheaper, faster models
• Apply caching strategies to frequently repeated prompts
• Detect abnormal consumption early through regular cost reviews

Cost and latency optimization is also closely tied to tool selection, which will be covered in the next section.

The AI observability tool market is expanding rapidly, with options ranging from OSS to commercial platforms. Choosing a tool that does not fit your use case or development structure carries the risk of inflated operational costs after adoption.

The key criteria for selection can be broadly organized into three points: "cost structure," "ease of integration," and "richness of evaluation features." The following H3 sections provide a detailed breakdown of the characteristics of OSS versus commercial options, along with a practical checklist for real-world use.

AI observability tools fall broadly into two categories: OSS and commercial platforms. Making the wrong choice can force a rebuild after adoption due to operational costs or insufficient functionality.

Key OSS (Open Source) Options

• Langfuse: An MIT-licensed OSS that provides tracing, evaluation, and prompt management in a single package. Fully supports self-hosting, with a cloud version also available.
• Phoenix (by Arize AI): Easy to launch locally, making it well-suited for validation at the PoC stage.
• OpenLLMetry (by Traceloop): OpenTelemetry-based and easy to integrate into existing monitoring stacks.

The strengths of OSS lie in customizability and cost control. However, it is important to note that infrastructure management and security patch handling during scale-out become the responsibility of your own organization.

Key Commercial Platform Options

• LangSmith (by LangChain): Deeply integrated with the LangChain ecosystem, providing tracing, evaluation, and dataset management in one place. Self-hosting is also supported under an Enterprise contract.
• Arize AI: Supports drift detection and A/B testing on production traffic.
• Datadog LLM Observability: Easy to integrate with existing APM and logging infrastructure.

Commercial tools offer SLAs and robust support, but usage-based billing tied to token volume tends to accumulate quickly (reference values at time of writing; check the latest pricing pages for current information).

Decision Guidelines

A practical approach is to start with a small-scale OSS implementation to understand the fundamentals, then consider migrating to a commercial solution as production scale demands grow.

To avoid tool selection failures, it is important to clarify your evaluation criteria in advance. Use the following checklist as a guide.

Integration & Compatibility

• Does it support the SDKs of the LLM providers you use (OpenAI, Anthropic, Google, etc.)?
• Does it offer native integration with frameworks such as LangChain and LlamaIndex?
• Can it connect with your existing monitoring stack, such as Datadog or Grafana?

Tracing Capabilities

• Can it record prompts, completions, and tool calls at the span level?
• Does it support distributed tracing across multiple hops in multi-agent systems?
• Can it provide real-time visibility into Context Window usage?

Evaluation & Quality Management

• Does it include built-in hallucination detection and grounding checks?
• Is it extensible enough to allow custom evaluation metrics to be defined?
• Can human feedback (HITL) be incorporated into the evaluation loop?

Cost & Security

• Can token consumption and costs be aggregated by model, user, and feature?
• Is there a clearly stated policy for handling sensitive information in prompts and outputs?
• Can you control where data is stored and how long it is retained?

Operations & Scale

• Does it support sampling rate adjustment during high-traffic periods?
• Is there a dashboard for sharing and managing alert thresholds across the team?

During the selection process, it is recommended to first use free tiers or OSS at the PoC stage and validate the above items against your actual workload. Since integration costs often balloon after moving to production, be sure to also assess the risk of vendor lock-in.

Trying to set up everything at once when adopting AI observability tends to lead to failure. An approach that starts with inserting traces into critical paths and then incrementally expands to evaluation pipelines and alert configuration tends to improve the rate of successful adoption.

The adoption process can be broadly divided into three phases:

• Step 1: Insert traces into critical paths
• Step 2: Build an evaluation pipeline
• Step 3: Set up alerts and dashboards

The details of each step are explained in the H3 sections that follow.

Trying to instrument traces everywhere at once inflates implementation costs and leads to burnout. Focusing first on the "critical path" is the fastest route to sustainable adoption.

How to Choose the Critical Path

• User-facing endpoints (chat responses, summarization, search)
• RAG retrieval and generation steps where errors have a large blast radius
• LLM call sites where cost and latency are concentrated

Basic Implementation Patterns

Most observability tools can be instrumented using decorators or context managers. In Python, for example, adding just a few lines to a function is enough to handle span start, end, and attribute assignment. The minimum set of data to record consists of these four items:

1. Full input prompt (including system prompt)
2. LLM-generated output
3. Token count and latency
4. Model name, version, and temperature parameter

Expanding Incrementally

For the first one to two weeks, run traces on the critical path only and confirm that logs are flowing correctly. Then gradually extend coverage to surrounding steps such as RAG chunk retrieval and tool calls. Rather than instrumenting everything at once, starting small and catching problems early tends to reduce rework.

Note that if input data contains personal information, masking must be applied before traces are saved. Governance requirements and instrumentation design should be considered together from the outset.

Once traces are in place, the next step is to use the collected data to build a system for continuously measuring quality. This is the evaluation pipeline.

The basic structure of an evaluation pipeline is easiest to organize across three layers:

• Online evaluation: Samples production requests in real time and automatically scores them for hallucinations and harmful output using an LLM judge
• Offline evaluation: Runs batch validation against a golden dataset before prompt changes or model switches
• Human review (HITL): Has humans label samples that fall below a score threshold, continuously refining the evaluation criteria

One important consideration during construction is to work backwards from business requirements when defining evaluation criteria. For a customer support use case, for example, "answer accuracy" and "appropriate tone" are the top priorities. For a code generation use case, "syntax error rate" and "test pass rate" tend to be the primary metrics.

A rough guide to implementation steps is as follows:

1. Narrow down the quality dimensions to evaluate to two or three items
2. Decide on the scoring logic for each dimension (LLM judge or rule-based)
3. Integrate offline evaluation into the CI/CD pipeline
4. Run a fixed percentage of production samples through online evaluation

Storing evaluation results linked to their corresponding traces makes it easier to set thresholds in the alerting step that follows. A phased approach of starting with weekly batch evaluation and expanding to production sampling once stable is effective for keeping operational overhead manageable.

Once traces and the evaluation pipeline are in place, the final step is to complete the "system for never missing anomalies" with alerts and dashboards. No matter how sophisticated a measurement infrastructure you have, it is meaningless if problems are not surfaced in a way that humans can notice.

Key Metrics to Visualize on the Dashboard

• Latency distribution: Display P50, P95, and P99 as time series to immediately spot spikes
• Error rate: Count of API timeouts, context window overflows, and guardrail activations
• Token consumption: Cost trends broken down by model and endpoint
• Quality scores: Trends in hallucination rate and relevance scores obtained from automated evaluation

By consolidating these onto a single screen, the goal is to enable on-call staff to assess the situation within 30 seconds.

Alert Design Principles

There is a common tendency in practice for alerts to be ignored when there are too many of them. Designing across the following three tiers is effective:

1. Critical (immediate response): Error rate exceeds threshold within the last 5 minutes. Notify via PagerDuty or Slack
2. Warning (next business day response): P95 latency has degraded significantly compared to the previous day
3. Informational (weekly review): Signs of rising cost trends or model drift

Tips for Sustaining Operations

After setting up alerts, review the noise rate (the ratio of actionable responses to total alerts fired) on a weekly basis and delete or adjust unnecessary alerts. The same applies to dashboards — sharing a team habit of cleaning up any metric that goes unused for two weeks tends to help maintain long-term operational quality.

Q1. What is the difference between AI observability and LLM monitoring?

LLM monitoring centers on uptime checks — confirming whether the system is running and whether errors are occurring. AI observability, on the other hand, combines tracing, evaluation, and cost analysis to create a system that lets you trace back to why a particular output was generated. The goal is to move from simple monitoring to "explainable operations."

---

Q2. Can small teams adopt this too?

Adoption can be approached incrementally. Even just instrumenting one or two critical paths with traces is enough to identify where hallucinations are occurring and where latency bottlenecks lie. Leveraging OSS tools helps keep initial costs low, and a realistic approach is to start at proof-of-concept (PoC) scale and expand gradually.

---

Q3. How granularly can costs be tracked?

By recording token consumption per request, you can understand the cost breakdown by model and by feature. Since how you use the context window and the length of your prompts directly affect costs, trace data makes it easier to prioritize optimization efforts. Because pricing fluctuates, it is recommended to check the latest pricing pages regularly.

---

Q4. How does monitoring AI agents differ from monitoring standard LLMs?

Because AI agents perform tool calls and external API integrations across multiple steps, multi-step reasoning traces that track which step led to an incorrect decision are essential. Without visibility into the entire agent orchestration, identifying the root cause of a problem becomes very difficult.

AI observability is the foundational technology for running LLM applications reliably in production. By making visible the aspects that traditional APM and MLOps could not fully capture — prompt quality, fluctuations in inference cost, and the chained behavior of agents — it enables early problem detection and a continuous improvement cycle.

The key points to take away from this article are as follows:

• Tracing: Record every step from input to output to identify where hallucinations and latency occur
• Evaluation: Automatically measure quality metrics and quantitatively track changes before and after releases
• Cost and latency management: Continuously monitor token consumption and response times to prevent runaway spending and degradation
• Alerts and dashboards: Establish the ability to detect anomalies immediately and share situational awareness across the team

A practical starting point is to begin small with instrumentation of the critical path. Trying to build a perfect monitoring setup from day one tends to inflate the effort required and stall adoption. Starting with traces on the main flows as an MVP and incrementally expanding the evaluation pipeline is an approach that tends to stick.

Each tool option — OSS and commercial platforms alike — has its own trade-offs. Use the checklist to make a decision that fits your team's size and existing infrastructure.

The more widely AI agents are adopted, the more important monitoring becomes. Instrumenting a single trace today is the first step toward trustworthy AI operations.