Integrated Management of LLM Inference Logs: Production Environment Observability with a Converged Database

When operating an LLM inference system in production, there are surely no small number of engineers who, during some incident, have found themselves shuttling between three dashboards—"input prompts in CloudWatch, output token counts in Datadog, error logs in Elasticsearch." When the information needed for root cause investigation is scattered, simply reconciling timestamp discrepancies and missing data can consume dozens of minutes.

A converged database is a database that natively handles multiple data models—relational, document/JSON, graph, vector, and text—within a single engine, managed under one optimizer and one transaction boundary. This article is aimed at operators of large-scale LLM inference systems, backend engineers, and MLOps engineers. After organizing the current challenges of input/output logs, latency, token usage, and error events being scattered across multiple storage systems, we will explain the implementation steps for unified management using a converged database. By the time you finish reading, you should have a clear picture of concrete visualization methods for improving anomaly detection responsiveness and drift detection accuracy.

In LLM inference, text/JSON data such as prompts and generated text, time-series metrics such as latency and token usage, and vector embeddings used for similarity search all occur simultaneously. Traditionally, it has been common to store these separately in storage systems dedicated to each use case (text DB, time-series DB, vector DB). However, this separation comes at a cost: even just tracking the information for a single inference call requires joining multiple queries on the application side.

A converged database is a mechanism that natively handles each data model—relational, document/JSON, vector, text, and graph—within a single engine, managed under one optimizer and consistency model. Rather than the type of data model itself, the practical criterion is "whether it can be handled within a single transaction boundary." With this design, there is no longer a need to separately retrieve the input/output, latency, token count, and error events for a single inference call and then join them afterward—correlation analysis can be performed within a single transaction boundary.

This unification of data models is the foundational concept for improving anomaly detection responsiveness and increasing drift detection accuracy. In the next section, we will organize exactly what kind of delays and degradation in detection accuracy distributed management can cause.

When the urgency of failure investigation is high, immediate access to latency and error logs takes priority; when the purpose is auditing or cost analysis, the comprehensiveness and retention period of token usage logs takes priority. Organizing the representative distribution patterns based on this decision axis yields the following:

Comparing the four rows in the table, both the text/JSON store and the time-series DB are fast on their own, but when trying to combine the two to determine "what was being input/output at the exact moment a given request's latency spiked," the overhead of joining becomes a bottleneck. The same applies to vector DBs: while similarity search itself is fast, situations arise where a separate query is needed to filter by token count or billing information alongside it.

Furthermore, in a distributed architecture, each store has its own consistency model, which can cause log timestamps to drift slightly out of sync. Even a discrepancy of a few hundred milliseconds can become a non-negligible error when tracing the sequence of events during failure investigation. If prioritizing anomaly detection responsiveness, this time drift and the number of round trips required for join queries are precisely the practical criteria for choosing a store architecture.

When input/output logs, latency, token usage, and error events are stored in separate stores, failure investigation requires issuing individual queries to multiple systems and manually joining the results. Each such round trip accumulates network I/O and processing wait time, adding hundreds of milliseconds to several seconds of delay even for simple filtering.

For teams that want to identify "what happened with which request" within minutes of an anomaly occurring, this cross-store round trip is precisely the biggest bottleneck. Consider a scenario investigating whether latency is degrading for a specific prompt pattern. This requires a two-stage query: extracting the target prompt IDs from the text store, then passing that ID list to the time-series DB to cross-reference latency. As data volume grows, the cost of passing intermediate results back and forth becomes non-negligible.

When each store has a different consistency model, discrepancies in write timing can cause logs for the same request to be temporarily referenced in an inconsistent state. This can lead not only to display delays in monitoring dashboards but also to false positives or missed detections in automated alerts. If input/output, metrics, and errors can be handled within a single transaction boundary, this join cost and consistency fluctuation can be structurally eliminated.

The root cause of delayed drift detection in production operations lies in the distributed structure of the logs themselves.

Drift detection is a process of tracking changes in the distribution of inputs and outputs over time and continuously comparing them against historical data. However, when input/output text is stored in a document store while latency and token usage are stored in a time-series DB, a join process to reconcile the two is inevitably inserted upstream of the detection pipeline. In configurations where this join depends on periodically scheduled batch processing, a time lag equal to the batch interval occurs between when an anomaly occurs and when the detection logic captures the difference. With hourly batches, in the worst case, detection could take nearly an hour.

Furthermore, when data retention periods and granularity differ between stores, the comparison periods can become misaligned, leading to false positives or missed detections. When investigating changes in prompt trends, the retention period of the text store becomes the starting point; when investigating latency degradation, the granularity of the time-series DB becomes the starting point. The reason detection timing varies depending on which store is referenced—even for the same incident—is this mismatch in starting points. For example, in a case where a change in prompt content and latency degradation occur simultaneously, even if a trend change begins to appear in the text store after a few minutes, if the time-series DB's granularity is only aggregated at 5-minute or 1-minute intervals, it takes even longer before the anomalies in both are linked together as "the same event." By the time the person in charge notices the alert, it is often already difficult to isolate the cause.

Observability platforms like Prometheus recommend using histograms for latency measurement, but this granularity may not match the log granularity on the text store side. In cases where the time-series DB retains values rounded to bucket units while the text store retains raw logs at the individual request level, one side's information may be too coarse during cross-referencing to detect a statistically significant difference. When granularity mismatches accumulate, distribution changes that are actually occurring cannot be statistically captured, and detection accuracy itself declines.

The quality of an integrated design changes significantly depending on what should be consolidated into a single table.

LLM inference logs contain a mix of data with different characteristics: input/output text, latency and throughput, and token usage and cost. If these remain scattered across separate tables or log files during operation, cross-referencing multiple sources becomes necessary every time a failure investigation occurs, which tends to extend the time required to pinpoint the cause. By designing the system to record everything centrally around a common request ID, this cross-referencing cost can be structurally eliminated. In what follows, we will focus primarily on input/output text and metadata as the core of the integration, and touch only briefly on the remaining types.

Whether to store input/output logs as-is in a text-type column or structure them as a JSON type depends on whether the prompt configuration is a simple single string or a multi-stage structure that includes system prompts or tool call results. For simple question-answering, a text type is sufficient, but when RAG search results or multi-turn conversation history are involved, the characteristics of a converged database—one capable of handling both relational and document/JSON types within a single engine—become valuable.

In anomaly detection scenarios, investigations often hit a wall when the full text of the input/output cannot be immediately retrieved while cross-referencing which part of the generated text is anomalous. If the prompt and the generated result are stored in separate storage systems, it takes time to go back and forth between multiple systems using the request ID as a clue, delaying the initial response. This round-trip cost cannot be dismissed lightly in production environments, where the initial response to an incident is often measured in minutes.

When storing data, it is necessary to decide on masking and encryption policies in advance, given the possibility that personal or confidential information may be included. In cases where the generated text is long, one option is to separate full-text storage from summary storage, referring to the full text only during detailed investigations. Since input/output logs serve as the common foundation for auditing, debugging, and drift detection, the design premise is that they can be linked to other metrics via the same request ID.

Latency and throughput need to be handled as time-series data in a table or set of columns separate from the input/output logs. This is because input/output text is data referenced on a one-off basis during investigations, whereas latency is a numerical value that is aggregated continuously to track trends. With a converged database capable of handling both within a single query engine, the latency at the time an anomalous generated text was detected can be immediately cross-referenced within the same transaction boundary.

When recording latency, it is important to retain the distribution, not just the average value. Prometheus's best practices also explicitly state that histograms should be prioritized over summaries for latency measurement, because this allows percentile values (such as p95 or p99) to be flexibly recalculated later. In inference systems, it is not uncommon for only a subset of requests to be extremely delayed, and looking only at the average value can cause this type of anomaly to be overlooked. A situation where p50 remains stable but only p99 spikes can be caused by specific prompt lengths or batch processing timing, and without retaining a histogram, it becomes impossible to even trace the cause afterward.

For throughput, recording the number of processed items per unit time separately by endpoint or model version makes it easier to isolate load balancing imbalances or performance degradation during model switching. In AWS SageMaker's InvokeEndpoint, there is a constraint that model processing must respond within 60 seconds, and monitoring the proportion of requests approaching this limit allows early detection of degradation just before a timeout occurs.

When inference costs surge month-over-month, can you identify which model call is responsible? If token usage logs are managed separately from input/output logs and latency, cross-referencing the time a cost anomaly occurred with the content generated tends to be time-consuming.

In OpenAI's API, the usage field of the response object contains prompt_tokens, completion_tokens, and total_tokens, which can be stored directly as structured data in a converged database. When using the Batch API, be aware that usage values are only recorded for batches created after a certain point in time, so systems that also use batch processing need to verify whether the recording is present.

Recording token counts alone is not sufficient. By storing them within the same transaction boundary, linked to the model name, the origin of the request, and the content of the generated text, you can immediately narrow down "which prompt design is the cause" when costs spike. By having per-request token counts as time-series columns and linking them to the input/output log ID via a foreign key, you can trace back from a cost anomaly alert to the corresponding generated content with a single query. This is a benefit specific to integrated management that is difficult to achieve when logs are scattered across different storage systems.

Attempting to rebuild everything for integrated inference log management all at once would have too great an impact on existing systems. In practice, therefore, an approach has become established of proceeding in three stages: selecting and initially building the converged database, capturing and formatting logs at inference execution time, and building the collection pipeline. Below, we will look at what to do and to what extent at each stage.

When the existing log volume is on the order of hundreds of gigabytes and consists mainly of structured data, a PostgreSQL-derived TimescaleDB extension setup becomes a candidate; when unstructured data such as generated text and embedding vectors predominates, a column-oriented engine like ClickHouse becomes a candidate. A converged database refers to a design that natively handles data types such as relational, document/JSON, vector, and text within a single engine, unifying the optimizer and transaction boundaries into one. This characteristic means that inference log input/output text, metadata, and vector representations do not need to be distributed across separate storage systems, and can instead be joined on the same query foundation—a key consideration in the selection process.

For initial construction, first identify the schema of existing logs and determine which data type (JSON, text, numeric, vector) will be used to hold each item—prompts, generated text, latency, token usage, and error events. Next, design an indexing policy based on expected write frequency and query patterns, and if there is a large volume of time-series data, consider partitioning along the time axis. Migration is safest when performed by gradually switching over production traffic; establishing a period of parallel operation with the old storage system to verify consistency helps reduce the risk of data loss during the switchover.

By defining in advance the log items to be captured at inference execution time, missing values and type mismatches become less likely to occur in the downstream collection pipeline. Specifically, the four categories—input/output text (prompts and generated results), latency, token usage, and error events—are formatted into JSON using a common schema.

For token usage, in cases like the OpenAI API where prompt_tokens, completion_tokens, and total_tokens can be obtained directly from the usage object in the response, mapping these directly to inference log fields keeps implementation costs low. However, for calls made via the Batch API, there is a specification whereby usage is only populated for batches created after a certain point in time, so configurations that also use batch inference need to build in backward-compatible handling that assumes fields may be missing.

For latency, the processing time from request receipt to response return should be recorded in milliseconds; if possible, measuring the model invocation portion separately from the pre-processing and post-processing portions makes it easier to isolate bottlenecks during later anomaly detection. For error events, timeouts, rate limits, output format anomalies, and the like should be recorded as enumerated status codes, kept in a separate column from free-text error messages—this simplifies aggregation queries on the converged database side.

Are there not many operators who hesitate over the timing of writing formatted logs to the converged database during the design phase of the collection pipeline?

The synchronous approach, writing directly to the database within the inference process, is simple to implement, but write latency tends to affect the inference response itself. For this reason, many implementations adopt an asynchronous approach that separates the inference process from log writing. Specifically, this involves a configuration where the application temporarily stores logs in a message queue or local buffer, and a separate collector process inserts them into the converged database in batches.

Batch size design involves a trade-off between collection frequency and database load. For anomaly detection use cases that prioritize real-time responsiveness, batch intervals need to be kept short, whereas for use cases like token usage aggregation that prioritize cost optimization, writing in bulk on a scale of several minutes often causes little practical harm. Therefore, it is practically effective to build conditional branching into the collection pipeline: logs requiring immediate detection, such as latency and error events, are processed at short intervals with small batches, while logs for which retrospective analysis suffices, such as token usage and cost aggregation, are processed at longer intervals with large batches.

There are also cases, such as AWS SageMaker's Data Capture, where the official documentation notes an operational caution that capture itself should be stopped when disk usage becomes high on the inference endpoint side—meaning the collection pipeline also needs to incorporate retry logic for write failures and fallback processing for when disk space becomes constrained.

Decision Axis: How the unified logs are utilized determines the success or failure of production operations.

Even if logs are centralized, quality monitoring does not function without anomaly detection design and dashboard-based visualization. Here, we explain specific methods for connecting accumulated input/output, latency, token usage, and error events to a real-time query and visualization infrastructure.

For detecting sharp spikes in latency, percentile monitoring via histogram aggregation is effective, while for detecting abnormal skew in token usage, comparison with a moving average against the recent window is suitable. Designing real-time anomaly detection queries as two separate systems along these lines makes it easier to suppress false positives.

For latency monitoring, it is useful to reference the fact that Prometheus best practices recommend using a histogram rather than a summary. Because histograms retain the distribution on a per-bucket basis, they offer the advantage of being able to flexibly recalculate percentiles such as p95 and p99 after the fact. On the converged database side as well, a design that aggregates the latency distribution over the past several minutes against a time-series table, and fires an alert when buckets exceeding a threshold occur consecutively, is easier to work with.

For anomaly detection in token usage, the basic approach is to record prompt_tokens, completion_tokens, and total_tokens from the OpenAI API's usage object over time, and run a query that extracts requests deviating beyond a certain range from the recent average. Correlation with error events should not be overlooked either. In cases where token usage spikes just before an error rate increase, the cause may be bloating of prompts on the input side; if the architecture allows input/output logs and metrics to be joined within the same transaction boundary, the number of round-trip queries needed to identify the root cause can be reduced.

How the operations team interprets anomalies detected by anomaly detection queries depends heavily on the quality of dashboard design. Even with integrated logs, without accompanying visualization, the work of reading "what is happening" from a list of numbers becomes necessary, which tends to delay response.

In dashboards, an effective configuration displays input/output logs, latency, token usage, and error events on the same screen in chronological order, allowing correlations to be checked at a glance. For example, overlaying whether specific prompt patterns or token counts are skewed at the moment when p95 latency spikes speeds up narrowing down the cause. With a converged database, there is an operational advantage in that these heterogeneous data types don't need to be retrieved via separate queries and manually joined—a single query result can be passed directly to the visualization tool.

Just as Google Cloud's Audit Logs provides a field indicating processing time, adding processing time and caller information as dashboard filtering conditions speeds up the initial response to incident investigation. By setting up rules to highlight threshold breaches, you can create a state where the person in charge notices the moment they look at the screen, without waiting for the results of the anomaly detection query. The dashboard's update frequency should be adjusted according to use case; designing separately for real-time monitoring screens and daily trend-reporting screens makes it easier to balance load and cost.

We organize the questions operators commonly have when implementing integrated inference log management into three points: database selection, migration effort, and cost optimization. Please use this as reference material for decision-making before implementation.

Decision axis: It is important to choose based on the type of logs and query characteristics.

A converged database refers to a product that can natively handle "relational, document/JSON, graph, vector, spatial, text" data within a single engine. The core of the selection lies in the ability to manage heterogeneous data—JSON structures like input/output logs, time-series metrics like latency and token usage, and similarity search via embedding vectors—under a single optimizer and a single transaction boundary.

Specifically, if there are many aggregation queries for time-series metrics, a configuration combining PostgreSQL with TimescaleDB is suitable, since this allows relational JSON operations and time-series aggregation to be handled within the same SQL. On the other hand, if the focus is on high-throughput analytical queries and aggregating large volumes of inference logs in a column-oriented manner, ClickHouse is well-suited.

In either case, judging based on three criteria—the frequency of composite queries involving vector search, the query language the existing operations team is familiar with, and the volume of logs to be monitored—helps keep the selection consistent. If a vector database is operated separately, limiting the scope of integration to the log management layer and clearly defining the division of roles with the search optimization concepts explained in What is Adaptive RAG? How to Balance Cost and Accuracy with Query-Driven Dynamic Retrieval can help avoid increased operational complexity.

If the scale of existing logs is small, you should expect a timeline of a few weeks; for large-scale distributed logs, a timeline of several months should be anticipated. The migration period is similar to "moving house"—not only does the amount of belongings (log data) matter, but the workload varies greatly depending on the compatibility of the wiring (data formats and query structures) between the old and new residences.

The migration work is mainly divided into three stages. First, investigating the schema of existing logs and designing the mapping; next, data reconciliation through parallel operation; and finally, the phased shutdown of the old system. If the parallel operation period is estimated too short, cases tend to arise where the anomaly detection thresholds shift immediately after migration, increasing false positives.

Factors affecting migration time are as follows:

• Diversity of log formats (JSON, CSV, proprietary binary, etc., proportional to the number of conversion logics required)
• Number of distributed existing storage systems (the more S3, RDB, and monitoring tools are separated, the more complex coordinating parallel operation becomes)
• Whether retraining of the drift detection model is necessary (since the data distribution changes after integration, retuning is likely to be required)

For a small-scale prototype environment, switching over within a few weeks is possible after going through a proof of concept (PoC). However, for a production environment spanning multiple regions, a well-margined plan is realistic, including phased traffic migration and preparation of rollback procedures.

Balancing retention period and cost is an area where a branching approach to decision-making is effective. The basic policy is to place logs that are frequently referenced for anomaly detection or audit purposes in a tier with fast access, while gradually moving older logs with reduced reference frequency to a lower-cost tier.

The decision criteria differ depending on the use case. Inference logs from the most recent few weeks are routinely referenced by anomaly detection and drift detection queries, so it is practical to keep them in a fast tier, while for logs older than that, only statistical summaries are retained and the raw logs are compressed and archived. Caution is needed here: if logs that require long-term retention for regulatory compliance and short-term logs used only for performance improvement are handled under the same retention policy, costs can balloon unnecessarily.

In OpenAI's Responses API, application state is retained for 30 days by default, and anomaly detection designs that rely on data beyond this period should be avoided. Even when managing logs in an integrated manner in-house, taking into account retention period guidelines similar to this, separating short-term analysis that completes within 30 days from long-term analysis that retains data beyond that at the design stage can reduce redesign costs later on.

For cost optimization, it is essential to periodically review the relationship between raw log compression rates and query frequency. While extending the retention period improves audit response capability, it creates a trade-off between storage cost and search performance, requiring operations to adjust thresholds according to business requirements.