What is Eval-Driven Development (EDD)? An Evaluation-First AI Development Process

Eval-Driven Development (EDD) is a development methodology that places the design of evaluation metrics at the starting point of AI development, ensuring the quality of models and products through continuous evaluation cycles.

In application development using LLMs (Large Language Models), there is a tendency to fall into a state where things "appear to be working, but quality cannot be measured." EDD addresses this challenge by defining multifaceted evaluation criteria—such as hallucination detection, response quality, and safety—before implementation, providing a framework that allows engineers and business stakeholders to move forward with a shared quality standard.

This article is intended for engineers and product managers involved in the development and operation of LLM applications.

Conclusion: EDD is a methodology that applies the philosophy of test-driven development to LLM development, enabling continuous quality assurance by placing evaluation metrics at the starting point of development.

We will examine in turn the similarities and differences between TDD and EDD, the inherent difficulty of testing LLMs, and the challenges caused by "development without evaluation."

TDD (Test-Driven Development) and EDD share the same philosophy of "defining quality standards before implementation." Just as TDD requires writing unit tests before writing code, EDD requires preparing evaluation metrics and evaluation datasets before building an LLM application.

Summary of similarities:

• Quality standards are defined upfront, and improvements are iterated until the implementation meets those standards
• Failure cases (test failures / drops in evaluation scores) serve as the starting point of the development cycle
• When combined with continuous integration, regressions can be detected early

It is tempting to think that "simply applying the TDD mindset directly to LLM development is sufficient," but in practice, the mechanisms for quality verification need to be fundamentally rethought, given the difference between deterministic code tests and probabilistic LLM outputs.

The core differences:

TDD asks "does the test pass?", whereas EDD asks "does the response accuracy exceed a threshold?" In MLflow evaluation examples, correctness/mean of 0.

In traditional software development, deterministic testing holds: "given input A, output B is returned." However, in applications using LLMs, this premise breaks down entirely. Because the same prompt can produce different text each time due to temperature parameters and internal sampling, pass/fail judgment based on exact matching does not work. Attempting to directly apply the frameworks of unit testing or E2E testing quickly runs into a wall.

The difficulties do not stop there. The definition of a "good response" varies entirely depending on the domain and use case, making it difficult to express quality as a single number. Hallucinations can slip through in fluent prose, making them undetectable by surface-level grammar checks. Furthermore, there is the problem of context window dependency, where response quality can vary significantly simply by changing the length or order of the provided context.

Faced with these characteristics, it becomes clear that quality assurance cannot be achieved through the intuitive confirmation of "it appears to be working."

However, what should be prioritized depends on the use case. For business applications in fields such as law, medicine, and finance—where output accuracy is the top priority—it is necessary to focus on grounding checks and quantitative evaluation of factual consistency. On the other hand, for creative generation or conversational use cases, greater weight should be given to qualitative evaluation criteria such as fluency, diversity, and user satisfaction.

The background against which Eval-Driven Development (EDD) is needed lies in the risk of production releases being made while still in a state where testing is not possible.

"I changed the prompt and it seemed to improve, but I can't tell whether it actually got better"—many developers have likely had this experience. In development without evaluation, the basis for improvement depends on subjectivity and impressions, making it impossible to objectively track changes in quality.

Consider what kinds of problems this causes. If "it feels like it got a bit better" becomes the standard of judgment every time a prompt is rewritten, you may not notice when behavior that previously worked correctly has quietly broken. As team members increase, the definition of "good quality" becomes fragmented, and the improvement cycle becomes dependent on individuals. Furthermore, while attention remains focused on technical operation checks, the gap between that and the response quality users actually need quietly widens.

EDD addresses these challenges by placing evaluation metrics at the starting point of the development cycle. Specifically, it involves defining "what constitutes success" before development begins, and incorporating a flow that measures all changes against that standard. In MLflow's evaluation workflow, the effect of a change can be confirmed quantitatively by comparing a baseline measurement (e.g., correctness/mean = 0.2) with the post-improvement measurement (e.g., correctness/mean = 1.0).

Furthermore, EDD has a structure that allows not only engineers but also business stakeholders to participate in discussions about evaluation criteria. By agreeing on evaluation metrics in advance, it becomes easier to prevent the classic disconnect of "it works technically, but it's not usable from a business perspective."

Even if you prioritize evaluation design, the evaluation loop cannot function without a foundation. Simply preparing metrics while "what to evaluate" remains vague often means that measurement results provide no actionable clues for improvement. Confirming three points in order—application type, golden dataset, and MLOps infrastructure—is a prerequisite for making EDD actually work.

Before introducing EDD, you first need to clarify "what to evaluate." Because the characteristics that should be evaluated differ significantly depending on the use case of an LLM application, sorting out the application type is the starting point.

It is tempting to think that "the same evaluation metrics can be applied to any app," but in practice, designing evaluations tailored to the characteristics of each application can dramatically reduce rework in later stages.

The main types of LLM applications and their respective evaluation characteristics are as follows:

• Q&A / Chatbots: Key metrics are answer accuracy, hallucination rate, and response speed. Quantitative evaluation against a golden dataset is straightforward.
• RAG (Retrieval-Augmented Generation) pipelines: Retrieval quality and generation quality must be evaluated separately.

Because evaluation accuracy is directly tied to data quality, designing the golden dataset is the first critical hurdle that determines the success or failure of EDD.

A golden dataset is a collection of input-output pairs with ground-truth labels, serving as the reference point for evaluation metrics. The preparation approach is broadly divided into two stages.

The fundamental decision criterion is: prioritize real data when sufficient real data is available, and supplement with synthetic data when it is not. Specifically, if 100 or more production logs or QA history records can be secured, build the dataset primarily from real data and have domain experts assign labels. On the other hand, during the pre-launch PoC stage or when coverage of rare cases is insufficient, using LLM-generated synthetic data to fill the gaps is an effective approach.

Key considerations when creating synthetic data are summarized below:

• Ensuring diversity: Intentionally design variations in difficulty, domain, and length—not just paraphrases of the same pattern.
• Quality filtering: After generation, remove obvious errors and duplicates manually or with rule-based methods.
• Managing the ratio to real data: If synthetic data makes up the majority, there is a risk that evaluations will diverge from real usage patterns, so record and manage the ratio carefully.

Additionally, a golden dataset is not something you create once and leave alone. If it is not updated regularly to reflect changes in user behavior and requirements, a gap will emerge where evaluation scores are high but actual operational quality has declined. It is important to decide on an update cycle in advance.

"I wrote the evaluation scripts, but I have no idea where to store the results"—this is a question frequently heard in teams just starting out with EDD. To run evaluation as a continuous cycle, two things are indispensable: an MLOps infrastructure capable of recording and comparing evaluation results, and AI Observability to visualize behavior at inference time.

The elements to put in place fall into three broad categories:

① Experiment Tracking Introduce an experiment management tool such as MLflow to centrally manage prompt versions, parameters, and evaluation scores. Note that MLflow's Evaluation Datasets feature requires a SQL backend—one of PostgreSQL, MySQL, SQLite, or MSSQL—so verify your database environment in advance.

② Pipeline Version Control Manage prompts, RAG (Retrieval-Augmented Generation) chunk sizes, and model configurations in Git, just as you would with code. The goal is to maintain the ability to trace "which change moved the evaluation score."

③ Embedding Observability In the production environment, collect metrics such as latency, token consumption, and hallucination rate in real time. Having a mechanism to feed collected logs back into the evaluation dataset ensures that evaluation and operations remain connected rather than siloed.

Conclusion: EDD is executed in 6 steps—"define evaluation metrics → create test cases → implement harness → measure baseline → improve → continuous loop."

Correctly understanding the role and order of each step, from evaluation design through to the continuous improvement cycle, is the key to practicing EDD. The following H3 sections walk through each phase in order.

The first pitfall of EDD is the approach of "build something that works first, then think about evaluation." In practice, trying to retrofit evaluation metrics after the fact makes it difficult to determine what to measure, and the direction for improvement tends to become vague. Defining evaluation metrics and test cases upfront serves as a compass for the entire development process.

Step 1: Defining Evaluation Metrics

First, decompose the product's goals into "measurable metrics." The choice of metrics varies depending on the use case.

• Correctness: Whether responses are consistent with facts. Particularly important in RAG (Retrieval-Augmented Generation) systems
• Grounding: Whether responses are supported by referenced sources
• Safety: Whether outputs contain harmful or inappropriate content
• Latency: Whether response times fall within an acceptable range for user experience

It is recommended to narrow metrics down to 3–5. Too many metrics diffuse priorities and slow down improvement decision-making. MLflow's evaluation examples demonstrate an approach of centrally managing correctness/mean as a baseline metric, and establishing a single primary metric first is an effective starting point.

Step 2: Creating Test Cases

Once metrics are defined, prepare test cases to validate them.

• Golden Dataset: Manually create pairs of expected inputs and correct outputs.

An evaluation harness is an execution infrastructure that feeds test cases into a model in bulk and automatically collects and aggregates the results. From the perspective of harness engineering, reproducibility and automation are the two pillars of its design.

Basic Structure of a Harness Implementation

• Input loader: Reads the golden dataset and combines it with prompt templates
• Execution layer: Calls the LLM API and saves responses as logs
• Evaluation layer: Calculates scores using predefined metrics (correctness, safety, etc.)
• Aggregation and visualization: Outputs per-metric averages and pass rates to a dashboard

Taking MLflow as an example, this entire workflow can be executed with a single call to mlflow.evaluate(). Empirical data from research notes reports cases where correctness/mean starts at a low baseline of 0.2, illustrating how important it is to quantitatively understand the current state before any improvements are made.

Decision Criteria for Baseline Measurement

If a project is in the PoC stage, it is practical to first measure only a single metric (e.g., correctness score) to establish a baseline. On the other hand, during the continuous improvement phase after a production release, it is necessary to measure multiple metrics in parallel—including safety, latency, and cost—and visualize the trade-offs.

"Is the score not improving because of a bad prompt, or is it a data problem?"—EDD provides answers to this question, commonly faced in the field, through its evaluation loop.

In Step 5, prompt engineering is carried out based on the results of baseline measurement. The following order of priority is generally recommended when considering improvements:

• Structuring the system prompt: Explicitly define roles, constraints, and output formats to target a reduction in hallucination rates
• Swapping few-shot examples: Select representative examples from the golden dataset and present them in CoT (Chain-of-Thought) format
• Chain decomposition: Break complex tasks into multiple steps and measure evaluation scores at each step

Each time a prompt is modified, the evaluation harness is run and the change in scores is recorded in a tracking tool such as MLflow. MLflow's evaluation examples show a case where the correctness/mean metric, which was 0.2 at baseline, improved to 1.0 after prompt refinement.

Only after prompt improvement reaches a plateau should fine-tuning in Step 6 be considered. Leveraging PEFT (Parameter-Efficient Fine-Tuning) with LoRA or QLoRA allows the model to be adapted to the task while keeping computational costs low (for details, see

Conclusion: The key to improving quality with LLM evaluation metrics is to selectively use quantitative and qualitative approaches depending on "what you want to improve."

The selection of evaluation metrics determines the precision of the EDD cycle. By correctly combining metrics with different objectives—such as text quality, safety, and user satisfaction—the direction for improvement becomes clear.

Safety metrics are the highest-priority evaluation dimension, as they directly affect the reliability of LLM applications.

There is a tendency to initially use fluency of responses as a proxy metric for quality, but in practice, fluency and factual accuracy are nearly independent, and fluent incorrect answers are more likely to go unnoticed. Designing safety evaluation first is the core of EDD's "evaluation-first" principle.

The following are representative safety metrics:

• Hallucination detection: Determines whether the model's output contradicts the given context or known facts. In a RAG (Retrieval-Augmented Generation) configuration, this checks whether information not contained in the retrieved results is being stated assertively.
• Grounding check: Scores whether each claim in the output is supported by source documents.

BLEU and ROUGE are reference-based quantitative metrics that have long been used in machine translation and summarization tasks. Because they calculate n-gram overlap rates against ground-truth text (reference sentences), they offer the advantages of fast, reproducible scoring and easy integration into CI/CD pipelines.

The key characteristics of each metric are summarized below:

• BLEU (Bilingual Evaluation Understudy): Measures, from a precision perspective, how much of the reference sentence's n-grams are contained in the generated text. Well-suited for evaluating translation quality.
• ROUGE-N: Measures the recall rate of how much of the reference sentence's n-grams are reproduced in the generated text. Well-suited for evaluating the coverage of summaries.
• ROUGE-L: Based on the Longest Common Subsequence (LCS), making it capable of handling word-order discrepancies.

However, these metrics have a fundamental limitation: they can only measure "surface-level overlap with reference sentences." Cases have been reported where an LLM returns accurate information phrased differently from the reference sentence, yet receives a low score.

The key decision criterion is the nature of the task. For tasks such as translation and summarization, where "a single correct text can be defined," BLEU/ROUGE serve as effective benchmarks. On the other hand, for tasks where "correct expressions are diverse"—such as chatbot response generation or RAG answer synthesis—it is recommended to combine these metrics with qualitative evaluation methods such as LLM-as-a-Judge, discussed in the next section.

"How do we score the appropriateness of a response's tone and context?"—this question comes up repeatedly in LLM application development. The LLM-as-a-Judge approach is attracting attention as a means of evaluating qualitative aspects of output quality—aspects that lexical matching metrics like BLEU and ROUGE cannot capture—without requiring manual review.

LLM-as-a-Judge is a technique in which a separate LLM (an evaluator model) scores generated outputs. The evaluator model is provided with a rubric (scoring criteria) and the text to be evaluated, and returns a score along with its reasoning. The primary evaluation dimensions are as follows:

• Correctness: Is the output factually accurate?
• Relevance: Does it align with the user's intent?
• Fluency: Is it written in a natural style?
• Safety: Does it avoid harmful or inappropriate expressions?

In MLflow's evaluation framework, this mechanism is available as a built-in evaluation metric, making it possible to link aggregated values such as correctness/mean with experiment tracking.

There are two practical considerations to keep in mind:

Conclusion: The most common pitfalls when adopting EDD are concentrated in three areas: metric design, data management, and feedback integration. Understanding each of these patterns in advance and implementing countermeasures is the key to continuous quality improvement.

In an attempt to cover all evaluation dimensions, it is not uncommon for teams to line up more than ten metrics and define the situation as "a problem if any one of them drops." At first, it seems reasonable to think that "the more angles we evaluate from, the safer we are," but in practice, narrowing down to three to five core metrics allows improvement cycles to move much faster.

Having too many metrics causes problems to cascade. For example, if metric A improves while metric B worsens, the team cannot reach consensus on which to prioritize, and decision-making stalls. Simply maintaining the evaluation harness for all metrics drains engineering resources, and as the number of metrics that are difficult for business stakeholders to interpret grows, evaluation results gradually stop being used to inform decisions.

An effective countermeasure is to organize metrics into two tiers: a "North Star Metric" and "diagnostic metrics." The North Star Metric should be narrowed down to one or two that serve as the ultimate criteria for development decisions—typical examples being the correctness score or task completion rate. Diagnostic metrics are supplementary tools for identifying root causes, and should be limited to roughly three to five—such as hallucination rate, latency, and grounding check pass rate.

Even in MLflow evaluation examples, a structure is demonstrated in which correctness/mean is tracked as the primary metric, while other metrics are treated as supplementary aids for drilling down into the causes of its decline.

Leaving a golden dataset untouched as a "finished product" after its initial creation is a failure pattern that recurs repeatedly in EDD practice.

A golden dataset is a collection of labeled samples that serve as the benchmark for evaluation. However, because the use cases and user expectations of LLM applications change over time, the dataset must be continuously updated at the same pace—otherwise, the evaluation criteria themselves become disconnected from reality.

There are several common causes of dataset staleness. As product features are added and domains expand, existing samples increasingly fail to cover new cases. User phrasing and question patterns shift, making the original reference answers feel "outdated." And revisions to laws or industry regulations can turn previously correct answers into incorrect ones.

As a rough guideline for update frequency, monthly sample reviews are advisable when a product undergoes frequent feature updates, while quarterly reviews may suffice for relatively stable use cases. What matters most is building the update schedule into the development process in advance, so that it does not get deferred.

A practical and effective countermeasure is to establish a pipeline that periodically extracts samples from production logs, manually labels them, and adds them to the dataset. It is equally essential to implement version control for the dataset, so that changes in evaluation scores can be attributed either to model improvements or to dataset updates. One staleness signal that is often overlooked is the gap where "evaluation scores are high, yet complaints about production quality are increasing." Explicitly setting up this discrepancy as a monitoring indicator makes it much easier to detect dataset degradation early.

Rather than treating a golden dataset as something that is finished once created, it is more natural to think of it as something that must be grown and maintained in parallel with the product itself.

"Evaluation scores appear in the weekly report, yet the product never seems to improve" — does this situation sound familiar?

"Siloing" — where evaluation results are never shared between the engineering team and the business side and therefore never translate into improvement actions — is one of the most common failure patterns observed after adopting EDD. Conducting evaluations becomes an end in itself, and scores end up never being used as input for decision-making.

The causes of siloing tend to converge on the same few points: evaluation results are confined to the technical team and never reach product owners or business stakeholders; there is no unified standard for interpreting scores and no defined threshold that signals "improvement is required"; and evaluation report cycles are simply not synchronized with development sprint cycles. When these three factors overlap, the result is a situation where reports are generated every week yet nobody acts on them.

Resolving this requires a mechanism that embeds evaluation results into the development cycle. By defining score thresholds in advance and configuring the workflow so that a ticket is automatically created whenever a threshold is breached, you can eliminate the person-dependent step of "someone notices and reports it." In addition, making the evaluation summary a standing agenda item in the weekly sprint review — held in a setting where business stakeholders also participate and improvement policies are decided — makes it easier to break down the structure in which only the technical team bears the burden of the problem.

Leveraging an evaluation platform such as MLflow makes it straightforward to visualize score trends on a dashboard and create an environment where all stakeholders can reference the same metrics. Reframing evaluation results from "something to be recorded" to "a trigger for action" — that is the fundamental countermeasure against siloing.

Conclusion: Because RAG and multi-agent systems involve multi-layered evaluation targets, the way EDD is applied must be redesigned to match their architecture.

Unlike a single LLM call, these architectures require independent evaluation axes for retrieval, generation, and inter-agent coordination respectively. This section explains the evaluation design for each layer and how to integrate them.

When evaluating a RAG (Retrieval-Augmented Generation) pipeline, it is tempting at first to think that "looking at the quality of the final answer alone is sufficient." In practice, however, separating the retrieval phase from the generation phase for evaluation allows you to identify the root cause of problems much more quickly.

Retrieval Quality Metrics

In the retrieval phase, you measure whether appropriate chunks are being retrieved for a given query. The representative metrics are as follows:

• Recall@K: The proportion of correct chunks included in the top K results
• MRR (Mean Reciprocal Rank): The average of the reciprocal ranks of the first correct chunk
• Context Precision: The proportion of retrieved chunks that were actually used in the answer

Measuring these metrics allows you to determine whether the vector database configuration or chunk size needs to be revisited.

Generation Quality Metrics

In the generation phase, you evaluate whether accurate answers are being generated based on the retrieved context.

Multi-agent systems require a different perspective from the quality evaluation of a single agent. Even if each individual agent is functioning correctly, if there are problems in instruction passing or result integration at the orchestration layer, the overall output quality of the system can degrade significantly.

When applying EDD to a multi-agent configuration, it is effective to design the evaluation granularity across the following 3 layers:

• Individual agent layer: Whether each agent is returning the correct output for the task it receives
• Inter-agent communication layer: Whether messages via A2A (Agent-to-Agent Protocol) are being interpreted and handed off as intended
• End-to-end layer: Whether the system as a whole is producing the expected results with respect to the user's final goal

When the evaluation targets are complex, a "bottom-up evaluation" approach — stabilizing the tests for individual agents first before moving up to higher-layer evaluation — is effective. On the other hand, when business requirement validation is urgent, a "top-down evaluation" approach — running end-to-end evaluation first to narrow down the problematic layer — is more appropriate.