Automating AI Red Teaming: Test Case Generation and Scalable Vulnerability Validation

In manual red teaming, a single engineer can only verify a limited number of prompt patterns per day. As more teams operate LLMs in production, this manual process increasingly becomes a bottleneck.

AI red teaming automation is an approach that continuously verifies LLM vulnerabilities—such as prompt injection and jailbreaks—using test case generation tools and CI/CD pipelines. This article is aimed at teams operating large-scale LLMs, DevSecOps personnel, and security engineers. Referencing frameworks such as the OWASP LLM Top 10 and NIST AI RMF, it provides a step-by-step explanation covering everything from test case generation to CI/CD integration and scalable vulnerability verification through parallel execution. By the end, you should have a grasp of the design principles for building a system capable of conducting vulnerability verification at 5 to 10 times the scale of manual testing. Note that this article is a general technical explanation; implementation decisions should be based on your own organization's risk assessment and expert supervision.

Manual red teaming is the work of security experts attempting adversarial inputs against a model to discover vulnerabilities. However, there are limits to securing specialized talent, and re-verifying everything each time a model is updated or a prompt is changed is not realistic in terms of time or manpower. In projects that update models weekly, it is not uncommon for the manual testing cycle to fail to keep pace with the update frequency.

Automation is an approach that compensates for this structural limitation. By combining test case generation with CI/CD integration, the scope and frequency of verification can be continuously expanded. By building a mechanism that mechanically generates a volume of input patterns beyond what can be covered manually and continuously checks the model's responses, vulnerability discovery can be incorporated as part of the development cycle.

When the scale of the target system is small, spot checks by a few experts may suffice. However, when the target becomes a large-scale LLM operation involving multi-agent systems or multilingual prompts, manual red teaming quickly reaches its limits.

The core of the problem lies in the asymmetry between the exponential increase in combinations of attack patterns and the inability to proportionally increase the personnel responsible for verification. For each vulnerability category listed in OWASP's "LLM Top 10"—such as prompt injection, sensitive information disclosure, and excessive agency—attempting to manually cover test cases that vary in phrasing, language, and context reveals a natural upper limit to the volume of verification a single person can handle.

Furthermore, every time a model or system prompt is updated, it becomes necessary to reconfirm whether the existing test cases remain valid. In manual operations, verification cannot keep pace with the update frequency, and periods of unverified exposure tend to occur. In a large-scale red teaming competition conducted by NIST (CAISI), over 250,000 attack attempts were made, and at least one successful attack was confirmed against every frontier model targeted. This scale of attack volume demonstrates that reproducing it manually—let alone maintaining continuous monitoring—is practically impossible.

Automation is not a replacement for spot checks; rather, it is a means of structurally transforming the frequency and comprehensiveness of verification. By incorporating test case generation and scoring into a CI/CD pipeline, the same set of vulnerability categories can be re-verified under the same criteria every time a model is updated or a system prompt is changed. A question that frequently arises in practice is how to handle a vulnerability discovered right before release—and in designing continuous verification, an answer to this question must be prepared in advance. Embedding a mechanism that quantifies verification results as a score, such as Attack Success Rate (ASR), and blocks builds that exceed a defined threshold within the pipeline, serves as a concrete answer to this question.

In a large-scale agentic red teaming competition conducted by NIST's CAISI, over 250,000 attack attempts by more than 400 participants confirmed at least one successful attack against every frontier model targeted. Reproducing this scale of trials manually is unrealistic; this volume of verification can only be achieved with automated test case generation and execution infrastructure. The purpose of continuous verification is to keep running such a large volume of attack patterns on a regular and consistent basis, enabling early detection of vulnerability recurrence or reproduction in new models.

In automated test case generation, prompts are diversified across attack categories—such as direct injection, indirect injection, and jailbreaks—to systematically expand coverage. Manually written test cases tend to be capped at dozens to a few hundred, which limits the ability to cover the full combination of attack patterns. With automated generation, existing attack prompts are turned into templates, and operations such as vocabulary substitution, sentence structure mutation, and translation into multiple languages are applied to produce large numbers of variations sharing the same attack intent. By classifying the generated prompts by category and visualizing the extent to which each attack technique is covered, it also becomes possible to target underrepresented areas for additional generation. Below, this generation pro

The key to automatically generating prompt injection verification cases is separating direct injection from indirect injection during generation. If this separation is neglected, only template mass-production progresses, and the cases that actually work end up buried.

Direct injection is an attack that drops wording overriding the system prompt's instructions directly into the user input field. By templating command phrases like "ignore all previous instructions" and turning word order, language, and honorific usage into variables, a small number of templates can generate a large volume of variant cases. In the OWASP LLM Top 10, prompt injection is positioned as LLM01, and MITRE ATLAS also organizes it as a technique category under "AML.T0051," making this an area that is easy to leverage as a classification axis for generation logic.

Indirect injection, on the other hand, refers to cases where malicious instructions are embedded in external documents, web pages, or email bodies referenced via RAG, and the model ends up executing them at retrieval time. The difficulty of verification rises a level here. An effective method is to mix attack phrases into part of the documents registered in the vector database and check whether grounding checks function properly, but detection accuracy varies significantly depending on the granularity at which the phrases are mixed in and which retrieval paths are covered. In operations dealing with RAG, the focus becomes whether poisoned context can be detected, so cases need to be built up in a separate track from ordinary input testing. For implementation details, Closing the "Invisible Attack Path" in AI Chat — An Implementation Guide for Defending Against DB-Mediated Prompt Injection is also a useful reference.

For short-term, single-shot attacks, a single-turn prompt is sufficient, but when defenses are gradually undermined using the model's memory or context, scenario design spanning multiple turns becomes necessary. Automated jailbreak verification requires an approach that captures the kind of breakdown that cannot be seen through single-turn banned-word checks alone—a way of probing for looseness by shaking the entire structure of the door, so to speak.

Specifically, three attack patterns are each templated and automatically generated: persona injection type, which disguises a fake role setting; hypothetical type, which pretends to be an authorized fictional context; and gradual escalation type, which stacks multiple harmless instructions to eventually lead to harmful output. These three types are often combined with one another—for instance, combining persona injection with gradual escalation tends to expose looseness in defenses more readily than a single template alone. Referencing benchmark designs like HarmBench, which places Attack Success Rate (ASR) as an evaluation axis, makes it easier to classify the generated set of cases into three tiers: "success, partial success, failure."

Judging harmful output using simple banned-word matching alone tends to cause both false positives and missed detections. For this reason, combining a judge that evaluates output content using a separate model is a realistic option. However, the judge itself is prone to bias, tending to become overly sensitive or overly dull to certain phrasings or topics. Since fully correcting this bias automatically is difficult, it is necessary to keep human spot-checking in the operational flow.

The basic approach is to use open source and enterprise platforms differently depending on the verification team's scope and budget. In academic verification or PoC stages, the extensibility of open source tools is effective, but for continuous integration into CI/CD or large-scale operations, platforms equipped with dashboard functionality and alert integration are better suited.

The first thing to check when selecting is whether the test target is a single prompt or multi-step agent behavior. This entirely changes the evaluation axis. When verifying agentic AI security, benchmarks that simulate task graphs, such as AgentDojo, are suitable; simple banned-word checks alone cannot capture issues like excessive agency or the confused deputy problem. For vulnerability diagnosis of single prompts, comprehensive scanning like Garak covers most of the initial diagnosis, but the behavior of agents operating across multiple tools can only be seen through task-based scenarios.

Given budget and the team's engineering capacity, a phased introduction—building a foundation with open source first and then expanding to a platform—is also a realistic option.

Which open source tools should be incorporated into your company's LLM operational environment to prevent gaps in vulnerability verification? Choosing the wrong tools can leave verification coverage skewed while only the appearance of "testing having been conducted" remains.

There are mainly three axes for selection. The first is ease of creating custom scenarios. PyRIT has a structure that allows attack templates to be extended in a Python-based framework, making it suited for research and PoC-stage teams that want to incorporate their own prompt injection verification cases. The second is comprehensiveness of known attack patterns; Garak is designed to systematically scan existing attack methods, making it efficient when you want to run through a full pass of typical vulnerability categories such as those listed in the OWASP LLM Top 10. The third is compatibility with benchmarks; using an evaluation framework that combines realistic tasks with security test cases, like AgentDojo, makes it easier to reproduce behaviors such as excessive agency or the confused deputy problem in agentic AI.

If you want to create your own attack scenarios, PyRIT is suitable; if you prioritize comprehensive scanning of known patterns, Garak is suitable; and if you want to verify risks specific to agents, AgentDojo-type benchmarks are suitable. Since all of these are updated in a community-driven manner, checking the repository's update status is essential for operations involving model additions or expansion of attack methods. It's worth noting that continuing to use a tool whose updates have stalled carries the risk of trusting verification results while gaps remain in coverage for new attack methods.

The dividing line between open source and enterprise depends on the scale of deployment and the operational structure.

Once operations enter a phase spanning multiple models and multiple teams, the cost of building test case management and report integration in-house often becomes larger than the tool usage fee itself. Enterprise platforms are characterized by providing, as an integrated package, attack scenario library management, batch execution across multiple models and multiple versions, and vulnerability severity classification through to ticketing.

Differentiating factors on the functional side include presets of attack categories aligned with the OWASP LLM Top 10, integration APIs with existing vulnerability management tools, and long-term audit log retention functionality. Since introduction costs vary depending on the provision format and licensing terms, it is practically important not to judge based on the price list alone, but to compare using total cost of ownership, including the number of models tested, execution frequency, and maintenance structure.

If your organization has personnel capable of designing verification scenarios in-house, combining this with open source tools can sometimes secure flexibility while keeping costs down. At a stage where decision-making material is insufficient, it is realistic to first verify the platform's fit through a small-scale PoC and then determine the contract scale based on that. Since pricing structures can change, obtaining the latest quote before contracting is essential.

If a small-scale, one-off verification is sufficient, manual execution remains acceptable. However, when continuous operation is the premise, integration into a CI/CD pipeline becomes essential. By designing the entire process—from test case generation to execution, reporting, and prioritization—as a unified automated flow, you can build a system in which vulnerability verification runs automatically whenever code changes or model updates occur.

The starting point for designing an automated flow is templating test cases around the categories of the OWASP LLM Top 10 (such as prompt injection and sensitive information disclosure). By referencing attack classifications such as MITRE ATLAS's technical category AML.T0051 (LLM Prompt Injection) and designing the system to automatically generate cases for each input pathway of the target application, you can reduce the gaps and omissions that were previously covered manually.

The flow generally proceeds in the following order:

1. Triggered by changes to code or the system prompt, a test case generation module detects the differences
2. Attack prompts generated using tools such as Garak or PyRIT are executed in batch
3. Responses are judged using a combination of rule-based methods and classification models, tagging results as success or failure
4. Results are passed to a report generation module in a format such as JSON and aggregated by vulnerability type

When turning this series of processes into a pipeline, isolating the execution environment at the container level helps prevent unexpected outputs during testing (such as malfunctions caused by excessive agent permissions) from affecting production systems. When the target of verification is a multi-agent system, it is also useful to include the inter-agent communication pathways discussed in What Is Multi-Agent AI? From Design Patterns to Implementation and Operational Insights within the scope of test generation, as this helps prevent gaps in verification.

When hundreds to thousands of test results come in every night, can you determine which failures should be reviewed first? Without a prioritization mechanism, critical vulnerabilities with high attack success rates can get buried among notifications for trivial formatting errors.

In result aggregation, test case outputs are first classified into three categories—"attack success," "partial success," and "defense success"—and attack success rates are aggregated by OWASP LLM Top 10 category. If this figure worsens compared to the previous run, it is likely caused by a code change or system prompt adjustment, so it is practical to prioritize checking the differences.

For prioritization, evaluating along two axes—scope of impact (sensitive information disclosure versus simple incorrect answers) and reproducibility (whether it reproduces with the same prompt or is a temporary fluctuation)—helps make judgments more consistent. For example, cases where direct injection leads to sensitive information disclosure can be assigned high priority, while minor output fluctuations found in boundary testing can be assigned low priority.

For alert settings, incorporate conditional branching that notifies the DevSecOps team and blocks builds on the CI/CD pipeline when the attack success rate exceeds a predetermined threshold. In practice, it is effective to set the threshold loosely during initial operation and gradually tighten it as you observe trends in false positives.

Decision criteria: As the scale of verification expands, the design of execution speed and result management increasingly determines quality. Let's examine methods for running test cases in parallel to efficiently utilize GPUs and API rate limits, as well as methods for visualizing and continuously tracking the accumulated results.

When hosting models on in-house infrastructure, the focus centers on adjusting GPU memory and batch size, whereas when using external APIs, the focus centers on managing rate limits and cost. For parallel execution, a practical approach is to divide test cases into queues by target model or Agent type, then gradually increase the number of workers to explore the upper limit of throughput. In Gray Swan Arena's large-scale public red teaming, approximately 1.8 million attack attempts were conducted, making it unrealistic to run verification at this scale using an operational setup close to manual execution. From a resource optimization perspective, it is effective to set priorities by attack category (direct injection, indirect injection, jailbreaking, etc.) and feed the highest-severity categories into the execution queue first.

For API-based targets, once the provider's rate limit is reached, failed requests accumulate, making the overall completion time for verification unpredictable. In such cases, combining throttling with a design that retries using exponential backoff when a 429 error occurs is essential. On the other hand, when hosting and verifying a local LLM, GPU occupancy time directly translates into cost, so a practical approach is to maximize throughput through batch inference while running lower-priority test cases whenever spare resources become available. Increasing the degree of parallelism too much tends to increase response latency and timeouts on the target side, so it is important to understand the upper limits for each resource type in advance and adjust gradually.

Simply staring at a large volume of attack test results as raw CSV or logs makes it impossible to determine where to start. Answering the on-the-ground question of "which vulnerabilities should we actually fix this week" requires well-designed visualization.

A dashboard should, at minimum, incorporate the following three axes:

• Time-series changes in Attack Success Rate (ASR): Track deterioration or improvement across releases
• Breakdown by OWASP LLM Top 10 categories: Understand the distribution of issues such as prompt injection or sensitive information disclosure
• Priority classification based on irreversibility and impact scope: Avoid treating minor output disruptions the same as sensitive information leaks

As for conditional logic, a practical decision framework is to treat vulnerabilities with broad impact scope and a rising ASR trend as immediate blocking targets, while routing those with limited impact and low occurrence frequency to the backlog.

Integrating the visualization infrastructure with an existing AI observability environment reduces operational overhead. By putting log collection and dashboard display on the same pipeline, red teaming results and anomaly detection during production operation can be compared on the same screen. This also serves as a handover document between team members, and when reporting to executive leadership, simply showing the ASR trend and category distribution makes it easier to explain the prioritization of countermeasures.

Here we address three questions frequently raised after introducing automation. We provide a concise organization of each: the limits of detection coverage, the timeframe and cost involved in adoption, and the policy for combined use with existing tools such as SAST/DAST.

Decision criteria: Automatically generated test cases excel at comprehensive coverage of known attack patterns and quantitative verification, but have limitations when it comes to detecting unknown, context-dependent vulnerabilities.

Because automatic generation diversifies test cases by combining existing attack dictionaries and templates, it is effective for verifying known categories identified by OWASP, such as prompt injection and harmful output. On the other hand, complex vulnerabilities—such as excessive agency that depends on industry-specific business workflows, or confused deputy problems that only surface through the interaction of multiple agents—tend to be difficult to reproduce with standalone test cases alone.

Additionally, attacks that pass through external documents or RAG retrieval results, such as indirect injection, see reduced detection accuracy unless conditions closely resembling the actual data environment are prepared. Attacks that accumulate information over multiple rounds of interaction, such as system prompt leakage or model extraction attacks, are also areas that are difficult to catch with single-shot test case generation.

For this reason, it is practical to position automation as merely a means of casting a wide net for initial verification, and to combine it with in-depth verification by human red teams with specialized expertise for high-risk scenarios identified through boundary testing. When designing the level of human involvement, the concepts introduced in What is Human-in-the-Loop (HITL)? The Fundamentals of "Human-Participatory" Design for Establishing AI-Driven Business Automation can serve as a useful reference.

When a small team uses an existing OSS tool as a PoC, setup can be completed in a few weeks. However, when scaling to an enterprise level that includes integration with CI/CD pipelines and coordination with existing vulnerability management infrastructure, the process can take several months. The comment "we don't know where to even start" is often heard on the ground during the early stages of adoption, but the key decision criteria are the number of target systems and the scope of test case diversification.

Initial setup involves three processes: selecting the test case generation logic, designing parallelization for the execution environment, and building a dashboard to accumulate results. Adopting open-source tools such as PyRIT or Garak tends to keep initial costs low, but requires more time for adjustment work to align with your organization's LLM configuration. Conversely, enterprise platforms tend to have higher adoption costs, but since templates and reporting features are typically built in as standard, the time to operational launch can sometimes be shortened.

Operating costs vary depending on the number of target models, testing frequency, and consumption of computing resources such as GPUs. When continuous verification is the premise, it is advisable to incorporate operational design—including resource optimization—from the initial stages. For specific costs, it is necessary to check each vendor's latest pricing page.

Traditional security testing tools such as SAST (static analysis) and DAST are optimized for detecting code vulnerabilities and known attack patterns in web applications, and the actual subject of verification differs fundamentally from AI red teaming. SAST statically identifies risks such as injection vulnerabilities and authorization gaps in source code, while DAST verifies vulnerabilities by sending actual requests to a running application. AI red teaming, on the other hand, targets vulnerabilities stemming from the model's behavior itself—such as an LLM's response behavior, including its resistance to direct injection or the potential to induce harmful output through jailbreaking.

In practice, it is rational to use both in combination. A reasonable division of roles is for SAST/DAST to verify application-layer code quality and API boundary security within the CI/CD pipeline, while an automated AI red teaming flow runs in parallel to verify prompt-based attacks and system prompt leakage. SAST/DAST alone cannot detect risks specific to the OWASP LLM Top 10, such as improper handling of LLM-generated output or excessive agency. Conversely, AI red teaming is not well-suited for verifying vulnerabilities in the application infrastructure itself. Combining both makes it possible to build a verification system that covers both the code layer and the model behavior layer.