How to Compare and Select AI Cybersecurity Evaluation Benchmarks

AI cybersecurity evaluation benchmarks are assessment standards designed to quantitatively measure the security strengths and weaknesses of AI systems under standardized conditions.

Traditional software security assessments centered on vulnerability scanning based on CVE (Common Vulnerabilities and Exposures) and Penetration Testing. However, as Generative AI and AI agents have become integrated into business systems, AI-specific attack methods have emerged—such as Prompt Injection, Jailbreak, and Data/Model Poisoning—making it increasingly difficult to address these threats with conventional evaluation methods alone.

Against this backdrop, dedicated benchmarks targeting AI behavior are being developed. A prominent example is CyberGym, presented in arXiv:2506.02548. CyberGym is a large-scale benchmark containing 1,507 real-world vulnerabilities and 188 software projects; even the top-performing methods achieve a success rate of only around 20%, illustrating just how difficult autonomous vulnerability discovery by AI remains.

The role of benchmarks can be broadly organized into three areas.

The methods used in AI-powered cyberattacks have rapidly grown more sophisticated over the past few years. Threats such as automated penetration testing, Prompt Injection exploiting LLMs (Large Language Models), and zero-day vulnerability discovery are increasingly difficult to address with conventional rule-based detection.

In response to these changes, evaluation benchmarks are attracting growing attention as an objective means of measuring the security resilience of AI systems themselves. This trend is driven primarily by three challenges:

• Subjectivity in evaluation: Assessment results vary depending on the experience and skills of security personnel, making cross-organizational and cross-product comparisons difficult.
• Diversification of attack methods: AI-specific attack vectors—such as Jailbreak, RAG poisoning, and model extraction attacks—are on the rise, creating areas that cannot be fully covered by conventional security evaluation frameworks alone.
• Strengthening of regulatory and governance requirements: Regulation (EU) 2024/1689 (EU AI Act) entered into force in August 2024, requiring high-risk AI systems to achieve compliance within 36 months of that date. Whether organizations can demonstrate objective evaluation evidence has become a critical compliance issue.

Furthermore, in the absence of a shared "common language" for AI security evaluation, there is an increased risk that vulnerabilities overlooked at the PoC (Proof of Concept) stage will surface in production environments.

When selecting a benchmark, it helps to organize your thinking around three axes: what is being measured, how it is measured, and whether the results are reliable. Let's work through these in order: scope of evaluation, scoring methodology, and reproducibility and update frequency.

Before selecting a benchmark, it is essential to clarify "what you want to evaluate." The scope of evaluation falls into three broad categories, and the appropriate benchmark will differ depending on your objective.

Vulnerability detection measures whether an AI can analyze source code or binaries to discover known vulnerabilities—such as those catalogued in CVE (Common Vulnerabilities and Exposures)—as well as unknown defects. CyberGym contains 1,507 real-world vulnerabilities and 188 software projects, giving it a scale well suited to evaluating detection accuracy. However, since even top-performing methods achieve a success rate of only around 20%, it is currently best suited for the purpose of "measuring capability as an assistive tool."

Defense evaluation verifies resistance to Prompt Injection and Jailbreak attacks, as well as the effectiveness of AI Guardrails. Whether a benchmark covers the threat categories defined by the OWASP Top 10 for Large Language Model Applications is one criterion for selecting a defense-oriented benchmark.

Attack simulation evaluates whether an AI can perform Penetration Testing and exploit code generation from an attacker's perspective.

"If a benchmark score is high, does that really mean it's secure?"—this is the first obstacle evaluators typically encounter. To correctly interpret what a score means, it is necessary to understand the definitions of the metrics and the structure of the scoring system.

The primary metrics used in AI cybersecurity evaluation benchmarks are as follows:

Scoring methods vary significantly across benchmarks. Some use simple Accuracy, while others apply weighting based on task difficulty or vulnerability severity (e.g., CVSS scores). It is important to note that comparing unweighted accuracy alone can result in benchmarks with a higher proportion of easy tasks receiving an unfairly high evaluation.

In the case of CyberGym, the success rate of top-performing methods stands at approximately 20%, a figure that reflects the high difficulty of the benchmark, which targets 1,507 real-world vulnerabilities.

The quality of a benchmark can be assessed along three axes that ask not "what is the score?" but "how trustworthy is that score?"

Reproducibility refers to whether the same results can be obtained when the benchmark is run under the same conditions. Benchmarks with undisclosed test cases or prompts are difficult to verify externally, and the meaning of their scores tends to be ambiguous. CyberGym published its test design in an arXiv paper (arXiv:2506.02548), allowing third parties to review the evaluation conditions covering 1,507 real-world vulnerabilities and 188 software projects. This transparency is what gives the reported success rate of "approximately 20%" for top-performing methods its credibility.

Transparency requires checking whether the evaluation dataset's sources, licenses, and exclusion criteria are clearly stated. Just like a food ingredient label, a benchmark whose contents are hidden cannot serve as a basis for safety judgments. Frameworks such as MITRE ATLAS™ that publicly map tactics, techniques, and procedures (TTPs) are particularly valuable because they allow evaluation results to be directly linked to security improvement efforts.

Update frequency determines whether a benchmark can keep pace with evolving threats. Since vulnerability information is continuously added as CVEs (Common Vulnerabilities and Exposures), static datasets drift further from reality over time. The NIST AI RMF Playbook is scheduled for semi-annual updates (approximately twice per year), making a framework's update cycle one of the criteria to consider during selection.

Benchmark evaluation proceeds in three steps: environment preparation, execution and data collection, and results analysis. Errors in procedure at any step will undermine the reliability of the comparison results. Work through each step in order to ensure the evaluation aligns with your organization's objectives.

Benchmark results can vary significantly depending on how the environment is configured. Whether to replicate conditions close to a production environment or to prioritize safety within an isolated sandbox depends on the purpose of the evaluation.

Deciding on an environment isolation policy

If you want to measure the accuracy of vulnerability detection, it is effective to reproduce a stack that closely resembles your actual development environment. On the other hand, evaluations that include attack simulations require a sandbox environment that is isolated from the production network. For benchmarks like CyberGym that target 1,507 real-world vulnerabilities, a container infrastructure with snapshot capabilities that allows the target software projects to be safely deployed and discarded is a prerequisite.

Dependency and version management

Build the environment with the versions of the AI models and analysis tools under evaluation pinned. If automatic library updates are left enabled, results may change on re-execution, making comparison impossible. Manage dependencies explicitly using requirements.txt or a Dockerfile to ensure environment reproducibility.

Configuring log collection

It is important to decide on the log verbosity level and storage destination before execution begins. To prevent situations where "the score from that run was never saved," set up a mechanism in advance to capture standard output, error logs, and model response logs together. When storage capacity is limited, a practical approach is to configure log rotation on a per-evaluation-session basis.

"I ran the benchmark but wasn't sure what to record or how much to capture" — this situation commonly arises in the early stages of an evaluation project.

The first task is to define the scope of log collection before execution begins. There are three broad categories of items to record:

• Execution logs: Input prompts for each task, model outputs, execution timestamps, and latency
• Scoring results: Comparison against ground-truth labels, attack success rate (ASR), and whether partial credit applies
• Environment metadata: Model name and version, inference parameters (temperature, maximum token count), and the API endpoint used

For real-world vulnerability-based benchmarks like CyberGym, running all 1,507 tasks takes considerable time and cost. For an initial evaluation, it is practical to narrow the scope and sample by CVE category or difficulty tier.

A key concern during data collection is ensuring reproducibility. Run the same task multiple times to observe output variance, and record results under both fixed and varied temperature parameter conditions; this will make your comparison baseline clear during later analysis.

Some benchmarks also include qualitative judgments such as "incomplete patch" or "zero-day detection." In these cases, it is recommended not to rely solely on automated scoring for model outputs, but to document the judgment criteria and combine automated scoring with manual review. Store collected data in a structured format such as CSV or JSONL so that it can be readily passed to downstream analysis tools.

Simply listing collected scores side by side yields no meaningful insight. Just as with sports results, a valid comparison requires aligning the context of "who measured what, under which conditions."

The first step in analysis is score normalization. Because the definition of a perfect score and the difficulty distribution vary across benchmarks, compare relative rankings or rates of change on a common task set rather than absolute values. In the case of CyberGym, it has been reported that even top-performing methods achieve an attack success rate (ASR) of only around 20%, meaning a high score does not necessarily imply "sufficient for practical use." It is important to weigh this figure against your organization's acceptable risk threshold.

Next, perform classification of failure patterns. Tallying results separately into false positives, missed detections, and incomplete patches reveals the model's weaknesses. CyberGym recorded 18 incomplete patch detections, so care must be taken not to overlook the residual risk present even in cases judged as "patched."

When creating a comparison table, aligning the following axes as columns makes decision-making easier:

Finally, it is essential to avoid making judgments based on a single metric. Even if ASR is high, a high false positive rate will drive up operational costs significantly.

Selecting a benchmark based solely on technical performance can lead to situations where operations break down after implementation. It is important to evaluate options along two axes: compatibility with your organization's use cases, and cost versus operational burden.

Before choosing CyberGym, it is important to first articulate "what you want to measure." If the evaluation criteria provided by the benchmark do not align with the challenges your organization faces, even a high-precision score will not be useful for practical decision-making.

CyberGym is a research benchmark covering 1,507 real-world vulnerabilities and 188 software projects, with strengths in evaluating offensive security capabilities such as vulnerability discovery, patch verification, and zero-day detection. As a result, its suitability varies significantly depending on the use case.

• If you want to measure the accuracy of vulnerability research or automated patch verification, CyberGym's evaluation criteria directly correspond to these needs, and a high degree of compatibility can be expected.
• If your primary objectives are network defense, incident response, or compliance audit automation, combining frameworks that systematize defensive tactics and techniques—such as MITRE ATLAS™ or OWASP Top 10 for Large Language Model Applications—will yield a more realistic evaluation.

Additionally, CyberGym is currently a research-stage benchmark. If you are planning direct application to a production environment, it is advisable to validate with a limited scope during the PoC (proof of concept) phase and confirm whether scores correlate with actual detection accuracy before making a decision.

Your organization's security maturity is also a factor in assessing compatibility. Organizations with a dedicated in-house security team can more readily leverage detailed evaluation results, whereas organizations with limited security expertise tend to face higher costs in interpreting those results.

"We want to adopt a benchmark, but have no idea how much effort and cost it will require" — this is a common concern in the early stages of an evaluation project.

Costs broadly fall into two layers: "environment setup costs" and "ongoing operational costs."

• Environment setup: Procurement of GPU instances, preparation of isolated environments for systems under test, and licensing fees
• Ongoing operations: Keeping up with periodic benchmark updates, re-running score measurements, and personnel costs for results review

Large-scale benchmarks like CyberGym, which cover 1,507 real-world vulnerabilities, require substantial GPU resources just to prepare the execution environment. Given that the success rate of top-performing methods currently stands at around 20%, it is important to note that computational costs accumulate as the number of trials increases.

From an operational burden perspective, the skill set of the personnel involved is a critical variable.

For small teams, a practical approach is to first conduct a PoC with a limited scope, measure the actual effort required, and then decide whether to proceed with full-scale adoption.

In benchmark selection, certain pitfalls tend to recur. Being aware of them in advance can significantly reduce rework after implementation.

Pitfall 1: Selecting based on scores alone

Choosing a benchmark simply because it shows high numbers for attack success rate (ASR) or detection rate is meaningless if its evaluation scope does not align with your organization's use cases. For example, it is common to see organizations adopt a benchmark specialized in traditional CVE-based vulnerability detection when what they actually want to validate is defenses against RAG poisoning or prompt injection. Developing the habit of writing down "what you want to measure" in a single sentence before selection is an effective practice.

Pitfall 2: Underestimating the reproduction environment

Benchmarks that handle real-world vulnerabilities, such as CyberGym, cannot be executed without a secure, isolated environment. There are cases where teams run them in near-production environments on a "let's just try it" basis, inadvertently creating unintended risks. Be sure to include the cost of preparing the evaluation environment in your scope before deciding whether to proceed.

Pitfall 3: Treating evaluation as a one-time exercise

AI threats evolve continuously. Knowledge bases such as MITRE ATLAS™ are also regularly updated, and models that scored highly six months ago may not be equipped to handle new attack techniques. Rather than conducting a one-off evaluation, it is important to incorporate a regular re-evaluation cycle into your operational plan.

AI cybersecurity evaluation benchmarks serve as a common measure for objectively assessing vulnerabilities in AI systems and prioritizing improvements. Benchmarks including CyberGym each differ in their evaluation scope, scoring methodology, and update frequency, making it important to select one that aligns with your organization's objectives.

The criteria for selection can be organized into the following three points:

• Alignment with objectives: The appropriate benchmark varies depending on whether the emphasis is on vulnerability detection, defensive evaluation, or attack simulation.
• Reproducibility and transparency: Verify whether the evaluation environment can be reconstructed and whether the basis for scores is disclosed.
• Cost and operational burden: Estimate in advance not only licensing fees, but also the effort required for environment setup, data collection, and results analysis.

A common pitfall is selecting a benchmark based solely on high scores, only to later discover a gap with the organization's actual environment. It is publicly known that even top-performing methods in CyberGym achieve a success rate of only around 20%. Rather than placing excessive trust in the results of a single benchmark, it is essential to combine multiple metrics when making judgments.

For practical implementation, a sound approach is to first build an evaluation environment through a small-scale PoC, analyze the results, and then decide whether to proceed with full-scale adoption. Clarifying your objectives and risks, then validating on a small scale, is the most direct path to keeping operational costs manageable after deployment.