How Safe Is AI Pentesting? Measuring ScopeWeaver's Isolated Lab and Its Challenges

In recent years, there have been reports of AI pentest tools and autonomous AI agents being misused in cyberattacks involving information theft. Anthropic's September 2026 threat report introduces cases in which attackers used public frameworks such as PentAGI, as well as cases in which AI was used to obtain credentials and exfiltrate data. AI-driven vulnerability discovery and tool execution are technologies that help the defending side with verification, while also being usable by the attacking side.

These are misuse cases observed by the reporting source, and do not indicate that ScopeWeaver was used in these attacks. Even so, those who introduce AI pentesting for defensive purposes also need to manage where the tool connects and what it can execute.

When evaluating pentesting by autonomous AI, in addition to whether it can find vulnerabilities, it is necessary to verify whether it can stop communication to out-of-scope targets, whether it can reliably terminate processing, and whether its reports can be cross-checked against evidence.

We ran ScopeWeaver v0.9 in an isolated verification environment and had it investigate an order API we created for testing. Both a local Qwen and GLM on Amazon Bedrock detected an authorization flaw that allowed retrieval of other users' orders. However, the AI's reports contained miscounted request numbers and definitive claims about areas that had not actually been examined, so the quality was not sufficient to submit as-is.

This article explains the design of the isolated lab, the measured results, and the remaining issues, for information systems and security personnel considering the adoption of autonomous AI. The measured values are based on our own verification records, and are not a benchmark indicating general detection rates or superiority/inferiority among models.

ScopeWeaver is an autonomous AI pentesting platform in which an LLM uses tools to carry out investigations. In the evaluation, we separately checked the accuracy of the AI's answers and the scope of operations the tools could execute.

Pentesting (penetration testing) is a form of verification that examines whether a vulnerability can actually be exploited. With autonomous AI, the model itself also decides what to investigate next and which tools to call. Compared to a case where a human operates step by step, incorrect judgments can more directly lead to external requests or file operations.

ScopeWeaver is an open-source investigation platform built on ARTEX. It uses Bash, Python, and tools via MCP, connecting the model's judgments to command execution. The configuration and terms of use can be found in the official README.

In ordinary chat, the main thing to check is errors in the answers, but for AI equipped with tools, the permissions of the executing user, the destinations it can communicate with, and the files it can read and write are also subjects of evaluation. Input information and tool output may also remain in logs or in content sent to the model.

Instructing it to "only investigate this URL" does not mean that OS-wide communication restrictions have been configured. Managing targets within the app is helpful, but it does not guarantee that a different execution path will pass through the same constraints. This time, we limited the permitted communications even outside the app, creating a configuration in which it is difficult for the scope to expand even if the AI misinterprets its instructions.

When evaluating the automation of penetration testing, it is necessary to check each of the following separately: instructions given to the model, controls on the application side, and controls on the OS/network side.

Before deployment, we conducted an intensive review of the source, which included 1,444 tracked files. We found no evidence that could be conclusively identified as an intentional backdoor, but the following issues remained. This does not mean that all files were audited at the same depth.

• The Bash, Python, and MCP tools use the executing user's permissions and network access
• The web-fetching process has a retry path that does not go through the logging proxy
• There is a location where the working-folder boundary check does not resolve symbolic links, which allowed reference to a verification file placed outside the working folder within the VM
• Items such as authentication token revocation, CORS, and login rate limiting require additional consideration if public deployment is assumed

The ScopeWeaver version used in verification was pinned to commit 08d48f8 in the v0.9 series. In the build copy, the go.mod and go.sum dependencies were updated. After updating, govulncheck found zero known vulnerabilities in the called paths and imported packages. However, this does not guarantee the absence of design flaws or unknown vulnerabilities.

In addition, the official README separately states the AGPL-3.0 license and usage restrictions specified by the author. The latter assumes research and technical verification in a local, isolated environment, and does not permit pentesting of live systems or production use. Even for targets for which you have the rights holder's permission, do not assume that you can use it while skipping over this statement. If you distribute a modified version to others, you must also check the licensing conditions, including the provision of corresponding source code.

We divided the roles of inspection, inspection target, and relay across three virtual machines, and in principle denied all communication outside the VMs. What was permitted was limited to the target API necessary for verification and a fixed inference path, among a few other things.

The isolated lab referred to here is a verification environment that does not connect to any real business system and handles only mock data. In case the settings within the VMs were to change, we also layered access control on the Hyper-V side.

We used Windows 11 Pro as the virtualization host and a DGX Spark with 128GB of memory for local inference. The memory allocated to the three VMs totaled 13GiB, and the virtual CPUs totaled 7.

We used a Private switch for the connection between the scan VM and target VM, and an Internal switch for management. On top of that, we configured Hyper-V's extended ACLs to deny anything other than the allowed destinations, communication directions, and TCP ports.

We did not set a default route to the internet on any VM, and also disabled IPv6. Management SSH is restricted to inbound access from outside via the gateway VM only. The dedicated inference path is used only when using GLM, and the configuration does not allow the scan VM to connect to arbitrary cloud services.

The purpose of separating roles was to make it easier to understand connection destinations and what to stop. Isolation strength is not determined by the number of VMs alone; it requires actually verifying which paths are allowed and denied.

We verified the signature and SHA-256 of the VM's initial image, and separated it into a read-only parent disk and differencing disks for each VM. During setup, we checked for overlaps with existing VMs, virtual switches, subnets, and listening ports, and configured the system to halt if a conflict was found. MAC address spoofing was disabled, and both DHCP Guard and Router Guard were enabled.

ScopeWeaver was built for Linux using Go 1.27, and we saved the SHA-256 of the deployed binary. On the scan VM, it was launched under a dedicated non-root user, with no sudo privileges granted. We used systemd to restrict write destinations, additional Linux capabilities, memory, CPU, and process count.

For shutdown, we used KillMode=control-group, designed to terminate subordinate processes along with the service. Automatic restart is also disabled, to prevent resumption immediately after a stop. Whether the task finishes on screen and whether the executing process remains are checked separately.

PostgreSQL does not listen on TCP; it uses peer authentication over a Unix socket, and the DB user is not granted administrator privileges either. The management screen only listens on the VM's loopback interface, and we operated it via SSH forwarding. In this way, we separated access to the management screen from the communication privileges used by the AI's tools.

In the evaluation, we fixed the inference settings and prepared a small order API with one type of authorization flaw. Cases where the source code was provided and cases where it was not were recorded as separate conditions.

By deciding in advance which flaws should be found and what a normal response should look like, we made it possible to judge results without relying solely on the AI's own explanations.

For local inference, we used a derivative model of Qwen's 27B model quantized with Q8. The model identifier used during the evaluation was huihui_ai/Qwen3.8-abliterated:27b-q8_0. The results below apply to this model and this configuration, and are not an evaluation of Qwen's overall performance.

On the DGX side, we set up a lab-dedicated Ollama service, and had the inference server listen only on loopback. We set OLLAMA_NO_CLOUD=1 to disable cloud features, and had it reference the model directory as read-only.

At the gateway, we fixed the model name and the digest identifying the model, and rejected the model fetch, create, and delete APIs. The request body is capped at 1MiB, and the response at 4MiB. Even if only the model name is aligned, if the input length or output limit changes somewhere along the relay, the conditions will not match.

Since the initial 8k context did not satisfy the operating conditions for ScopeWeaver this time, we aligned the inference server, gateway, and application all to 32k. For reproducibility, it is necessary to record the settings of the entire path along with which model was used.

The target under inspection is a mock API that receives an order ID and returns order information. We set a business rule stating "user alice can only read her own orders." On the vulnerable path, alice can also retrieve bob's order, while on the comparison path with corrected authorization, this is denied.

This flaw corresponds to BOLA (Broken Object Level Authorization). It is a state in which, beyond just checking whether the user is logged in, there is no verification of whether that user is permitted to read the specified order. OWASP's explanation of BOLA also emphasizes the importance of verifying access rights to data specified by ID.

For judgment, we did not treat HTTP 200 alone as evidence; we also checked the owner in the response body. We also examined normal use by the legitimate user, retrieval of another user's data, denial on the fixed path, and 404 for nonexistent IDs, cross-referencing these with the target's HTTP logs.

In the no-source condition, we provided the target URL and the business rule. In the with-source condition, we additionally provided the mock API's source code, and had flaws inferred from the code reproduced via HTTP as well. By separating the conversation, the evidence, and the aggregation, we ensured that results aided by the source were not mixed into the no-source results.

Note that "no source" does not mean "no credentials and no business rules either." This evaluation was conducted under fixed mock-user conditions, and it was not a test that broke the login mechanism itself.

Qwen obtained HTTP evidence for the same single type of BOLA in all 7 attempts. This is not a result of discovering 7 different vulnerabilities.

On the other hand, errors remained at the stage of summarizing detection findings into text. We evaluate detection success and reporting accuracy separately.

Each attempt had a time limit, and we enforced a limit of 1 request per second, up to a maximum of 100 requests, in front of the target VM. This was a configuration where, in addition to instructing the AI to reduce requests, an upper limit was also set externally.

The source is our verification records. In all attempts, HTTP evidence was obtained for all four conditions: retrieving one's own orders, retrieving other users' orders, rejection of other users' orders via the patched route, and normal use of one's own orders via the patched route.

The average for A1–A3 was approximately 178 seconds, and the average for B1–B3 was approximately 112 seconds. For this target, the with-source condition took less time, but we do not calculate a general time-reduction rate. This is because these are repetitions using the same model, the same small target, and the same data, not measurements that varied the type or difficulty of the flaws being searched for. R1 is excluded from the average as a separate condition.

The fact that there were zero tool errors does not mean that all searches or judgments were correct. The success or failure of API calls and the adequacy of the investigation scope are separate evaluation items.

The report created by the AI contained descriptions that did not match the evidence. For example, there were errors such as writing "approximately 18 requests" when the target log showed 21 requests, or writing that a route's response was 404 when it was actually 200.

There were also cases where, despite having confirmed only a few IDs, the AI asserted definitively that "there are only 2 orders," and cases where it concluded that "no other endpoints or authentication methods exist" based solely on the scope explored via HTTP. Treating unexamined scope as "nonexistent" causes users to perceive the inspection scope as broader than it actually was.

We retained the AI's original text and saved the results of cross-referencing it against the target's HTTP logs in a separate review record. This is because if only the erroneous text is corrected and the original text is deleted, it becomes impossible to trace later where the judgment diverged. Errors in the report, including hallucinations, are also recorded separately from the detection results.

Additionally, expressions that describe a retry within the same conversation as an "independent reproduction" are also subject to correction. R1 re-acquired evidence without reading the original conversation or report store, but the model and data were the same. This does not constitute evidence as strong as independent verification by a different operator or a different model.

For the 14 prohibited routes confirmed this time, communication timed out even when defenses within the VM were temporarily disabled. We also confirmed termination of descendant processes upon stop, and restoration of the target VM.

For the communication verification, we temporarily disabled the nftables restrictions within the VM and added a default route. Even so, the prohibited routes were blocked by the Hyper-V ACL outside the VM, while the permitted route to the target HTTP and the fixed LLM passed through. After completion, the settings within the VM were restored to their original state.

What this result shows is that the external controls functioned for the 14 routes tested. This is not an evaluation that covers all bypass routes or escape techniques from the virtualization infrastructure. We recorded "communication restrictions were configured" and "blocking was verified for the intended routes" as distinct items.

Regarding stopping, we confirmed that descendant processes launched from ScopeWeaver's custom tools were terminated along with the stopping of the systemd service. Auto-start was disabled for the VMs, and a restoration checkpoint was created with all three VMs in a stopped state.

Restoration was actually attempted only for the target VM. We confirmed that a marker file created in advance disappeared after restoration, and that the target service was restored. Checkpoints also exist for the inspection VM and the relay VM, but actual restoration verification was not performed for these. An operational caution point is not treating the creation of a restore point as equivalent to the completion of a restoration test.

The cloud-based GLM was also able to detect the same authorization flaw in the simulated API. However, because the number of attempts and execution limits differ from those for Qwen, this cannot be used for comparing model performance.

For the cloud connection, we designed the system so that AWS credentials were not passed to the inspection VM. On the other hand, the prompts and tool results needed for inference are sent to AWS. The location where credentials are stored and the destination to which data is sent need to be considered separately.

This time, ScopeWeaver's OpenAI-compatible connection assumed Bearer authentication and was not configured to directly perform SigV4 signing for AWS. Therefore, we placed a relay with a fixed destination on a Mac, where we handled the authentication process for AWS.

The model identifier specified in the verification was global.zai.glm-5.3. According to AWS's official specification for the target model, Chat Completions, streaming, and client-side tool calling are supported. The reasoning strength is configurable, and this time it was fixed at reasoning_effort=medium.

The relay connects only to the designated API and model on Bedrock. We set upper limits of 128KiB maximum input, 4,096 tokens maximum output, concurrency of 1, a maximum of 32 calls, a 60-minute deadline, and a reasoning budget of 1 USD. Requests to arbitrary URLs, other models, or unauthenticated requests are rejected.

Even when usage cannot be determined due to failures or disconnections, the design does not readily refund the reserved budget. This is because even if no response is received, processing may still proceed on the provider side, incurring costs. This limit management was implemented in this relay for this occasion, and was not evaluated as a standard feature of ScopeWeaver.

First, without going through ScopeWeaver, we verified normal responses, tool calls, the final answer after returning a simulated tool result, streaming, and continuation of reasoning. Requests to other models, other API paths, and unauthenticated requests were rejected.

Next, when instructed by ScopeWeaver to verify 4 routes, GLM called the Bash tool 3 times and produced a report in Japanese in 18 seconds. However, the target side recorded 6 HTTP requests. This was because it first retrieved only the response code, then retrieved the body again to check the owner. Although it stayed within the outer request limit, it failed to strictly comply with the instruction of "4 requests."

Afterward, under a no-source condition where a URL and business rules were given, we conducted 1 trial including exploration. The result was 125 seconds, 34 HTTP requests, 15 LLM calls, and 0 tool errors. GLM explored order IDs and reported retrieving its own order, retrieving another person's order, and being denied on the patched route. These matched the target's HTTP logs.

On the other hand, there was also duplication where the same order on the patched route was retrieved 4 times. This behavior was not observed in Qwen's record this time, but since the number of trials is small, it cannot be concluded as a general tendency of the model.

The 18 seconds for the connection check and the 125 seconds for the trial including exploration are separate measurements. Also, the request limits, output limits, and number of executions differ from Qwen. We do not simply line up the times and conclude that GLM is faster, or that Qwen is superior.

Even if the connection destination visible from the inspection VM is set to loopback, if the relay beyond it forwards to the cloud, the processing does not complete locally. This time, the configuration is such that prompts, tool definitions, and tool results are sent to AWS.

local_llm_only was maintained as a connection restriction on the inspection VM side, but it is not a setting that guarantees the data location of the entire route. When considering a design that handles internal information or real data, it is necessary to check the destination after the relay, not just judge based on the app's setting name alone.

According to AWS's official specification, the model used does not support In-Region inference, which fixes processing within a single region. This time, since we used the Global cross-Region inference profile from the Tokyo endpoint, this is not a configuration that limits the processing region to Japan.

Regarding cost, based on conservative calculations on the relay side, the first measurement frame including the connection check was approximately 0.052 USD, and the record including the no-source trial was approximately 0.356 USD, both within the set 1 USD limit. These are calculated values for limit management based on published unit prices, and are not confirmed amounts from an AWS invoice. Since they include measurement frames and reserved amounts, they cannot be directly converted into a per-diagnosis unit price either.

Prices and provided regions of external services may change. When using them, please check the model specifications and Amazon Bedrock pricing.

These results alone cannot determine the safety of production use, the general vulnerability detection rate, or reproducibility in actual projects. What was verified was a small case in which a fixed simulated user reads an order via GET.

The target is one type of authorization flaw, and does not include complex authentication or state changes. The fact that one authorization flaw was detected does not mean that other types of vulnerabilities, such as SQL injection, can also be detected. Long-running autonomous execution and comparison with other tools have not been evaluated either.

For the next stage, we prepared a target including the following conditions, along with an HTTP verification script that judges results separately from the AI. We distinguish between the preparation of implementation and the completion of trials, and treat the latter as an evaluation item not yet conducted here.

When evaluating updates or deletions, the policy is to check the state before and after the change not only from the response of the user who performed the operation, but also from the legitimate owner. No-source and with-source conditions will continue to be separated, and when comparing between models, the prompts, limits, and number of trials will be aligned. Only by aligning the conditions can we consider where the differences originate from.

There is a difference between verifying operation in an isolated environment and judging that it can be used for business purposes. Regarding this result, we organize the scope of use, model comparison, and handling of the report.

From this verification, it cannot be concluded that it can be used as-is. Even when a target is specified within the app, this does not translate into permissions for the execution tool or restrictions on the entire network.

In addition, the official README states usage restrictions that disallow pentesting on real systems or production use. Adding isolation settings does not remove that restriction. What this article covers is a technical verification using a mock API that does not connect to a real system.

When designing a verification environment for autonomous AI, decide on the communication destinations, execution permissions, stop method, restoration method, and evidence storage location, and confirm the actual behavior.

These results do not allow a judgment of superiority or inferiority. Although both obtained the same HTTP evidence of the authorization flaw, the number of attempts, request limits, output limits, and so on were not aligned.

When making a selection, in addition to detection results, it is necessary to compare the availability of external transmission, the computing resources that can be operated, duplication of requests, the accuracy of reporting, and the cost ceiling. The model name or the time required for a single run alone cannot determine suitability for business use. To measure performance differences, multiple verifications with matched target data and execution conditions are required.

This time, the report was not of a quality that could be submitted as-is. This was because, in addition to errors in the number of requests and response codes, there remained definitive conclusions about areas that had not been investigated.

Before submission, the reproduction steps, requests and responses, affected users/data, the confirmed scope, and the unconfirmed scope should be cross-checked against the target logs. In particular, a conclusion such as "there are no other problems" requires support from the scope that was actually confirmed.

By storing the AI's original text and the human corrections/judgments separately, the basis and errors of the report can be traced. Succeeding in detecting a vulnerability and producing a report that fulfills accountability are separate evaluation criteria.

In an isolated lab using ScopeWeaver v0.9, both the local Qwen and the cloud GLM were able to obtain evidence of an authorization flaw set up in a mock order API. At the same time, it was also confirmed that restrictions on communication destinations, stopping, restoration, and report verification need to be prepared separately from the instructions given to the AI.

What can be taken away from this case is less about the superiority of a specific model and more about how to proceed with verification. Decide on the target and business rules first, collect evidence for both normal and abnormal cases, distinguish whether sources exist, and cross-check the AI's report against the target logs. It is also important to actually test stopping and restoration, and to record the scope that was confirmed versus the scope that was not.

What has been confirmed at this point is the behavior within this isolated environment and with a small target. These are not results that demonstrate a general detection rate, reproducibility in real-world cases, or safety for production use. When starting your own verification of having an LLM use tools, first narrow the target to a scope where communication, permissions, stopping, and evidence can be managed, and then gradually expand the evaluation conditions.