Validating AI Pentesting with ARTEX: Results and Limitations Found with Juice Shop

AI pentesting is a mechanism in which AI devises verification procedures, sends requests using tools, and examines the weaknesses of a web application based on the responses. At our company, we have deployed the first-generation ARTEX locally and are conducting verification using the security training application "OWASP Juice Shop."

OWASP Juice Shop challenges are rated by difficulty using stars, on a six-level scale where ★1 is the easiest and ★6 is the most difficult. In this verification, challenges of difficulty ★1 to ★3 were solved smoothly, while for more difficult challenges, how the task goal was set determined whether progress was made. With focused goal-setting, we obtained successful results in around 10–15 minutes for NoSQL-related challenges, and about 40 minutes for "Multiple Likes," the highest difficulty level at ★6.

This article presents the results by challenge and the goal-setting approaches used, and organizes the design lessons learned from issues related to input handling and concurrent updates. We separate the cases where high-difficulty challenges were successfully handled from the areas still under verification, and consider how AI pentesting should be evaluated in practical use.

We deployed the first-generation ARTEX in a local environment and used GLM-5.3 as the reasoning model. The target was a Juice Shop verification environment prepared by our company. The "local deployment" referred to here concerns the placement of ARTEX itself, and does not mean local inference of the LLM.

ARTEX is a penetration testing platform in which multiple LLM-based agents divide and carry out tasks. Penetration testing refers to verification that examines weaknesses through operations simulating an attack, and confirms what kind of impact may result. A key feature is that, rather than merely proposing countermeasures in text, the AI repeatedly performs actual tool operations and checks the results.

In this verification, agents were made to investigate the target, execute requests with varied inputs and submission conditions, and compare the results. Humans monitored progress and narrowed down the task goals, clarifying which functions or vulnerabilities to verify. This was not a trial in which all challenges were solved autonomously based solely on an initial broad goal; rather, it is an operational case that also involved human goal-setting.

Specifying the verification target, preparing the execution environment, and judging whether a result should be treated as a vulnerability all remain areas for which humans are responsible. In addition, this verification used a configuration based on ARTEX for testing purposes, and is not a trial that measured the performance or safety of the standard distribution alone. The results should be regarded as having been obtained through the combination of this configuration, model, and goal-setting.

OWASP Juice Shop is a web application with intentionally built-in weaknesses, designed for learning about vulnerabilities and verifying security tools. It is well suited as material for examining flawed implementations and linking behavior to underlying causes.

However, the number of practice challenges solved cannot be used to calculate a detection rate for real-world services. The solutions to the challenges are publicly available, so it is possible that the AI made use of existing known knowledge. The ability to discover unknown weaknesses and the ability to understand complex business specifications need to be evaluated separately.

The fact that the training material is publicly available is also distinct from whether it is acceptable to test any arbitrary site on the internet. When trying this out, use a target for which the permission and scope of execution are clear, such as a verification environment you have set up yourself. The results covered in this article are also those obtained in an environment that satisfies these conditions.

In addition to challenges of difficulty ★1 to ★3, successful results were obtained for the high-difficulty NoSQL-related challenges and for Multiple Likes, the highest difficulty level at ★6. For the high-difficulty challenges, the turning point was narrowing down the task goal for verification, rather than having the AI broadly investigate the target.

For difficulty ★1 to ★3, the challenges attempted within the scope of this effort were successfully solved without issue. However, this does not mean that all challenges at each difficulty level were completed. The results for the high-difficulty challenges are shown below, along with the difficulty level according to the official current guide.

The star ratings follow the notation in the official NoSQL challenge list, Multiple Likes, and the RCE DoS challenge list. The conditions for using these challenges need to be checked depending on the version and settings used for verification. In particular, the official guide notes that Successful RCE DoS cannot be used in a Docker environment, so the current environment adjustment work is not included among the successful results.

These times are approximate figures from this particular run. They are not a guarantee that the same amount of time will be needed in a different environment, or that all challenges of the same difficulty level can be solved. When making comparisons, the starting point of measurement, the carryover of prior investigation, and how environment adjustments are handled must all be aligned.

Being able to pass an array or object to a field that expects a string is a reason to review input validation. However, whether the database command itself changed, or whether an authentication condition was bypassed, must each be judged by separate evidence.

In this login check, normal valid credentials succeeded, and incorrect credentials failed. On the other hand, when the email address was sent as a single-element array, login succeeded as long as the correct password was used together with it. What this shows is that an array was not rejected by an input that was originally meant to require a string. It does not mean that a user who did not know the password was able to log in.

In JavaScript, converting a single-element array to a string can result in the same string as the element itself. For example, the result of String(["user@example.test"]) is "user@example.test". If the input is converted to a string partway through processing, it is not surprising that the matching result ends up the same as for a normal email address.

This behavior alone is not grounds to claim that the ORM converted the array into an IN operation, that a NoSQL operator was interpreted, or that administrator authentication was bypassed. To claim that authentication was bypassed, one must also verify that a protected function or data was reached using input that did not satisfy the authentication condition. Not conflating a flaw in input format with a breach of authorization is the first thing to check when reading a report.

The fact that a 500 error was returned for object-format input is also a result that should be interpreted carefully. For example, calling a string-specific method on a non-string value will raise an exception. If processing fails before reaching the database, that is not evidence that the meaning of the query was changed.

In fact, in Juice Shop's public source code, both the login process and the product search process use SQL. The search side also has processing that treats the search term as a string. Simply because an input in a form commonly used with NoSQL was sent is not sufficient reason to classify the resulting behavior as NoSQL injection.

On the other hand, the "NoSQL Exfiltration" that succeeded this time is an official challenge involving retrieval of order data including that of other users, and "NoSQL Manipulation" is an official challenge involving the bulk update of multiple product reviews. These successful results should be organized separately from the confirmation of input types in login and search. Succeeding at the former does not mean that the 500 error in the latter also constitutes a successful NoSQL injection.

The public source code is a helpful reference for understanding the processing, but it is not a substitute for identifying the version and configuration of the test environment. When determining the cause, one should check by matching the implementation of the target environment against the communication logs.

The starting point for countermeasures is to define, on the server side, the types of input that will be accepted. If a string is required for the email address, password, or search term, then arrays, objects, null, and so on should be rejected before being passed to business logic. Not just the type, but also the length, required conditions, and acceptable formats should be decided for each field. Restricting input in the screen's input field does not cover the case of sending directly to the API.

However, restricting to strings alone does not complete SQL injection countermeasures. Queries should be constructed using methods such as passing values as parameters, rather than embedding them via string concatenation. Likewise for NoSQL, an object sent by a user should not be used directly as a search condition; only fields and operations permitted by the application should be used. The OWASP Input Validation guidelines also distinguish between input validation and the defenses required at the point of use, such as in queries.

Error handling is also a separate target for remediation. Input with an invalid type should be returned as a managed error, and the response should not include exception details or internal paths. Rather than simply changing 500 to 400, one should verify both that internal processing does not run on invalid input, and that details are not exposed even in the case of unexpected exceptions.

In the ★6 Multiple Likes challenge, the "once per person" limit was exceeded in about 40 minutes. The difference observed—where the same operation was rejected when sent sequentially but accepted as duplicates when sent simultaneously—became a clue for considering the cause.

In this verification, executing "like" twice in sequence resulted in HTTP 200 the first time and HTTP 403 the second time. When 20 requests were sent simultaneously to a review that had already been liked, all of them were rejected as well. The logic for rejecting duplicates does exist.

However, when 30 requests were sent simultaneously to a review that had not yet been liked, all of them succeeded, and the like count increased from 0 to 30. In the verification log, the array of users who had liked the review also retained duplicate entries for the same user.

Here, it is important not to draw a conclusion from the mere fact that HTTP 200 was returned 30 times. It is possible to design an API that accepts duplicate requests but limits the effect on storage to just once. In this case, since the count increased even in the saved state, this shows not a difference in responses, but a violation of the business constraint of "once per person." By examining both the response and the saved result, we can distinguish this from a mere display or communication issue.

The mechanism resembles a situation where multiple counters simultaneously hand out the very last numbered ticket. If each counter checks "there's still one left" and only records the distribution afterward, everyone can pass through at the confirmation stage.

In Juice Shop's public source code for like processing as well, the checking of whether a review has already been liked, the incrementing of the count, and the recording of user information are all separated. Furthermore, as an exercise for learning about race conditions, an intentional delay has been inserted in the middle. This structure can explain the difference between sequential execution and simultaneous execution.

The problem where a confirmed state is not necessarily maintained until the update is called TOCTOU (Time-Of-Check to Time-Of-Use). However, from this result of 30 requests, we cannot definitively conclude that "all processes read the same empty array at the exact same moment." Identifying the actual order of execution in between would require additional logging. Also, the fact that the training material is designed to easily trigger race conditions must be taken into account when evaluating AI capability.

In the fix, the duplicate check and the recording of the like are combined into a single process that remains consistent even under contention. The implementation method varies depending on the database and data model used.

For example, in a design where each like is saved as an independent record, a unique constraint can be placed on the pair of "review ID and user ID." This is because the database will reject duplicates even if the same pair is added simultaneously. If the count is kept separately, the addition of the record and the update of the count must also be handled consistently, such as through a transaction.

In a design where data is saved within a single document, one approach is to combine the condition check and the addition into a single atomic update. However, MongoDB's atomicity specification cannot be directly applied to a different implementation as-is. For saving reviews, Juice Shop uses MarsDB rather than MongoDB itself.

Also, even with MongoDB, a unique index on an array alone does not necessarily prevent duplicate elements within the same document. The scope of the uniqueness guarantee must be checked before choosing an approach. Measures such as preventing repeated button clicks or rate limiting are supplementary measures and are not a substitute for guaranteeing consistency under concurrent updates.

In this verification, narrowing the task's objective proved helpful in advancing the high-difficulty challenge. When evaluating ARTEX's performance, it is necessary to check not only the model itself, but also how much should be entrusted to a single task.

In this local environment, when ARTEX was given a broad objective, findings—the detection results—were slow to emerge, resulting in empty turns where no notable progress was recorded. Even though exploration was continuing, it took time for the results to be compiled into a report.

From this observation alone, it cannot be determined that the cause lies in the Mac's processing power or the model's reasoning ability. Factors such as task division, the scope of investigation, and synchronization between agents also affect progress. It was necessary not to judge the model's limitations based solely on execution time, but to check which task was causing the stall and what had not been reported.

Therefore, a human narrowed the task's objective, clarifying the target functionality or vulnerability, and proceeded with the verification. An important lesson gained from this operation is that changing how objectives are set made it possible to handle even high-difficulty challenges. However, since this was not a strict controlled experiment, it cannot be definitively stated that narrowing the objective alone was the cause of the improvement.

Narrowing down the goal does not simply mean allotting more time. It means breaking down a goal such as "search the entire app for problems" into units where the target feature, the nature of what is being checked, and the conditions for considering it complete are all clear. For example, to examine whether an operation's repetition limit is maintained even under concurrent execution, you first decide on the target operation and the saved state to check before and after.

For this round's NoSQL-related task, a successful example was obtained in about 10–15 minutes, and for Multiple Likes, in about 40 minutes. Having reached the point of succeeding on a ★6 task gives a sense that, by making tasks concrete and working out individual conditions one by one, the range of coverage for high-difficulty tasks can be expanded. On the other hand, Successful RCE DoS is still being adjusted within the Docker environment and is not included among the tasks counted as successful.

To conclude that "given enough time, ★6 tasks in general can be solved with almost no problem" would require success and reproducibility across other tasks as well. The difficulty stars are a guideline within the teaching material, and the required techniques and environmental conditions differ from task to task. At this stage, we are treating "having a successful example at the highest difficulty" and "being able to reliably handle the highest difficulty in general" as two separate evaluations.

To turn an AI-written report into actual remediation work, reproduction conditions, evidence, and the scope of what has not yet been verified are needed. We design the process as a single verification that spans from setting the scope before execution through to re-confirmation after the fix.

First, decide on the target URL, the app's version and configuration, the privileges to be used, the operations to be permitted, and the upper limits on concurrency and execution time. For verifications that rewrite data, also prepare a way to restore the initial state. Check both the scope of the instructions given to the agent and the settings that actually restrict the destinations of communication.

Simply separating the environment into containers is not enough to conclude that communication has been isolated. Check communication to non-target destinations, tool execution privileges, mounted files, and credentials. If the configuration uses an external LLM service, data transmission accompanying inference is also within the scope of management. Treat "communication to the target app" and "communication to the inference service" as distinct.

Tie the results to the baseline normal-state communication, the communication that reproduced the problem, and the saved state before and after. This time as well, there were cases where the results in the execution log and the corresponding communication serving as evidence could not be matched automatically. The condition for using this in actual work is a record that avoids leaving only an AI-generated summary behind, allowing the person in charge to trace the conclusion back to the evidence. When sharing, mask tokens and personal information.

The completion criterion for a fix is not that the error display has changed. For an input-related issue, confirm that unexpected types are rejected before business logic processing, that normal input can still be used as before, and that exception details are not exposed externally. If fixes to query construction or authorization are also needed, verify those conditions separately as well.

For duplicate likes, repeat the concurrent execution that broke the limit before the fix, and confirm that the saved effect is now limited to a single instance. Beyond that, also check whether other users can still operate correctly, and whether no duplication occurs even if communication is interrupted midway and the request is resent. This is to see whether only the rejection has been strengthened in a way that breaks normal usage.

The same way of thinking can be applied to reviews of things like coupon usage or inventory allocation as well. However, the necessary constraints differ by feature. For balances, prevention of insufficient balance and consistency across multiple records are also needed. Rather than assuming "adding a uniqueness constraint solves everything," first put into words the business conditions that must be preserved, and then choose the remediation approach.

When evaluating AI pentesting, it is necessary to separately consider the number of tasks achieved and the actual safety of the real service.

cannot be judged sufficient. Juice Shop has intentional weaknesses and published solutions. In production, test conditions must also account for permissions, understanding of business specifications, acceptable load, and how records are handled. The evaluation should begin in a test environment close to your own, including the work needed for false-positive checks and report reviews. Today's results are one piece of material for thinking through that design.

In this verification using the first-generation ARTEX, success cases were obtained not only for difficulty levels ★1 through ★3, but also for a high-difficulty NoSQL-related challenge and the highest-difficulty ★6 Multiple Likes. With focused, goal-specific execution, the time required was roughly 10–15 minutes for the NoSQL-related challenge and about 40 minutes for Multiple Likes. Successful RCE DoS is still being adjusted in the Docker environment.

Input-type deficiencies, exposure of detailed errors, authentication bypass, and injection are not the same thing. Concurrency issues, too, must be judged not just from response codes but from the saved state. On the development side, this means separating input validation from the defenses at the point of use, and designing data constraints that remain protected even under contention.

In this operation, when broad goals led to stalling, having a human concretize the target and success conditions was what led to the next result. When considering the adoption of AI pentesting, narrow the scope of what you delegate, and evaluate whether you can proceed through to fixes and re-verification based on reproducible evidence. Beyond the number of findings, what creates practical value is leaving behind results that development staff can actually use for remediation.

Reference materials (confirmed: October 21, 2026)

• OWASP Juice Shop: Purpose and Project Overview
• Official Guide: Challenge Definitions such as NoSQL Manipulation
• Official Guide: Multiple Likes Challenge Definition
• Official Guide: RCE DoS Challenge Definition and Execution Environment Restrictions
• OWASP: Input Validation Guidelines
• MongoDB: Atomicity of Updates, Scope of Unique Index Guarantees

References for the implementation are shown in the links within the text. Since public sources are updated over time, please check the version of the code in use at the time of re-verification.