Which Open-Weight Models Excel at Tool Use on DGX Spark? Benchmarking 6 Models

We assigned 6 open-weight models on a DGX Spark (128GB) three tasks that involve reading files, listing their contents, and extracting information. Of all the models, only Qwen's 27B (Q8_0) passed all three tasks, and only Nemotron Bolt (74B-A9B) produced zero hallucinations (plausibly stating content that differs from the facts). We ran the same tasks against each model 3 to 6 times through our own lightweight agent harness and scored the answers against a table of correct answers.

What separated the results was two things: whether the model would go read the file using a tool, and whether it could write out what it read without dropping anything. We've put together this summary so that developers and IT staff who want to delegate business tasks combining natural language instructions with tool operations to a local LLM can judge what to check before choosing a model.

The measurement was carried out by running the models one at a time on a single DGX Spark (128GB), running three tasks with predetermined correct answers through the same harness, and checking the answers against the correct ones.

The machine used is a DGX Spark (128GB). In our previous speed comparison, we measured how the time required changed depending on the inference engine and lookahead settings; this time, instead of speed, we compared 6 models focusing solely on whether the answers were correct.

The harness repeats the exchange of passing the request to the model and, whenever the model calls a tool, executing it and returning the result, continuing until the final answer. Unless the model opens the file itself, it cannot obtain the material needed for the answer. What is being measured here is not the standalone capability of the model, but the results when used through this harness.

LLM-jp's 33B, GPT-OSS-Swallow, and the Qwen-based Swallow were run with a tuned version in addition to the baseline, and the article presents the results of that tuned version. Qwen, Nemotron Bolt, and LLM-jp's 32B-A3B were only run with the baseline settings.

All three tasks require reading a file to answer, and each has a single, fixed correct answer.

Task 1 is the same task used in the previous article, listing 47 APIs. It can be solved by reading a single file to the end, but because there are many APIs, stopping partway through reading or skipping something while writing the table results in omissions.

Task 2 cannot be solved without reading and comparing both the file defining the APIs and the screen-side code. Since the correct answer is only 2, listing even one extra API that is actually called results in failure, making this the strictest criterion among the three tasks.

Task 3 is a task of extracting path-and-section pairs from a manual written in Japanese. It tests the ability to read Japanese text rather than English code, and the ability to stay within the scope of what should be extracted.

The figures in the table are counted as follows.

• Accuracy rate: The proportion of correct answers included in the response. We take the median across all runs for each model-task combination.
• Hallucination: The proportion of runs (out of all runs), across the three tasks, in which at least one item was judged a "fabrication" during scoring.
• Execution time: The elapsed seconds from the start of a run to the end of the final response, including time spent executing tools.

For execution time, we take the median per task using only the runs that ended in "completion," and report the minimum to maximum of the three tasks' medians. Runs that timed out, hit the output limit, errored out, or ended without calling a tool are excluded, and seconds are rounded to the nearest whole number.

As a result, each column has a different base count. Accuracy rate and hallucination are counted over all runs, while execution time is counted only over completed runs. Also, since accuracy rate is a median, the variation between runs does not appear in the table. Pass/fail is a determination that aggregates the results of all runs for each task; even Qwen, which passed all three tasks, had 2 failing runs each in Task 2 and Task 3.

Qwen's 27B was the only model to pass all 3 tasks, and Nemotron Bolt passed Task 1 with zero hallucination verdicts. The other 4 models did not pass any of the tasks.

Under this round's conditions, the two strong candidates are Qwen's 27B and Nemotron Bolt.

• The percentages for each task are the median (across all runs) of the proportion of correct answers included in the response
• "—" for execution time means the run completed in only 1 out of 9 runs, so no figure is given
• "Indeterminate" means that in most runs the process stopped due to environment issues, so the model's capability could not be measured (discussed later)

The correspondence between the names in the table and each model's scale/configuration is as follows.

Only in the Hallucination column did Nemotron Bolt outperform Qwen. Qwen's 11% reflects 2 out of 18 runs that were flagged as fabrications. Looking at the runs where Qwen failed, there were 2 runs in Task 2 where it listed APIs that were actually called, and 2 runs in Task 3 where it picked up lines outside the extraction range — but in no run did it fabricate something that did not exist in the document.

For completed runs, execution time ranged from 68–194 seconds for Qwen and 59–292 seconds for Nemotron Bolt — both roughly 1 to 5 minutes per run. Although Nemotron Bolt has fewer active parameters per token (9B), it sometimes took longer than Qwen depending on the task.

LLM-jp's 33B took 510–706 seconds, or 6 to 26 minutes per run. In some runs, the model had read a considerable number of the correct lines but still ran out of time — a duration too heavy for use cases where a person waits for the result.

LLM-jp's 32B-A3B was short at 54–146 seconds, but as discussed below, many runs stopped due to environment issues, so this figure alone cannot be compared with the other models. For the two Swallow-family models, only 1 out of 9 runs completed, with most runs either hitting the output limit or ending without ever calling a tool.

Execution time can also vary greatly depending on the inference engine and prefetch settings. In the previous article, simply changing the prefetch length reduced the time required by about 20%. Treat the seconds given here as a rough guide for comparison on the same machine and the same harness.

The differences among the 6 models arose from two factors: whether they went to read the file using a tool, and whether they could correctly write out what they had read.

The first dividing point is whether the model goes to read the file at all. None of the tasks can be answered correctly without reading the file.

Qwen, Nemotron Bolt, and LLM-jp's 33B called a tool to read the file every single time. The two Swallow-family models, on the other hand, wrote out the table in many runs without ever opening the file. The accuracy rate was 0–2% for GPT-OSS-Swallow and 0% for the Qwen-based Swallow. Since these tables were written without reading the file, many runs were also flagged as fabrications, with hallucination rates of 89% and 67% respectively.

What should be noted here is the reason the tool was not called. This round did not distinguish between whether the model's capability was insufficient or whether the way the tool was invoked simply did not match this harness. For the Swallow-family models, what can be said is only "under this harness, the tool was rarely called" — not that "the model is weak."

The second fork in the road is whether the model can take what it read and put it into the answer table without errors. Even among the 3 models that called the tool every time, differences showed up here.

Qwen's 27B read every correct row in all 3 tasks, and its accuracy rate was a median of 100% across all 3 tasks as well. The runs that failed were 2 cases in Task 2 where an API that was actually being called was listed as unused, and 2 cases in Task 3 where rows outside the extraction range were also picked up—both failures on the "wrote too much" side.

Nemotron Bolt, conversely, failed on the "under-writing" side. In Task 3, it had read all 14 target paths for extraction via the tool, yet rows 4 through 11 were missing from the answer table. Task 2 also failed due to omissions, extras, and hitting the output limit.

LLM-jp's 33B read a fair number of the correct rows, but still missed the answer or ran out of time. In Task 2, it did not read a single one of the 71 calls present in the screen-side code. Without that, there's no way to judge which APIs are unused.

The fact that a model can read the material correctly but still fail at the write-out stage is an easy thing to overlook when choosing a model. Looking only at the number of tool calls or the number of files read, the gap between Qwen and Nemotron Bolt doesn't show up.

LLM-jp's 32B-A3B stalled on 5 out of 9 runs at a path to a different development machine written in the instruction file (AGENTS.md) in the working folder. This path doesn't exist on the verification machine and has nothing to do with the task. For that reason, the table marks this not as a failure but as "unjudgeable."

This is a case of stumbling over an environment flaw before the model's actual capability could even be measured. Instruction files like AGENTS.md are sometimes automatically loaded by the harness and passed to the model. If settings written for a development machine are carried over as-is to the verification machine, some models will stall at the path written there.

When comparing models in-house, first check whether instruction files or paths meant for a different machine remain in the verification working folder. An environmental flaw can end up looking exactly like a difference in model performance. We are holding off on evaluating this model until the environment is fixed and it is measured again.

Nemotron Bolt used the tool every time and produced no fabrications, but it lost points at the write-out stage after reading. Since failures at the write-out stage leave room for the harness side to compensate, we see significant upside potential here. That said, this is a hypothesis we have not yet verified in this round.

Nemotron Bolt is an open-weight model released by NVIDIA. Of its total 74B parameters, only 9B are active per token, making it a MoE (Mixture of Experts) model.

MoE itself is not a special configuration. The Qwen-based Swallow (30B-A3B) that failed this time, as well as LLM-jp's 32B-A3B that couldn't be measured, are also MoE models. What's different about Nemotron Bolt is that it combines Mamba-2, which works on a different mechanism from attention. However, whether this configuration is what caused the difference in results this time cannot be determined from this verification alone.

Another point of interest is computational cost. The parameters active per token are 9B, one-third of Qwen's 27B, which is a Dense model. One could view this as: with less computation, it matched Qwen in calling the tool every time, and outperformed it in having fewer fabrications. That said, 74B worth of weights still need to be loaded into memory.

Nemotron Bolt's failure pattern is the opposite shape of the Swallow family's. The Swallow family got stuck at the reading stage, whereas Nemotron Bolt made it past the reading stage but fell down at the write-out stage.

• It executed the tool in all 12 runs, and it did see the correct rows as tool results
• Its fabrication count was 0 in all 12 runs—the only model among the 6 to achieve this
• The causes of failure were under-writing in Task 3 (rows 4 through 11), omissions and extras in Task 2, and hitting the output limit (once, in Task 2)

Models that fail at the reading stage don't even possess the raw material for the answer, so it's hard to rescue them by changing how they write. In contrast, a model that reads the material correctly but fails at the write-out stage may well produce different results if the output assembly method is changed.

Another important factor is the type of failure. Under-writing can be caught simply by checking the count against the total. Plausible-sounding fabrications, on the other hand, can only be caught by checking each answer one by one against the original document. Since the latter is the more troublesome problem in business operations, we consider Nemotron Bolt—which fails through under-writing rather than fabrication—to be a strong candidate worth trying harness-side remedies on.

In light of the write-out failures, the next verification will try the following remedies one at a time. Whether any of them actually works has not yet been confirmed.

Beyond this, we also see room to fine-tune the harness settings to match the characteristics of a model combining Mamba-2 and MoE. Since any of these remedies could also benefit Qwen's 27B, when comparing the two, the same remedy must be applied to both and the change in the gap observed.

This result alone does not let us conclude that "Nemotron Bolt will improve if the configuration is adjusted." What we can say is only this: because the failures were concentrated at the write-out stage, it has become clear where to aim when trying remedies.

This round's results come from a small-scale verification in which three tasks—reading files and then listing/extracting information—were each run 3 to 6 times. When reading the numbers, keep the following points in mind.

Because the number of trials is small, a single-trial difference can swing the percentages considerably. For example, for a model run 9 times, the hallucination rate changes by about 11 points with just one trial's difference. The differences shown in the table are not statistically confirmed differences, but rather tendencies observed in this round's trials.

This round's 3 tasks are limited to processing that involves "reading a file and listing/extracting from it." This verification does not tell us about superiority or inferiority in summarizing text, or in processing that spans multiple steps to operate screens or data. Before using this in actual business operations, it is most reliable to measure in the same way using one's own documents and tasks for which one's own company has prepared correct answers.

Since the model that passed all 3 tasks was Qwen's 27B, one might want to conclude that "Qwen-family models are strong." However, Swallow (30B-A3B), which is also based on Qwen, rarely called the tool and had a 0% accuracy rate on all 3 tasks.

Even when the base model is the same, the way the tool is used can differ greatly depending on additional training, scale, and whether it is Dense or MoE. What can be called a leading candidate is "Qwen's 27B quantized at Q8_0," as measured this round—not every model with "Qwen" in its name. This verification also did not measure accuracy when the quantization level is changed.

Using the model name's lineage as a guide is convenient, but after narrowing down candidates, each one must be measured individually on the same tasks. This round's results show that even models bearing the name of the same lineage can differ dramatically—one passing all tasks, and the other scoring 0% on all tasks.

Here we answer, to the extent this round's results allow, the questions that commonly arise when entrusting tool operation to open-weight models on local machines such as DGX Spark.

In this round of 3 tasks (reading files to list and extract data), Qwen's 27B (Q8_0) passed all tasks, and Nemotron Bolt had zero instances of fabrication. However, this is based on only 3–6 small trials per task, so please measure performance in the same way using your own company's documents and tasks before making a choice.

It's not a simple case. The Qwen-based Swallow (30B-A3B), which failed this time, is also an MoE model, but Nemotron Bolt, also an MoE model, used tools every time and produced no fabrications. What's different about Nemotron Bolt is its configuration combined with Mamba-2, but it's unclear whether that is the cause of the difference. LLM-jp's 32B-A3B is also an MoE model, but it could not be measured due to environment issues, so we cannot say it failed because it was MoE.

Not producing fabrications and extracting everything without omission are two different things. Nemotron Bolt showed no instances of fabrication across all 12 runs, but in the extraction task from a Japanese procedure manual, it dropped 4 to 11 lines that it should have read. If you're relying on it for extraction, combine it with mechanisms to catch dropped content, such as cross-checking counts or review procedures.

When we tasked 6 open-weight models running on a DGX Spark (128GB) with 3 tasks involving reading files to list and extract data, Qwen's 27B (Q8_0) passed all tasks, and Nemotron Bolt had zero instances of fabrication. For processes that combine natural language instructions with tool operations, these two models are the strongest candidates under the conditions tested this time.

What separated the results was whether the model read the file using tools, and whether it could write out everything it read without dropping content. The two Swallow-series models wrote tables with barely any reading, while Nemotron Bolt did read the content but dropped some of it during output. The latter issue leaves room for improvement on the harness side, which we'll verify in our next round of testing.

When selecting a model, don't decide based solely on the model family name—measure separately, using your own company's documents and tasks with known correct answers, whether it "actually read" the content, "didn't fabricate" anything, and "didn't drop" anything. For a speed comparison, see How to Speed Up Local LLMs on DGX Spark?, and for an overview of how to choose a model, see Local LLM / SLM Introduction Comparison.

We also support verification and PoC work for local LLMs and agents. For inquiries, please reach out via AI/DX Support.

References (accessed at time of writing): For Nemotron Bolt's configuration, see the Artificial Analysis model page, the ApX model page, the DataLearner comparison page, and the NVIDIA NeMo AutoModel model page.