How to Speed Up Local LLMs on DGX Spark? Benchmarking Ollama, SGLang, and Speculative Decoding

To speed up local LLM on DGX Spark, configuring speculative decoding (look-ahead) to accelerate generation is the most effective lever: in our measurements with a coding agent, simply lengthening the look-ahead cut the time required by about 20%. We ran our own coding-agent LLM on a single DGX Spark–equivalent machine, feeding the same real task through 8 combinations of Ollama, llama-server, SGLang, and look-ahead settings, and compared the time taken and the quality of results. We also measured a comparison of 3 types of quantization, and how to fix the problem where, in long conversations, the input cache (prompt cache) gets silently discarded.

We put this together so that developers using local LLMs for agents on DGX Spark or Ollama can judge, with numbers, which engine and settings to choose.

The measurements were taken by loading one inference engine at a time onto a single DGX Spark–equivalent machine, running the same read-only task 3 or more times for each combination, and observing the time taken and the correctness of the answers.

The models used are two series: the dense 27B from the Qwen family, and the MoE Flash-Next.

All weights are quantized models published by the community. For the 27B, Ollama and llama-server use Q8_0 while SGLang uses NVFP4, so differences in quantization are also mixed into the engine comparison.

The task is to have the agent generate a list of APIs from a single server-side file of our own web app. The target is 47 routers written with if statements, and the agent calls file search/read tools to check the contents and writes a table of "method, path, line number." It's a read-only task of roughly 10 steps per run.

Answers were mechanically checked against the correct table (line numbers and methods), and the same request was issued 3 times per combination, each time in a new conversation. The context length was 262,144, thinking mode was off, and the sampling values were the same across all engines.

Since this agent can only talk to LLMs via the Ollama API, we placed an approximately 300-line relay in front of the OpenAI-compatible llama-server and SGLang to convert the request/response formats. Because the 27B's conversation format only accepts a single system message at the start (two or more causes an HTTP 500), the relay merges them into one.

When comparing look-ahead lengths, to prevent the differences from being buried in step-count variance (8–16 steps), we saved 11 requests from a single run that finished normally, and replayed them 3 times for each setting.

Each time, the agent resends the entire conversation so far. If input reuse (prefix cache) works, only a small amount of input needs to be newly computed. Across our 44 recorded work sessions, of the 21.17 million input tokens sent, only 9.3% were new, and about 75% of the model's time was spent on generation. In this task as well, the difference in input speed (600–1,070 tok/s) amounts to only about ten-odd seconds per run, but the difference in generation speed (20–38 tok/s) widens to tens of seconds or more.

The means for speeding up generation is speculative decoding, which we refer to in this article as "look-ahead." It's a mechanism where a draft is written several tokens ahead, and the main model verifies them all at once in a single computation; discarded drafts that miss are thrown away, so the output distribution doesn't change. This time, we tried MTP (Multi-Token Prediction), which uses the prediction layer built into the weights, as well as DSpark and DFlash2, which use a separate draft model.

For speed, SGLang's MTP (lookahead of 8 tokens) and DFlash2 were tied for first place, with median times for the normally completed runs of 60.2 seconds and 62.5 seconds. For Ollama's 27B as well, simply extending the lookahead from 4 to 12 brought the time down from 109.6 seconds to 85.5 seconds.

These are the results comparing the original lookahead lengths as-is (MTP at 4, DSpark and DFlash2 at 8).

Elapsed is the value for the normally completed runs, and generation tok/s is the median of 3 runs. Table correctness is the number of runs out of all 47 lines where the line numbers and method matched.

Without lookahead, the MoE Flash-Next is faster, since the weights read per token are small, about 5.8GB, giving a bandwidth-based upper limit of about 47 tok/s. The dense 27B Q8_0 reads about 28.6GB every time, so its upper limit is about 9.5 tok/s (measured at 7.9), but MTP hits its predictions well, and adding lookahead reversed the standings.

Since MTP was essentially pinned at its upper limit of 4 tokens and hitting its predictions well, we extended the lookahead and reran the same 11 requests. SGLang's MTP plateaus at 8 tokens.

The generation time per run is a calculated value obtained by dividing the approximately 700 tokens of a tool call and the approximately 1,050 tokens of a long answer by their respective speeds and adding them together.

In Ollama, the lookahead length is determined by the model setting draft_num_predict. If you create an alias using the following Modelfile, the weights will be shared with the original model.

FROM <original 27B model name>
PARAMETER draft_num_predict 12

Changing it from 4 to 12 lowered the acceptance rate from 0.886 to 0.619, but since the amount advanced per verification increased, the tool call speed rose from 25.9 to 37.4 tok/s. In actual tasks, SGLang's MTP 8 shrank from about 75 seconds to 60.2 seconds, and Ollama's lookahead of 12 shrank from 109.6 seconds to 85.5 seconds. MTP 8 and DFlash2 (62.5 seconds) are nearly the same speed, but MTP has the advantage of not needing a separate draft model.

SGLang's adaptive lookahead adjustment was slower than the fixed value of 8, and continuous decoding only differed by 1〜2%, so neither was effective.

While lookahead hits its predictions well for outputs with a fixed form, such as tool calls, it almost never hits for freely written Japanese text, resulting in the same speed as having no lookahead at all. So, we also measured how much quantization could speed up Japanese text.

Using Ollama's 27B , we counted by output length from the logs of the llama-server running underneath (408 cases combining testing and regular use). The Japanese text consists of 3 types of requests to write about 600 characters without headings, each done twice.

For Japanese text, the value is the total of 6 cases; for the others, it is the median. The 7.7 tok/s is unchanged from the 7.9 tok/s without lookahead. The difference is that tool calls and tables have a fixed form, so the draft hits its predictions, while freely written text does not. The llama-server option --spec-draft-p-min, which cuts off lookahead when it's not confident, cannot be passed through Ollama's model settings, and we have not tried it.

We compared Q6_K and Q4_K_M with lookahead 12 added, from the same distributor as the 27B Q8_0. With Q4_K_M, the elapsed time of the actual task is 0.88x, and Japanese text generation is 1.39x faster.

The gains were 1.1x to 1.4x, smaller than the upper bound estimated from the ratio of weight sizes (about 1.3x, 1.7x), and we have not measured the reason. For Japanese text, which showed the largest gain, we see this as the difference in the amount of weights read having a direct effect, since lookahead doesn't hit as often. The number of outlier runs was 2 for Q8_0, 1 for Q6_K, and 0 for Q4_K_M, but with only 6 runs each, we cannot say whether this is due to the quantization difference or just variance.

Before switching quantization, we check whether the GGUF has MTP layers. We fetch only the first 12MB of the body (the tensor list) from the distributor and look for the layer name (blk.64.nextn.* for this model).

When continuing a long work conversation in Ollama, simply inserting a short, unrelated request would cause the conversation's cache (prompt cache) to be discarded, forcing the full text to be re-entered with every subsequent request.

For each request, our agent first sends a short "decision request"—without conversation history—to decide whether work or a web search is needed.

Ollama's 27B runs the internal llama-server with a single cache slot (-np 1). When a different request comes in, llama-server moves the current conversation's cache to a RAM eviction area, and on the next request restores the best-matching cache. The default limit for this eviction area is 8 GiB.

In the 27B model that mixes in linear attention, the linear attention state is cached as checkpoints so that computation can be resumed partway through. Each one is about 150MiB, and up to 32 can accumulate for a single conversation; for a work agent that sends a request at every step, the limit is reached within a few dozen steps.

Caches exceeding 8 GiB are simply discarded, with only exceeds cache size limit appearing in the log—no error occurs.

Following a script that extends the conversation in the same shape as the actual task (adding 6,000 bytes of source per step, reaching about 48,000 tokens over 30 steps), we inserted one short request in the same shape as the decision request, then sent the continuation of the conversation.

In exchange, the inserted short request's time grew from about 1 second to 4.5 seconds. We take this to be the time to move the roughly 11 GiB cache into RAM, though we have not measured it directly.

The limit is determined by llama-server's --cache-ram, but it cannot be passed from Ollama's model settings. Since llama-server accepts many arguments via environment variables as well (LLAMA_ARG_CACHE_RAM), and Ollama passes its own environment variables through as-is, writing it into the service configuration works.

We set it to 16 GiB. This value accommodates conversations up to about 80,000 tokens, and still fits within 121GB even combined with the usage after using Flash-Next (105GB). We also tried OLLAMA_NUM_PARALLEL=2 to use two cache slots, but Ollama 0.40.4 does not support parallelism for this 27B model type (qwen35), and it remained at -np 1.

Repeatedly re-reading the same file with slightly shifted ranges occurred in about 30% of runs, regardless of engine or quantization. Having the agent itself count the number of reads and add a caution note allowed it to break out of the repetition.

The rate of outliers (26+ repeated steps, or stopping due to an error) was roughly the same across engines. SGLang's 27B NVFP4 had 7 out of 24 runs (29%), Ollama's 27B Q8_0 had 3 out of 11 runs (27%), and Flash-Next had 2 out of 6 runs.

Many of the repetitions involved re-reading a target file with read_file using slightly different ranges each time. In one case, out of 150 calls, there were 25 different sets of arguments—a watchdog that only stops "if the same call continues" wouldn't catch this. Since switching engines didn't solve the problem, we addressed it on the agent side.

The watchdog treats read_file and any search specifying that file as the same target if it's the same file, even if the range or search term differs (it also counts read-only commands such as grep and sed).

• From the 12th call onward, a note to the effect of "You have read this file 12 times. Please write your answer based on the results so far" is appended to the read result.
• From the 20th call onward, the read is not executed; instead, the next step is sent without tools, prompting the model to write an answer.

The thresholds were determined based on the fact that runs which ended normally (34 cases) read the same file at most 10 times, while repetitive runs ranged from 20 to 150 times. When we took over a conversation that had run without the watchdog up to 150 steps, resuming from step 30 and running it 3 times, in 2 cases the model moved to an answer in the step following the 12th-call warning, and in 1 case it answered before the warning. All 3 times, the table was 47/47.

Even after stopping, there was a pitfall. Ollama returns a tool call written by the model even for requests that don't send tools. So we made it so that such requests are not executed, and instead a note saying "Tools cannot be used" is appended before sending once more. In testing, of the 7 times it was stopped, tools were requested 5 times, and in all 5 cases the model answered on the next step.

If speed is the top priority, SGLang's MTP 8; if you want to make do with Ollama, 27B with a lookahead of 12—these are the answers from our actual measurements. Whichever you choose, you'll separately need a mechanism to stop outliers and settings for the context cache in long conversations.

• If speed is the top priority, SGLang's MTP 8. Normally completed runs took about 0.55 times as long as Ollama's 27B (lookahead 4). Startup takes about 3 minutes, and you'll need to avoid running it alongside Ollama on the same machine at the same time. For models that mix in linear attention, add --mamba-radix-cache-strategy extra_buffer to make lookahead and input reuse compatible.
• If you want to make do with Ollama, set the 27B's lookahead to 12. This single setting makes it about 20% faster, only about 25 seconds slower per run than SGLang's MTP 8.
• If you need the model to write a lot of Japanese text, use Q4_K_M. It's about 10% faster on real tasks and about 40% faster on Japanese text.
• If you're continuing a long work conversation, raise the upper limit of the RAM cache.

Our default is Ollama's 27B Q8_0 (lookahead 12), with the RAM cache set to 16 GiB. Although this is slower than SGLang, it avoids keeping a separate engine resident. Since Q4_K_M only speeds things up by about 10% and we've only evaluated quality on 6 runs each, we've added it to the model list and are comparing it while using it.

These are the pitfalls we encountered during measurement.

• When running Ollama and another engine on the same machine, an agent's read request caused Ollama to start loading Flash-Next (about 67GB), and llama-server was forcibly terminated due to insufficient memory.
• fa3, which often appears in SGLang startup examples, is exclusive to sm_90 and cannot be used on GB10. We used --attention-backend flashinfer instead.
• When Ollama's model import stalled at 91% for over 25 minutes, retrying caused it to resume from where it left off and finish within seconds.

There are also limitations to our measurements. The task was limited to a single type involving only reading, and the number of runs per combination was 3 to 8. The 27B model's quantization differs by engine, and SGLang's input speed was measured differently from the others. Even with 16 GiB, conversation caches exceeding about 80,000 tokens won't fit.

Here are answers, based on what we found in this round of measurements, to the questions that commonly come up when running agents on DGX Spark or Ollama.

Extend the length of the lookahead. The only setting Ollama exposes for lookahead is draft_num_predict, and in our actual tasks, changing it from 4 to 12 made things about 20% faster. When switching quantization, first check whether the GGUF includes the MTP layers.

It's possible that the input cache exceeds the RAM offload limit (default 8 GiB) and is being silently discarded. If exceeds cache size limit appears in the llama-server logs, that's the cause. Raising the limit with LLAMA_ARG_CACHE_RAM reduced, in our environment, the time for continuing input of 48,000 tokens from 62.6 seconds to 3.75 seconds.

If you're using a local LLM for an agent on a single DGX Spark, the key point is how to speed up generation, which takes up most of the time. In our measurements, lengthening the lookahead (MTP) was the most effective: SGLang's MTP 8 was the fastest, and even with Ollama, a lookahead of 12 cut time by about 20%. For Japanese text where lookahead doesn't hit, Q4_K_M is about 40% faster.

An easy thing to overlook is the problem of input cache being silently discarded during long conversations, which you can check for via exceeds cache size limit in the logs. Repeated outliers aren't fixed by switching engines, so our conclusion is to combine this with monitoring on the agent side. For the overall picture of how to choose a local LLM, please also see Local LLM / SLM Adoption Comparison.

We also support verification and PoC work for local LLMs and agents. Please reach out via AI/DX Support for consultations.

References (as of the time of writing): llama.cpp's RAM cache (#16391) and checkpoint (#15293), Ollama's Modelfile documentation, SGLang startup instructions for GB10 (MiaAI-Lab).