Which Size Should You Choose for PLaMo 3 Translate? Benchmarking 4 Sizes and Claude on DGX Spark

PLaMo 3 Translate is a translation-specialized LLM with weights released by Preferred Networks (PFN). It comes in four sizes—0.8B, 2B, 8B, and 31B—and supports translation across 53 languages. We ran all four sizes on a single DGX Spark (128GB), had them translate the same 240 sentences in English, Japanese, and Thai as Claude Haiku and Claude Sonnet, and scored the results using two metrics: chrF and COMET-22. The 8B model produced accuracy indistinguishable from Claude Sonnet, with a median of 1.7 seconds per sentence. The 31B model slightly outperformed the 8B model but was nearly four times slower. Lao, which is not among the supported languages, did not produce usable translations at any size.

For information systems and DX staff at companies with operations in Thailand or Laos who want to translate documents locally that they don't want to send outside the company, we've summarized in numbers which size to choose and where to use PLaMo versus Claude.

After reviewing the model's characteristics and the license conditions for business use, we ran the four sizes in sequence on a DGX Spark (128GB), had them translate the same 240 sentences, and scored the results.

PLaMo 3 Translate is built on "PLaMo 3 NICT," a pretrained model obtained through joint research between PFN and the National Institute of Information and Communications Technology (NICT), with additional translation-focused training on tens of billions of tokens. It can handle up to 32,768 tokens combined across input and output.

How the translation is performed is specified via arguments in chat_template_kwargs of the Chat Completions API. You can pass the target and source languages (target_lang and source_lang; the source language can be omitted), a glossary (glossary), context such as surrounding text or scene descriptions (context), and the register (plain or polite form) to use when translating into Japanese (form)—all without writing them into the source text or instruction prompt.

The license is the PLaMo Community License. For personal or academic use, it can be used free of charge without registration. For business use—including use only within an organization—both of the following conditions must be met:

• Register via PFN's commercial use registration form before starting use
• The user's or affiliated company's revenue or sales for the most recent fiscal year is 1 billion yen or less

If revenue exceeds 1 billion yen, a separate contract must be concluded with PFN. There are also rules requiring that translations be identified as PLaMo-generated when published, and that "Built with PLaMo" be stated when providing output to third parties. Before building this into internal systems, check the terms with your legal department.

Inference was performed using llama-server from llama.cpp, loading one Q4_K_M (quantized to 4 bits) GGUF model distributed by PFN per size. According to PFN's model card, for all four sizes, the difference between Q4_K_M and unquantized BF16 is within 0.0011 on English→Japanese COMET-22.

We did not specify temperature for Claude, because the API used this time did not accept values other than the default. Claude was run only once, so we did not measure variability across repeated runs. Claude's response time includes the round-trip communication from Bangkok to the Tokyo endpoint.

Evaluation sentences were selected from WMT24++ (Apache-2.0), published by Google. From sentences that had both Japanese and Thai translations available for the English source, we randomly selected 15 sentences each from 4 domains—literature, news, social media, and speeches—totaling 60 sentences (600 characters or fewer), and had them translated in 4 directions: English→Japanese, Japanese→English, English→Thai, and Japanese→Thai, for a total of 240 translations. For the two directions using Japanese as the source, the Japanese text used was itself translated from English, so the phrasing may differ from sentences originally written in Japanese.

Two metrics were used for scoring. chrF counts how much the character sequences (up to 6 characters) overlap between the translation and the reference translation (the translation treated as correct). Since Japanese and Thai do not have spaces between words, we used chrF rather than chrF++, which also counts word-level overlap. However, chrF tends to score paraphrased translations low. So we also scored using COMET-22 (Unbabel/wmt22-comet-da), which reads the source text, translation, and reference translation and scores semantic closeness. COMET-22 is also the metric PFN uses in its model card.

Differences between models were verified using paired bootstrap resampling with 1,000 iterations at the sentence level, and differences with a p-value below 0.05 were considered "significant."

Accuracy rises sharply from 0.8B to 2B, and the gap between 8B and 31B narrows. Speed drops by roughly 3 to 4 times with each step up in size.

8B offers the best balance between accuracy and latency. While 31B beats it by 0.003 on COMET-22, it takes about 4 times as long per sentence.

Processing 240 sentences took about 10 minutes for 8B and about 41 minutes for 31B.

We believe 31B's slowness comes from hitting the ceiling of memory read speed. Assuming 19.5GB of weights are read for every token generated, 10 tokens per second corresponds to roughly 195GB/sec of reads, which is close to the GB10's nominal memory bandwidth (273GB/sec) (estimate). While 128GB of memory is enough headroom to load large models, it doesn't make 31B any faster.

For Thai translation, every size took roughly 3 times as long. This is because the output has more tokens (for 8B, the median token count is 52 for English→Japanese versus 145 for English→Thai), and translation time scales almost proportionally with this.

Since the chrF value level differs depending on the language, we compare sizes within the same direction. The largest difference due to size is in English to Japanese, where chrF increased from 36.3 for 0.8B to 39.9 for 31B. In English to Thai, 8B (52.9) exceeds 31B (52.2). Since the comparison for each direction is based on 60 sentences each, no statistical testing was performed.

The difference in size was visibly apparent in the formatting of dates and numbers. The literary example "JANUARY 14th, 10:26 PM, 2543." was translated by the four sizes as follows:

Only 31B rearranged the date into the Japanese order.

The COMET-22 (GGUF Q4_K_M) for English to Japanese in the WMT24++ section of the PFN model card is 0.8716–0.8810, and our English to Japanese results (0.8866–0.8976) were 0.012–0.017 higher for all sizes. We attribute this (estimated) to the difference in how the 60 sentences of 600 characters or less were selected. The ranking of the sizes and the gap from 0.8B to 31B (published value +0.0094, our result +0.0110) are consistent.

In terms of accuracy, 8B and 31B were on par with Claude Sonnet, while 2B was on par with or slightly above Claude Haiku. Looking at median time per sentence, PLaMo's smaller sizes were faster, but Claude showed less slowdown on long sentences.

For both chrF and COMET-22, no statistically confirmed difference was found between 8B/31B and Claude Sonnet.

The results of the difference tests were as follows (p-values are given in the order chrF, COMET-22).

Whether 0.8B and 2B rank above or below Claude Haiku flips depending on the metric, so the accurate reading is that the difference between these two sizes and Haiku is small.

In terms of median, PLaMo's smaller sizes were faster, while Claude showed smaller delays (95th percentile) when long sentences were mixed in. Per-sentence times were a median of 1.44 seconds and a 95th percentile of 2.76 seconds for Claude Haiku, and 1.76 seconds and 3.58 seconds for Claude Sonnet (PLaMo figures are in the table in the previous section).

For translation into Japanese or English, models up to 8B were faster than Claude Haiku in terms of median (for English→Japanese, 8B took 1.32 seconds versus Haiku's 1.43 seconds, a narrow margin). For translation into Thai, only models up to 2B (1.31–1.34 seconds) were faster than Haiku (1.85–1.94 seconds). Since PLaMo's processing time increases with output length, the 95th percentile grows larger when long sentences or Thai sentences are mixed in.

The cost for 240 cases was about $0.024 for Claude Haiku and about $0.48 for Claude Sonnet. These are reference values calculated at the time of writing, converting the number of tokens used based on Anthropic's API list prices (per million tokens: Haiku at $0.10 for input and $0.50 for output, Sonnet at $2 for input and $10 for output); the actual billed amount on Bedrock was not verified. Please check each provider's pricing page for the latest rates. PLaMo does not incur a per-case charge, but does incur hardware and electricity costs.

The glossary and re

Lao is not among the supported languages of PLaMo 3 Translate; when set as the target language, it was translated into a different language, and when set as the source language, the text was unreadable. Claude produced actual translations in all 4 directions.

Passing the language code lo does not result in an error—a translation is still returned—so there is a risk of using it without noticing the problem.

For the evaluation sentences, 60 sentences were taken from Microsoft's publicly available NTREX-128 (CC BY-SA 4.0; 1,997 English news sentences human-translated into 128 languages), yielding 240 cases across 4 directions: English→Lao, Japanese→Lao, Lao→Japanese, and Lao→English. Since 31B took up to 101 seconds per sentence, only the first 15 sentences (60 cases) were run for that model.

PLaMo, at all 4 sizes, was unable to produce a single correct Lao translation.

The numbers in the table are chrF scores. Even when restricted to the same 15 sentences as 31B, the ranking does not change.

There was a consistent pattern to the deviation. At every size, the models produced translations with coherent meaning, but written in a different language. When asked to translate "He is free." into Lao, the following was returned:

The substituted languages—Hebrew, Thai, Khmer, and Myanmar—are all languages supported by PLaMo 3 Translate. The chat template bundled with the model (chat_template.jinja) only checks the format of the language code (2–3 lowercase letters). This suggests that when given an unsupported code, the model substitutes one of its supported languages and translates into that instead, with the substituted language varying depending on the model size.

When the source text was in Lao, PLaMo was unable to read Lao in any size. I compared the 30 samples from the 31B model against the reference translations one by one, and not a single one conveyed the meaning of the original text. The returned translations fall into three main categories.

The first is returning the original text as is. For the 0.8B model, 39 out of 60 Lao-to-Japanese samples remained in Lao characters.

The second is a form where meaningless katakana or Roman characters are strung together. The 2B model repeated the same sounds, such as "Juhara Nyaunchai Rapa Rapa Rapa...", while the 31B model created spellings like "Lehtiewulaba Kedsaiehalahmeihailekefufengjin." For the 0.8B, 2B, and 8B models, 29 to 36 out of 120 samples did not stop and were cut off at the output limit (1,024 tokens).

The third is a form that creates sentences unrelated to the original text, which is the hardest to notice. For an original text where the reference translation is "The boy was airlifted to Rady Children's Hospital in San Diego and is said to be in critical condition," the 31B model translated it as "Rady Children's Hospital in San Diego, California, specializes in pediatric cancer treatment." It picked up only the hospital name and location that were already in English in the original text and fabricated the rest. Since it is plausible-sounding Japanese, someone who cannot read Lao would not realize it is an error. This is also a type of hallucination. The reason the Lao-to-English chrF for 31B is as high as 19.2 is because these English proper nouns matched.

Claude also had mistranslations. Claude Sonnet read "ລາວ," which can mean both "he" and "Laos," as the country name and translated it as "Laos has freedom," and it translated a word meaning "scuffle" as "the bombing."

Even when the translation is into a different language, COMET-22 will still award points as long as the meaning matches.

PLaMo scored 0.27–0.39 lower than Claude at every size, and all of these differences were significant (p < 0.001). However, the 31B's En→Lao score was 0.718 even though it did not write a single Lao character. The chrF for that same translation is 4.5, so if you only looked at COMET-22, you'd mistakenly conclude it was "translating reasonably well." Because COMET-22 measures how close the meaning of the translation is to the source, it will still award points if the meaning matches, even if the output is in Khmer or Thai.

When evaluating languages with few speakers, you should separately check whether the translation is actually written in the correct script. In this case, we counted the most common character type in the translation by Unicode range, and that is how we caught this mismatch.

This measurement was taken from a single run of 60 sentences, so the selection should be treated as a rough guide. We also list the conditions that were not measured, so please verify with your own company's documents before making a decision.

If you are translating Japanese, English, and Thai on your own hardware, under the conditions tested here, it is reasonable to start by trying 8B.

No statistical test was run between the sizes, so the gain from 31B is a difference observed in this set of 240 cases.

0.8B is the fastest at 0.20 seconds per sentence and requires only 737MiB of GPU memory. However, it falls below Claude Haiku on chrF, and fine details such as date ordering tend to break down, so for rough drafts of documents meant to be read by people, we recommend 2B or larger.

The following conditions were not measured.

• Formats intended for vLLM (BF16, FP8, NVFP4, etc.). Only the GGUF Q4_K_M format was measured
• Throughput when multiple requests are sent at the same time. All requests were sent one at a time, in sequence
• Long documents. Only sentences of 600 characters or fewer were used; lengths close to the 32,768 token limit were not tested
• Human evaluation, and trained metrics other than COMET-22 (MetricX, COMET-Kiwi, etc.)
• Comparison with translation tools other than Claude (TranslateGemma, DeepL, Google Translate, etc.)
• The effect of two other inference servers running on the same machine (they were nearly idle during measurement)
• The remaining 45 sentences for 31B in Lao

In-house documents differ from the evaluation sentences in the amount of specialized terminology and sentence length. Including how well a glossary works, measuring your own company's documents translated using the same procedure as this time will make it easier to judge which size is sufficient.

Here, based only on what was learned from these results, we answer the questions that commonly come up when considering whether to run PLaMo 3 Translate locally.

All four sizes ran. With the Q4_K_M GGUF, even the 31B model uses only about 19.8GB of GPU memory, leaving plenty of headroom in the 128GB unified memory. However, the 31B model is slow, with a median of 6.5 seconds and a 95th percentile of 30 seconds per sentence, so for use in a conversation, 8B or smaller is more practical.

For translating Japanese, English, and Thai documents locally, the 8B model delivers accuracy on par with Claude Sonnet, with no per-request fees. However, for languages outside its support, such as Lao, or when you want the requested text aligned to plain/informal style as well, Claude proved better suited based on this round of results.

It can be used under certain conditions. If the revenue or sales of the user and its affiliated companies for the most recent fiscal year is 1 billion yen or less, you can use it commercially for free by registering through PFN's commercial use registration form. Registration is required even for internal use only. If the amount exceeds 1 billion yen, a separate contract with PFN is required.

When PLaMo 3 Translate was run on a DGX Spark (128GB), for English, Japanese, and Thai, the 8B and 31B models achieved accuracy on par with Claude Sonnet, while the 2B model matched or slightly exceeded Claude Haiku. Balancing accuracy and latency, 8B is the more manageable size to work with.

Lao, which falls outside the supported languages, could not be used. Since passing a language code does not trigger an error, be sure to check the list of supported languages before use, and also verify whether the translation is actually written in the correct language. COMET-22 alone will miss this kind of error.

For related articles, please also see How to speed up local LLMs on DGX Spark?, Which open-weight models excel at tool use on DGX Spark?, and Local LLM / SLM Adoption Comparison.

We also support verification and PoC work for translation and agents using local LLMs. For inquiries, please reach out via AI & DX Support.

The translation examples of PLaMo 3 Translate included in this article are outputs generated by the model itself (Built with PLaMo).

References (as of the time of writing): PLaMo 3 Translate 8B GGUF model card, PLaMo Community License, PFN's Tech Blog, WMT24++, NTREX-128.