How to Estimate Token Usage Before AI Implementation — Steps to Calculate Costs in Advance

The amount of tokens consumed by AI agents and chatbots can be calculated before a PoC, once the text volume for the usage scenario and the model's unit price are determined. This article is written for system staff and planning staff tasked with pre-calculating AI adoption costs, and walks through the process from confirming the prerequisites to Step 1 (estimating input/output text volume), Step 2 (converting to token counts), Step 3 (calculating consumption per request), and Step 4 (calculating monthly cost). By the end, you will be able to calculate the monthly cost yourself from your organization's anticipated usage scenarios and reflect it in your PoC budget decisions.

Token volume calculations will fluctuate unless you first fix the model and pricing structure to be used, as well as the unit of the usage scenario. Even for the same processing volume, the estimate will change depending on the model selection and unit design, so these two points must be settled before beginning the calculation.

Selecting the LLM and checking the pricing table is the first task in token estimation. Unit prices differ by model for input tokens, output tokens, and cached input respectively, and the monthly cost can vary by several times depending on the model chosen, even for the same processing volume. Before beginning the estimation, the following four points must be settled.

• The name of the model to be used (including version. Since the discontinuation of a model or migration to a successor model may be announced, the target model as of the time of adoption should be specified)
• The unit price for input tokens and output tokens respectively (on many platforms, this is expressed as "per 1M (million) tokens")
• Whether a discounted unit price exists for reusing the same context, such as with prompt caching
• Whether there are regional surcharges or pricing differences based on contract type

Since the unit price of tokens changes every time the pricing table is updated, keeping the date referenced and a link or screenshot of the relevant model's pricing table in your estimation materials will make it easier to compare differences later when prices change. When comparing multiple models, you need to check not only the unit price but also the tokenization efficiency discussed later. If these are not settled at the early stage of model selection, the very premise of the monthly cost calculated in Step 4 will collapse.

Designing the unit for processing volume begins with enumerating all the text contained in a single request. In chat-style exchanges, all text sent—including the system prompt, user statements, past conversation history, document chunks inserted as search results via RAG, and the definition text for tool calls—is subject to token billing. You need to identify these components for each usage scenario and define "all text sent in one request."

Specifically, organize usage scenarios into the following units:

• Simple Q&A-style chatbot: system prompt + user statement + output only
• Search-combined type using RAG: the above plus search chunks (Top-K count and chunk size)
• Type including examples via few-shot: the above plus the token volume of example text
• Multi-turn agent type: the above plus the cumulative amount of past conversation history

If the unit is mistaken at this stage, both the per-request calculation in Step 3 and the monthly cost calculation in Step 4 will be thrown off. If you use RAG, tentatively deciding the chunk size and Top-K count in advance, and if you use few-shot, settling the number of example texts in advance, will make the subsequent token count calculation go smoothly. If there are multiple scenarios, you need to separate the units by scenario and estimate each individually.

Token consumption for a single request is determined by both the amount of text sent and the amount of text returned. On the input side, you need to write out the character count of the system prompt and user statements, and on the output side, the expected length of the response, as numerical values for each usage scenario.

Estimating the character count of a system prompt starts with counting fixed text and variable text separately. Since the system prompt contains the same content attached to every request throughout the conversation, you only need to count its characters once. However, since the length of user messages varies depending on the content of the inquiry, you need to work out expected character counts for both short and long messages.

Specifically, quantify the following three items separately.

• The fixed character count of the system prompt (the full text including role definitions, output format instructions, prohibitions, etc.)
• The character count of the tool definition text when using tool calls (including function names and argument descriptions)
• The expected character count of user messages (divided into the shortest, longest, and average expected inquiry text)

In scenarios that use RAG, at this stage you do not count the character count of the chunk body itself; you only record that there is a slot for inserting search results. Chunk size and count are to be added in Step3, and counting them here as well would result in double-counting in subsequent calculations. If you are using few-shot examples, treat the example text the same way, as fixed character count.

When counting characters, accuracy improves if you prepare multiple actual inquiry text patterns and count characters separately for the shortest, longest, and average patterns. For the system prompt as well, if you plan to change the output format instructions or prohibitions, you need to recount using the revised text. At this stage, do not yet convert to token counts—completing only the character count tally makes the conversion work in Step2 easier to organize.

Estimating response tokens needs to be handled separately from the character count tally on the input side. Since response length varies greatly depending on the model and the content of the prompt's instructions, it should not be treated as a fixed character count; instead, expected ranges should be separated by type of inquiry.

Specifically, record response length separately for the following three patterns.

• Settings that return short answers in a one-question-one-answer format (FAQ responses, standard replies, etc.)
• Medium-length responses that include explanations or summaries (responses including step-by-step instructions or enumeration of multiple items)
• Responses involving long-form generation (document creation assistance, report generation, etc.)

If the output settings limit the maximum token count, also record that upper limit. If the configuration generates multiple response candidates, you also need to include in your estimate the fact that output tokens increase according to the number of generations.

Since it is difficult to accurately estimate the average response character count at this stage, a reliable method is to run sample executions with representative inquiry text and measure the actual response character count. Completing this measurement at the design stage before the PoC improves the accuracy of the token conversion work in Step2. The overall approach to PoC design is covered in the AI Adoption PoC Design Guide.

Character count and token count do not correspond one-to-one; the conversion ratio varies by language and character type. Applying the English rule of thumb of 1 token ≒ 4 characters directly to Japanese produces large errors, so actual measurement using a BPE tokenizer is necessary.

Japanese and English have different token counts even for the same character count. English is centered on Latin characters, and words or parts of words are often treated as-is as tokens, so the number of characters per token tends to be relatively large. On the other hand, in non-Latin-character languages such as Japanese, Chinese, and Korean, the number of characters corresponding to one token tends to be smaller, meaning that the same character count tends to produce a higher token count—a structural tendency.

The reason this difference arises depends on which language's corpus the tokenizer's vocabulary is primarily built from. A BPE tokenizer registers frequently occurring character string combinations as vocabulary, but the higher the frequency of English appearance in the training data, the more easily English words and subwords are grouped into a single token. Japanese hiragana, katakana, and kanji have diverse character combination patterns, so even with the same vocabulary size, they tend to be split closer to the single-character level, resulting in increased token consumption.

This difference also varies depending on the encoding used (such as cl100k_base or o200k_base), so you should avoid applying a fixed conversion ratio for calculations. When anticipating Japanese-centered inquiries, you should not simply apply the English rule of thumb of 1 token ≒ 4 characters; instead, you need to check the converted token count using actual inquiry text.

For actual measurement of token counts, use the method of directly running a BPE tokenizer rather than guessing a conversion ratio. For OpenAI models, use the tiktoken library; for Hugging Face-based models, use the tokenizers library—doing so gives you an accurate token count based on the target model's encoding.

The measurement procedure is as follows.

1. Check the encoding name corresponding to the target model (e.g., o200k_base or cl100k_base)
2. Prepare each piece of character count data tallied in Step1 (the system prompt's fixed text, the shortest/longest/average patterns of user messages, and response samples)
3. Pass the text to the tokenizer and record the returned token count for each pattern

By running this code for each of the system prompt, user messages, and response samples, and creating a correspondence table between character counts and token counts, you eliminate the need to re-estimate token counts from character counts in the calculations from Step3 onward. The same check can be done with online Tokenizer tools, but when there are many items, batch processing with a library is more efficient.

It's best to keep the correspondence table organized by inquiry text pattern, and to operate in a way where, whenever the system prompt text is changed, you re-take the measurements using the same procedure—this way, the premises of your estimate won't drift out of sync.

The token consumption of a single request is the combined total of the system prompt, user input, RAG chunks, few-shot examples, and generated tokens. Since single-turn Q&A and multi-turn conversations that retain history accumulate tokens differently, they need to be calculated separately.

In requests that incorporate RAG, in addition to the system prompt and user input, the chunks retrieved through search are added directly to the input tokens. Since this added amount is determined by "chunk size × Top-K count," increasing the chunk size raises the token count per chunk, while increasing Top-K proportionally raises the count based on the number of chunks. These two variables represent a trade-off between accuracy and cost: setting Top-K too narrow risks missing relevant documents, while setting it too wide inflates input tokens and strains the monthly budget.

Calculation example: If the chunk size is 300 tokens and Top-K is 5, the RAG addition is 300 × 5 = 1,500 tokens. Adding the token counts of the system prompt and user input to this gives the total input tokens for the RAG configuration.

When few-shot examples are also used, the token count of the example text is further added on top of this amount. When chunks approach the upper limit of the context window, a design that limits the amount inserted through summarization or relevance filtering becomes necessary. Optimization of chunk design is covered in What Is Adaptive RAG? How Query-Driven Dynamic Retrieval Balances Cost and Accuracy.

In a multi-turn design that retains conversation history, all past user input and responses are resent with every subsequent request. Unlike single-turn Q&A, the key point to grasp in cumulative calculation is that input tokens keep piling up as the number of turns increases.

Calculation example: If the system prompt is 200 tokens, and each turn consists of 50 tokens of user input and 150 tokens of response, then for the request at the 3rd turn, "system 200 + (50+150) × 2 turns of history + 50 tokens of user input for the 3rd turn" accumulates as input tokens. As turns increase, this history portion grows linearly, and as it approaches the upper limit of the context window, a decision must be made to either summarize older messages or switch to a sliding-window approach that retains only the most recent N entries.

At the estimation stage, tentatively setting an expected average number of turns and session continuation rate, and calculating "cumulative token volume per session" as a separate line item from the per-request unit price of single-turn requests, prevents the two from being conflated. In scenarios where topics frequently switch within a single session, the cumulative volume varies significantly depending on where the retained history is cut off, so whether or not switch detection is in place should also be included as an assumption in the estimate. If a design that compresses history through summarization is adopted, the average post-compression token count should be measured directly before reflecting it in the monthly cost.

Once the token consumption for a single request is settled, the monthly cost can be found simply by multiplying the expected number of requests by the unit price. However, relying only on normal-time figures overlooks the cost during peak access periods and API-side limits.

Monthly cost is calculated as "token consumption per request × monthly request count × unit price." Since input tokens and output tokens have different unit prices, the key point in the calculation is to compute them separately before summing, rather than combining them from the start.

Calculation example: Suppose input tokens are 1,800 per request (system prompt + user input + RAG chunk addition), output tokens are 300 per request, and the monthly request count is 50,000. In this case, the input cost is calculated as "input token count (1,800) × monthly request count (50,000) ÷ 1,000,000 × input unit price," and the output cost is calculated as "output token count (300) × monthly request count (50,000) ÷ 1,000,000 × output unit price." Summing these two gives the monthly cost. Refer to the official pricing page of the model being used for unit prices.

For multi-turn designs, multiply the cumulative token volume per session by the number of sessions before proceeding to the unit price calculation. In systems where single-turn and multi-turn designs coexist, calculating the monthly cost separately by scenario and then summing them at the end makes the breakdown easier to track.

For models with a set unit price for cached input, separating fixed, reused portions such as the system prompt as cache targets and calculating the regular unit price and cache unit price separately brings the estimate closer to the actual cost. Keeping a list of monthly costs by scenario allows you to verify, using your own data, which components account for the bulk of the cost.

Calculating monthly costs uses average request counts, but actual traffic varies significantly by time of day. If you budget based solely on off-peak RPS (requests per second), you overlook traffic concentration during business hours or specific events, which can lead to hitting the API's rate limits or having unexpected burst costs exceed your budget.

In your estimate, record off-peak RPS and peak RPS separately, and calculate the monthly conversion for both. Since unit prices may differ for burst usage during peak periods depending on the model or plan, checking how bursts are handled in the pricing table helps ensure your calculation assumptions remain valid.

Calculation example: Assume 2,000 requests per hour during off-peak times, and 8,000 requests per hour during peak hours (2 hours per day). The daily request count would be 44,000 for the 22 off-peak hours plus 16,000 for the 2 peak hours, totaling 60,000 requests. Based on this total, you calculate the monthly cost, and by additionally isolating just the 2 peak hours for unit price calculation, you can determine what percentage of the total monthly cost the peak period accounts for.

In scenarios where this ratio is high, it becomes a useful basis for identifying which time periods reach capacity limits first, and for deciding whether to include throttling or a switch to asynchronous processing as a PoC requirement. Note that budgeting based solely on the off-peak unit price means the overage from peak periods will appear as an unexpected additional cost.

Estimation errors more often stem from "forgetting to count" something than from mistakes in the calculation formula itself; if you overlook part of the text being sent and proceed to unit price calculation, the actual billed amount can far exceed expectations. Reviewing and separating elements that can be treated as fixed costs from elements that can be compressed through discounts is a practical way to reduce this error.

The failure of overlooking tokens from system prompts or tool definitions occurs when the scope of estimation is limited to only user utterances and responses. The system prompt is a fixed portion whose content remains the same throughout the conversation and is attached to every request; when the text is long, it can account for the majority of the input tokens in a single request. If this fixed portion is left uncounted when proceeding to unit price calculation, the actual measured monthly cost ends up higher than the estimate.

In configurations that use tool calls, the tool definition text—including function names, argument descriptions, and return value type information—is also sent with every request and added to the input tokens. As the number of tools increases, this additional amount accumulates, and in AI agents with multiple registered tools, the tool definition text can sometimes consume more tokens than the user's utterances. In configurations with many registered tools, it is worth checking whether definitions for unused tools are still being sent every time.

To prevent oversights, from the stage of counting characters in Step 1, treat the system prompt's fixed text and tool definition text as separate line items, and make it standard practice to always add them into the per-request calculation in Step 3. It also helps to establish a procedure for recounting token counts whenever the text or tool configuration changes, so the estimation's assumptions don't drift.

The failure of not including prompt caching in estimates occurs when fixed portions, such as the system prompt, are calculated at full price every time. In configurations where the same context is sent repeatedly, some platforms apply a lower-than-normal unit price to cached input, and ignoring this discount results in overestimating the monthly cost. Fixed system prompts, tool definition text, and configurations where RAG references the same document every time are all portions likely to qualify for caching.

Conversely, in designs that shrink history through summarization or compression, there is a failure of estimating the monthly cost using the pre-compression assumed value rather than recalculating with the actual post-compression token amount. Since the summarization process itself also generates input and output tokens, even if compression reduces the volume of the main transmission, overlooking the cost of the summarization calls can cancel out the savings.

At the estimation stage, record the fixed portions eligible for caching and the variable portions not eligible for caching separately, and follow the procedure of checking whether the pricing table has a cache pricing item before reflecting it in the monthly calculation. When introducing summarization or compression, you need to measure the actual token counts both before and after compression, and compare them including the cost of the compression process itself. Specific implementation patterns for caching and compression by model are covered in the LLM Cost Optimization Guide.

Estimation results only become meaningful once they are incorporated as numerical values into the PoC's verification items. Rather than using the calculated monthly cost directly as approval material, include in the PoC implementation plan a step that compares the token counts actually measured using representative inquiry samples against the assumed values used in the estimate.

If the comparison reveals that the measured values significantly exceed the assumed values, identify whether the cause lies in system prompt bloat, RAG chunk design, or history accumulation in multi-turn conversations, then adjust the relevant design variable and recalculate. In a PoC comparing multiple models, line up the monthly costs that include not just unit prices but also measured token counts, and judge which model has the advantage in total cost.

Including the monthly cost that reflects peak RPS within the PoC's budget allocation helps avoid a situation where the plan falls apart due to rate limits or cost overruns at the time of production rollout. Keep the assumptions used in the estimate (model, unit price, unit design) recorded in the PoC report, so that any discrepancies can be traced when remeasuring with production data.

We cover two points: how to choose a tool for actually measuring token counts, and the impact of model changes on cost.

There are tools that accurately measure token counts. OpenAI's online Tokenizer lets you quickly check by simply pasting in your text, and when dealing with a large number of items, batch processing with the tiktoken library is practical. Hugging Face's tokenizers library can also be used for actual measurement, provided it has an encoding that supports the target model. In all cases, the common point is that measurement is done directly using the target model's encoding, rather than estimating from character count.

The per-unit price for each model is clearly stated in the pricing table, but the price difference alone is not enough to determine the total cost. Even for the same text, different models have different tokenizer vocabulary structures, causing the token count itself to increase or decrease, so the price difference alone is insufficient as a basis for comparison. You need to check both the unit price and the tokenization efficiency before making a comparison.

Estimating token volume begins with establishing the premises, proceeding in the following order: in Step 1, estimate the amount of input and output text; in Step 2, convert the character count into a token count; in Step 3, calculate the consumption per request; and in Step 4, calculate the monthly cost based on the expected number of uses. Within this flow, the main sources of error are failing to count system prompts and tool definition text, and overlooking the reduction effects from caching and compression, so the habit of recording fixed and variable portions separately determines accuracy. The calculated figures should not be used as-is as approval material; instead, they should be compared against actual measured values from representative inquiry samples, and after identifying the design variables where discrepancies appear, they should be incorporated into the PoC budget and verification items. Premises such as the model, unit price, and unit design should also be kept on record so that discrepancies can be tracked when re-measuring with production data. First, choose one expected use case for your own company and start tallying character counts from Step 1.