Multilingual Invoice OCR Evaluation: Ground Truth Data, Accuracy Calculation, and Accounting Integration Checks

Evaluating multilingual invoice OCR in manufacturing means extracting characters and items from received invoices, and verifying—using your own company's ground truth data—the entire process through confirmation, correction, and handoff to the accounting system. What accounting and DX staff in manufacturing want to determine is not just how many characters were read correctly in a demo, but whether incorrect amounts or currencies can be caught, and whether the workload is actually reduced once confirmation work is included. This article starts from the question of what to decide and what to keep before evaluation, shows an approach for splitting real invoices into a tuning set and a final evaluation set and fixing the ground truth data and matching rules together with accounting staff, and explains how to design this as a small-scale multilingual OCR PoC.

Evaluation proceeds in the order of ①character recognition, ②item extraction, ③accounting integration, and ④operations and cost, with the target invoices and ground truth fixed in advance. We assume scenarios involving Japanese, English, Thai, and Lao, and explain using calculation examples and decision tables. We do not set a universal passing score for every company; instead, it should be decided based on the errors your company can tolerate and the conditions for manual confirmation. The numerical examples in the text are fictional data used to illustrate the calculations. For tax treatment and compliance with statutory retention requirements, please check with your accounting manager or a specialist for each applicable country.

First, decide on the target business partners, languages, and invoice forms, as well as the endpoint of the processing you will leave to OCR. "Creating input candidates," "transcribing approved data," and "passing data to payment processing" each carry different impacts from errors and require different tests. Record the scope, the definition of ground truth, the passing conditions, and the decision-maker in the evaluation plan, and get approval before looking at the product's results.

Take inventory of the received invoices and list them by business partner, language, currency, layout, input method, and page count. Even if Japanese transactions make up the majority, if a small number of Thai or Lao invoices are included, make sure those conditions are not dropped from the evaluation table. Classifying by language name alone can cause you to overlook the difference between, say, a single-page English document with one currency and a multi-page document with continuing line items.

In addition to a set of invoices representative of normal receiving conditions, gather invoices on which you want to test how the process handles difficulties—skewed scans, faint printing, stamped seals, folds, handwritten additions, missing pages, and so on. Use the normal set to estimate the overall workload burden, and use the difficult-condition set to check how exceptions are handled, reporting the results separately. This is because the correction rate of a set loaded with many difficult cases should not be treated as-is as the company-wide monthly correction rate.

Split the prepared invoices into a tuning set—used to adjust settings, extraction rules, and confirmation thresholds—and a fixed final evaluation set. Manage these by original document unit so that the same invoice's PDF and scanned image, or near-duplicate copies of the same invoice, do not end up on both sides. If adjustments were made while looking at the answers in the final set, re-evaluate using a separate, unused set of invoices. Google's explanation of data splitting also covers the separation of training, validation, and test sets, the exclusion of duplicates, and the need for a test set that represents real data.

Plan the required number of samples only after deciding on the target conditions and the tolerable error rate. JAISA's Image Recognition Technology Introduction Guidelines, page 26 describes the difficulty of uniformly determining sample size, and presents an approach of communicating customer requirements and consulting together to decide on a target. Do not make collecting a specific number of sheets itself a passing condition; instead, record how many documents and items were tested under each condition, and what remains unevaluated.

The deliverable is an invoice ledger containing "document ID, condition, tuning or final-evaluation use, ground-truth creator, and version." Even when using images processed for evaluation purposes, keep the correspondence to the originals. Before handing data to an outside service, check the purpose of use, the recipient, storage and deletion practices, and access rights, and apply any necessary masking.

Ground truth data is a comparison baseline that a person determines by looking at the original. If you copy OCR output directly into the ground truth, you will overlook the same misreadings. Have different people handle creation and verification, and for places that cannot be read or where judgment is split, record the reason for holding rather than filling in a guess. What matters more than the number of people involved is being able to distinguish between finalizing the ground truth and tuning the tool.

Each item should carry the original text value, a value for machine processing, and a conversion rule. For example, an invoice number should be a string that preserves leading zeros, an amount should be a representation that can be handled precisely as a decimal number, and currency should be a separate item. If a date reads "03/04/2026," hold off on conversion until the interpretation can be fixed based on the document's region and transaction terms. Do not determine date order or currency based on language alone.

Uniformly rounding unit prices or quantities to two decimal places loses information at the stage of matching against the original. Define decimal places, digit grouping, negative numbers, discounts, unit conversions, and rounding on an item-by-item basis, and record the before-and-after of conversions to match the accounting side's intake specifications. Even if amounts match, a mismatch in currency is still a mismatch. Also distinguish between a tax category that is simply not recorded and one that failed to be extracted. The determination of journal entries and tax codes should be finalized by accounting staff as a business rule separate from the recognized characters.

Line items should carry correspondence via "document ID, page, ground-truth line ID, item number, quantity, unit, unit price, tax/discount, line amount." Relying solely on line order means that once one line is missing, all subsequent lines become mismatched. Decide in advance how to handle continuation lines across multiple pages, separate lines with the same item number, and per-page subtotals.

Roles can be organized so that accounting staff approve values and business rules, technical staff record settings, outputs, and integration, and the evaluation lead approves scope and decisions. Even when roles are combined, do not adopt a procedure of fixing settings using the ground truth from the final evaluation. Fix the ground-truth file, item definitions, and decision rules under the same version number, and redo any affected aggregation whenever changes are made.

Character recognition compares the ground-truth text and the output text using the same rules. CER and WER are metrics for examining error tendencies. Even if the overall invoice values look good, a single-character error in the invoice amount can remain, so business-critical items should be evaluated separately in Step 2.

CER is the character error rate, and WER is the word error rate. The reference and output are aligned using the minimum edit distance, counting substitutions, deletions, and insertions. Following the definitions in OCR-D's evaluation specification, the calculation takes the following form.

CER = (Substituted characters S + Deleted characters D + Inserted characters I) ÷ Reference character count N

WER = (Substituted words S + Deleted words D + Inserted words I) ÷ Reference word count N

As an illustrative calculation example, if the reference is "AB1234" and the output is "AB1284," there is 1 substitution, 0 deletions, 0 insertions, and 6 reference characters, so the CER is 1÷6 = approximately 16.7%. For "AB12345," there would be 1 insertion. Since CER can exceed 100% when there are many insertions, it is not treated as "100% minus CER equals the character recognition rate." For regions where the reference is empty, the denominator becomes zero, so the number of unnecessary character outputs is recorded separately from the normal CER.

As an example of WER calculation, consider the reference "TOTAL USD 100" and the output "TOTAL USD 1000," split into 3 words by whitespace. Since there is 1 word substitution, the WER is 1÷3 = approximately 33.3%. This value changes depending on the word segmentation rules. If comparing Japanese, Thai, and Lao using the same method, fix the word tokenizer, dictionary, and version used. When counting by characters as well, record the Unicode normalization, the character units including combining characters, and the handling of whitespace and line breaks. OCR-D uses NFC-normalized grapheme clusters.

Strict comparison against the original text and comparison that only adjusts agreed-upon notational differences should be presented separately. A conversion that standardizes whitespace handling is not the same as a conversion that removes currency symbols, decimal points, and minus signs. To make evaluation easier, check whether important errors have been removed by comparing the original text with the converted text.

In actual aggregation, record the number of documents, reference character count, S/D/I, and CER for each language, form type, and image quality. The overall CER is calculated by summing the error counts across all documents and dividing by the total reference character count. Distinguish this from a value obtained by simply averaging the CER of short and long documents. Do not silently exclude documents that failed to be read from the successful ones; record both the number of failures and the reasons for exclusion from evaluation alongside the results. This narrows down the conditions under which character recognition needs correction and the points that should be checked during item extraction.

In item extraction, not just the values but also "which document, and which item or line item the value belongs to" is cross-checked. By separating the precision, recall, and F1 of key items, and the proportion of documents where a full set of important items is present, it becomes easier to identify amount and currency errors that would otherwise be hidden within average values.

First, the matching conditions are determined. In the evaluation design of this article, an output is counted as a TP (True Positive, i.e., correct extraction) when the document ID, item name, and required line item correspond, and the value matches the reference. An output that does not correspond to any reference is an FP (False Positive, i.e., extraneous or incorrect extraction), and a reference that could not be obtained is an FN (False Negative, i.e., a miss). If an incorrect value is output, it counts as both an FP for the erroneous output and an FN for failing to capture the original correct value. Matching is one-to-one, and multiple outputs are not matched to the same reference item. Any remaining duplicate outputs are counted as FP. This counting method is also used in Google Document AI's evaluation description.

Precision = TP ÷ (TP + FP)

Recall = TP ÷ (TP + FN)

F1 = 2TP ÷ (2TP + FP + FN)

As a calculation example, consider a case where, out of 100 reference items, 98 were output, of which 95 were correct, 3 had incorrect values, and the remaining 2 were not output. TP = 95, FP = 3, FN = 5. Precision is 95÷98 = 96.94%, recall is 95÷100 = 95.00%, and F1 is 190÷198 = 95.96%. These figures are for illustrating the method and are not actual measured values of a product. Whether the denominator is zero is determined separately for each metric. When a reference exists but there is no output, precision cannot be calculated, and recall and F1 are 0. When there is neither a reference nor an output, all metrics are treated as not calculable, and a perfect score is not assigned.

First, aggregate by item: invoice number, business partner, date, amount, currency, tax notation, product number, quantity, and unit. The overall F1 is calculated after summing TP, FP, and FN separately, and this is distinguished from a simple average of F1 by language or by item. Matches on the original text and matches on approved converted text are kept in separate columns. The process of mapping business partner name abbreviations to the same company should also be evaluated separately from OCR character recognition.

The evaluation table should be structured so that errors in important items can be read first.

The model's confidence score is kept separate from the accuracy rate measured on hand with actual invoices. The threshold for routing to review is determined using tuning data, and the fixed threshold is then tested on the final set. Aggregate examples where errors occur despite high confidence, and also test review conditions other than confidence, such as amount/currency combinations and missing line items. Microsoft's Transparency Note addresses evaluation for one's own use case and design that includes human review.

In integration testing, the success response of the transmission is cross-checked against the content saved on the accounting side. Connect to a test business entity/environment, and track from recognition through registration using the document ID. Even before the OCR settings are finalized, integration specifications can be verified in advance using input created from reference data.

Record the target product, version, and integration method of the accounting software, and verify the required items, character encoding, date, currency, tax codes, authentication, and limitations from the official specifications actually used. Do not assume common specifications such as "all APIs use OAuth" or "all retransmissions are automatically deduplicated." If both CSV and API are adopted, test each separately.

In the normal-path test, cross-check the original document, the reference data, the OCR output, the converted data, and the value saved on the accounting side, one by one. For CSV, check the delimiter, quotation marks, line breaks within cells, leading zeros, decimals/negative numbers, and multilingual characters. For API, check the field types, required values, authentication expiration, request limits, and error responses. The pass criteria should include not just the number of records imported, but also that the amount, currency, business partner, and document correspondences all match.

For failure-scenario testing, include the following perspectives. This is an example of evaluation item design and does not mean that all accounting software behaves identically.

Different business partners may issue invoices with the same invoice number. Duplicate determination should not be based on the number alone; design how the original document, business partner, transmission history, and accounting-side ID are linked. Also account for corrected invoices and cancellations, and confirm that a new transaction is not erased just because it has "similar values" and is treated as a duplicate.

After testing, cross-check the list of input documents against their status: correctly registered, pending review, failed, or cancelled. Create a tracking sheet that leaves no single record unaccounted for, and retain results even after resuming following an error fix. Even if the API success rate alone is high, if there are unsent records, duplicates, or incorrectly saved values, the integration should not be passed.

The final decision is made by combining accuracy, mechanisms to stop errors, the burden on staff, and recovery procedures. The acceptance criteria used here are determined by each company; we do not set an industry-wide standard stating that unattended operation becomes possible once a certain F1 is exceeded.

Before evaluation, decide the "conditions" in the following table together with the business owner, and finally fill in the measured values and reasons for not meeting them.

"No critical errors were observed during testing" and "critical errors do not occur" are different things. Even if there are zero errors in a small number of forms, this does not guarantee future safety. Introduce the system only within the scope that has been evaluated, and for areas with significant impact—such as amounts, currencies, trading partners, and tax handling—either keep approval or verification in place, or clearly define the conditions for stopping.

Manual labor hours cannot be measured solely by the number of items corrected after OCR. Sum up the time spent on confirmation, correction, exception investigation, resending, and reconciliation with accounting for each document, and compare it with the time taken to process the same forms using the conventional procedure. In an illustrative calculation example, if confirmation/correction takes 60 minutes, exception investigation takes 20 minutes, and reconciliation takes 10 minutes, the total manual work is 90 minutes. Do not multiply this already-summed time by the number of corrections again. Also separate the one-time costs of initial setup and creating correct answers from the monthly usage fees and confirmation labor hours.

The judgment should be one of: "start within the scope where all conditions are met," "start with limited scope while retaining manual confirmation," or "fix the cause and re-evaluate," and unevaluated languages or forms should be explicitly noted. If, after introduction, a trading partner's format or the model's version changes, confirm the impact using the fixed evaluation set and the new forms.

Even when using the same forms, the conclusion changes if the denominator for aggregation, the scope covered, and how confirmation work is counted change. Before filling in the comparison table, check whether the selection method retains only favorable results, or whether the operational burden has been overlooked.

Even if the overall F1 is high, the results may be skewed by the large number of Japanese-language forms, causing errors that occur only in a small number of Lao-language forms to be overlooked. Show the TP, FP, FN, and document count for each language and form type, and do not extend the conclusion to assume that "each condition is good because the overall result is good." Even when multiple languages are mixed within the same invoice, do not fold this into the overall aggregate as "language unknown"; record it as a mixed-language condition.

Use the same fixed set and correct answers for the products being compared, and keep records of the version of the features/models adopted, preprocessing, settings, and evaluation date. Check the languages supported for character recognition and the languages supported for invoice item extraction separately for each feature. Do not use the fact that a language is handled in the OCR portion as grounds for guaranteeing that it also covers extraction of trading partner names or line items.

In the results table, list the number of documents input, the number of successful processes, the number of failures, and the number of exclusions. If accuracy is calculated only from forms that were successfully processed, clearly state this condition and also show the overall processing failure rate at the same time. It is necessary not to mix forms used for tuning into the final evaluation, and not to present values after removing difficult forms as the accuracy of the entire actual operation.

If the number of cases cannot be increased, report the confirmed scope narrowly. For example, design the system so that input assistance begins only with evaluated trading partners, while unevaluated languages/formats are routed to the conventional manual process. By expanding the scope after additional data becomes available, there is no need to approve the unexamined scope all at once.

Even if the text created by OCR, the converted CSV, and the accounting-side slips are all in place, if the received original cannot be traced, it becomes difficult to make corrections or respond to inquiries. Retain the original file, and test that it is possible to trace from the document ID to the extraction results, the reviewer, the changes made, the transmission history, and the storage destination ID. Also ensure that the staff can check the current status of processing even when an image is missing, ingested multiple times, or processing stops midway.

When considering Japan's retention system, distinguish between scanner storage, which involves digitizing paper documents received in physical form, and the storage of data received through electronic transactions, such as PDFs attached to emails. The National Tax Agency publishes relevant materials for each on its Electronic Books, etc. Retention System Special Site. Do not use OCR accuracy or successful import into accounting alone as grounds for compliance with retention requirements. Please verify the specific requirements, such as retention periods and search/correction history, according to the method of receipt and your company's conditions. Since overseas offices have their own local systems, do not apply Japan's requirements as-is.

When testing external OCR services, make sure to check contractual matters such as data usage, storage location, subcontracting, handling of use for training, retention period, deletion methods, and access permissions. Since these vary by product and contract, do not judge based solely on general feature names; verify them according to the configuration you adopt. Including the deletion of evaluation data and revocation of permissions as part of the completion criteria helps prevent test copies from remaining indefinitely.

Assign responsibility for handling exceptions as well. Decide who is responsible for confirming cases of low confidence, unknown currency, total mismatches, duplicate candidates, and integration failures, and test the handover procedure and pending deadline for when the responsible person is absent. If a backlog of pending confirmations builds up at month-end, include that processing time in the calculation of the implementation's effectiveness. Do not finish merely by looking at the recognition results on screen; also verify that you can return to the original document, that staff can halt the process, and that reconciliation is possible after recovery.

What tends to be confusing in the evaluation plan is the schedule estimate and how far to proceed with automated processing. Both become easier to judge once the scope to be evaluated and the confirmation work to be retained are determined.

First, confirm the day samples will be gathered, the day correct answers will be approved, and the day the integration environment becomes available, then estimate based on the volume of work. A method that sets the evaluation period uniformly at some number of weeks will fail to account for waiting on suppliers to provide materials or waiting on accounting staff for confirmation.

Break the estimate down into "collecting and confirming use of documents," "creating and inspecting correct answers," "OCR configuration and tuning," "integration implementation," "testing on a fixed set," and "handling defects and retesting." Record the working hours needed for each task and any external approval/provision waits separately, and organize which tasks can be done in parallel. First measure the actual time required for creating and confirming correct answers on a small sample, then update the schedule based on the remaining volume and difficulty. If you add target languages or integration methods, revisit the corresponding workload as well.

F1 alone cannot determine this. Check the number of cases where errors in amount, currency, or trading partner passed through, the correctness of the full set of key fields, duplicates and omissions, how approval-pending items are handled, and recovery from failures.

Decide the conditions for routing to automated processing, and measure the errors within that set alone as well as the proportion routed to manual handling. If raising the confidence threshold causes the number of automated cases to drop drastically, then even if accuracy appears high, it may not translate into reduced workload. For operations that affect payment or tax determinations, design the necessary approvals, permissions, and reconciliation together with the business owner. In a small-scale evaluation, you can also start with limited input assistance and make the decision to expand gradually within a range where stopping and returning items functions properly.

To evaluate multilingual invoice OCR, first determine the target documents and the endpoint of processing, then prepare correct-answer data approved by accounting staff. Separate the data used for tuning from that used for final evaluation. For character recognition, check CER and WER; for item extraction, check per-field F1 and agreement on the full set of key fields; for accounting integration, check stored values, duplicates, omissions, and recovery.

Formulas alone do not determine whether to adopt the system. The core of the decision lies in which errors to stop, who will check them, and whether the burden is actually reduced once checking and exception handling are included. Decide the required number of documents, the period, and the passing criteria yourselves based on your target conditions and the impact you are willing to accept, and avoid generalizing results beyond the scope you evaluated.

First, create a ledger of invoices that actually arrive along with field definitions, and fill in your company's conditions into the judgment table in this article. For the basic scope of processing, see the AI-OCR / Intelligent Document Processing Implementation Guide; for designing the confirmation workflow, see Fundamentals of Human-in-the-Loop (HITL) AI Automation. Combining these will help you concretize operations beyond the evaluation stage.