Why 95% extraction accuracy is not good enough

Do the multiplication

A document intelligence vendor quotes 95% accuracy. It sounds strong. It is the number that appears in the proposal, and it is usually true.

It is also field-level accuracy, and the number that governs your operation is document-level accuracy.

Take a reasonably typical invoice with twelve extracted fields: supplier name, supplier tax ID, invoice number, invoice date, due date, PO reference, currency, net amount, tax amount, gross amount, payment terms, bank details.

If each field is independently 95% accurate, the probability that all twelve are correct is 0.95 to the twelfth power. That is approximately 0.54.

Roughly 46% of your invoices contain at least one error.

At 98% per field, the document-level figure is about 0.98^12 ≈ 78% — still nearly a quarter of documents carrying an error. At 99.5% per field, you reach approximately 94%.

The relationship is multiplicative, and it punishes field count. This is why a vendor demonstration on a three-field document looks so much better than the same system on a full invoice, and why extraction quality that seemed adequate in a pilot becomes an exception queue in production.

Two caveats in the interest of honesty. Field errors are not perfectly independent — a badly scanned document tends to produce several errors at once, which clusters failures into fewer documents and makes the real figure somewhat better than the pure calculation. And not every field carries equal consequence: a wrong payment term is an annoyance, a wrong bank detail is a fraud incident. The arithmetic is a floor for thinking, not a precise forecast. It is still a great deal closer to your operational reality than the number in the proposal.

The metric that actually matters

The useful measure is straight-through processing rate: the proportion of documents that pass from intake to system of record with no human touch and no subsequent correction.

STP rate is a business metric. It maps directly to cost per transaction, and it is the number that moves the line your CFO looks at. Field accuracy is a component of it, alongside three things vendors rarely discuss:

Confidence calibration. A system that knows when it is uncertain is far more valuable than one that is marginally more accurate but uniformly confident. Well-calibrated confidence lets you route the doubtful 8% to a human and let the rest through untouched. Poorly calibrated confidence means either routing far too much to review, or letting errors through with high confidence attached — which is worse, because nobody checks them.

Validation rules. Most extraction errors are catchable without a human. Does net plus tax equal gross? Does the PO reference exist? Does the supplier tax ID match the supplier record? Is the total within the tolerance of the purchase order? Arithmetic and lookups catch a substantial share of errors at effectively zero marginal cost, and they catch exactly the errors that matter most.

Exception routing. What happens to the documents that fail. If they land in a queue that grows faster than it drains, your STP rate is irrelevant, because the backlog is the cost.

The validation layer is the product

This is the point we would most like a buyer to take away.

Model selection matters less than most procurement processes assume. The models available today are good, they are improving, and they are broadly interchangeable at the accuracy levels enterprise document work requires. Differences between vendors on raw extraction are real but rarely decisive.

What determines whether a deployment earns its cost is the layer around the model: the confidence thresholds, the validation rules, the routing logic, the review interface, and the feedback path that turns a corrected exception into an improvement rather than into a one-off fix.

That layer is where the engineering effort goes in every document intelligence build we have delivered. It is also the part that a demonstration never shows, because it only becomes visible at volume and under real supplier variance.

Questions to ask before signing

• "Is that accuracy figure field-level or document-level, and on how many fields?" If the answer is field-level, do the multiplication in the meeting.
• "What is your straight-through processing rate in production, on documents like ours?" Not accuracy. STP.
• "How is confidence calibrated, and can we set our own thresholds by field?" Bank details and payment terms should not share a threshold.
• "What validation rules ship with it, and can we add our own?" The arithmetic checks are usually the cheapest accuracy improvement available.
• "What is the exception rate, and what does the review interface look like?" Ask to see the operator's screen, not the dashboard.
• "When a human corrects an extraction, what happens to that correction?" If the answer is nothing, your STP rate will not improve after go-live.

Set the target on STP rate, measure it weekly against a baseline you established before anything was deployed, and watch the exception queue depth alongside it. Accuracy is a component. Straight-through processing is the outcome you are actually buying.