Image: File:Hapag-Lloyd shipping container.jpeg by 20px|link=User:Kcida10|Kcida10 Kcida10 (talk) (Uploads)., licensed under CC0 1.0.
Your freight audit platform reports that 71 percent of invoices went straight through last month. That number tells you how often nobody looked. It does not tell you how often nobody needed to.
The distance between those two statements is where the money sits. An invoice that cleared because every charge matched the contract is a win. An invoice that cleared because the system could not resolve the load and quietly took the carrier's figure is an overpayment you will never go looking for, since nothing in your reporting records that it happened.
The touch rate rewards the same behaviour a benchmark does
Straight-through processing is the number the whole accounts payable profession is graded on. Ardent Partners' Accounts Payable Metrics that Matter in 2026, as summarised by Basware, puts best-in-class teams at 55.2 percent of invoices moving through with no manual touch, against 31.4 percent for everyone else. Nothing is wrong with measuring it. The trouble starts when it becomes the target.
A touch rate carrying no error budget is a coverage figure dressed as a quality figure. You can lift it any afternoon: widen a tolerance, let a fuzzy match count as a match, tell the system to take the closer of two plausible loads. Each move reads as progress on the dashboard, and each converts a visible exception into an invisible payment.
Current models make the incentive worse rather than better. Adam Tauman Kalai, Ofir Nachum, Santosh Vempala and Edwin Zhang argued in September 2025, in Why Language Models Hallucinate, that models confidently produce plausible falsehoods because training and evaluation reward guessing over admitting uncertainty. Most benchmarks grade in a way that penalises an uncertain response, so a system optimised to score well learns to answer anyway. Point something shaped by that pressure at a freight invoice, then grade it on touch rate, and you have reproduced the scoring flaw inside your back office.
The confidence score is not the safety net
The standard reassurance is a threshold. Let the agent attach a number to each extraction, approve above 0.9, route the rest to a person. That only works if the number means what it appears to mean, and on invoice documents it often does not.
Spotify's engineering team published a case study in December 2024 on building confidence scores for precisely this job: parsing vendor invoices that arrive in many languages and layouts, under ITGC and SOX review obligations. They tested averaged token log probabilities and found no clear correlation between that score and accuracy. Majority voting across several models did correlate, and they still ran Platt scaling over the result before anyone was allowed to trust it.
A measured version of the same finding landed last month. Yichao Jin and colleagues, in Perception, Layout, and Validation: Calibrated Confidence for Reliable Straight-Through Processing of Financial Documents (September 2026), evaluated vision language models on real invoices, synthetic invoices and ad-buy forms. Confidence the model reported about itself scored 0.54 to 0.74 AUROC, where 0.5 is a coin toss. Their decomposed score, built from perception, layout and validation signals, reached 0.90 to 0.99.
The operational consequence is in the next pair of figures. Holding the target error rate below 10 percent, the models' own confidence cleared between 0.1 and 7.0 percent of fields for automatic approval. The calibrated score cleared 49 to 72 percent. Same documents, same extraction, and the defensible automation rate moves by an order of magnitude depending on what sat behind the threshold. A touch rate quoted without the error budget it was measured against is not a result you can act on.
Needs data is an outcome, not a failure
Most audit systems offer two exits: approve or dispute. Whatever fits neither gets pushed through whichever door is cheaper to operate, and in practice that door is approve. A third exit fixes it, provided the exit is a named result rather than a dumping ground.
Penny treats needs data as a decision with reasons attached. She reaches it on the cases where freight genuinely withholds the answer.
- A detention charge arrives with no gate-in or gate-out timestamp from either party. Nothing in the file establishes who held the truck, which is the only question the charge has to answer.
- An accessorial is billed under a code that appears in no contract version you hold, because the amendment adding it was signed regionally and never loaded.
- A PRO number resolves to two loads that both moved that week, on that lane, for that shipper.
In the third case the agent could pick. The candidates differ by a few hundred dollars, one scores marginally higher on the match, and selecting it would nudge the touch rate up. It would also bind an audit result to the wrong shipment, the failure described in the freight audit bottleneck is load resolution. Choosing is worse than declining, and only one of the two appears on any report.
Do not expect the model to supply that restraint unaided. AbstentionBench, released by FAIR at Meta in June 2025, tested 20 frontier models across 20 datasets of unanswerable, underspecified and false-premise questions, and concluded that abstention remains unsolved. Scale did not help: larger models were no better at declining. Reasoning fine-tuning actively hurt, degrading abstention by 24 percent on average. Holding back is something you construct around a model, from rules, evidence checks and a queue with somewhere for the answer to land.
What the result has to carry
A queue of items reading "manual review required" costs more than no queue at all, because an analyst now rebuilds from nothing the work the agent already performed. The trace is the deliverable.
| What the item settles | Bare exception | Needs data with a trace |
|---|---|---|
| The question | Manual review required | The single fact that would resolve it, in one sentence |
| What was considered | Nothing shown | Each candidate, with the fields that agree and the fields that do not |
| Evidence | The analyst re-pulls every document | Rate confirmation, bill of lading and delivery receipt already attached |
| Stakes | Unknown until someone opens it | The dollar spread between the readings, against the invoice total |
| Owner | The queue | The named party holding the missing artefact |
| Deadline | Whenever somebody notices | A stated date, after which the item pays or disputes by rule |
That final row carries more weight than it looks. An unresolved item that sits forever is a slower version of approving by default, and payment terms do not pause for your backlog. Every item needs a clock and a written rule for what happens when the clock expires.
Four numbers that replace the one
Retire the standalone touch rate and report these instead, with a standing rule that the first is never quoted on its own.
- Auto-approval rate at a stated error budget. "71 percent" is unfalsifiable. "71 percent, at a measured error rate under 1 percent on sampled review" is a claim somebody can test.
- Silent error rate. Pull a random sample of auto-approved invoices each week and audit them by hand. Nothing else tells you what the automation cost. A team that never samples does not have a low error rate, it has no measurement.
- Resolution time, split by reason. Items waiting on a carrier and items waiting on your own contract repository are different problems with different owners. Reporting them together hides both.
- Share of items answered by a document you already held. When that fraction runs high, the shortfall is not uncertainty, it is plumbing, and plumbing is the cheaper of the two to repair.
The fourth metric is the cheapest to act on. A missing contract amendment, a POD sitting in a carrier portal nobody connected, a rate table loaded for one business unit and not the other: none of those require a better model, and every one of them raises the honest automation rate rather than the reported one.
What to do next
Take last month's automatically approved freight invoices and pull 30 at random. Audit each by hand against the contract and the execution evidence, then sort the findings into two piles: charges that were correct, and charges that passed because nothing in the system could tell either way. The second pile is your actual automation rate, and it will not match the one on the dashboard.
Bring that sample to a walkthrough and we will run the same invoices through Penny, with the needs-data items shown as they would reach your analysts: candidates, evidence, owner and clock, all on the item.