Why the Orchestrator Is a Language Model and the Workers Are Not

A carrier sends you an invoice carrying a charge code your audit rules have never seen. One kind of system stops, drops the invoice into an exception queue, and waits for a person to reach it on Thursday. Another kind reads the code, decides it resembles detention, and approves $840 against a contract clause that does not exist.

Both answers are wrong, and they sit at opposite ends of one design question. Hardcoded pipelines cannot adapt to freight. Fully generative pipelines cannot be trusted with money. Our answer is to split the system: the orchestrator is a language model, and the workers underneath it are not.

Two architectures, two different failures

The rules engine came first. Someone enumerated the charge codes, the document layouts, the contract structures and the carrier quirks, then wrote a branch for each. The result runs fast, costs almost nothing, and reproduces perfectly. It is also a photograph of what freight looked like on the day it was written.

Freight does not hold still for one. A detention charge on a dry van and a demurrage charge on an import container are different clocks with different owners, and a carrier may bill both under a code that reads DET. Contract pricing lives in a matrix on one lane and a formula on the next. Enumerate every case this quarter and a carrier amends a tariff rule in April.

The gap widens from there, surfacing as an exception queue that grows faster than the team working it. Gartner's press release of June 25, 2025 predicts that more than 40 percent of agentic AI projects will be cancelled by the end of 2027, citing escalating costs, unclear business value and inadequate risk controls. Repainting a rules engine does not move it off that list.

The generative pipeline is the fashionable opposite. Hand a model the invoice, the contract and the event record, ask for a verdict, and it returns one for anything you give it. Coverage stops being the problem. What arrives instead is a figure nobody can trace back to an input and nobody can rebuild six weeks later, when the carrier's billing desk asks where it came from.

What the agent benchmarks actually measure

The case for keeping the model away from the money is not a hunch. It is measured behaviour from agents asked to carry a whole job through tools.

τ-bench, published by Shunyu Yao, Noah Shinn, Pedram Razavi and Karthik Narasimhan in June 2024, sits a function-calling agent in front of a simulated user with domain APIs and written policy, then grades it by comparing the final database state against an annotated goal state. State-of-the-art function-calling agents succeeded on fewer than half the tasks. The second metric matters more to an audit team: pass^k asks whether the agent solves the same task on all k independent attempts, and the paper records pass^8 below 25 percent in the retail domain.

Read that as a finding about consistency rather than capability. An agent that gets a task right on Monday and wrong on Tuesday is not partly reliable. It is unreliable with good days.

METR frames the same limit as duration. Its time horizon measure, introduced by Thomas Kwa and colleagues in March 2025 and updated on metr.org through September 8, 2026, asks how long a task can run, in human expert time, before a model stops succeeding at a chosen rate. METR publishes it at 50 percent and 80 percent success, and the 80 percent horizon is the shorter one. The lesson holds at either threshold: the longer the stretch you hand a model without checking it, the worse your odds.

Span is the lever, then. Every step removed from the generative stretch is a step that cannot drift.

A worker is code that refuses rather than guesses

Beneath the orchestrator sits a collection of small, dull, typed functions. A free time calculator. A rate lookup against the contract version in effect on a given date. A revenue ton allocation across four house bills sharing one container. A completeness check against the fields 46 CFR 541.6 requires on a demurrage invoice.

None of it is clever, and each function carries four properties. A system that skips the fourth turns generative again without anyone deciding to.

• Determinism. The same inputs return the same output, on any hardware, in any year the claim is still open.
• Typed inputs. A missing gate-out timestamp raises an error instead of collecting a plausible default.
• A version. A finding from March can be rebuilt with March's logic rather than with today's.
• The ability to return nothing. A function that cannot answer says so, and hands the problem back up.

That last property carries the design. When the day counter refuses because no availability date exists, Penny cannot substitute a date that looks about right. She opens a data request against the load instead, the behaviour we set out in why load resolution is the real audit bottleneck. Miles calls a different set of workers at tender time, Chase calls another against the event stream, and one rule binds all three: no agent writes a number it was not handed. The arithmetic case is argued in our post on demurrage day counts.

The interface is where the failures live

The obvious objection is that all of this is engineering taste. Research on multi-agent systems says otherwise, and locates the risk precisely at the seam.

In "Why Do Multi-Agent LLM Systems Fail?", presented at NeurIPS 2025, Mert Cemri and colleagues analysed execution traces across seven popular multi-agent frameworks and built MAST, a taxonomy of 14 failure modes in three groups: system design issues, inter-agent misalignment, and task verification. The taxonomy reached a Cohen's kappa of 0.88 between annotators, over a released trace dataset of 1,642 executions. Their conclusion is the part worth keeping: failures stem primarily from system design and misalignment between components, not from limitations of the underlying language models.

Upgrading the model does not repair an architecture. The work sits in the contract between the layer that decides and the layer that computes, and that contract is specific enough to write down.

How to tell which one you are buying

Gartner returned to this on May 20, 2026, warning the supply chain planning market about agent washing, which that release defines as relabelling conventional automation as agentic. Jan Snoeckx, speaking at the Gartner Supply Chain Symposium in Barcelona, told planning leaders to separate genuine capability from market noise.

The test runs in both directions, and the second direction attracts far less attention. A rules engine with a chat window bolted to the front is one kind of relabelling. A language model wrapped around a total and presented as a calculation engine is the other. Four questions tell them apart.

• Which decisions does the model make, and which numbers does it never produce? A vendor who cannot draw that line has not drawn it in the code either.
• Show me one finding as a computation. What were the inputs, which function consumed them, and which version of that function ran?
• Run the same audit twice over the same files. Do both passes return the same figure and the same disposition?
• What happens when a required input is missing? A default, an inference, or a stop?

Answers to the first two can be rehearsed. The last two are harder to stage, which is why they belong in the room rather than in email.

What to do next

Pull thirty recent invoices for one charge type your team knows well, then ask whatever system you run today for the finding and the derivation side by side. Where no derivation exists, you are holding generated evidence rather than computed evidence.

Book a walkthrough to see exactly where Penny's reasoning hands off to a deterministic engine, traced against your own invoices.