The workflow you already run#
Three-way match is tedious: purchase order, goods receipt, invoice — reconcile, flag the mismatches, approve the rest. So a model does it, reading across an SAP instance, an older ERP from an acquisition, and a pile of supplier PDFs. Approvals speed up. The backlog shrinks. It looks like a clean win.
The risk you can't yet see#
But the systems don't agree with each other. One ERP stores amounts in local currency, another in USD; units differ; a supplier's bank details sit in the invoice free-text. The model does its best across that mess — and "its best" sometimes means a confident match that's quietly wrong, or a supplier's account number travelling to the model with everything else. When finance asks which reconciliations are trustworthy and whether any account data left the tenant, there's no trail to answer with.
Why nobody misconfigured anything#
Reconciliation across legacy ERP is genuinely hard, and that is the point. Supplier names differ between systems. Units differ. A purchase order and an invoice describe the same goods in two vocabularies written a decade apart.
A model is good at that. It is good at it in a specific and awkward way: it returns a confident match whether or not the match is right, because it is doing resemblance, not verification.
Meanwhile the account numbers reached the model for the same dull reason they always do. They were in the same record as the amounts, and the query took the record.
What a wrong one costs here#
This is the workflow where the error-cost arithmetic bites hardest, because reconciliation runs continuously and its output is trusted by design. That is the whole purpose of a reconciliation: to be the thing you stop checking.
A confident wrong match does not announce itself. It clears an exception that should have stayed open. The cost is not the match — it is the exception that never got looked at, discovered in a quarter-end review or by a supplier who was not paid.
So the useful measure is not the match rate. It is what happens to a match nobody verified, and how long a wrong one survives before anything catches it.
One more thing worth measuring while you are there: how many exceptions the process raises that a person then clears without investigating. Every reconciliation accumulates a category of routine exception that gets waved through because it is usually nothing. Usually is doing a lot of work in that sentence, and nobody has checked it recently.
The smallest version that produces a number#
One supplier group, one period, six weeks. Let the reconciliation run exactly as it does today, and record what it matched, what it did not, and what it was confident about.
Then sample the confident matches — not the exceptions. The exceptions already get attention. The number that changes the decision is how often a confident match was wrong, because that is the population nobody is checking.
Six weeks at real volume is usually enough to find the answer. If the confident-match error rate is near zero, the automation is doing its job and you can widen it. If it is not, you have found a class of error that was silently clearing exceptions, and you found it before it compounded across another quarter.
What an Inspector caught#
Started in Watch mode, an Inspector observed the same reconciliation run across both systems:
On record#
Account numbers it would have masked before the model saw them. Currencies and units normalized so the match was arithmetic, not a guess. Three low-confidence matches held back for a human instead of auto-approved. And 240 lines reconciled with every match traced to its source — signed, on the company's own stack, nothing run. Watch first: see where the guessing was, before you trust the match.
The reconciliation kept running at the same speed, and no exception was cleared or reopened by the Inspector.
What the team gained is visibility into the population they were not looking at: the confident matches. Exceptions get attention by definition. Confident matches get trusted by definition. Only one of those two groups was ever being checked, and it was not the larger one.
What to check in your own stack#
Four questions. None of them needs a tool to answer, and all four are worth asking before you buy one.
- What is in the payload, not what is in the prompt? The prompt is written by a person and reviewed. The payload is assembled by a query and never read by anyone.
- Who would notice a wrong one, and when? Name the person and the moment. If you cannot, the review path does not exist yet.
- Where does the model run, and what leaves the tenant? "In our cloud" and "in our tenant" are different sentences.
- What record survives the conversation? If the answer is application logs, the answer is no.
What this does not tell you#
An Inspector in Watch mode reports what it would have caught. It does not tell you whether the workflow was worth automating, and it does not find a rule you have not written. Your policy is the ceiling.
It also will not fix data it cannot reach. If the access is not there, Watch mode says so in week one — an uncomfortable finding rather than a deliverable, and better in week one than month nine.
Related