"Checking the LLM" usually means the wrong thing#
Search for an "LLM checker" and you'll mostly find tools that grade an answer — is it accurate, is it toxic, does it hallucinate. That matters, but in a regulated workflow it's the easy part. Grading the output does nothing about the data that reached the model, the fields that shouldn't have, or the absence of any record you could hand an auditor. The output was never the exposure. The pipeline was.
The real job, in three moves#
A checker worth running in a regulated enterprise sits in front of the model, not just after it, and does three things on every call:
- Filter. Block or scrub anything that breaks policy before it reaches the model — PII, secrets, privileged content — so the exposure never happens rather than getting flagged after the fact.
- Reconstruct. Normalize and repair malformed inputs so what the model sees actually meets the rules, instead of failing quietly and producing confidently wrong results.
- Detect. Flag data problems and governance mismatches in the surrounding systems, and write them down — logged, attributable, on record.
Grading the answer is a fourth, downstream step. Useful, but it's the 10%, not the job.
The part a "checker" usually skips: proof#
The difference that matters in regulated work is the record. When an auditor asks what personal data reached the model and whether any of it left your walls, "we think it's fine" is not an answer. A real checker produces a signed trail — what came in, what was filtered, what the model returned, where it ran — so the answer is evidence, not assurance.
That's what an Inspector is: not an output grader bolted on at the end, but a governed worker in front of the call that filters, reconstructs, detects, and keeps the record — starting read-only in Watch mode, so you can prove it before you trust it with anything.
Why output grading feels like the whole job#
Grading is satisfying. You can see the answer, judge it, and get a number. It maps onto how people already think about quality, and there are plenty of tools that do it well.
It also happens at the only point in the pipeline where it is too late to matter. By the time you are grading an output, the data has already reached the model, whatever was going to leave has left, and the only question remaining is whether to use the result.
That is a useful question. It is just not the governance question.
What sits in front of the model#
Filter. Decide what may reach the model before it does. This is the only control that prevents an exposure rather than reporting one, and it is the reason position matters more than accuracy.
Reconstruct. Repair what arrives malformed. Dates in three formats, numbers as strings, a field that is empty in a way that means something. A model handles all of these by guessing, confidently and invisibly.
Detect. Flag the mismatches between what the workflow assumes and what the data actually contains. Not errors in the output — errors in the world the output was drawn from.
The record is the deliverable#
All three of those produce something a grader never does: an account of what happened. What was sent, what was withheld, what was repaired, what left.
That account is what an auditor asks for, what a client asks for, and what you need yourself in nine months when someone asks why a decision was made. It is also, awkwardly, the part that produces no visible improvement on day one — which is why it is the thing that gets cut from scope.
What this means for how you buy#
Ask where the checker sits, not how accurate it is. A tool that only reads outputs cannot answer any question about inputs, however good its scoring.
Then ask what it writes down, and whether that record can be cleared by an administrator. A record that can be quietly emptied is doing the appearance of the job.
Watch mode is this whole argument made concrete. An Inspector sits in front of the call, records what it would have filtered, repaired and flagged, and changes nothing while it does it. You find out what the pipeline is really doing before you decide what to control.
The failure that looks like success#
The hardest case is not the obviously wrong answer. It is the plausible one built on the wrong input — the right format, the expected shape, drawn from a document eleven months out of date or the wrong version of a policy.
A grader reads that output and passes it, correctly, because the output is fine. Nothing downstream objects. The error is not in the answer; it is in the premise, and by the time anyone traces it back, the record of what the model was given no longer exists.
This is why "what went in" is not a lesser question than "what came out." It is the only one that explains the other.
Related