What it takes to trust a model with someone's books

A demo that reads an invoice correctly proves almost nothing. I know this because I built one, watched it work, and then watched it confidently misread a freight invoice as a sales invoice and post the entry the wrong way round. Nothing crashed. No error surfaced. The books were simply wrong, quietly, until someone went looking.

That is the actual problem with putting a language model near an accounting ledger. It is not that the model makes mistakes — every system makes mistakes. It is that this particular kind of system makes them silently, with the same confident tone it uses when it is right. In Indian accounting the cost lands later and larger: a wrong input tax credit becomes a misfiled GSTR-3B, which becomes a notice, which becomes a penalty three months after anyone could have caught it cheaply.

So the question I had to answer on GeniusCFO was not "how do I make the model better?" It was "how do I make sure the model's errors always land somewhere I can see them?" Those turn out to be very different engineering problems.

The only layer that can be perfect

The framing that unlocked this for me: 100% accuracy is achievable, but only in the deterministic layer. No amount of prompt engineering makes an LLM reliably correct. But arithmetic is reliably correct. A GSTIN check digit either validates or it does not. A set of line items either sums to the invoice total or it does not. Debits either equal credits or they do not.

So rather than trying to push the probabilistic layer toward certainty, I made the deterministic layer as complete as I could, and then arranged things so every error the AI layer makes has to pass through it. What survives is either verifiably consistent or explicitly flagged. What it cannot decide, a person decides.

The goal stopped being an accurate model. It became a system where being wrong is loud.

Check, then repair, then give up honestly

Every extracted document runs through a harness before anything is posted. It works in three stages, and the order matters because each one is cheaper than the next.

• Free deterministic checks first. GST arithmetic identities, the GSTIN check digit, IFSC format, date plausibility, line-item sums, and grounding every extracted identifier back against the raw OCR text. These cost nothing and catch a surprising share of failures.
• Targeted repair second. Whatever the checks flag goes back to the model — but only the flagged fields, and at most twice. Re-running the whole extraction would be slower and would risk disturbing fields that were already right.
• Honest failure last. Anything still failing gets its confidence demoted and goes to human review. The harness never raises an exception and never deletes a field. Its job is to route, not to guess.

The same shape shows up in the natural-language SQL agent. Generated SQL runs read-only; if the database rejects it, the real error message goes back to the model for at most two repair rounds. After that the user gets "I could not answer that" rather than a fabricated number. An honest failure is a feature. A plausible wrong answer is the thing you are actually trying to prevent.

The blind spot I nearly missed

Here is the failure mode that took me longest to see. Documents that go to review generate correction signal — a human fixes them, and now you know something. Documents the system auto-posts with high confidence generate nothing at all. If the model is confidently wrong, that error produces no signal whatsoever. You cannot measure it, so you assume it is not there.

The fix is unglamorous: route a small, configurable fraction of confidently auto-posted documents to a quick human confirm anyway. It costs a little review time and it buys the one thing you cannot get otherwise — an unbiased estimate of how often you are wrong when you are sure you are right.

Alongside that, a nightly sweep runs plain SQL invariants over the actual rows rather than over the model's opinion of them: every posted entry balances, each tenant's trial balance nets to zero, general-ledger rows mirror their journal lines, no posted line is missing its GL row, inventory on hand equals the sum of its movements. This is the one place where full coverage of what it checks is literally achievable, so it is checked every night and it fails loudly.

A number you can actually defend

None of the above means anything without measurement, and measurement is where most RAG and extraction work quietly stops. So there is a golden pack of documents with known-correct fields, audited so the GSTINs are checksum-valid and the arithmetic reconciles, including one deliberately planted duplicate. Extraction accuracy is scored per document type against that pack, on the live code path rather than in a notebook.

The first honest baseline for invoices was 91.4% field accuracy. That number is more useful than a higher one would have been, because the same run surfaced a classification gap on goods-receipt notes that I would otherwise have shipped without noticing. A measurement that only ever confirms your assumptions is not a measurement.

If a number in a system like this has never once told you something you did not want to hear, it is probably not measuring what you think it is.

Getting better without retraining anything

The last piece is a nightly loop that turns production corrections into improvements — grade, distil, gate, promote. Resolved review corrections become typed per-field grades. Corrections that repeat consistently become distilled rules injected into the extraction prompt. Observed correction rates recalibrate the per-field thresholds that decide what goes to review at all, so you get fewer needless reviews and fewer wrong auto-posts at the same time.

Two constraints kept this from becoming a liability. First, nothing is promoted on vibes: a candidate rule is re-run over the golden set and promoted only if the score does not regress, and it is retired automatically if it does. Second, every improvement is a versioned, reversible row in a database rather than a weight change somewhere — inspectable, auditable, and undoable. In an accounting product, an improvement you cannot explain is not an improvement.

The whole thing sits behind environment flags that default off, and it fails open. A broken improvement job degrades to "the agents are as good as they were yesterday" and never blocks a live request. That is deliberate. A self-improvement system that can make things worse is just an outage with extra steps.

What I would tell myself at the start

• Build the measurement before you build the improvements. Every hour spent tuning prompts before you can score them is an hour spent guessing.
• Spend your effort making the deterministic layer complete rather than making the probabilistic layer smarter. It is the only part that can actually be finished.
• Assume your confident successes are hiding failures, and sample them. The errors you cannot see are the expensive ones.
• Prefer an honest refusal to a plausible answer, everywhere, without exception.
• Make every automated improvement reversible and inspectable, or you have traded one silent failure mode for another.

None of this is exotic. It is mostly ordinary engineering discipline applied to a component that happens to be probabilistic. But that is rather the point — the interesting work in production AI is not the model. It is the structure you build around it so that the model being wrong is survivable.