Clarobix

AI Systems

Nobody has read your AI-generated codebase

It runs, it demos, and no human has traced a request through it end to end. Here is what that costs, and the four things worth checking first.

Drawn from

The code works. That is usually the problem.

A team ships something in six weeks that would have taken six months. It demos well, the tests pass, and the first users are happy. Then someone asks a question nobody can answer: what happens if this request arrives twice? And it turns out that no person has ever read the path that request takes, from the route to the row it writes.

This is now the most common state we find a system in. Not badly built — unread.

Working is not the same as reviewed

Traditional code review happens because a human wrote something and another human has to approve it. The review is a side effect of the workflow. When a model writes the code and a human accepts the diff, that step quietly disappears. Accepting is not reading. Nobody decides to skip the review; there is simply no longer a moment where it would have happened.

The gap does not show up in tests either, because the tests were generated from the same understanding as the code. If the model misread the requirement, it misread it consistently, and the suite agrees with the bug.

Generated code fails in a particular shape

It is not sloppy. It fails differently from the way tired humans fail, and knowing the shape tells you where to look.

It is locally plausible everywhere. Every function, read on its own, looks like something a competent developer would write. The errors live between functions — in the assumption one makes about what another already checked.

Conventions drift across files. One controller validates in a form request, the next validates inline, a third trusts the caller. A reader's mental model never stabilises, so reviewing file four does not make file five faster to review.

Security decisions get made implicitly. Asked to list records, a model returns records. It does not ask whose. The scope is not deliberately omitted — it was never a decision, so nothing records that it was skipped.

Duplication stands in for abstraction. The same rule appears in four places. Someone finds a bug and fixes it in three.

Four things worth checking first

Not a full audit. These are where the expensive failures concentrate.

1. Authorisation boundaries

Find every query that returns a list and ask what scopes it. Generated code reliably produces the filter that makes the feature work and omits the one that makes it safe, because only the first is visible in testing.

The version of this that catches experienced teams is retrieval. A vector search has no concept of a tenant. If the tenant filter is applied after the nearest-neighbour lookup rather than inside it, the top matches are drawn from the whole corpus first and filtered second — so a customer's question gets answered out of another customer's documents, or returns nothing at all while the logs show a successful search. Both failures look like a ranking problem. Neither is.

2. Money paths

Every write that moves value needs an answer to: what happens if this runs twice? Retries, double-clicks, webhook redelivery and queue redrives all make it run twice eventually.

Generated code almost never produces an idempotency key unless asked, because the happy path does not need one. What the ledger says after the second attempt is the entire question, and it is not visible in a demo.

3. Data integrity under partial failure

Trace one multi-step write — charge, record, notify — and ask what state the system is in if step two fails. Whether that sequence is inside a transaction is a decision somebody has to make. If nobody made it, it is not.

Related: updated_at is not an audit trail. A column that records when a row last changed cannot say what it changed from, who changed it, or that it changed twice. If a regulator, a clinician or a finance team will ever ask what a value used to be, the history has to be written as its own record. Retrofitting that after the fact means the answer for everything before today is that you do not know.

4. Failure modes at the edges

List every third-party call. For each: what is the timeout, what happens on a 500, and what does the user see. Generated integrations are typically written against the documented success response. Providers do not fail in documented ways, and the default timeout on most HTTP clients is long enough to hold a web worker open until the queue backs up behind it.

What this actually costs

Nothing, until it does. These are not bugs that degrade gracefully. An authorisation gap is fine until the day it is a disclosure. A missing idempotency key is fine until a payment provider retries a webhook. An absent audit trail is fine until the first time somebody asks a question about the past.

The cost is also asymmetric in time. Adding a tenant scope during a build is a line of code. Adding it after two years of data means proving which existing rows were exposed, and to whom.

Reading it before someone else does

The work is unglamorous: trace the real paths, write down what is actually true, and separate what must change now from what can wait. The output is a document, not a rewrite. Most of what it finds is small. The value is knowing which small things are the ones that matter.

If this describes a system you are responsible for, our production readiness review is fixed in scope and ends in that document — yours to act on with us or without us.

More

Related reading.

AI Systems

Someone still has to answer for it

Generation gets cheaper every year. Being the party who answers when a regulator asks who approved the data model does not, and that gap is widening.

AI Systems

Engineer the boundary, generate the implementation

The useful split is not humans versus agents. It is that the decisions live upstream and the code lives downstream, and only one of those can be handed over.

Recognise the problem?

If this describes where you are, the first conversation is usually short and tells you whether we are useful.