At a glance

The software had been built with AI, worked perfectly well, and had been in front of paying customers for months. Nobody had done anything careless. The founder was capable and conscientious.

And any logged-in user could read any other company's invoices by changing a number in the address bar.

Names and details are left out here deliberately. The pattern is the point, because we find some version of it in almost everything we look at.

How it surfaced

Not through a security scan. Not through a penetration test. Through a customer being helpful.

They had been sent a link to an invoice, edited the number in the address bar out of ordinary curiosity, and landed on a document belonging to a different company entirely. They mentioned it in passing, the way you would mention a typo.

The founder called us that afternoon, ostensibly about page speed. The invoice thing came up about ten minutes in, described as "probably nothing".

It was not nothing.

What had actually gone wrong

In plain English: the software checked that you were logged in, and never checked whether the thing you asked for was yours.

Those are two separate questions, and only the first had been asked. The code took the invoice number from the address bar, fetched that invoice from the database, and displayed it. There was no line anywhere asking whether that invoice belonged to the account making the request.

Every screen in the application worked this way. Invoices, customers, uploaded files, reports. Consistently, thoroughly, invisibly wrong.

Why the code looked completely fine

This is the part worth sitting with, because it explains why it survived months of use.

The code was tidy. It was readable, sensibly named, and did exactly what it appeared to do. Reading it, you would see a function that fetches an invoice and shows it, and nothing about that would look wrong, because the fault is not in what is written. It is in what is absent.

You cannot spot a missing check by reading for mistakes. You can only spot it by knowing that the check should be there and noticing that it is not. That is knowledge, not attention, and it is precisely what AI does not bring to the job.

Why it took forty customers

With one account, the flaw is undetectable. Every record in the database is yours, so fetching any record returns something you are entitled to see. The software behaves identically whether the check exists or not.

With five friendly early customers, nobody is editing address bars.

At forty, with real businesses using it daily, somebody eventually does. The vulnerability was there on day one. It simply required an audience to become visible, which is a horrible property for a flaw to have, because it means the exposure grows quietly in proportion to your success.

How common is this?

It is the single most frequent serious problem we find in AI-built applications. Not occasionally. Routinely.

There is a reason for that. AI is trained on enormous amounts of example code, and example code is written to demonstrate one idea clearly. Tutorials show you how to fetch a record. They do not clutter the example with the authorisation logic a real application needs, because that is not what the tutorial is teaching. The model learned the shape of the example, not the shape of production.

What happened next

The fix itself was not dramatic. A day to establish where ownership checks belonged, a few days to apply them everywhere and add tests that would catch a regression, and a careful review of the logs to work out what had actually been accessed.

The harder conversation was the other one: this is personal data, it was exposed, and there are obligations that come with that. Fixing the code is the easy half.

The full write-up of what we found, and what it cost to put right, is here: we fixed an AI-built app, and this is what we found.

If this made you slightly uneasy

Good. That instinct is worth acting on while it is cheap.

There is a thirty-second version of this test. Log into your own application, open a record, and change the identifier in the address bar to a neighbouring number. If you can see something that is not yours, you have the same problem, and you would rather know today than hear it from a customer.

As ever: AI is a brilliant typist and a terrible architect. It typed a function that fetches an invoice. Deciding who is allowed to have it was never its job.

Want someone to check properly rather than guessing? Our AI Code Audit is a fixed-price £495 review that tells you plainly what you are dealing with, or get in touch for a no-pressure conversation.