By Zahi Raifer
Disclosure: I build Orivox, the AI website builder this post describes.
Most writing about LLM features starts with the prompt. When you are the one who gets the incident email, it makes more sense to start with what breaks. Our product lets small-business owners change their live website by typing a request in Hebrew or English. A model plans the change and Python carries it out. Before we tuned a single prompt, we listed the ways that pipeline could hurt a customer's site, and for each one we chose a control that does not depend on the model behaving well.
This is that list, written the way I would want to read it at 3 a.m.
1. The model is slow, or the provider is having a bad day
Symptom: pages hang, request workers pile up, and the site feels down even though nothing has crashed.
Control: no web request ever waits on a model. Edit turns and other long work go onto job queues backed by Redis, and workers pick them up. The system runs as four processes: a web app and its job worker, and an editing service and its worker. A slow model call can only tie up a queue worker, never a page load.
2. Two writers touch the same site at once
Symptom: an AI edit and a manual change land on top of each other, and one quietly undoes the other.
Control: one lock per project, stored in Redis and held for the whole turn by whoever is writing. That might be an edit, a manual commit, a site copy or a background job. The web app never runs Git against a project. When it has to write files itself, it takes the lock, writes, and asks the editing service to commit. There is always exactly one writer, so there is always exactly one version.
3. The model reports a change that never happened
Symptom: the assistant says "done", the owner reloads, and nothing is different. This erodes trust faster than any outright error.
Control: a claim of success needs a real change on disk. The model never writes files. It returns a short list of operations, and the server applies them, checks them and decides whether the turn is saved. If nothing changed, the turn does not count as a success, whatever the model says.
4. A save would wipe out a live page
Symptom: a confused edit replaces a working page with an empty or near-empty one.
Control: a save may never empty a live page. It is one of eight deterministic checks that run on every save, in plain Python, regardless of what was proposed. Others protect the owner's logo, keep the contact form's privacy notice next to the form, and keep every section of a page the owner did not mention when the page is rewritten.
5. A retry creates a duplicate
Symptom: an external AI client gets an error after the work actually succeeded, retries, and the owner ends up with two copies of a project.
Control: a completed write is always reported as done. When the files are written but usage metering fails afterwards, the tool still returns success, so the client never retries. A missed usage record is a small accounting problem. A duplicate site the owner never asked for is a support ticket.
6. The AI connector dies quietly
Symptom: owners who edit through their own AI client (we run a remote MCP server for that) get failures, and nobody on our side notices until someone complains.
Control: watch it from the outside. Every five minutes a probe walks the connector end to end, from discovery and sign-in to a real tool call. It emails an incident report when something fails and another when it recovers. The connector keeps no session state, so a restart costs at most the one request in flight.
7. Nobody is sure what is actually deployed
Symptom: a bug "fixed last week" is back, and the version label says the fix is live.
Control: verify a release by its content. We confirm what is running by hashing the files that are actually deployed, not by trusting a label. Code never changes on a production server by hand. It goes to the repository, is pulled into a development environment and checked there, then is promoted from an admin dashboard that records every release and keeps a backup of what it replaced.
8. An old failure comes back
Symptom: a regression in a guard that someone "simplified" months later.
Control: every live failure becomes a replay test, and every guard is pinned by one. The contract between the two services is pinned by tests on both sides, and a contract change is made additive first, released, and only then used.
What the list taught us
Two patterns kept coming up.
Put the decision in code, not in the prompt. Each control above sits outside the model. The model can be wrong in new and creative ways, and the checks stay the same, because they look at the result rather than at the reasoning.
Make every failure boring and bounded. A queue bounds the cost of a slow call. A lock bounds who can write. A per-turn Git commit, with the original site committed as a baseline before the first edit, bounds how bad a bad edit can get, because undo is a restore. When something does go wrong, the worst case is already known.
If you are about to put an LLM in front of production data, try writing this list before you write the system prompt. For each failure, name the symptom, the control and the code that enforces it. Any row you cannot fill in is the incident you have not had yet.
More about how the editor is built: https://www.orivox.pro/
Top comments (0)