Most writing about incident response comes from companies with on-call rotations and status pages. This is the other kind: what an incident looks like when the entire engineering department is one person, and why handling it properly matters more at that size, not less.
What happened
In May 2026 one of the studio's scheduled jobs, part of the automation that sends operational email, over-fired. Instead of sending once for the month, it sent repeatedly, and by the time I caught it the studio had burnt through roughly double its monthly email allowance. No client data was touched. The cost was quota, noise, and the risk of annoying the people the automation existed to serve.
The response, in order
Detection came first, from reading the morning's logs rather than from an alert, which is itself a finding. Mitigation took under an hour: the misfiring job was stopped with a single-file deploy, which is the advantage of infrastructure simple enough to patch surgically. Then the part that mattered most: I contacted the affected client and explained what had happened before they noticed anything was wrong. They had not noticed. That is the best possible time to tell someone.
The root cause was two bugs, not one
Digging in, the over-fire needed two separate defects to line up. First, the job was missing a per-month state gate: nothing recorded that this month's send had already happened, so every run believed it was the first. Second, a filter was missing, so the job selected more recipients than it should have. Either bug alone would have been a small anomaly. Together they compounded.
That pattern shows up in nearly every honest post-mortem ever written: real incidents are rarely one mistake, they are two or three small ones stacked. The lesson is not to write perfect code. It is that every automated job needs its own independent guard, because you cannot predict which two imperfections will meet.
What changed permanently
The fixes went in the same day: the state gate, the filter, and then the generalisation, because a fix that only patches the job that failed is half a fix. Every scheduled job in the studio now follows the same defensive pattern: it checks state before acting, acts on the minimum set, and records what it did. Sends got a hard daily cap at the infrastructure level, a circuit breaker that makes the worst case boring regardless of which future bug fires. And the incident got a written post-mortem, which felt faintly ridiculous for a one-person studio and turned out to be the most useful document of the quarter.
Why bother with process at this size
Because the write-up is where the compounding happens. The post-mortem forced the question "where else could this class of bug live", and the answer improved jobs that had never failed. When I built KeyProof's automations a year later, the daily brief, the retention sweeps, the nurture emails, every one of them was born with a state gate, a cap, and a log line, because the pattern was written down rather than remembered. Process is how a small operation stops paying for the same lesson twice.
The uncomfortable finding
The honest one: detection was luck. The job misbehaved quietly, and reading the logs that morning was routine, not alerting. The durable fix for that is failure-surfacing: the studio's jobs now email a digest when anything errors repeatedly, so silence means quiet success instead of unobserved failure. A system that fails loudly is worth more than a system that rarely fails.
What this looks like from outside
If you hire a small studio, or you are one, the test is not whether incidents happen. They do, everywhere, at every size. The test is whether the operator can tell you what broke, why, what changed, and show you the write-up. An incident handled in the open, with the client told before they noticed, did more for that client relationship than the automation ever did. The full delivery approach, including how the studio builds now, is on the [about page](/about).