What my first production incident taught me about logging
The log line I needed didn't exist. Now every service I own has it.
There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.
The story so far
The first version worked, which is exactly why it was hard to see how wrong it was. Everything looked fine in the demo, and everything looked fine until traffic arrived on a Tuesday afternoon and the graph started doing something I'd never seen before.
By the end of this post you should be able to reproduce the whole thing yourself in under an hour, including the parts I got wrong the first time — those are marked so you can skip them if you're in a hurry.
[deploy]
strategy = "rolling"
healthcheck = "/healthz"
max_unavailable = 1The measurement
What surprised me most was how small the final change was. Most of the work was understanding the problem precisely enough that the fix became obvious, and that understanding took far longer than the fix itself.
The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.
A risky operation or a likely mistake.
The change
By the end of this post you should be able to reproduce the whole thing yourself in under an hour, including the parts I got wrong the first time — those are marked so you can skip them if you're in a hurry.
Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.
- Capture the current behavior before touching the code
- Make the smallest change that preserves it
- Delete the workaround you were sure you'd need
Numbers
The first version worked, which is exactly why it was hard to see how wrong it was. Everything looked fine in the demo, and everything looked fine until traffic arrived on a Tuesday afternoon and the graph started doing something I'd never seen before.
There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.
By the end of this post you should be able to reproduce the whole thing yourself in under an hour, including the parts I got wrong the first time — those are marked so you can skip them if you're in a hurry.