The day I learned to fear feature flags
Dead branches, stale config, and the cleanup ritual nobody scheduled.
The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.
Setting the scene
The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.
I kept a log of every decision during this project, and reading it back a month later was humbling: the choices that aged well were the boring ones, and the clever ones all came with a footnote I hadn't written yet.
fn retry<T>(mut f: impl FnMut() -> Result<T>, times: u32) -> Result<T> {
let mut last = None;
for _ in 0..times {
match f() {
Ok(v) => return Ok(v),
Err(e) => last = Some(e),
}
}
Err(last.unwrap())
}The naive approach
I want to be honest about the trade-offs here. The approach I settled on solves the problem I had, not every problem of this shape, and there is a clear point where you should reach for something more heavy-duty.
The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.
A note for context that helps the reader.
What the profiler said
By the end of this post you should be able to reproduce the whole thing yourself in under an hour, including the parts I got wrong the first time — those are marked so you can skip them if you're in a hurry.
By the end of this post you should be able to reproduce the whole thing yourself in under an hour, including the parts I got wrong the first time — those are marked so you can skip them if you're in a hurry.
- Latency at p50, because most users live there
- Latency at p99, because someone always lives there
- Error rate, because a fast failure is still a failure
- Saturation, because averages hide the cliff
The rewrite
I kept a log of every decision during this project, and reading it back a month later was humbling: the choices that aged well were the boring ones, and the clever ones all came with a footnote I hadn't written yet.
The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.
There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.