The cost of a cache: measuring before optimizing
updated #performance#caching
Hit rates, eviction, and the latency numbers behind a cache that made things worse.
Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.
Where I started
By the end of this post you should be able to reproduce the whole thing yourself in under an hour, including the parts I got wrong the first time — those are marked so you can skip them if you're in a hurry.
What surprised me most was how small the final change was. Most of the work was understanding the problem precisely enough that the fix became obvious, and that understanding took far longer than the fix itself.
What went wrong
There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.
The first version worked, which is exactly why it was hard to see how wrong it was. Everything looked fine in the demo, and everything looked fine until traffic arrived on a Tuesday afternoon and the graph started doing something I'd never seen before.
A note for context that helps the reader.
[deploy]
strategy = "rolling"
healthcheck = "/healthz"
max_unavailable = 1The fix
What surprised me most was how small the final change was. Most of the work was understanding the problem precisely enough that the fix became obvious, and that understanding took far longer than the fix itself.
I kept a log of every decision during this project, and reading it back a month later was humbling: the choices that aged well were the boring ones, and the clever ones all came with a footnote I hadn't written yet.
The best engineering advice I've received is unglamorous: make it measurable, make it reversible, and only then make it fast.
What I'd do differently
Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.
I kept a log of every decision during this project, and reading it back a month later was humbling: the choices that aged well were the boring ones, and the clever ones all came with a footnote I hadn't written yet.
| Approach | Best for | Watch out for |
|---|---|---|
| Baseline | Quick comparisons | No tail visibility |
| Instrumented | Finding the real hotspot | Overhead of the probes |
| Production | Truthful numbers | Harder to reproduce |
Where I started
What surprised me most was how small the final change was. Most of the work was understanding the problem precisely enough that the fix became obvious, and that understanding took far longer than the fix itself.
There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.
A risky operation or a likely mistake.
What went wrong
Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.
I want to be honest about the trade-offs here. The approach I settled on solves the problem I had, not every problem of this shape, and there is a clear point where you should reach for something more heavy-duty.
def load_config(path: str) -> dict:
with open(path, encoding="utf-8") as f:
return {k.strip(): v.strip() for k, v in
(line.split("=", 1) for line in f if "=" in line)}The fix
What surprised me most was how small the final change was. Most of the work was understanding the problem precisely enough that the fix became obvious, and that understanding took far longer than the fix itself.
Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.
- Latency at p50, because most users live there
- Latency at p99, because someone always lives there
- Error rate, because a fast failure is still a failure
- Saturation, because averages hide the cliff
What I'd do differently
By the end of this post you should be able to reproduce the whole thing yourself in under an hour, including the parts I got wrong the first time — those are marked so you can skip them if you're in a hurry.
Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.
I want to be honest about the trade-offs here. The approach I settled on solves the problem I had, not every problem of this shape, and there is a clear point where you should reach for something more heavy-duty.
Wrap-up
If you remember one thing, make it this: hit rates, eviction, and the latency numbers behind a cache that made things worse compounds. The cost of getting it right early is a afternoon; the cost of getting it late is a migration.