We run most of the operational layer of a small software company with automation: scheduled checks, monitors, a morning summary of what needs attention. Over a few months, the failures that cost us the most were not the loud ones. A crash gets noticed. What did not get noticed was a check that quietly stopped checking, while still reporting that everything was fine.
Here are three of them, with the identifiers removed and the dates moved. The stories are otherwise as they happened.
1. The monitor that said zero for months
A monitor counted problems of a certain kind and reported the count every day. The count was zero, day after day, and zero was the number we wanted to see.
It was zero because the monitor was not looking anymore. A change upstream had moved the data it read, and from then on it read an empty source and reported, correctly, that the empty source contained no problems. Nothing errored. The report looked exactly like a good day.
What gave it away in the end was not the monitor but a problem we found by hand that the monitor should have counted.
2. The summary that picked 4 items out of 23
Every morning a summary listed the items waiting on the owner of the business. One day it listed four. The underlying board had twenty-three.
The other nineteen had been left out "for readability". One of them had been untouched for more than two days and was blocking two other pieces of work. The shortened list was easier to read and quietly wrong: the one item that mattered most was in the part nobody saw.
The fix was not a better selection rule. It was to stop selecting: the list is generated from the board, every time, and if the board says 23, the summary says 23. Length is information. A long list is the honest answer to the question "how much is waiting on me?"
3. "42 passed, 0 failed", with a test that never ran
A test suite reported 42 passed and 0 failed. One of the tests depended on a directory that existed on the main machine but not in the clean copy the suite ran in. In the clean copy it skipped itself, and the summary counted a skip the same way as a pass.
On the main machine, the same suite reported 41 passed and 1 failed. The failure was real: a safety check was missing from one of our configurations. The green run had been green because the one test that could see the problem had not run.
The rule behind all three
In each case, two different states produced the same output:
- "we checked and found nothing", and
- "we did not actually check".
Zero problems, a short list, an all-green summary. Each of them looks the same whether the check worked or not.
What we changed is small and mechanical. Every check in our setup now has three possible answers instead of two:
- ok: it looked and found nothing,
- finding: it looked and found something,
- not measured: it could not look, and it says why.
The third answer is never allowed to look like the first. A check that cannot see is never shown as green. Lists are generated from their source, not written from memory. Summaries say how many things were skipped, and the number is printed even when it is zero, because a zero that is printed is a measurement, and a zero that is missing is just silence.
None of this is new as an idea. What was new for us was how easy it was to lose without noticing, and how much of our setup had been reporting the first answer when it meant the third.