senn-techA host with apt exit 100 stopped the patch round for the entire fleet
The report ended with failed. That is good news first: The run reported a result. The bad news follows this line. A file host stopped with apt-get and exit code 100 on the night of September 16, 2026. 19 packages were open, five of them security updates. The runner treated this as an abort of the entire round, not as a response for this one host. Every host behind it received neither patches nor a CVE check that evening. A queue whose failure mode is "the rest never runs" is not a queue. It is a hidden dependency.
What happened overnight
The nightly runner works through the fleet host by host, in a fixed order, doing two things: patching and checking CVEs. The order is not arbitrary. It keeps database hosts at the start, where an error is most costly. This decision stays.
Nobody wrote down the second branch. apt-get exits with 100 if it fails, as stated in the man page. A third-party repository held back two packages, the runner received exit 100 and treated it as a hard failure. 19 packages remained unprocessed, five of them security-related. Behind this: nothing. No patch, no scan, no line stating "work was never started". In the morning, failed appeared in the report, and for the system the matter was closed.
Success and unpatched in one minute
On September 17 we reproduced the issue, which yielded the second finding. A manual single run on the same host, triggered when writing an ops note, ended with ok. apt still reported 14 packages as "not upgraded" at the same time. Success and unpatched, one minute apart. The lock (flock) only protects the nightly wrapper; a direct call bypasses it.
Four commands from this window, expected result versus real consequence:
| Command | Expected result | Actual consequence |
|---|---|---|
apt-get in nightly run | Result for this host, rest continues | Exit 100, 19 packages open, rest never started |
| Manual single run on the same host | ok or failed, both reliable | ok reported, while apt counts 14 packages as "not upgraded" |
ssh with StrictHostKeyChecking=accept-new | accept new key, report changed key | abort in ssh call, host without line, 13 runs blind |
git push with token in the command line | Update runs, access remains private | Token was in a June record, invalid after rotation |
The pattern is not negligence, but the gap between what a command returns and what we read into it. A credential in a command line ends up in every log that records that line.
Missing is worse than red
The third finding is quieter and more unpleasant. The CVE-Watcher maintains its own pin file for host keys, independent of our ssh-Config, and runs with StrictHostKeyChecking=accept-new. It accepts a new key without comment. A changed key causes a hard abort, and because this happens within the ssh call, it does not result in a red entry: The host is missing. A production node was blind for thirteen consecutive runs, while dashboards remained quiet.
Our backstory: 34 machines share a host key from a golden template. A changed key is routine for us, not an attack. Monitoring that cannot handle this normal case reports no security; it disables itself. Cleanup did not help either: Cleaning up our config was not enough because the job reads a second pin file, and there is a third, versioned copy. Tools with their own known_hosts must be in the inventory, otherwise the repair starts from scratch every week.
This is the lesson from our DNS SERVFAIL monitoring. The check showed green there because it measured reachability only, not resolution. Here a host disappears because the check cannot report that it did not run. A check that cannot state "I did not run today" is theater. The question "did every host get an entry today?" is therefore a separate metric, per host, as a timestamp, not as an aggregate. A sum value across a fleet does not distinguish a checker failure from a quiet evening.
Our decision: fail-open for the fleet
We closed the incident instead of continuing to manage it. Fail-open for the fleet, fail-stop per host. One host's apt issue must not cost more than thirty hosts their patching night again. A failure is no longer an abort, but a separate line with its own alert: host, exit code, pending packages, timestamp. The order remains justified. Only its end is now a normal case.
If you build this, you need four lines of code; that takes one evening:
- Expected hosts versus delivered rows. The list is known, the report gets one row per host, even for omissions. Report the difference, not the aggregate value.
- Host errors as a row, not a failure. The exit code belongs in the report, the loop continues. One alert per failure, not one per night.
- Timestamp of the last successful check per host. A missing entry becomes visible without anyone having to read the report. This is the answer to thirteen blind runs.
- No tool without a version number in the image tag. If a patch tool runs in a container with a mutable tag, the question "are you patched?" cannot be answered, because the tag does not carry the answer. We documented this for patch management in SMEs, but never applied it to the tools themselves.
Point 4 pays the most: a tag change, and a question that no one else answers in the audit.
Process documentation is not monitoring
Now the uncomfortable side. Our fleet has a documented patch process with twelve steps, including maintenance windows. It still produced a night when a security update sat on thirty machines while the report failed was written and nobody read it. Twelve steps on paper are an intention, not monitoring.
Fourth finding: The nightly job schedule file is generated from shell history, and its commands have drifted from what is actually needed over the months. We now trust the numbers from pre-checks more than the plan text. A plan that nobody verifies is just a reminder.
The fifth point sounds like cosmetics, but it is not. The report colors its badge based on wording: "uebersprungen" and "nicht erreichbar" are deliberately yellow, and an excluded word is the string "0 Befehle fehlgeschlagen". Rewording a message changes the color signal of the report. Wording is load-bearing structure, not text maintenance. An alarm that waits for someone to read it is not a control. That is why CrowdSec instead of fail2ban blocks on its own, and backups are verified and restored instead of trusted. For patching, we settled for failed.
Status: September 17, 2026, second night. The file host still has open packets, the third-party repository is holding them back. This time a single line with its own alert, no abort for all lines behind it.
Further Reading
Why does a single host stop the entire patch round?+
Because the runner has no dedicated branch for the error case. It executes a fixed sequence and treats an error exit code as an abort of the entire run rather than as a result for that specific host. The path was never incorrectly programmed, it was never decided. The sequence was documented, not the behavior on failure.
What is the difference between a red host and a missing host in the report?+
A red host triggers a response, a missing one does not. The CVE-Watcher run failed during the ssh call, so the host didn't appear as an error but simply vanished. Dashboards stayed quiet, and a production node was blind for thirteen runs. Red is a complaint, missing is withdrawal.
How do you know every host was actually checked?+
Two metrics unrelated to patching: expected host count versus delivered rows, and a timestamp of the last successful check per host. Report the difference, not the aggregate value. This reveals check system failures, not just the failures it reports.