DNS monitoring: why a green uptime check proves nothing

Brandon Lee published an article on virtualizationhowto.com titled "I Thought My DNS Was Fine Until I Started Monitoring This". His point: a DNS server can be online, listening on port 53, and still fail recursive lookups. We took that as a prompt to run the numbers on our own resolvers. The result was uncomfortable enough to be worth writing up. Thanks for the nudge, Brandon.

The blind spot

Our monitoring checked what most monitoring checks: is port 53 reachable, does the resolver's web interface respond — Technitium, in our case. Both had been green for months.

Neither answers the question that matters. A resolver that rejects every query with SERVFAIL is reachable. It even answers quickly. It just doesn't resolve anything anymore. Measuring reachability alone measures past the problem — a point we already made when writing about Uptime Kuma: reachability is not health.

What the numbers showed

Two of our internal resolvers had not had a single clean day in thirty. Roughly 20,000 SERVFAIL responses per server per month, between 0.1 and 1.1 percent of daily volume. Never zero.

The distribution was more interesting than the total. A typical day: hours at zero, then 142 failures in one hour, then 152, then hours at zero again, a spike to 121 around midday, another 38 in the evening. Transient bursts that recover on their own — precisely the pattern an interval check running every five minutes will most likely miss.

The third resolver looked unremarkable day to day: eight failures in 24 hours across nearly 58,000 queries. The monthly view, however, contained a single day with 56,341 SERVFAIL responses out of 102,375 queries — 55 percent. It had built up over two days, from 0.9 to 2.9 to 55 percent. Query volume that day ran at two and a half times normal, because clients retry failed lookups. Afterwards the server was clean again.

Nobody noticed. Nothing alerted. We found it four weeks later, doing the arithmetic.

The cause was external

The interesting part came with attribution. A query log tells you not just how many lookups fail, but which. Around 95 percent of all SERVFAIL responses came down to three hostnames under a single foreign domain — a telemetry endpoint belonging to third-party software whose authoritative nameservers simply stopped answering.

We verified that before going looking inside our own servers:

The same name fails identically across three independent public resolvers.
It fails with DNSSEC validation switched off as well.
The nameservers of the relevant top-level domain time out on the delegation query.
A control lookup for a healthy name returned NoError in 3.62 milliseconds over the same path.

That settled it. Nothing is broken in the resolver. It is correctly reporting that it cannot obtain a valid answer for that name.

SERVFAIL is usually not your server's fault

This is where a lot of troubleshooting heads off in the wrong direction. SERVFAIL does not mean "my server is broken", it means "I could not obtain a valid answer". The usual causes sit elsewhere: dead authoritative nameservers, failed DNSSEC validation, upstream timeouts.

The rate is still a metric worth tracking — not because every failure is fixable, but because the shape of the curve carries information. A flat baseline of broken third-party domains is noise. A jump from one percent to fifty is an incident. You can only tell them apart if you know what your baseline looks like.

One side effect helped us during attribution: when two resolvers carrying very different query volumes produce almost identical failure counts on the same days, the cause is neither load nor any individual server. The same names are simply failing on both.

What we changed

Three checks, essentially the layers from Brandon's article:

Reachability — port 53 over TCP, as before.
Application — does the resolver's admin interface respond.
Function — a real recursive lookup against external names, evaluated on the RCODE rather than merely on "something answered".

Alongside that, the SERVFAIL rate as its own metric, with the threshold set on deviation from the normal baseline rather than on an absolute number. And a query log with adequate retention: without one, you cannot establish after the fact which names failed. The server with the 55-percent day lacked exactly that, which is why that incident remains unexplained.

A practical warning to close on: test with single queries at a calm cadence. Running a diagnostic loop across dozens of names will, depending on your firewall, trip flood protection — after which you are measuring its effects rather than your DNS health. That reliably costs you an evening.

Conclusion

Our resolvers were reachable the entire time. That was never in question. The question was whether they still resolved, and for months nobody asked it, because the dashboard was green.

The third layer of checking costs a handful of lines. The difference is that an outage becomes visible while it is happening, rather than four weeks later in a monthly statistic.