❓ What does SERVFAIL mean?
- SERVFAIL
- general server error
- means
- the server tried and failed
- the code does not say why
- not
- "that name does not exist"
- (that one is NXDOMAIN)
- why
- ask for the reason: EDE
- run
dig @RESOLVER NAME TYPE- then
same again with +cd- code
- RCODE 2 (RFC 1035 4.1.1)
Step 1: read the config#
Read what this box is told to do before asking it anything.
side I am asking
systemd-resolved. Read the DNS Servers line, and DNSSEC on or off (resolvectl(1)).
resolvectl statusGlobal
Protocols: +LLMNR -mDNS DNSOverTLS=opportunistic DNSSEC=yes/supported
resolv.conf mode: foreign
Current DNS Server: 1.1.1.1#cloudflare-dns.com
DNS Servers: 1.1.1.1#cloudflare-dns.com 8.8.8.8#dns.googleAny Unix. Read the nameserver lines, in order.
cat /etc/resolv.confnameserver 1.1.1.1
nameserver 8.8.8.8
options timeout:2 attempts:2macOS. Read the first resolver block: that one is the default (scutil(8)). The addresses, search domain and interface below are documentation values.
scutil --dnsDNS configuration
resolver #1
search domain[0] : example.net
nameserver[0] : 2001:db8::1
nameserver[1] : 192.0.2.1
if_index : 13 (en0)
flags : Request A records, Request AAAA recordsWindows. Read the DNS Servers line on the adapter carrying the default route (ipconfig).
ipconfig /allside I run the resolver
BIND. Read forwarders and dnssec-validation as named loaded them (named-checkconf(8)).
named-checkconf -p /etc/bind/named.conf dnssec-validation auto;
recursion yes;
allow-query {
"localhost";
};
forward only;
forwarders {
1.1.1.1;
8.8.8.8;
};
};Unbound. Read the forward zones in use (unbound-control(8)).
unbound-control list_forwards. IN forward 1.1.1.1 8.8.8.8Unbound. Read the anchor it validates against; an empty line means unset (unbound-checkconf(8)).
unbound-checkconf -o auto-trust-anchor-file/var/lib/unbound/root.keyside I run the zone
A zone I own. The parent's DS has to match a key the zone serves.
side I run the zone
Asks for the DS the parent publishes for this zone.
dig +https @1.1.1.1 dnssec-failed.org DSGO a DS whose key tag and digest match a key the zone publishes
NO GO a key tag no published key has, which is this one
DO zone owner: Fix the DS at the registrar
side I run the zone
Asks the zone for the keys it publishes, validation off so the answer arrives.
dig +https @1.1.1.1 +cd dnssec-failed.org DNSKEYGO a 257 key whose tag and digest the DS above carries
NO GO no 257 key matches the DS (EDE 9)
DO zone owner: Fix the signing so a published key matches the DS
Not my resolver and not my zone: go to step 2.
Step 2: dig each layer#
Ask each layer directly, and find the first one that answers differently.
side I am asking
Asks the resolver this client is configured to use, by address, for the name that is failing.
dig +https @1.1.1.1 dnssec-failed.org AGO NOERROR and an answer, so this resolver is not where it stops
NO GO SERVFAIL, with an EDE line if the resolver sends one
DO you: Ask a resolver you do not run, below
Swap in the address from step 1. The id, Query time, WHEN and a cached TTL move between runs. +https is DoH and needs dig 9.18 or newer: where port 53 is intercepted, the middle box answers for any @server I name. The interception check.
side I am asking
Asks a resolver you do not run, same name, same type.
dig +https @8.8.8.8 dnssec-failed.org AGO an answer here and SERVFAIL above puts it on my resolver or its forwarder
DO you: Open check-a-resolver
NO GO the same SERVFAIL puts it on the zone
DO you: Read the EDE below
check-a-resolver, then check-a-forwarder.
side I run the resolver
Read the EDE the resolver returned:
| EDE | Name | Reads as |
|---|---|---|
| 6 | DNSSEC Bogus | validation ran and failed |
| 7 | Signature Expired | an RRSIG has expired |
| 9 | DNSKEY Missing | the parent's DS matches no published key |
| 10 | RRSIGs Missing | the signatures are missing |
| 13 | Cached Error | a failure is being served from cache |
| 14 | Not Ready | the server was not fully functional when the query arrived |
| 22 | No Reachable Authority | nothing answered for the zone |
| 23 | Network Error | the resolver's path upstream broke |
Causes below, in the order the query travels. RFC 1035 section 4.1.1 gives all of them one code: "Server failure - The name server was unable to process this query due to a problem with the name server".
| Family | Who emits it | The dig that separates it | GO and NO GO | The move |
|---|---|---|---|---|
| nothing authoritative answered (EDE 22, 23) | my resolver | each NS, +norecurse | GO: one aa. NO GO: all silent | zone owner: Check every nameserver in the NS set |
| the resolver hit a fetch quota | my resolver | a second resolver, then mine again | GO: mine recovers. NO GO: mine fails under load; fetches-per-zone and fetches-per-server answer SERVFAIL when set to fail (BIND 9 ARM) | resolver operator: Check the fetch quotas |
| a cached failure replayed (EDE 13) | my resolver | the same query after the cache window | GO: it answers within the 5 minute ceiling (RFC 9520 section 3.2). NO GO: it still fails | you: Wait out the cache, then ask again |
| the forwarder failed and nothing falls back | my resolver, forwarding | the forwarder, asked directly | GO: it answers. NO GO: it does not, and forward only (BIND 9 ARM) or forward-first: no (unbound.conf(5)) means no fallback | resolver operator: Fix the forwarder, or drop only |
| the delegation names servers that do not serve the zone (RFC 9499 section 7) | the parent, and the servers it names | each NS, +norecurse | GO: aa from each. NO GO: REFUSED, or NOERROR with no aa | zone owner: Compare the parent NS set with the child's |
| the server has the zone and cannot serve it (EDE 14) | that authoritative server | that server, +norecurse | GO: aa and the SOA. NO GO: SERVFAIL, or a secondary past its SOA expire, no longer authoritative (RFC 1035 section 3.3.13) | zone owner: Check the primary and the last transfer |
| DNSSEC validation failed (EDE 6, 7, 9, 10) | my resolver | the same query with +cd | GO: +cd changes nothing. NO GO: +cd answers (RFC 4035 section 5.5) | zone owner: Fix the DS, or re-sign the zone |
Each row names its own dig. The authoritative server itself: check-an-auth.
Step 3: compare against the path#
Compare where it should have gone with what answered.
you -> stub resolver -> your resolver -> forwarder, if the config named one
-> root -> TLD -> the zone's NS setThe first mismatch is the fault.
The ;; SERVER: line names the socket that answered. +nsid names the anycast instance behind it, for a ticket, where the operator publishes one (RFC 5001).
side I run the zone
Compare what the parent hands out with the NS set the zone serves. check-a-delegation.
side I run the zone
Asks a TLD server which nameservers the parent delegates this zone to.
dig -4 +norecurse @a0.org.afilias-nst.info dnssec-failed.org NSGO a referral, no aa and no ra, with the NS set in AUTHORITY
NO GO ra set, so a recursive server answered in the TLD server's place
DO you: Run it from a network that leaves port 53 alone
An answer carrying ra came from a recursive server, whatever the ;; SERVER: line says. A +trace on an intercepting network walks someone else's path. Trace from outside it, or in the dig tool.
Then wait. A failure is cached 1 second to 5 minutes (RFC 9520 section 3.2), so a correct fix still answers SERVFAIL for a few minutes.
See also#
- Extended DNS Errors: every code, and the failures that arrive bare
- DNSSEC troubleshooting: once
+cdhas proved the signer is at fault - dig flags:
+cd,+norecurse,+nsid,+httpsand the rest - DoT, DoH and DoQ: transports a port-53 interceptor cannot answer for
- SERVFAIL diagnosis: the EDE, the same question with +cd, then the chain