Network troubleshooting order: IP and CIDR, DNS, TCP and UDP ports, WHOIS and RDAP

Start from the layer: address, name, port, or registration

Every network complaint arrives as a symptom, and the symptom already names a layer whether or not the reporter knows it. An address that will not answer, a name that resolves to the wrong host, a port that refuses, and a domain that expires next week are four different investigations. Starting on the wrong one produces evidence that looks conclusive and is not.

Ask what the reporter can already observe. If a literal address answers and the name does not, the fault sits above the address layer: resolution, not reachability. If the name resolves and the connection is refused, the path is working and the listener is absent or filtered. If both fail at the same hop, the question is routing or the local stack. Each branch removes explanations you no longer have to test.

Address questions are arithmetic: does this prefix contain that host, how many hosts does it leave, is it private space. Name questions are about delegation and caching. Port questions separate reachability from listening. Registration questions are about ownership, state and dates, and they are the only layer whose answer can be months stale while still looking authoritative.

The cost of starting at the wrong layer is time and a false negative. A port probe aimed at a hostname proves nothing about the service, because a proxy, a load balancer or a stale DNS answer can reply on its behalf; a WHOIS record cannot tell you whether a host responds. Settle the layer first, then pick the tool: literal address, name, port, registration, in that order.

IPv4 arithmetic: prefixes, host counts, and the two reserved addresses

An IPv4 prefix states how many leading bits are fixed, so the block holds 2^(32-p) addresses. A /24 holds 256, a /25 holds 128, a /26 holds 64, a /30 holds 4. Subtract two for the network address and the directed broadcast and a /24 leaves 254 usable hosts, a /30 leaves 2. RFC 4632 defines this classless notation and the longest-prefix-match forwarding that goes with it; RFC 1918 reserves 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16 for private use.

Two exceptions break the subtract-two rule, and they are where hand arithmetic usually fails. RFC 3021 allows /31 on point-to-point links, where both addresses are usable hosts and there is no broadcast; a /32 is a single host route. Two further blocks get misread as ordinary private space: 100.64.0.0/10 is carrier-grade NAT space under RFC 6598, and 169.254.0.0/16 is link-local under RFC 3927. Hosts in either range can appear on the wire while being unreachable from where you are testing.

The default that hurts is assuming a cloud subnet obeys plain RFC 1918 counting. AWS reserves five addresses in every subnet: the network address, the VPC router, the DNS resolver, a future-use address and the broadcast. A /28 therefore hands out 11 addresses, not 14, and a size chosen from the formula alone runs out early. Providers differ in how many they reserve, so read the vendor page before you size a subnet for a fixed host count.

What is irreversible here is overlap. Two networks claiming the same range both work until the day they must exchange traffic, and by then the fix is renumbering, with downtime, DNS changes, ACLs and monitoring to follow. Check a proposed range against every existing route, peering and VPN before it is typed in. A subnet calculator confirms the arithmetic; it cannot see your other networks.

IPv6: prefix lengths, interface identifiers, and what compression hides

IPv6 prefixes are counted the same way and sized very differently. A /64 holds 2^64 addresses, about 1.8 times 10^19, which is the subnet size stateless autoconfiguration expects; RFC 4291 defines the architecture and the 64-bit interface identifier that fills the lower half. A /128 is one address, a /127 a point-to-point pair under RFC 6164, a /48 the usual site allocation, a /32 the usual provider allocation.

Compression is the trap. RFC 5952 fixes one canonical text form: lowercase hex, leading zeros suppressed, the longest run of zero groups replaced by one double colon, and never a single zero group. So 2001:db8:0:0:0:0:0:1 and 2001:db8::1 are the same address written twice. Any dedupe, allowlist, grep or log join that compares IPv6 as text fails silently. Normalise first, and remember that a zone suffix such as %en0 belongs to the interface, not to the address.

Scope lives in the prefix rather than in a separate field. 2000::/3 is global unicast, fe80::/10 is link-local and always needs a zone identifier, fc00::/7 is unique local, and since the L bit is set in practice what you see is fd00::/8; ff00::/8 is multicast. Treating a link-local address as routable produces the familiar case of a host that reaches its neighbour and nothing else.

Misuse: slicing a /64 into /120s to look tidy. SLAAC needs all 64 interface bits; subnets longer than /64 break stateless autoconfiguration and stable privacy addressing, and most routers will not route or advertise a /65 anyway. When you want small subnets for point-to-point links, use /127, which RFC 6164 exists to bless. Do the subnetting between /48 and /64 instead, where 16 bits buy 65536 subnets without touching a single host identifier.

DNS: record types, TTL, and why a cached answer disagrees with the tool

Start with the record type, because it decides what question you are allowed to ask. A returns IPv4 addresses (RFC 1035 section 3.4.1), AAAA returns IPv6 (RFC 3596), CNAME aliases a name, MX names mail exchangers with a preference, TXT carries policy and verification strings, SOA holds the zone primary, serial and negative-caching timers, NS delegates. RFC 8482 lets an authoritative server answer ANY with a minimal reply, so a thin ANY response is not evidence that other records are absent.

TTL is the field people care about and misread: it is a caching permission in seconds, not a schedule. A record published with TTL 86400 may sit in a resolver for a day, and resolvers clamp values further, since BIND caps cached TTLs at max-cache-ttl, 604800 seconds by default. Lower the TTL one old TTL before the change, never at the moment of the change: reducing it then does not evict answers that were cached under the old value.

Negative answers are cached as well, per RFC 2308, using the smaller of the SOA MINIMUM field and the SOA record TTL. That explains the standard report of a record created minutes ago that still does not exist: the NXDOMAIN is cached and adding data at the zone cannot shorten it. If you need a fast cutover, plan the negative TTL as well as the positive one.

What the tool cannot see: the browser cache, the OS resolver, a stub at 127.0.0.53, and any split-horizon view that serves intranet addresses inside the network. Query the authoritative server directly with recursion disabled, then query your local resolver, then compare. A difference means cache or view, never propagation, because DNS has no propagation step at all, only caches expiring.

Ports: TCP versus UDP, and what a closed port actually tells you

TCP yields three distinguishable outcomes; UDP yields one clear outcome and a great deal of ambiguity. A completed TCP handshake means something accepted. A reset means the host is reachable and nothing listens on that port, which is genuinely useful evidence: a RST proves the host is up. Silence, usually after retries, means a firewall dropped the segment, and then you cannot tell open from closed at all.

UDP has no handshake to observe. An ICMP port-unreachable message (type 3, code 3) means closed; its absence means open or filtered, since many networks rate-limit or suppress ICMP. Treat a silent UDP probe as inconclusive and confirm at the protocol level, with a DNS query or an NTP request, rather than reporting an open port you never saw answer.

Port numbers are a convention, not a contract: 0 to 1023 are well known and need privilege to bind on Unix, 1024 to 49151 are registered, 49152 to 65535 are dynamic (RFC 6335). A service listening on 8080 while 80 is closed is ordinary; a database documented on 5432 answering on 3306 is worth explaining before you close the ticket.

Bind address beats firewall in the false-negative rankings. A process bound to 127.0.0.1 is unreachable from every other host no matter what the firewall permits, and a process bound to 0.0.0.0 or :: is exposed on every interface, including the one you forgot. Test loopback and the routed address separately; loopback passing and remote failing points at the bind address or the host firewall, not at routing.

Scanning production is not free. It trips intrusion detection, gets source addresses blocked, and leaves logs that read like reconnaissance. Ask before scanning, and prefer one targeted probe over a range sweep.

Registration data: WHOIS versus RDAP, redaction, and staleness

WHOIS is a line protocol on port 43 with no defined response format; RFC 3912 describes the transport and stops there. Each registry invents its own field names, so parsers break and the same query returns different keys from different servers. RDAP replaces it with HTTP and JSON: RFC 7480 for transport and redirects, RFC 7481 for security, RFC 7482 for query format, RFC 7483 for responses, RFC 7484 for bootstrap. Prefer RDAP where a bootstrap entry exists; fall back to WHOIS only for registries that never deployed it.

Redaction is not a bug. Since the GDPR, registrars withhold registrant name, email, phone and address for natural persons unless the requester shows a legitimate purpose, and RFC 9537 defines how RDAP marks each redacted field. A blank in WHOIS output means redacted far more often than missing.

Freshness. Two authorities describe the same domain: the registrar holds customer records and the registry holds delegation. A contact update lands at the registrar at once and at the registry on its next sync, so the two answers disagree for hours or days. Creation and expiry dates are registry fields and are the more reliable ones; contact fields are frequently cached and can be months old.

Read the state machine, not the record. A domain inside the redemption grace period, typically 30 days after deletion, can be restored only by the previous holder, and a delete is followed by a pending-delete window of about 5 days before release. ICANN transfer policy also locks a domain for 60 days after registration or transfer. Those states and dates decide whether a domain is about to move; the prose around them does not.

A triage order that rules out local causes first

Order the checks by how many explanations each one eliminates. Local causes kill more hypotheses per minute than remote ones, so they go first: interface address and carrier state, default gateway, resolver configuration. A host with two default routes, a leftover VPN route, or a duplicate address claimed by a second MAC will fail every remote test for reasons unrelated to the remote side.

The sequence that works: link and interface, the longest matching route to the destination, name resolution queried twice against local resolver and authoritative server, a TCP connect to a literal address on one port, TLS and HTTP status, and only then registration. Never open at the last step, because a domain can look healthy while its DNS is broken.

Capture before you change. Each step destroys the evidence for the ones after it: restarting a resolver flushes the cache holding the bad answer, and reconnecting the VPN replaces the routes you still needed to read. Save the route table, the resolver configuration, a packet capture and a UTC timestamp before touching anything, so the next person inherits facts instead of a working system.

Clock skew belongs in the first pass, not the last. A host two minutes off fails TLS validation and Kerberos, and DNSSEC validation rejects signatures outside their validity window, which appears as an unexplained SERVFAIL.

Finally, separate what you proved from what you assumed. A successful request from the server itself proves the service answers on loopback with that name; it does not prove the service is reachable from the internet, because NAT and split-horizon DNS let an internal test pass while the external path is broken.

Try it now