Routing failures are quiet. There is no alarm for "the packet went somewhere plausible and died." You get a timeout, a one-way ping, or a page that loads its text and hangs on the images.

The method below is deliberately boring: state what should happen, prove each hop, then prove the return path. It works on a four-line static table and on a campus core.

The method

  1. Write the expectation. Source address, destination address, expected next hop at each device. If you cannot write it, you are about to debug by superstition.
  2. Prove layer 2 to the first hop. ARP entry present, MAC reachable, port up.
  3. Prove the forward path. traceroute from the source, then from each router along the path.
  4. Prove the return path. Ping with a source address from the destination back to the source. This is the step people skip, and it is where half of all failures live.
  5. Prove the payload. Ping with a large, unfragmented packet to test MTU.
Where a route silently drops: the forward path succeeds, the return path has no route.

The tools, and what each one actually proves

Tool Proves Does not prove
ping Round trip for small packets That large packets pass
ping -s 1472 -M do MTU end to end Application behavior
traceroute Forward hops that answer Return path
mtr Loss and latency per hop, over time Which direction dropped
ip route get <addr> The exact route chosen That the next hop forwards
show arp / ip neigh Layer 2 resolution Anything above layer 2
Packet capture What actually arrived Why it did not leave

The single most useful command in a static routing environment is ip route get. It answers the router's question directly, including which source address it will use:

$ ip route get 10.50.7.20 from 10.40.12.57
10.50.7.20 from 10.40.12.57 via 10.0.0.9 dev vlan100 src 10.40.0.1

If that output surprises you, stop debugging and fix the table.

Five failures worth memorizing

1. One-way reachability (missing return route). Pings from the new campus reach the district; the district cannot reach the campus. The forward path has a summary; the return path has no default. The symptom is "the server is down" from one side and "everything is fine" from the other.

2. The summary black hole. A route for 10.48.0.0/14 is installed and quietly covers 10.50.0.0/16, which belongs to a different site. Traffic goes to the wrong next hop and dies. This is why summaries must be allocated, not estimated.

3. MTU black hole. ping works. SSH connects and then freezes. HTTPS hangs after the handshake. The path MTU is smaller than the sender believes — usually a tunnel — and ICMP "fragmentation needed" is being filtered, so nothing learns. Test with a large DF packet, then fix the tunnel MTU or clamp MSS.

4. Asymmetric return. Outbound goes over fiber, return comes over the VPN, and a stateful firewall drops the reply because it never saw the request. mtr looks fine in one direction. Fix the routing so both directions use the same path, or make the firewall stateful on both.

5. Duplicate subnets. Two sites both use 10.10.40.0/24. The more specific route wins and one site silently loses. This is the argument for a documented address plan more than any other.

Ping is not a routing test

A successful ping proves a 64-byte ICMP echo made a round trip. It does not prove TCP, it does not prove MTU, and it does not prove the application's source address is routable. "I can ping it" has closed more tickets incorrectly than any other sentence in networking.

A worked investigation

Report: "The new campus can't reach the print server at 10.0.0.30, but internet works."

  1. Expectation: campus 10.50.40.0/21 → district 10.0.0.30, via 10.0.0.9.
  2. From a campus laptop: traceroute 10.0.0.30 — dies after the campus router.
  3. On the campus router: ip route get 10.0.0.30 — matches 0.0.0.0/0 via 10.0.0.9. Good.
  4. On the district core: ip route get 10.50.40.57 — matches the summary. Good.
  5. Return test: from the print server, ping -S 10.0.0.30 10.50.40.57 — fails.
  6. On the core: the print server's VLAN has no route to 10.50.0.0/16 because its gateway is a different device with an older table. Fixed by adding the summary there.
  7. Re-test with a large packet to confirm MTU.

Total time: eleven minutes, because every step had a stated expectation.

Change discipline for routing

  • One route change at a time, with the expected table state written down first.
  • Change window after 17:00, because a wrong route is an outage for a whole subnet.
  • Snapshot the table before and after (ip route, or the UniFi config backup) and diff them.
  • Keep a floating static for the old path for a week after a cutover.
  • Document each route with a reason, an owner, and a date.

Checklist

  1. You can write the expected path before touching a keyboard.
  2. You test the return path with a sourced ping, not just forward reachability.
  3. You test MTU with a large DF packet on any tunnel.
  4. Every summary is allocated from a documented block.
  5. Route changes happen in a window, with a before/after table diff.
  6. The route table lives in the documentation next to the topology diagram.

Next

Static routing between sites is the smallest complete routing design. The next series moves up the stack: DNS, DHCP, and Core Services, where the failures are even quieter.

  • Troubleshooting
  • Traceroute
  • MTU
  • Routing