Network Diagnostics Deep-Dive¶
When a ping succeeds but a service still won't connect, or a connection works "sometimes," you need tools that show you sockets, paths, and packets directly. This guide covers the four you reach for most: ss, traceroute/tracepath, mtr, and tcpdump.
Tested on
AlmaLinux 9 / RHEL 9 (also applies to Rocky Linux 9). Debian 12 / Ubuntu 22.04+ differences are noted inline. Commands are run as root via sudo.
If you haven't already, skim Networking Basics for the addressing and routing concepts these tools assume.
Installing the tools¶
ss ships with the base system (part of iproute2), so it's almost always present. The rest you usually install yourself.
What each package gives you
tcpdump— packet capture.mtr(Debian:mtr-tinyfor the CLI-only build) — combined ping + traceroute.bind-utils/dnsutils—dig,host,nslookup(handy alongside the rest; see Network or DNS Isn't Working).tracerouteis often already installed; if not,dnf install traceroute/apt install traceroute.
ss — socket statistics (the modern netstat)¶
ss reads socket data straight from the kernel. It's faster than the old netstat and is the tool to use today.
What's listening?¶
The single most useful invocation lists every listening TCP and UDP socket, numerically, with the owning process:
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
tcp LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=941,fd=3))
tcp LISTEN 0 511 *:80 *:* users:(("nginx",pid=1402,fd=6))
tcp LISTEN 0 4096 127.0.0.1:6379 0.0.0.0:* users:(("redis-server",pid=1188,fd=6))
udp UNCONN 0 0 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=712,fd=14))
The flags break down as:
-tTCP,-uUDP-llistening sockets only-pshow the owning process (needs root to see all processes)-nnumeric — don't resolve ports to names or addresses to hostnames
Read the Local Address carefully
0.0.0.0:80 (or *:80) means the service listens on all interfaces. 127.0.0.1:6379 means it listens on localhost only — a very common reason a service is unreachable from another host. The Recv-Q on a LISTEN socket is the current backlog; Send-Q is the configured backlog limit.
Established connections¶
Drop -l and filter by state to see live connections:
Recv-Q Send-Q Local Address:Port Peer Address:Port
0 0 10.0.0.12:22 10.0.0.5:51234
0 0 10.0.0.12:443 203.0.113.9:44120
Other states you can filter on include time-wait, syn-sent, fin-wait-1, and the shortcuts connected, synchronized, and bucket.
Filtering by port¶
ss has its own filter syntax. Match the destination (remote) port:
Or the local (source) port:
You can combine expressions with and/or and use comparisons (<, >, =):
Summary¶
For a quick health snapshot of all socket types and TCP states:
Total: 287
TCP: 24 (estab 11, closed 6, orphaned 0, timewait 5)
Transport Total IP IPv6
RAW 1 0 1
UDP 6 4 2
TCP 18 15 3
INET 25 19 6
A large and growing timewait count is normal under load; a runaway orphaned count or a backlog stuck near the Send-Q limit on a listener points at a saturated service.
traceroute / tracepath — the path to the destination¶
traceroute reveals the router hops between you and a destination by sending packets with increasing TTL and recording which router replies at each step.
traceroute to example.com (93.184.216.34), 30 hops max, 60 byte packets
1 _gateway (10.0.0.1) 0.412 ms 0.380 ms 0.371 ms
2 100.64.0.1 (100.64.0.1) 4.221 ms 4.180 ms 4.150 ms
3 * * *
4 ae-1.edge2.example.net (198.51.100.7) 11.9 ms 11.8 ms 11.7 ms
5 93.184.216.34 (93.184.216.34) 12.4 ms 12.2 ms 12.1 ms
Why hops time out¶
Seeing * * * at hop 3 above looks alarming but usually isn't. Common causes:
- The router deprioritizes or rate-limits the ICMP/UDP "time exceeded" replies traceroute relies on — it forwards your real traffic fine but won't waste CPU answering probes.
- A firewall drops the probe packets (UDP high ports by default) but allows the actual service ports.
Timeouts mid-path are normal
A * * * line in the middle of a trace does not mean traffic is lost there. What matters is whether you reach the final destination and whether every hop after a problem point fails. Use -T (TCP SYN, often -T -p 443) or -I (ICMP echo) to probe with packets firewalls are more likely to allow:
tracepath is a simpler, unprivileged alternative that also discovers the path MTU:
It needs no root and is handy for spotting MTU black holes (look for the reported pmtu value dropping mid-path).
mtr — traceroute and ping, continuously¶
mtr runs traceroute repeatedly and aggregates the results, giving you live per-hop packet loss and latency. It's the single best tool for "the connection is flaky" complaints.
This opens an interactive, refreshing display. For logs, tickets, or scripts, use report mode:
Start: 2026-06-07T10:15:02+0530
HOST: web01 Loss% Snt Last Avg Best Wrst StDev
1. AS??? _gateway 0.0% 100 0.4 0.5 0.3 1.9 0.2
2. AS64500 100.64.0.1 12.0% 100 4.2 4.6 4.0 9.1 0.8
3. AS64500 198.51.100.1 0.0% 100 9.8 10.1 9.7 14.2 0.6
4. AS15133 ae-1.edge2.net 0.0% 100 11.9 12.0 11.7 13.0 0.3
5. AS15133 93.184.216.34 0.0% 100 12.4 12.5 12.1 13.4 0.2
The flags mean:
-rreport mode (run, print, exit)-wwide output (don't truncate hostnames)-zshow the ASN for each hop-bshow both hostname and IP-c 100send 100 cycles before reporting
How to read mtr¶
This is where most people misdiagnose. Read the Loss% column from the bottom up:
Loss at an intermediate hop is usually not a problem
In the report above, hop 2 shows 12% loss but hops 3–5 show 0%. That loss does not persist to the destination, which means hop 2 is simply rate-limiting the ICMP replies directed at the router itself — it forwards transit traffic perfectly. This is the single most common false alarm in mtr output.
Loss is real only when it starts at some hop and continues all the way to the final hop. For example, if hops 4 and 5 both showed ~8% loss, that's genuine loss affecting your traffic, and the problem lives at or before hop 4.
Also watch the latency columns: a steadily climbing Avg from one hop onward (and high StDev/Wrst) indicates congestion or a saturated link starting at that hop.
TCP and UDP modes¶
By default mtr uses ICMP. If a path filters ICMP, probe with the actual protocol and port your service uses:
TCP mode often gives a cleaner picture for web traffic, because routers and firewalls that drop ICMP will happily forward TCP SYNs to a real service port.
tcpdump — see the actual packets¶
When you need to know whether a packet actually arrived or got answered, nothing beats capturing it. tcpdump is the workhorse.
tcpdump needs root and can expose sensitive data
Capturing requires sudo. A capture can contain credentials, tokens, cookies, and personal data in plaintext — anything sent unencrypted. Capture the minimum you need with a tight filter, store .pcap files securely, and delete them when you're done. Never paste raw payload captures into a public ticket or chat.
Capture basics¶
Pick an interface with -i. List interfaces with tcpdump -D (or ip -br link):
Three flags you'll use on almost every capture:
-nn— don't resolve hostnames or port numbers (faster, and avoids DNS lookups polluting your own capture)-c <count>— stop after N packets-v/-vv— more verbose decode
10:42:01.110233 IP 10.0.0.5.51234 > 10.0.0.12.443: Flags [S], seq 1138...
10:42:01.110540 IP 10.0.0.12.443 > 10.0.0.5.51234: Flags [S.], seq 882..., ack 1139...
10:42:01.110612 IP 10.0.0.5.51234 > 10.0.0.12.443: Flags [.], ack 1...
Snaplen
Modern tcpdump captures the full packet by default (snaplen 262144). If you only care about headers — for example, to watch a connection handshake without storing payload — limit it with -s, e.g. -s 96 grabs roughly the IP/TCP headers and keeps .pcap files small.
BPF filters¶
The expression after the options is a Berkeley Packet Filter. The kernel applies it before packets reach tcpdump, so it's cheap. Primitives you'll use constantly:
host 10.0.0.12— traffic to or from that addresssrc host 10.0.0.5/dst host 10.0.0.12— one direction onlyport 443— to or from that port;src port/dst portfor directionportrange 8000-8100net 10.0.0.0/24tcp,udp,icmp— protocol- combine with
and,or,not(and parentheses, quoted to protect them from the shell)
Common recipes¶
# Just the TCP SYN packets — see who's trying to open connections
sudo tcpdump -i any -nn 'tcp[tcpflags] & tcp-syn != 0'
Inspecting payload¶
-A prints the packet payload as ASCII — perfect for plaintext HTTP:
-X prints hex and ASCII side by side, for binary protocols:
Payload inspection only works on unencrypted traffic
For HTTPS (port 443) the payload is TLS-encrypted, so -A/-X show ciphertext. You can still confirm the handshake happens and watch the SYN/SYN-ACK/ACK and TLS record headers, but you won't see request contents without the session keys.
Write to a pcap, analyze in Wireshark¶
For anything non-trivial, capture to a file with -w and analyze later. Writing to a file also avoids the cost of decoding on a busy box.
# Capture to a file (note: -w output is binary, not text)
sudo tcpdump -i any -nn -w /tmp/cap.pcap host 10.0.0.12 and tcp port 443
Read it back on the CLI with -r (the same filters apply):
Then open /tmp/cap.pcap in Wireshark for full protocol dissection, stream following (Follow TCP Stream), and graphs. Copy it off the server first:
Ring buffers for long captures
To capture over a long window without filling the disk, rotate files: -w cap.pcap -C 100 -W 10 keeps ten 100 MB files, overwriting the oldest. Combine with a tight BPF filter so you only keep relevant packets.
A combined troubleshooting flow¶
Symptom: "the service on port 443 is unreachable from a client." Work outward from the server, layer by layer.
1. Is the service even listening, and on the right address? (on the server)
If there's no row, the service is down — start there. If it shows 127.0.0.1:443 instead of 0.0.0.0:443, it's bound to localhost only and no remote client can reach it, regardless of the network.
2. Is the network path between client and server healthy? (from the client)
Read loss from the bottom up. Loss that reaches the final hop is a real path problem (escalate to the network team / provider with this report). Clean path? Move on.
3. Are the packets actually arriving and being answered? (on the server)
Now interpret what you see:
- No packets at all → traffic isn't reaching the host. Suspect a firewall or routing in between — check the host firewall and Security Groups. See Firewalls.
- SYN arrives, no SYN-ACK back → the host received the connection attempt but the service or firewall isn't accepting it. Recheck
ssand the local firewall rules. - SYN, then a RST from the server → something actively refused the connection (no listener on that port, or a reject rule).
- Full SYN / SYN-ACK / ACK handshake → the network and service are fine; the problem is higher up (TLS, application, auth). Capture with
-Aon the relevant unencrypted port or check application logs.
This ss → mtr → tcpdump progression isolates the failure to a layer in three commands.
Verify your work¶
Run these to confirm the tools are installed and you can read their output:
# Tools present
ss -V
tcpdump --version
mtr --version
dig -v
# You can list listening sockets with process names
sudo ss -tulpn | head
# You can capture a few packets (generate some with a ping in another shell)
sudo tcpdump -i any -nn -c 5 icmp
You should see version strings, a table of listeners with process names, and five ICMP packets captured. If ss -tulpn shows no process names, you forgot sudo.
Summary¶
ssreplacesnetstat:ss -tulpnfor listeners,ss -tn state establishedfor live connections,ss -sfor a summary. Always check the bind address —127.0.0.1vs0.0.0.0explains a huge share of "unreachable" tickets.traceroute/tracepathshow the path; mid-path* * *is usually rate-limited ICMP, not lost traffic. Use-T -p 443to probe through ICMP-filtering paths.mtrcombines ping and traceroute with live loss/latency. Read loss bottom-up: only loss that persists to the final hop is real. Use--tcp --portfor a realistic picture of web paths.tcpdumpcaptures the packets themselves. Learn-i,-nn,-c, BPF filters (host,port,tcp[tcpflags] & tcp-syn), payload inspection (-A/-X), and-w/-rfor pcap files you analyze in Wireshark. It needs root and can expose secrets — filter tightly.- The flow:
ss(is it listening?) →mtr(is the path healthy?) →tcpdump(are the SYNs arriving and being answered?).
See also: Networking Basics, Network or DNS Isn't Working, Firewalls.