Skip to content

How to diagnose a host you cannot reach

Use this guide when ssh, curl or another client on a Linux machine can't connect to a remote host and you need to find out where the connection breaks. You work from the local machine outward: interface, address, route, gateway, name resolution, the path, the port, and finally the local firewall. Each check either clears a layer or tells you where the fault is.

Before you begin

  • You have a shell on the machine where the connection fails, and sudo rights.

  • Write down the exact failure first: the command, the target, the port and the full error message. The error alone often points at a layer:

    ssh: Could not resolve hostname app.example.com: Name or service not known   <- name resolution
ssh: connect to host 203.0.113.10 port 22: Network is unreachable              <- local routing
ssh: connect to host 203.0.113.10 port 22: No route to host                    <- gateway, neighbour or a reject rule
ssh: connect to host 203.0.113.10 port 22: Connection timed out                <- path or a firewall dropping packets
ssh: connect to host 203.0.113.10 port 22: Connection refused                  <- nothing listening, or a reject rule

  • The commands use these placeholders. Replace them with your own values:

    Placeholder Meaning Example
    <TARGET> The name of the host you can't reach app.example.com
    <ADDRESS> The IP address you expect <TARGET> to have 203.0.113.10
    <PORT> The TCP port of the service 22
    <INTERFACE> Your machine's network interface enp0s3
    <GATEWAY> Your default gateway 192.0.2.1
    <DNS_SERVER> A DNS server you are configured to use 192.0.2.53

  • Install the diagnostic tools used below (dig, mtr, nc and nft) if they're missing:

    sudo apt-get update
sudo apt-get install -y bind9-dnsutils mtr-tiny netcat-openbsd nftables

    sudo dnf install -y bind-utils mtr nmap-ncat nftables

    On these distributions nc is Ncat, from the Nmap project. It accepts the options used in this guide.

    sudo pacman -S --needed --noconfirm bind mtr openbsd-netcat nftables

    sudo emerge --noreplace --getbinpkg net-dns/bind net-analyzer/mtr net-analyzer/openbsd-netcat net-firewall/nftables

    This needs a synchronised ebuild repository (emerge --sync). --getbinpkg uses Gentoo's binary package host where a matching package exists; the rest are built from source.

Steps

  1. Check that your interface is up and has an address:

    ip -br link
ip -br addr

    Look for your interface in state UP with an address in the subnet you expect. DOWN or NO-CARRIER means a link problem: a cable, a virtual switch, or an interface that was never brought up. No IPv4 address usually means DHCP failed. Look at the error and drop counters too:

    ip -s link show <INTERFACE>

    Counters under errors or dropped that grow each time you run the command point at the link or the driver.

  2. Ask the kernel which route it would use to reach the target address:

    $ ip route get <ADDRESS>
203.0.113.10 via 192.0.2.1 dev enp0s3 src 192.0.2.25 uid 1000
    cache

    The output shows the gateway (via), the outgoing interface (dev) and the source address (src). Check that they're the ones you expect. RTNETLINK answers: Network is unreachable means there is no route at all: usually a missing default route. If the target is on your own subnet, there is no via and the packets go straight to the target.

  3. Check that the gateway answers:

    ip route show default
ping -c 3 <GATEWAY>
ip neigh show <GATEWAY>

    If ping fails, look at the neighbour entry. REACHABLE with a link-layer address (lladdr) means the gateway answered ARP (or IPv6 neighbour discovery) just now, so it exists on the link even if it ignores pings. STALE means it answered at some point but hasn't been confirmed recently; run ping again and recheck, and the entry should turn REACHABLE (or DELAY while it's being confirmed). FAILED or INCOMPLETE means nothing on the link answers for that address: wrong gateway, wrong VLAN or subnet, or a link fault.

  4. Check that the name resolves, and to the address you expect:

    getent hosts <TARGET>
dig +short <TARGET>

    getent resolves the name the way most programs do, through /etc/nsswitch.conf, so it includes /etc/hosts. dig asks DNS only. If the two disagree, look for an entry in /etc/hosts. If neither returns an address, query a DNS server directly to separate "the resolver is unreachable" from "the record doesn't exist":

    dig @<DNS_SERVER> <TARGET>

    In the dig output, status: NXDOMAIN means the name doesn't exist on that server, status: SERVFAIL means the server couldn't get an answer, and connection timed out; no servers could be reached means the DNS server itself is unreachable.

    Then check which DNS servers your machine uses:

    resolvectl status
resolvectl query <TARGET>

    These work when systemd-resolved is running, which is the default on Ubuntu and on Debian cloud images. Otherwise the servers are listed in /etc/resolv.conf.

    cat /etc/resolv.conf
nmcli device show <INTERFACE>

    NetworkManager writes /etc/resolv.conf. systemd-resolved isn't installed by default; it's a separate package of the same name, so resolvectl (which comes with systemd) reports The name is not activatable. Look at the IP4.DNS lines in the nmcli output.

    resolvectl status
resolvectl query <TARGET>

    These work when systemd-resolved is running. If resolvectl status lists only Fallback DNS Servers, the network configuration hasn't supplied any DNS servers and resolved is using its built-in fallbacks.

    With OpenRC, systemd-resolved isn't used. The DNS servers are in /etc/resolv.conf. For an interface configured with config_<INTERFACE>="dhcp" in /etc/conf.d/net, netifrc starts dhcpcd, and its 20-resolv.conf hook writes the file from the DHCP lease:

    cat /etc/resolv.conf

    With the systemd profile (not verified):

    resolvectl status
resolvectl query <TARGET>

  5. Trace the path towards the target:

    sudo mtr --report --report-cycles 5 -n <ADDRESS>

    Use the address rather than the name, so the trace works even if name resolution is broken. mtr sends probes with increasing TTL values and lists each router that answers, with its packet loss and latency. Read it from the top: the last hop that answers, and the first hop after which every line shows ??? or 100% loss, bracket the place where packets stop. Loss shown at one intermediate hop but not at the hops after it is usually that router rate-limiting its replies, not real loss.

  6. Test the port itself:

    nc -vz -w 5 <ADDRESS> <PORT>

    If nc isn't available, Bash can open a TCP connection itself:

    timeout 5 bash -c 'exec 3<>/dev/tcp/<ADDRESS>/<PORT>' && echo "port <PORT> is open"

    Read the result:

    The wording depends on the tool: OpenBSD netcat (Debian, Ubuntu, Arch, Gentoo) prints succeeded!, failed: Connection refused or timed out; Ncat (RHEL family) prints Connected to, Connection refused. or TIMEOUT.; Bash prints Connection refused, or nothing when timeout stops it (exit status 124).

    • Succeeded or open: the network path and the service's listening socket are fine. The problem is in the application or its configuration (for example authentication, TLS or the HTTP response).
    • Connection refused: nothing listens on that port, or a firewall rule on the target or along the path rejects the connection. Either way, routing works up to the point that answered.
    • Timed out: packets are dropped somewhere, by a firewall on the target, on the path or on your own machine.

    To see what a refusal looks like, test a port on your own machine that nothing listens on. This is the OpenBSD netcat output; Ncat prints Ncat: Connection refused.:

    $ nc -vz -w 5 127.0.0.1 9
nc: connect to 127.0.0.1 port 9 (tcp) failed: Connection refused

  7. If you can reach the target another way (a console, or ssh from another host), check that the service listens on the expected address and port there:

    sudo ss -ltnp 'sport = :<PORT>'

    A service listening on 127.0.0.1 only accepts connections from the target itself. It needs to listen on 0.0.0.0, [::] or the target's own address to accept connections from you.

  8. Check your machine's own firewall rules:

    sudo nft list ruleset

    Look for drop or reject rules in output chains, or a restrictive output policy (policy drop), that match the target address or port. An empty output means no nftables rules are loaded.

    Rules created with iptables appear in this output only when iptables uses the nf_tables backend; they show up in tables named ip filter and ip6 filter. Rules created with the legacy backend (iptables-legacy) don't appear, so list them as well:

    sudo iptables -S
sudo ip6tables -S

    iptables uses the nf_tables backend by default (sudo iptables -V reports nf_tables), so these rules also appear in the nft output; listing them here also catches rules added with iptables-legacy.

    iptables isn't installed by default. The only iptables available is the nf_tables variant (iptables-nft), whose rules already appear in the nft output. If firewalld is in use, sudo firewall-cmd --list-all shows its configuration in its own terms.

    sudo iptables -S
sudo ip6tables -S

    iptables uses the nf_tables backend by default (sudo iptables -V reports nf_tables), so these rules also appear in the nft output; listing them here also catches rules added with iptables-legacy.

    sudo iptables -S
sudo ip6tables -S

    net-firewall/iptables uses the legacy backend unless it was built with the nftables USE flag (sudo iptables -V reports legacy). Legacy rules never appear in the nft output, so these commands are the only way to see them.

    Run the same commands on the target, if you can, to check its input chains.

Verify

The diagnosis is complete when you can name the layer at which the connection fails, with the command output that proves it. After you fix the cause, run the client command that originally failed; it should now connect. Then repeat the port test from step 6:

nc -vz -w 5 <ADDRESS> <PORT>

Troubleshooting

ping fails but ssh or curl works
Many hosts and firewalls drop ICMP echo requests. A failed ping proves nothing on its own; use the port test from step 6.
The name resolves with getent but not with dig (or the reverse)
getent also reads /etc/hosts and other sources in /etc/nsswitch.conf; dig reads only DNS. An old /etc/hosts entry pointing at a previous address is a common cause.
mtr shows ??? for every hop after the first
Some networks drop the ICMP time-exceeded replies mtr relies on. Try TCP probes to the service port with sudo mtr --report --tcp --port <PORT> -n <ADDRESS>.
nc reports success but the client still fails
The network is fine. Run the client with verbose output (ssh -v, curl -v) and check the service's logs on the target.