A troubleshooting method¶

Most outages are fixed by someone who already knows the answer, or by someone who works through the problem in an orderly way. The first case is luck and experience; the second can be learned. This page explains the second: a method that turns "it's broken" into a short series of questions, each of which the system can answer.

None of the ideas here is new. They come from the scientific method, from network engineering's layered models, and from the hard-won habits of people who have spent nights restoring services. What they share is a refusal to change things at random.

Why method beats intuition¶

Intuition is fast and often right, which is exactly what makes it dangerous. An experienced administrator who sees "connection timed out" thinks "firewall" and goes straight to the rules. When that guess is right, it saves minutes. When it's wrong, it costs far more: rules get loosened, the problem remains, and now there are two changes to undo and no clear record of which one mattered.

A method doesn't replace intuition; it disciplines it. Your hunch becomes the first hypothesis, tested cheaply, rather than the first change you make. If it's confirmed you have lost nothing; if it's refuted you have learned something and still have a known-good system to reason about.

The loop¶

The method is a loop that you repeat until the fault is found:

flowchart LR
    A["Define the symptom<br/>precisely"] --> B["Establish what<br/>changed"]
    B --> C["Form hypotheses"]
    C --> D["Test the cheapest,<br/>most likely first"]
    D -->|"refuted"| C
    D -->|"confirmed"| E["Change one thing"]
    E --> F["Check the symptom<br/>again"]
    F -->|"still broken"| G["Roll back,<br/>record, rethink"]
    G --> C
    F -->|"fixed"| H["Record the cause<br/>and the fix"]

Each stage exists to stop a particular kind of wasted effort.

Define the symptom precisely¶

"The website is down" is a report, not a symptom. A symptom is something you can reproduce and measure: which client, which name or address, which port, what error, since when, and for whom. "From the office network, curl https://app.example.com/ fails with Could not resolve host since about 09:40; from a phone on mobile data it works" already rules out most of the stack.

Precision matters because different errors point to different layers. A name that doesn't resolve, a connection that is refused, a connection that times out and a server that answers with an HTTP 500 are four different faults in four different places, even though a user may describe all of them as "it doesn't load".

A precise symptom also gives you the test you will use at the end: you know the fault is fixed when that exact command succeeds.

Establish what changed¶

Systems that worked yesterday and don't today usually changed in between. Before forming any theory, ask what changed: a package update, a configuration change, a certificate expiry, a reboot (which applies changes that were made long ago but never activated), a new firewall rule, a DNS record, a cloud security group, a full disk, a colleague's "small tweak".

Package manager history, configuration management logs, version control, change tickets and journalctl --since are the places to look. "Nothing changed" is rarely true; more often the change happened somewhere you weren't watching, such as upstream, at a provider, or in the passage of time itself (expiring certificates, leases, passwords and log partitions that finally filled).

If a recent change lines up with the start of the symptom, it becomes your strongest hypothesis, and rolling it back may be the fastest safe fix even before you fully understand it.

Form hypotheses¶

A hypothesis is a specific, testable claim: "the resolver configured on this host is unreachable", not "something is wrong with DNS". A good hypothesis predicts what a particular test will show. If you can't think of a test that would prove it wrong, it isn't yet useful.

Write down more than one. Holding several candidate explanations at once protects you from the most common trap in troubleshooting, which is falling in love with the first idea and interpreting every result as support for it.

Test the cheapest, most likely first¶

Order your tests by two things: how likely the hypothesis is, and how much the test costs. Cost includes time, but also risk: reading a routing table is free and harmless; restarting a database is neither.

Cheap, read-only observations come first: is the interface up, is there an address, is there a route, does the name resolve, is anything listening. Each takes seconds and changes nothing. Expensive or disruptive tests, such as packet captures on busy links, service restarts or reboots, come later, when the cheap tests have narrowed the field enough to justify them.

A test is only useful if it can separate your hypotheses. Running a command whose result wouldn't change your next step is activity, not diagnosis.

Change one thing at a time¶

When you do change something, change one thing, then check the symptom again with the same test you defined at the start. If you change three things and the problem goes away, you don't know which one fixed it, and two unnecessary changes now live on the system, possibly weakening it. If you change three things and the problem remains, you can't tell whether one of them helped and another made things worse.

Record what you did¶

Keep a running log as you work: the time, the command, what it showed, what you concluded and what you changed. A text file or a ticket is enough. The log stops you repeating tests, lets a colleague take over without starting again, makes rollback possible, and becomes the raw material for a post-incident review. It also forces you to state your reasoning, which is often when its gaps become obvious.

Know when to roll back¶

Every change you make while troubleshooting should have a known way back, decided before you make it. If a change doesn't fix the symptom, undo it before trying the next idea, unless you have a clear reason to keep it. If the incident is dragging on and a recent change is the likely trigger, restoring the previous known-good state is often better than diagnosing it live; you can investigate the cause afterwards, without users waiting.

Set yourself checkpoints. If an hour of work hasn't narrowed the problem, stop, reread your log, and consider escalating, rolling back or asking someone with fresh eyes.

Two ways to divide the problem¶

The loop tells you how to test; you still need a way to decide where to look. Two complementary strategies help.

Layer by layer¶

Network communication is built in layers, each relying on the one below. If a lower layer is broken, everything above it fails too, often with misleading errors. Checking layers in order means each test runs on a foundation you have already confirmed:

Working bottom-up is thorough and suits problems where you have little information, such as a newly built host or a symptom that affects everything. Working top-down, starting with the application error, is faster when the lower layers are clearly fine (other services on the same host work, say). Starting in the middle is a common shortcut: if a name resolves and a TCP connection to the port succeeds, everything below the transport layer is proven to work in one step, and you can move up or down from there. This "divide and conquer" approach is the fastest when you choose the split point well.

Notice how errors map to layers. Connection refused means a packet reached the host and it answered "nothing is listening here", or a firewall on the way rejected it; either way, routing works up to that point. A timeout means nothing answered at all: a dropped packet, a missing route back, or a host that is down. These two messages alone often halve the search.

Along the request path¶

Layers describe one host. Real requests cross several: a client, a resolver, perhaps a proxy, a load balancer, a firewall, the server, and the server's own dependencies such as a database. The second strategy is to draw that path and test at each hop, noting where the request is last seen working and where it is first seen failing. The fault lies between those two points.

Packet captures and logs are the tools for this: if the request appears in the load balancer's log but never reaches the backend, the problem is between those two, and nothing on the client needs to change. Testing from different vantage points (from the client, from a host on the server's subnet, from the server to itself) is the same idea: each vantage point removes part of the path from suspicion.

The two strategies combine naturally. Follow the path to find the hop where things go wrong, then work through the layers on that hop.

Common traps¶

Fixing the symptom report instead of the symptom. Users describe effects ("email is slow"); confirm the actual behaviour yourself before acting on their diagnosis.

Confirmation bias. Once you suspect the firewall, every timeout looks like the firewall. Ask what result would prove your hypothesis wrong, and run that test.

Trusting a single vantage point. A command that works from your laptop says little about the server's view of the network. Run tests from where the failure actually happens.

Testing the wrong thing. ping succeeding proves ICMP works, not that TCP port 443 is reachable; a hostname that resolves in your shell may not resolve inside a container or through a different resolver; /etc/hosts can mask DNS problems. Make sure the test exercises the same path as the failing request.

Changing several things at once, or not undoing failed changes. The system drifts away from any known state, and the eventual fix becomes impossible to explain.

Disruptive "fixes" that destroy evidence. Rebooting or restarting a service may clear the symptom, but it also clears the state that would explain it: in-memory caches, connection tables, the process that was misbehaving. When it's safe to do so, capture logs and state before restarting.

Ignoring intermittency. A fault that happens one time in ten is still deterministic somewhere: a single bad backend in a pool, a flapping link, one of two DNS servers returning stale data. Repeat tests enough times to see the pattern.

Stopping at the first fix. Restoring service is not the same as understanding the cause. If you don't know why it broke, record that, and come back to it; otherwise it will break again.

Locking yourself out. Changes to firewall rules, routes, addresses or SSH configuration on a remote host can cut off the connection you are using to fix it. Keep a second session open, use a console where one exists, or schedule an automatic rollback before applying such changes.

Further reading¶

• RFC 1122, "Requirements for Internet Hosts -- Communication Layers", for how the host-side layers fit together
• Brendan Gregg, "The USE Method": utilisation, saturation and errors, a checklist for performance problems in the same spirit
• Google, Site Reliability Engineering, chapter "Effective Troubleshooting"
• OpenRC and other init systems, for why service status and logs look different on Gentoo

Related¶

• How-to: How to diagnose a host you cannot reach
• Reference: Diagnostic commands by layer