Idempotency and configuration management¶

Every server starts the same way: someone logs in and types commands. Install a package, edit a file, restart a service. The first time, that's the quickest way to get the job done. The problems appear later: when a second server has to be made to match the first, when nobody remembers which of the twenty edits to /etc/ssh/sshd_config mattered, or when a server that "hasn't been touched" behaves differently from its supposed twin.

Configuration management is the discipline of writing down what a system should look like, in a form a program can apply and check, so that the description, not anyone's memory, is the source of truth. The central idea that makes this work is idempotency.

Two ways to describe a change¶

An imperative description lists steps: do this, then this, then this. A shell script is imperative by nature:

apt-get install -y nginx
echo "server_tokens off;" >> /etc/nginx/conf.d/hardening.conf
systemctl restart nginx

A declarative description states the result and leaves the steps to the tool: nginx is installed, this file has exactly this content, the service is running, and it's restarted whenever that file changes. In Ansible the same intent looks like this:

- name: nginx is installed
  ansible.builtin.package:
    name: nginx
    state: present

- name: hardening snippet is in place
  ansible.builtin.copy:
    dest: /etc/nginx/conf.d/hardening.conf
    content: "server_tokens off;\n"
  notify: restart nginx

- name: nginx is running and starts at boot
  ansible.builtin.service:
    name: nginx
    state: started
    enabled: true

The difference isn't the syntax. It's what happens when you run each one a second time.

Run the script again and echo … >> appends a second copy of the line, and nginx is restarted for no reason. Run it when the package mirror can't be reached and apt-get fails, and the script, which has no set -e, carries on with commands that assume nginx is installed; they fail in turn, and what the script leaves behind depends on which commands happened to succeed. Every run assumes a starting state, and the script is only correct if that assumption holds.

Run the declarative version again and nothing happens: the package is already present, the file already has the right content, so no handler is triggered and nginx isn't restarted. Run it on a server that is half configured, and it does only the missing parts. The description is correct whatever the starting point, because each step first compares the actual state with the desired one.

Idempotency¶

An operation is idempotent if applying it once or many times has the same effect. "Set the file's content to X" is idempotent; "append X to the file" isn't. "Ensure user deploy exists" is idempotent; useradd deploy isn't, because the second run fails with an error that the user already exists.

Idempotency is what makes it safe to apply configuration repeatedly, and repeated application is what keeps servers correct over time. It also changes how you use the tool: instead of asking "which steps haven't I run on this machine yet?", you just apply the whole description, and the tool works out what, if anything, needs to change.

Well-behaved configuration tools report which steps changed something. That report is useful in its own right: a run that reports zero changes proves the server already matches the description, and a run that reports a change on a server nobody was supposed to touch tells you something changed it.

Not everything can be made idempotent automatically. A tool can compare a package list or a file's contents with what it wants, but it can't know whether an arbitrary command such as a database migration has already run. Configuration tools provide escape hatches for running commands, and with them the author is responsible for adding a guard: a file the command creates, a query that checks first, or a condition on an earlier result. Most of the effort in writing good configuration code goes into these edges.

Shell scripts can be written idempotently too. install -d instead of mkdir, grep -q … || echo … >>, writing a whole file instead of editing one, useradd guarded by id -u: careful scripts use these patterns all the time. The difference is that in a script each guard is hand-written, and each one is a chance to get it wrong; in a declarative tool, most guards come built into the modules.

Drift¶

Configuration drift is the gap that opens over time between what a server is supposed to be and what it actually is. It comes from manual fixes during an incident, packages updated on one machine and not another, a colleague's quick experiment that was never reverted, or a script that ran on some servers but failed on others.

Drift matters because it makes servers unpredictable. Two servers that should be identical respond differently to the same update; a server rebuilt from the documented procedure doesn't behave like the one it replaces. Each difference is small, but they accumulate, and "pet" servers that nobody dares to rebuild are often the result.

A declarative description with an idempotent tool gives you two answers to drift:

Detection. Running the tool in check mode (Ansible's --check, Puppet's --noop) reports what it would change without changing it. Any reported change is drift.
Correction. Running it for real puts the server back to the described state, touching only what differs.

Neither works for things the description doesn't mention. A configuration tool manages the resources you declared and ignores everything else, so a file someone added by hand, or a package installed outside the description, stays where it is. This is the main difference from immutable infrastructure, where servers (or containers) are never changed in place at all: any change means building a new image and replacing the old instances, so there is nothing to drift. The two approaches are often combined, using a configuration tool to build the image.

Push and pull¶

Configuration tools differ in how a change gets from the description to the servers.

flowchart LR
    subgraph push["Push (Ansible)"]
        ctl["Control node<br/>playbooks + inventory"] -->|SSH| h1["server 1"]
        ctl -->|SSH| h2["server 2"]
        ctl -->|SSH| h3["server 3"]
    end
    subgraph pull["Pull (Puppet, Chef, CFEngine)"]
        srv["Server<br/>catalogue / policy"]
        a1["agent on server 1"] -->|"every N minutes"| srv
        a2["agent on server 2"] -->|"every N minutes"| srv
    end

In the push model, an operator (or a CI job) runs the tool on a control node, which connects to each target and applies the changes there. Ansible is the best-known push tool: it connects over SSH, copies small programs (modules) to the target, runs them with the target's Python interpreter, and removes them again. Nothing has to be installed or kept running on the targets beyond SSH and Python, which is why Ansible is called agentless. Changes happen when someone runs them, so it's easy to see who changed what and when; the flip side is that a server only converges when someone (or something) runs the playbook, so drift between runs goes unnoticed unless runs are scheduled.

In the pull model, every managed server runs an agent that periodically fetches its configuration from a central server and applies it locally. Puppet, Chef and CFEngine work this way by default. Pull scales well and corrects drift automatically on the agent's schedule, at the cost of running and securing an agent on every host and a central server they all trust. A mistake in the central configuration also reaches every server on its next check-in without anyone pressing a button, so pull setups need careful staging of changes.

The distinction is a default rather than a law. Ansible can be run in pull mode (ansible-pull checks out a repository and applies it to the local machine, often from a timer), and Salt supports both a long-running agent (a "minion") and an agentless mode over SSH.

Version control for configuration¶

Once the desired state is written down in files, those files deserve the same care as program source code, and the same tool: a version control system, usually Git. Version control gives configuration management several things it can't do without:

History. Every change is recorded with who made it, when and, in its commit message, why. "When did we turn off password logins, and why?" has an answer.
Review. Changes can be proposed and reviewed (for example as merge requests) before they reach any server, and checked automatically by linters and test runs.
Rollback. Going back to the configuration of last Tuesday is a git revert followed by a run, not an attempt to remember what was changed.
One source of truth. The repository is the authoritative description. Any difference between a server and the repository is either drift to correct or a change that should be committed.

Version control also works for configuration you don't manage with a tool. Tools such as etckeeper keep /etc itself in a Git repository and commit automatically around package upgrades, so even hand-maintained servers gain a history.

Secrets are the exception that needs special handling. Passwords, private keys and API tokens shouldn't be committed in plain text, even to a private repository, because a repository is copied to every clone and keeps every old version. Configuration tools provide encrypted stores for them (Ansible Vault, Puppet's eyaml back end, or an external secrets manager), so the repository holds only encrypted values or references.

Where scripts still fit¶

None of this makes shell scripts obsolete. Configuration tools are good at converging a known set of resources to a described state. They are clumsy for procedures: one-off data migrations, tasks with complex control flow, or anything that has to run on a machine where the tool isn't available. Scripts remain the right tool for those, and for scheduled jobs that do the same small task every night. What carries over from configuration management is the discipline: write scripts so that a re-run is safe, so that a failure stops the script instead of letting it continue in an unknown state, and keep them in version control.