All lessons
DevOps & cloudIntermediate12 min readWorth knowing

Ansible

Configure many servers the same way with readable YAML playbooks, over plain SSH.

  • Playbooks
  • Inventory
  • Roles
  • Idempotency
Step 1/6
Infrastructure as code: you describe the servers and the database you want in a file — main.tf — and keep it in Git.

In 30 seconds

Ansible is an automation tool that connects to your servers over SSH and brings them to a described state: packages installed, files in place, services running, users created. You write the desired state in YAML playbooks and list the servers in an inventory. It needs no agent on the servers, and a well-written playbook is idempotent: running it twice changes nothing the second time.

Not in my daily stack — I know the concepts and would learn the details when a project needs it.

Why it matters

  • It replaces hand-typed server setup with code you can review, repeat and version, so the tenth server is configured exactly like the first.
  • It is agentless: only SSH and Python on the target, which makes it easy to adopt on existing servers without installing anything.
  • YAML playbooks are readable by people who are not full-time programmers, so operations and development teams can share them.
  • It complements Terraform well: Terraform creates the machines, Ansible configures what runs on them.

How it works

01The idea: desired state over SSH

Imagine writing a checklist for setting up a server: install nginx, copy the config, open port 443, start the service. Ansible turns that checklist into code. The control node, your laptop or a CI runner with Ansible installed, connects to managed nodes over SSH, copies small Python modules, runs them and removes them. Each task calls a module such as apt, copy, template, user or service, and describes the end state (state: present, state: started) rather than the commands to get there. The module checks the current state first and only changes what differs, reporting ok or changed. That property, idempotency, means you can run the same playbook every day as a drift check. Ansible is distributed as ansible-core (the engine and built-in modules) plus collections, packages of modules for clouds, databases and network devices, bundled together in the ansible community package.

  • Control node: where Ansible runs. Managed nodes: the servers.
  • Module: a unit of work (apt, template, service).
  • Idempotent: second run reports ok, not changed.

02Install, inventory and ad-hoc commands

Install Ansible on the control node with pipx or pip into its own environment, which avoids conflicts with system Python. The inventory lists managed hosts and groups them, in INI or YAML. Groups let you target web servers or databases separately, and group_vars/ and host_vars/ folders hold variables per group or host. Before writing playbooks, test connectivity with ad-hoc commands: one-off module calls from the command line. ansible all -m ping does not use ICMP; it logs in over SSH and checks Python. Use SSH keys and a non-root user with sudo; Ansible's become: true escalates privileges for tasks that need it. Keep ansible.cfg in the project to set the inventory path and sensible defaults.

yaml
# install on the control node
pipx install --include-deps ansible
 
# inventory.yml
all:
  children:
    web:
      hosts:
        web1: { ansible_host: 203.0.113.10 }
        web2: { ansible_host: 203.0.113.11 }
  vars:
    ansible_user: your_user
 
# test connectivity and gather facts
ansible -i inventory.yml all -m ping
ansible -i inventory.yml web -m ansible.builtin.setup -a "filter=ansible_distribution*"

03Your first playbook

A playbook is a list of plays; each play targets hosts and runs tasks in order. The example installs nginx on the web group, deploys a site configuration from a Jinja2 template, and makes sure the service runs. The notify line triggers a handler only when the template actually changed, so nginx reloads once at the end of the play instead of on every run. Always use fully qualified module names such as ansible.builtin.apt, which make it clear which collection a module comes from. Run it with --check --diff first to see what would change without changing anything, then for real. A second real run should show changed=0; if it does not, a task is not idempotent and should be fixed.

yaml
- name: Configure web servers
  hosts: web
  become: true
  tasks:
    - name: Install nginx
      ansible.builtin.apt:
        name: nginx
        state: present
        update_cache: true
    - name: Deploy site config
      ansible.builtin.template:
        src: templates/site.conf.j2
        dest: /etc/nginx/sites-available/default
        mode: "0644"
      notify: Reload nginx
    - name: Ensure nginx is running
      ansible.builtin.service: { name: nginx, state: started, enabled: true }
  handlers:
    - name: Reload nginx
      ansible.builtin.service: { name: nginx, state: reloaded }

04Variables, templates and secrets

Variables make one playbook work for many environments: the same role configures staging and production with different values from group_vars/staging.yml and group_vars/production.yml. Templates use Jinja2 syntax, {{ variable }} and simple loops or conditions, to render configuration files from those variables. Facts are variables Ansible gathers automatically from each host, such as the OS version, memory and IP addresses, useful for conditions like when: ansible_distribution == 'Ubuntu'. Variable precedence has many levels; keep it simple by using role defaults for sane values and group_vars for environment overrides. For secrets, Ansible Vault encrypts files or single values with a password, so they can live in Git safely. In CI, supply the vault password from the CI secret store, or fetch secrets at run time from a manager such as HashiCorp Vault with a lookup plugin.

yaml
# group_vars/production/vars.yml
app_domain: example.com
app_workers: 4
db_password: "{{ vault_db_password }}"
 
# group_vars/production/vault.yml (encrypted)
# ansible-vault create group_vars/production/vault.yml
# vault_db_password: change-me
 
# run with the vault password
ansible-playbook -i inventory.yml site.yml --ask-vault-pass

05Roles and collections: structure that scales

When playbooks grow, split them into roles: a standard folder layout with tasks, handlers, templates, files, defaults and meta. A role does one thing well, such as nginx, postgres or a base hardening baseline, and a site playbook composes roles per host group. ansible-galaxy role init creates the skeleton. Collections package roles, modules and plugins together under a namespace, and you install them from Ansible Galaxy or a private hub with a requirements.yml pinned to versions, the way you would pin npm packages. Test roles with Molecule, which starts a container or VM, applies the role, runs it again to check idempotency and verifies the result. Lint with ansible-lint in CI to catch deprecated syntax, missing names and risky patterns.

yaml
# requirements.yml
collections:
  - name: community.general
    version: ">=10.0.0,<11.0.0"
  - name: ansible.posix
    version: ">=2.0.0"
 
# site.yml
- hosts: web
  become: true
  roles:
    - base_hardening
    - nginx
 
# ansible-galaxy collection install -r requirements.yml

Use it when

  • Configuring a fleet of Linux servers or VMs consistently: packages, users, configs, services, hardening.
  • Repeatable provisioning of servers created by Terraform or a cloud console.
  • Orchestrated operations: rolling restarts, patching in batches, certificate rotation across hosts.
  • Network devices and appliances, which have strong Ansible collections from vendors.

Skip it when

  • Creating and tracking cloud infrastructure with dependencies and state; Terraform or OpenTofu fits better.
  • Workloads already fully containerised on Kubernetes, where manifests, Helm and operators manage configuration.
  • Very large fleets needing continuous enforcement every few minutes, where agent-based tools or immutable images may suit better.

Alternatives

  • Terraform / OpenTofu

    Declarative infrastructure provisioning with state; often used together with Ansible.

  • Chef / Puppet

    Agent-based configuration management with continuous enforcement.

  • Salt

    Fast, event-driven configuration and remote execution with an agent or SSH mode.

  • Immutable images (Packer, cloud-init)

    Bake configuration into images and replace servers instead of changing them.

Best practices

  1. Write idempotent tasks with modules; use shell or command only when no module exists, with creates: or changed_when.
  2. Use fully qualified collection names (ansible.builtin.copy) and pin collection versions in requirements.yml.
  3. Organise code into small roles with defaults, and keep environment differences in group_vars.
  4. Encrypt secrets with Ansible Vault or fetch them from a secret manager; never commit plain passwords.
  5. Run --check --diff before risky changes and use serial: for rolling updates.
  6. Lint with ansible-lint and test roles with Molecule in CI.
  7. Use SSH keys, a dedicated automation user and become only where needed.

Common mistakes

  • Using shell: for everything, so every run reports changed and nothing is safely repeatable.
  • Mixing tabs, wrong indentation or unquoted values that YAML reads as booleans or numbers, such as yes or 0644.
  • Relying on complex variable precedence until nobody knows where a value comes from.
  • Running a change on all hosts at once without serial, taking the whole service down together.
  • Storing vault passwords next to the encrypted files in the repository.

Deep dive

06How a task actually executes

For each task, Ansible renders the module arguments with Jinja2 on the control node, packages the module code (AnsiballZ) and sends it over SSH, then runs it with the remote Python and reads JSON back. By default it does this for forks hosts at a time (default 5) under the linear strategy, where every host finishes a task before any host starts the next. Performance tuning matters at scale: raise forks, enable pipelining in ansible.cfg (it removes extra SSH round-trips, but requires that sudo does not demand a TTY), keep SSH ControlPersist multiplexing on, and turn off or cache fact gathering when you do not need facts. The free strategy lets fast hosts run ahead. Long tasks can run asynchronously with async and poll so connections are not held open.

ini
# ansible.cfg
[defaults]
inventory = inventory.yml
forks = 25
gathering = smart
fact_caching = jsonfile
fact_caching_connection = .ansible_facts
fact_caching_timeout = 3600
 
[ssh_connection]
pipelining = True
ssh_args = -o ControlMaster=auto -o ControlPersist=60s

07Rolling updates with serial and health checks

Production changes should never hit every host at once. serial accepts a number, a percentage or a list such as [1, 5, "25%"], which gives you a canary host, then a small batch, then larger ones. delegate_to runs a task on another machine, typically to take the current host out of a load balancer. Combined with until and retries, a health check waits for the service to become healthy before moving on. max_fail_percentage aborts the play when too many hosts in a batch fail, which stops a bad release from spreading. any_errors_fatal stops everything at the first failure, useful for database clusters where partial changes are dangerous.

yaml
- hosts: web
  become: true
  serial: [1, 5, "25%"]
  max_fail_percentage: 0
  tasks:
    - name: Deploy new release
      ansible.builtin.include_role: { name: app_release }
    - name: Wait until healthy
      ansible.builtin.uri:
        url: "http://{{ ansible_host }}:8080/healthz"
        status_code: 200
      register: health
      until: health.status == 200
      retries: 20
      delay: 3
      delegate_to: localhost

08Dynamic inventory, AWX and Execution Environments

Static inventory files go stale in the cloud. Inventory plugins such as amazon.aws.aws_ec2, azure.azcollection.azure_rm or google.cloud.gcp_compute query the provider and build groups from tags, so a server tagged role=web appears in the web group automatically. To run Ansible as a shared service rather than from laptops, AWX (the open-source upstream of Red Hat Ansible Automation Platform) provides a web UI, role-based access, credential storage, schedules, an API and audit logs. Execution Environments are container images that bundle ansible-core, collections and Python dependencies, built with ansible-builder and run with ansible-navigator or AWX. They fix the classic "works on my control node" problem by making the automation runtime as reproducible as the application images it deploys.

Real-world scenarios

01

Hardening twenty existing servers consistently

Problem

Twenty Ubuntu servers were set up by hand over several years. SSH settings, firewall rules and users differ between them, and an audit is coming.

How to solve it

Create a base_hardening role: disable SSH password and root login through a template for sshd_config with a validate: sshd -t check, install and configure ufw with default deny incoming, enable unattended security upgrades, manage admin users and their keys from a variable list, and install fail2ban. Run it with --check --diff against all hosts to see the drift, review it, then apply to one host, then the rest with serial. Schedule a weekly check-mode run from CI and alert on any changed result, so drift is detected before the next audit.

02

A playbook always reports changed

Problem

A team's deploy playbook shows changed on every task every run, so nobody can tell when something really changed, and handlers restart services needlessly.

How to solve it

Replace shell commands with modules: apt instead of apt-get install, copy or template instead of echo into files, systemd instead of systemctl restart. Where a command is unavoidable, add creates: for one-time setup, or register the output and set changed_when based on it. Move restarts into handlers triggered by notify so they run only after real changes. Run the playbook twice in Molecule with the idempotence test, and add ansible-lint to CI, which flags no-changed-when patterns. The second run should report changed=0.

03

Patching a database cluster safely

Problem

Three database replicas and a primary need kernel updates and reboots. Rebooting more than one at a time, or the primary first, would cause an outage.

How to solve it

Write a play with serial: 1 and any_errors_fatal: true, targeting replicas first in their own group, then the primary in a later play. For each host: confirm replication is healthy with a check task, apt upgrade, reboot with the ansible.builtin.reboot module (which waits for the host to return), then wait until replication lag is back under a threshold with until and retries. Before the primary, trigger a controlled failover to a replica through the database's tooling, then patch the old primary as a replica. Run on staging first.

Cheat sheet

Check connectivity
ansible -i inventory.yml all -m ping
Dry run with diff
ansible-playbook site.yml --check --diff
Limit to one host
ansible-playbook site.yml --limit web1
Run only tagged tasks
ansible-playbook site.yml --tags nginx
Create encrypted file
ansible-vault create group_vars/production/vault.yml
Encrypt one value
ansible-vault encrypt_string 'change-me' --name 'vault_db_password'
New role skeleton
ansible-galaxy role init nginx
Install pinned collections
ansible-galaxy collection install -r requirements.yml
Lint playbooks
ansible-lint
Show module docs
ansible-doc ansible.builtin.template

Questions & answers

Q1What does idempotent mean in Ansible?

An idempotent task produces the same end state no matter how many times you run it, and changes nothing when the system is already correct. For example, ansible.builtin.apt with state: present installs nginx only if it is missing; on the next run it reports ok. That lets you run playbooks repeatedly as a safety net and drift check. Modules are idempotent by design; raw shell commands are not, because Ansible cannot know what they did. When you must use command or shell, add creates: or removes: or a changed_when condition so the task reports honestly. A good test is running the playbook twice: the second run should show changed=0.

Q2Ansible or Terraform?

They solve different layers and work well together. Terraform is declarative provisioning with state: it creates cloud resources such as networks, VMs, databases and DNS records, tracks them in a state file, plans changes and can destroy them. Ansible is procedural configuration over SSH: it configures the inside of machines, packages, files, users and services, and orchestrates operations such as rolling restarts. Ansible has cloud modules, but without a state file it is weaker at tracking and deleting resources. A common pattern is Terraform creates servers and outputs their addresses, and Ansible uses a dynamic inventory to configure them.

Q3Why is Ansible called agentless and what are the trade-offs?

Ansible needs no daemon on managed servers. It connects over SSH, copies small Python modules, runs them and cleans up. Requirements on the target are just SSH and Python, which nearly every Linux server has. That makes adoption simple and the attack surface small. The trade-offs: it is push-based, so configuration is only enforced when someone or something runs the playbook, not continuously. SSH connections cost time on very large fleets, though forks, pipelining and SSH multiplexing help a lot. Agent-based tools like Puppet enforce state every half hour by themselves. With Ansible you get similar behaviour by running playbooks on a schedule from CI or AWX.

Q4How do you handle secrets in Ansible?

The built-in answer is Ansible Vault: it encrypts whole files or individual variables with AES-256, so secrets can be committed alongside playbooks. A common layout keeps vault.yml encrypted next to a plain vars.yml that references vault_ prefixed variables, so you can still search for variable names. The vault password itself comes from a prompt, a password file outside the repository or a script that reads it from a secret store; in CI it is a protected secret. For larger setups, lookup plugins fetch secrets at run time from HashiCorp Vault or cloud secret managers, so they never sit in Git at all. Use no_log: true on tasks that would print secrets.

Q5How do you roll out a change to 50 web servers without downtime?

Use a rolling update in one play. Set serial: to a batch size, for example 5 or 10%, so Ansible completes all tasks on one batch before the next. In pre_tasks, remove the host from the load balancer; in tasks, deploy and restart; then run a health check with ansible.builtin.uri until the app responds, and add it back in post_tasks. Set max_fail_percentage so the rollout stops if a batch fails rather than breaking the whole fleet. Test with --check --diff on staging first and limit to one host with --limit for a canary. This turns a risky change into a controlled, observable process.

Senior-level questions

S1Mutable configuration with Ansible or immutable images: which do you prefer?

Immutable means building a server image with all configuration baked in, for example with Packer, and replacing servers rather than changing them. It gives identical machines, fast scaling and easy rollback to the previous image, and eliminates drift. Mutable configuration with Ansible changes running servers in place: cheaper to start, works with existing long-lived servers and stateful systems like databases, and is faster for small changes. Many mature setups combine them: Ansible runs inside the Packer build to produce the image, so the same roles serve both. For stateless app servers in an autoscaling group I prefer immutable; for databases, bastions and a handful of long-lived VMs, Ansible in place is pragmatic. The real enemy is unmanaged manual change, whichever model you pick.

S2How would you structure Ansible for multiple teams and environments?

Separate reusable code from environment data. Roles and modules live in versioned collections in their own repositories, tested with Molecule and released with semantic versions. Each environment has an inventory repository or directory with group_vars, host_vars and a requirements.yml that pins collection versions, so staging can try a new role version before production. Execution Environments freeze ansible-core and Python dependencies. Runs happen from CI or AWX, not laptops, with credentials stored there and role-based access controlling who can run which job template against which inventory. Keep variable precedence flat: defaults in roles, overrides in group_vars, and avoid extra-vars except for one-off parameters. The trade-off is ceremony: for a team of two with five servers, a single repository is perfectly fine.

S3What limits Ansible's performance and how do you scale it to thousands of hosts?

The bottlenecks are SSH connection setup, per-task round-trips, fact gathering and the control node's CPU, since each fork is a process. Scaling steps: increase forks in line with the control node's cores and memory; enable pipelining and ControlPersist to reuse connections; cache facts or set gather_facts: false where not needed; reduce tasks by using modules that accept lists, for example installing many packages in one apt call; and use async for long operations. Split very large runs by group or region and run them in parallel from several control nodes or AWX execution nodes close to the hosts. Mitogen-style accelerators exist but track ansible-core versions unevenly, so test carefully. At very large scale, consider pull mode with ansible-pull or immutable images instead of pushing to every host.

Glossary

Control node
The machine where Ansible is installed and from which playbooks run.
Managed node
A server Ansible configures, reached over SSH (or WinRM for Windows).
Inventory
The list of managed hosts and their groups and variables, static or dynamic.
Playbook
A YAML file of plays mapping host groups to ordered tasks.
Module
A unit of work Ansible runs on a host, such as apt, template or service.
Handler
A task that runs only when notified by a changed task, usually a restart or reload.
Role
A reusable folder structure of tasks, templates, handlers and defaults for one purpose.
Collection
A versioned package of modules, roles and plugins under a namespace, from Galaxy or a hub.
Ansible Vault
Built-in encryption for secret files and values stored alongside playbooks.
Facts
Information Ansible gathers about each host, such as OS, memory and network addresses.

Have a product in mind?Let's build it.

Tell me about your idea, your product or your team. I usually reply within one working day.