Ansible for dotfiles: the introduction I wish I've had

14 min.

A dotfiles repo will help you manage configuration over time and synchronize them across machines, but it won’t help you with your dependencies – you still have to install git, tmux, vim, etc. Let’s see how Ansible can help with that.

Ansible would often come up as a solution to dependency management, but I’d quickly dismiss it because it looked too complex and enterprise-y, an impression that the landing page certainly contributed to. In my opinion, It doesn’t help convey the idea of what Ansible turned out to be for me: a tool to automate tasks in a declarative way.

I wish I was introduced to Ansible differently, so this post is an attempt to write the introduction I wish I’ve had. We’ll deliberately abstract away the more complex applications of Ansible and focus on solving one problem: how to set up a machine with your preferred programs, appearances, settings etc.

Installing Ansible #

First things first, let’s install Ansible. The official docs has an extensive list with installation instructions for many Unix-like operating systems.

Another option, available at every operating system, is to use pip, the package manager for Python, which was my choice:

$ pip install ansible

How Ansible clicked for me #

If I remember correctly, it was by looking at the source code of a dotfiles repo that used it. The README.md explained that you needed to run a one single command to set up everything:

$ ansible-playbook --ask-become-pass bootstrap.yml

So I went and looked at what was going on at the bootstrap.yml file:

- name: Bootstrap development environment
  hosts: localhost

  tasks:
  - name: Install packages with apt
    become: yes
    ansible.builtin.apt:
      name:
        - git
        - tmux
      state: present

Things immediately clicked for me once I saw it, like, it’s clear that this is using apt to download packages!

Of course, at this level, how is that much better than directly running apt install git tmux in a shell script called, say, bootstrap.sh? In this particular example, it’s not much better but it’s better already, as we’ll see. We’re just getting started.

There’s still a lot going on that we have to understand, e.g. what is become, and hosts? But I think the terminology already shows what is Ansible’s primary goal: to leave a machine in a desired state by playing a bunch of tasks. In this example, the desired state is to have git and tmux installed (or “present”), but the desired state will likely not be that simple. Also, what if we’re on macOS? Or Fedora? We won’t be able to use apt in this case. We’ll see that Ansible makes it easy to handle this.

Playbook #

The file bootstrap.yml is, in Ansible terminology, a playbook, which is, in my own words, a way to declare which tasks should run in a machine in order to achieve the desired state. Playbooks are written in YAML, which is easy to read and learn.

A playbook may contain various plays, and each play may contain various tasks. For example, the bootstrap.yml playbook has a single play, named “Boostrap development environment”, with one task, “Install packages with apt”.

Hosts and inventory #

Ansible can execute tasks on remote and local machines. This is managed with the help of an “ inventory”, where these machines (hosts) are classified with patterns like “prod”, “test”, “webserver” etc..

We’re not interested in that here since we just want to run tasks on our local machine. Passing localhost as a playbook’s hosts will do what we want, without any inventory, because Ansible will implicitly define localhost to match the local machine.

- name: Bootstrap development environment
  hosts: localhost

Tasks #

Now, let’s dissect the “Install packages with apt” task. Here it is again:

  tasks:
  - name: Install packages with apt
    become: yes
    ansible.builtin.apt:
      name:
        - git
        - tmux
      state: present

Become #

Let’s start with the keyword become. It is used so that you can “become another user while this task is executing”, that user being by default root. So it’s, effectively, the equivalent of using sudo. This is also why you need to run the playbook with the option --ask-become-pass (-K for short), because it’ll be necessary to prompt for your password.

If you need to become another user other than root, use the become_user keyword.

Modules #

ansible.builtin.apt is an Ansible module, which is the preferred way to interact with another program in an Ansible playbook.

Ansible has several built-in modules, here’s a full list. It has modules for most Linux distributions package managers, like dnf. But there are also community-developed modules, like one for Homebrew, if you’re on macOS, which we’ll have to download separately, which we’ll learn how shortly.

An Ansible module takes arguments to accomplish a task. In the ansible.builtin.apt case, we pass a list of packages to the name argument and the desired state we want them to be in to the state argument.

  tasks:
  - name: Install packages with apt
    become: yes
    ansible.builtin.apt:
      name:
        - git
        - tmux
      state: present # could also be 'latest' or 'absent'

Idempotency #

Ansible playbooks are meant to be idempotent, meaning that, no matter how many times they run, the machine will end up in the same desired state.

An Ansible module can tell if something changed between runs. For example, let’s say I already have git and tmux installed and run the bootstrap.yml playbook:

$ ansible-playbook -K bootstrap.yml
BECOME password:

PLAY [Bootstrap development environment] *****************************************************************************

TASK [Gathering Facts] ***********************************************************************************************
ok: [localhost]

TASK [Install packages with apt] ******************************************************************************************
ok: [localhost]

PLAY RECAP ***********************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

The task “Install packages with apt” was labelled “ok”, as in “nothing changed”: git and tmux were already present. If either were absent, the task would have been labelled as “changed”, because now both are present.

This feature is module-dependent. The task could be idempotent by nature, yet Ansible wouldn’t know about it unless the module is smart enough to tell it.

To illustrate this, let’s use the ansible.builtin.shell, which is used to run arbitrary commands into a shell. Let’s run a task using this module in an ad hoc way (outside of a playbook):

$ ansible localhost -m ansible.builtin.shell -a 'echo $SHELL'

[WARNING]: No inventory was parsed, only implicit localhost is available
localhost | CHANGED | rc=0 >>
/bin/bash

Ansible says something changed, but why? Running echo $SHELL cannot change anything in a system (I guess?). But Ansible has no way to know this, since it could be any arbitrary command, such as rm -rf $HOME.

Most of Ansible built-in modules are idempotent and smart enough to tell if something changed between runs. ansible.builtin.shell is kind of a low-level one, so it’s best to avoid it whenever possible. There are ways to make it smarter with parameters such as creates, removes, or, at the task level, changed_when.

Facts and conditionals #

Now let’s imagine you use both macOS and Ubuntu. You want to install git and tmux in both operating systems with Ansible. You’ll need to add some sort of conditional logic to your playbook since apt isn’t available in macOS, and you probably don’t want to use Homebrew in Linux, even though you could.

We can run a task conditionally with the when keyword:

  tasks:
  - name: Install packages with apt
    become: yes
    ansible.builtin.apt:
      name:
        - git
        - tmux
      state: present
    when: ansible_distribution == "Ubuntu"

  tasks:
  - name: Install packages with brew
    become: yes
    community.general.homebew:
      name:
        - git
        - tmux
      state: present
    when: ansible_distribution == "MacOSX"

The task will run if the expression passed to when evaluates to True. This expression is a Jinja2 expression, a template engine written in Python, so it’s not strictly Python syntax but it looks a lot like it.

The ansible_distribution variable is an Ansible fact: information about your system gathered and provided by Ansible, for convenience. There are a LOT of facts, you can run ansible localhost -m ansible.builtin.setup to check what’s available, but here’s the most commonly used ones: ansible_distribution, ansible_distribution_version and ansible_os_family.

Ansible Galaxy #

The module community.general.homebrew is not a built-in module, you have to install it separately via Ansible Galaxy, which is a hub for Ansible content.

Ansible Galaxy provides a command-line interface to install stuff, like collections and roles (which we’ll learn about shortly). community.general is a collection of modules, which we can install by running:

$ ansible-galaxy collection install community.general

Loops #

You may need a loop to accomplish a task. For example, the following task will build dwm, slock and dmenu from source, by running sudo make install in each directory containing the source code:

- name: Build and install suckless tools
  become: true
  loop:
    - dwm
    - slock
    - dmenu
  community.general.make:
    target: install
    chdir: "suckless/{{ item }}"
    make: /usr/bin/make

Another example is if a module do not accept a list of strings as an argument, just a string. community.general.npm is one of such modules. Here’s an example of how to install some npm modules globally:

- name: Install npm global packages
  loop:
    - yalc
    - npm-merge-driver
    - diff-so-fancy
  community.general.npm:
    name: "{{ item }}"
    state: present
    global: true

Jinja2 #

Jinja2 was mentioned briefly in conditionals, but because it’s a huge component of Ansible, it deserves a few more words than that.

We’ve been using Jinja2 throughout this post already, e.g., {{ item }} is a Jinja2 expression to expand the value in the variable item.

It’s possible to manipulate this value with filters, e.g. {{ item | upper }}.

There are a lot of built-in filters. The upper filter is kind of an innocuous one, there are some more powerful, e.g. product to build a cartesian product from lists:

- name: Give users access to multiple databases
  community.mysql.mysql_user:
    name: "{{ item[0] }}"
    priv: "{{ item[1] }}.*:ALL"
    append_privs: yes
    password: "foo"
  loop: "{{ ['alice', 'bob'] | product(['clientdb', 'employeedb', 'providerdb']) | list }}"

Ansible uses Jinja2 tests to express conditional logic. The syntax for tests is slightly different than the filter syntax:

vars:
  url: "https://example.com/users/foo/resources/bar"

tasks:
    - debug:
        msg: "matched pattern 1"
      when: url is match("https://example.com/users/.*/resources")

The usual comparison operators are available, like ==, !=, <, <=, >, >=, and it’s, of course, also possible to combine or transform multiple boolean expressions with and, or, not etc.

Roles #

An Ansible role is a way to organize your tasks logic in a file structure. A role is usually the format in which you use other people’s Ansible tasks. For example, there is an Ansible role to install Visual Studio Code. It’s available in Ansible Galaxy, so we can install it like this:

$ ansible-galaxy role install gantsign.visual-studio-code

A role is usually customizable via parameters. For example, the gantsign.visual-studio-code extension has a parameter that allows you to install extensions:

    - role: gantsign.visual-studio-code
      users:
        - username: phelipe
          visual_studio_code_extensions:
            - "eamodio.gitlens"
            - "kahole.magit"
            - "PhilHindle.errorlens"
            - "sleistner.vscode-fileutils"
            - "vscodevim.vim"

In practice, creating a role is a matter of organizing things like tasks, default variables, templates, files etc. in a file structure that Ansible can understand.

You start by creating a tasks/main.yml file, which is the role’s entry point, from which all tasks will derive.

Let’s check a role I made to install Neovim, to illustrate this:

$ tree roles/nvim
roles/nvim
└── tasks
    ├── fedora.yml
    ├── macos.yml
    ├── main.yml
    └── ubuntu.yml

1 directory, 4 files

The contents of main.yml simply import tasks from the other files, depending on the host’s operating system:

- name: Build nvim from source in Ubuntu
  import_tasks: ubuntu.yml
  when: ansible_distribution == "Ubuntu"

- name: Build nvim from source in Fedora
  include_tasks: fedora.yml
  when: ansible_distribution == "Fedora"

- name: Install nvim with Homebrew in macOS
  import_tasks: macos.yml
  when: ansible_distribution == "MacOSX"

Ansible will make it easy for you to use templates and files stored in the role’s templates and files directory. You can provide variables’ default values in the default/main.yml file. The list could go on, but I won’t go any further. Learn more about it in the official docs about roles.

dotfiles use cases #

Now let’s look at some examples at how I used Ansible to manage my dotfiles.

Building Neovim from source #

Let’s build Neovim from source with Ansible, in Ubuntu, with the help of the official guide.

I say Ubuntu because Fedora and Homebrew usually have an up-to-date Neovim stable, so I can’t be bothered.

I already have Neovim source code as a git submodule in deps/neovim, but it’s also possible to add a task to clone its repository with the ansible.builtin.git module.

We’ll need to install build dependencies, run some make commands, and that’s it!

# In case you don't like git submodules
- name: Clone nvim repository
  git:
    repo: https://github.com/neovim/neovim
    dest: "{{ ansible_env.HOME }}/src/nvim"
    clone: true
    version: "v6.0.0"

- name: Install nvim build dependencies
  become: true
  apt:
    name:
      - ninja-build
      - gettext
      - libtool
      - libtool-bin
      - autoconf
      - automake
      - cmake
      - g++
      - pkg-config
      - unzip
      - curl
    state: present

- name: Build nvim release version
  community.general.make:
    chdir: deps/neovim
    params:
      CMAKE_BUILD_TYPE: Release

- name: Install nvim release version
  become: true
  community.general.make:
    chdir: deps/neovim
    target: install

Installing efm-langserver #

efm-langserver is a Language Server to make any LSP client understand the output of an arbitrary linter or formatter.

Let’s install it using Ansible too! It’s a Golang app, so we could use the Go toolchain to install it, but I had too much trouble going down this path, so now I just download the tarball from GitHub, unpack (with ansible.builtin.archive module) it and put the binary into my PATH (in $HOME/.local/bin):

- name: Create directory ~/.local/bin/
  file:
    path: "{{ ansible_env.HOME }}/.local/bin"
    state: directory

- name: Install efm-langserver in Linux
  unarchive:
    src: https://github.com/mattn/efm-langserver/releases/download/v0.0.37/efm-langserver_v0.0.37_linux_amd64.tar.gz
    dest: "{{ ansible_env.HOME }}/.local/bin"
    remote_src: true
    extra_opts:
      - "--strip-components=1"
      - "efm-langserver_v0.0.37_linux_amd64/efm-langserver"
  when: ansible_distribution == "Ubuntu" or ansible_distribution == "Fedora"

It isn’t as easy to do this in macOS though, just because it’s a zip file instead, which lacks the convenience of the extra_opts parameter… But I might be missing something.

Installing linters and formatters #

I also install the linters and formatters I use with Ansible. Here’s a snippet of it:

- name: Install linters and formatters with dnf
  become: true
  dnf:
    name:
      - ShellCheck # this is shellcheck in apt and brew
    state: present
  when: ansible_distribution == "Fedora"

# ...

- name: Install linters and formatters with pip
  ansible.builtin.pip:
    name:
      - flake8
      - black
      - yamllint
      - git+https://github.com/Vimjas/vint@master
    state: present
    executable: pip3

- name: Install linters and formatters with npm
  loop:
    - eslint_d
    - prettier
  community.general.npm:
    name: "{{ item }}"
    global: true
    state: present

If you’re interested, you can look at the file on GitHub.

Stow #

Stow is a huge part of dotfiles management: it’s responsible to put the contents of my dofiles into my $HOME directory, as symlinks, while preserving the same folder structure.

I also install and run Stow in my playbook. There isn’t an Ansible module for Stow though, so I have to resort to ansible.builtin.shell, which isn’t great because it’s dumb. But not all is lost, we can make it smarter by tweaking the task parameters. We can determine if something changed between runs by running Stow in verbose mode (--verbose=2), to make it describe everything it did, and then analyze its output.

stow dir is /home/phelipe/dotfiles
stow dir path relative to target /home/phelipe is dotfiles
Planning stow of package ....
--- Skipping .profile as it already points to dotfiles/.profile
Planning stow of package .... done
Processing tasks...

By looking at the output, we see that Stow says it skipped a file if there is already a symlink for it in $HOME. But in the case there isn’t, it’ll say it linked that file, like so:

LINK: .profile => dotfiles/.profile

So if the string LINK is in the stderr, it means a symlink was created, something changed:

- name: Run stow
  shell: "stow . --target {{ ansible_env.HOME }} --verbose=2"
  register: result
  changed_when: 'result.stderr is search("LINK: ")'

This is a small thing, but it’s just nice to run a playbook and have everything labelled as “ok”.

GNOME #

We can’t customize GNOME with configuration files. We normally configure it with GUIs, which is not a great thing if we’re interested in automating the customization step.

Fortunately, there is a command line interface to make this possible called dconf (there’s also gsettings but we won’t use it). The collection community.general has a module for dconf, so let’s use it.

For now, I just made two tweaks to my GNOME DE:

  • Click by tapping on the touchpad.
  • Show battery percentage next to the battery icon.

Here are the equivalent tasks:

- name: Enable tap to click on touchpad
  community.general.dconf:
    key: "/org/gnome/desktop/peripherals/touchpad/tap-to-click"
    value: "true"

- name: Show battery percentage
  community.general.dconf:
    key: "/org/gnome/desktop/interface/show-battery-percentage"
    value: "true"

But how did I find that I needed this specific key /org/gnome/desktop/peripherals/touchpad/tap-to-click? That’s the hard part, but it’s easier if we use dconf-editor, a GUI to explore GNOME applications’ internal settings. Navigate through it until you find the desired setting that seems responsible for what you want, grab its path and pass the value you want to it.

Conclusion #

You should now be able to automate downloading your favorite programs with your preferred configuration across operating systems, with tasks/roles written by you or by the community (which is the nicest part).

The next step should be to test if it really works by using a virtual machine – the Vagrant and Virtual Box combo seems to be a popular choice.

It’s not wise to assume that your playbook will work on a fresh machine because you might be relying on a dependency your playbook does not ensure it’s true, even though it is in your current host.