Ansible for dotfiles: the introduction I wish I've had

A dotfiles repo will help you manage configuration over time and synchronize them across machines, but it won’t help you with your dependencies — you still have to install git, tmux, vim, etc. Let’s see how Ansible can help with that.

Ansible would often come up as a solution to dependency management, but I’d quickly dismiss it because it looked too complex and enterprise-y, an impression that the landing page certainly contributed to. In my opinion, It doesn’t help convey the idea of what Ansible turned out to be for me: a tool to automate tasks in a declarative way.

I wish I was introduced to Ansible differently, so this post is an attempt to write the introduction I wish I’ve had. We’ll deliberately abstract away the more complex applications of Ansible and focus on solving one problem: how to set up a machine with your preferred programs, appearances, settings etc.

Installing Ansible

First things first, let’s install Ansible. The official docs has an extensive list with installation instructions for many Unix-like operating systems.

Another option, available at every operating system, is to use pip, the package manager for Python, which was my choice:

$ pip install ansible

How Ansible clicked for me

If I remember correctly, it was by looking at the source code of a dotfiles repo that used it. The README.md explained that you needed to run a one single command to set up everything:

$ ansible-playbook --ask-become-pass bootstrap.yml

So I went and looked at what was going on at the bootstrap.yml file:

- name: Bootstrap development environment  hosts: localhost   tasks:  - name: Install packages with apt    become: yes    ansible.builtin.apt:      name:        - git        - tmux      state: present

Things immediately clicked for me once I saw it, like, it’s clear that this is using apt to download packages!

Of course, at this level, how is that much better than directly running apt install git tmux in a shell script called, say, bootstrap.sh? In this particular example, it’s not much better but it’s better already, as we’ll see. We’re just getting started.

There’s still a lot going on that we have to understand, e.g. what is become, and hosts? But I think the terminology already shows what is Ansible’s primary goal: to leave a machine in a desired state by playing a bunch of tasks. In this example, the desired state is to have git and tmux installed (or “present”), but the desired state will likely not be that simple. Also, what if we’re on macOS? Or Fedora? We won’t be able to use apt in this case. We’ll see that Ansible makes it easy to handle this.

Playbook

The file bootstrap.yml is, in Ansible terminology, a playbook, which is, in my own words, a way to declare which tasks should run in a machine in order to achieve the desired state. Playbooks are written in YAML, which is easy to read and learn.

A playbook may contain various plays, and each play may contain various tasks. For example, the bootstrap.yml playbook has a single play, named “Boostrap development environment”, with one task, “Install packages with apt”.

Hosts and inventory

Ansible can execute tasks on remote and local machines. This is managed with the help of an ”inventory”, where these machines (hosts) are classified with patterns like “prod”, “test”, “webserver” etc..

We’re not interested in that here since we just want to run tasks on our local machine. Passing localhost as a playbook’s hosts will do what we want, without any inventory, because Ansible will implicitly define localhost to match the local machine.

- name: Bootstrap development environment  hosts: localhost

Tasks

Now, let’s dissect the “Install packages with apt” task. Here it is again:

  tasks:  - name: Install packages with apt    become: yes    ansible.builtin.apt:      name:        - git        - tmux      state: present

Become

Let’s start with the keyword become. It is used so that you can “become another user while this task is executing”, that user being by default root. So it’s, effectively, the equivalent of using sudo. This is also why you need to run the playbook with the option --ask-become-pass (-K for short), because it’ll be necessary to prompt for your password.

If you need to become another user other than root, use the become_user keyword.

Modules

ansible.builtin.apt is an Ansible module, which is the preferred way to interact with another program in an Ansible playbook.

Ansible has several built-in modules, here’s a full list. It has modules for most Linux distributions package managers, like dnf. But there are also community-developed modules, like one for Homebrew, if you’re on macOS, which we’ll have to download separately, which we’ll learn how shortly.

An Ansible module takes arguments to accomplish a task. In the ansible.builtin.apt case, we pass a list of packages to the name argument and the desired state we want them to be in to the state argument.

  tasks:  - name: Install packages with apt    become: yes    ansible.builtin.apt:      name:        - git        - tmux      state: present # could also be 'latest' or 'absent'

Idempotency

Ansible playbooks are meant to be idempotent, meaning that, no matter how many times they run, the machine will end up in the same desired state.

An Ansible module can tell if something changed between runs. For example, let’s say I already have git and tmux installed and run the bootstrap.yml playbook:

$ ansible-playbook -K bootstrap.ymlBECOME password: PLAY [Bootstrap development environment] ***************************************************************************** TASK [Gathering Facts] ***********************************************************************************************ok: [localhost] TASK [Install packages with apt] ******************************************************************************************ok: [localhost] PLAY RECAP ***********************************************************************************************************localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

The task “Install packages with apt” was labelled “ok”, as in “nothing changed”: git and tmux were already present. If either were absent, the task would have been labelled as “changed”, because now both are present.

This feature is module-dependent. The task could be idempotent by nature, yet Ansible wouldn’t know about it unless the module is smart enough to tell it.

To illustrate this, let’s use the ansible.builtin.shell, which is used to run arbitrary commands into a shell. Let’s run a task using this module in an ad hoc way (outside of a playbook):

$ ansible localhost -m ansible.builtin.shell -a 'echo $SHELL' [WARNING]: No inventory was parsed, only implicit localhost is availablelocalhost | CHANGED | rc=0 >>/bin/bash

Ansible says something changed, but why? Running echo $SHELL cannot change anything in a system (I guess?). But Ansible has no way to know this, since it could be any arbitrary command, such as rm -rf $HOME.

Most of Ansible built-in modules are idempotent and smart enough to tell if something changed between runs. ansible.builtin.shell is kind of a low-level one, so it’s best to avoid it whenever possible. There are ways to make it smarter with parameters such as creates, removes, or, at the task level, changed_when.

Facts and conditionals

Now let’s imagine you use both macOS and Ubuntu. You want to install git and tmux in both operating systems with Ansible. You’ll need to add some sort of conditional logic to your playbook since apt isn’t available in macOS, and you probably don’t want to use Homebrew in Linux, even though you could.

We can run a task conditionally with the when keyword:

  tasks:  - name: Install packages with apt    become: yes    ansible.builtin.apt:      name:        - git        - tmux      state: present    when: ansible_distribution == "Ubuntu"   tasks:  - name: Install packages with brew    become: yes    community.general.homebew:      name:        - git        - tmux      state: present    when: ansible_distribution == "MacOSX"

The task will run if the expression passed to when evaluates to True. This expression is a Jinja2 expression, a template engine written in Python, so it’s not strictly Python syntax but it looks a lot like it.

The ansible_distribution variable is an Ansible fact: information about your system gathered and provided by Ansible, for convenience. There are a LOT of facts, you can run ansible localhost -m ansible.builtin.setup to check what’s available, but here’s the most commonly used ones: ansible_distribution, ansible_distribution_version and ansible_os_family.

Ansible Galaxy

The module community.general.homebrew is not a built-in module, you have to install it separately via Ansible Galaxy, which is a hub for Ansible content.

Ansible Galaxy provides a command-line interface to install stuff, like collections and roles (which we’ll learn about shortly). community.general is a collection of modules, which we can install by running:

$ ansible-galaxy collection install community.general

Loops

You may need a loop to accomplish a task. For example, the following task will build dwm, slock and dmenu from source, by running sudo make install in each directory containing the source code:

- name: Build and install suckless tools  become: true  loop:    - dwm    - slock    - dmenu  community.general.make:    target: install    chdir: "suckless/{{ item }}"    make: /usr/bin/make

Another example is if a module do not accept a list of strings as an argument, just a string. community.general.npm is one of such modules. Here’s an example of how to install some npm modules globally:

- name: Install npm global packages  loop:    - yalc    - npm-merge-driver    - diff-so-fancy  community.general.npm:    name: "{{ item }}"    state: present    global: true

Jinja2

Jinja2 was mentioned briefly in conditionals, but because it’s a huge component of Ansible, it deserves a few more words than that.

We’ve been using Jinja2 throughout this post already, e.g., {{ item }} is a Jinja2 expression to expand the value in the variable item.

It’s possible to manipulate this value with filters, e.g. {{ item | upper }}.

There are a lot of built-in filters. The upper filter is kind of an innocuous one, there are some more powerful, e.g. product to build a cartesian product from lists:

- name: Give users access to multiple databases  community.mysql.mysql_user:    name: "{{ item[0] }}"    priv: "{{ item[1] }}.*:ALL"    append_privs: yes    password: "foo"  loop: "{{ ['alice', 'bob'] | product(['clientdb', 'employeedb', 'providerdb']) | list }}"

Ansible uses Jinja2 tests to express conditional logic. The syntax for tests is slightly different than the filter syntax:

vars:  url: "https://example.com/users/foo/resources/bar" tasks:    - debug:        msg: "matched pattern 1"      when: url is match("https://example.com/users/.*/resources")

The usual comparison operators are available, like ==, !=, <, <=, >, >=, and it’s, of course, also possible to combine or transform multiple boolean expressions with and, or, not etc.

Roles

An Ansible role is a way to organize your tasks logic in a file structure. A role is usually the format in which you use other people’s Ansible tasks. For example, there is an Ansible role to install Visual Studio Code. It’s available in Ansible Galaxy, so we can install it like this:

$ ansible-galaxy role install gantsign.visual-studio-code

A role is usually customizable via parameters. For example, the gantsign.visual-studio-code extension has a parameter that allows you to install extensions:

    - role: gantsign.visual-studio-code      users:        - username: phelipe          visual_studio_code_extensions:            - "eamodio.gitlens"            - "kahole.magit"            - "PhilHindle.errorlens"            - "sleistner.vscode-fileutils"            - "vscodevim.vim"

In practice, creating a role is a matter of organizing things like tasks, default variables, templates, files etc. in a file structure that Ansible can understand.

You start by creating a tasks/main.yml file, which is the role’s entry point, from which all tasks will derive.

Let’s check a role I made to install Neovim, to illustrate this:

$ tree roles/nvimroles/nvim└── tasks    ├── fedora.yml    ├── macos.yml    ├── main.yml    └── ubuntu.yml 1 directory, 4 files

The contents of main.yml simply import tasks from the other files, depending on the host’s operating system:

- name: Build nvim from source in Ubuntu  import_tasks: ubuntu.yml  when: ansible_distribution == "Ubuntu" - name: Build nvim from source in Fedora  include_tasks: fedora.yml  when: ansible_distribution == "Fedora" - name: Install nvim with Homebrew in macOS  import_tasks: macos.yml  when: ansible_distribution == "MacOSX"

Ansible will make it easy for you to use templates and files stored in the role’s templates and files directory. You can provide variables’ default values in the default/main.yml file. The list could go on, but I won’t go any further. Learn more about it in the official docs about roles.

dotfiles use cases

Now let’s look at some examples at how I used Ansible to manage my dotfiles.

Building Neovim from source

Let’s build Neovim from source with Ansible, in Ubuntu, with the help of the official guide.

I already have Neovim source code as a git submodule in deps/neovim, but it’s also possible to add a task to clone its repository with the ansible.builtin.git module.

We’ll need to install build dependencies, run some make commands, and that’s it!

# In case you don't like git submodules- name: Clone nvim repository  git:    repo: https://github.com/neovim/neovim    dest: "{{ ansible_env.HOME }}/src/nvim"    clone: true    version: "v6.0.0" - name: Install nvim build dependencies  become: true  apt:    name:      - ninja-build      - gettext      - libtool      - libtool-bin      - autoconf      - automake      - cmake      - g++      - pkg-config      - unzip      - curl    state: present - name: Build nvim release version  community.general.make:    chdir: deps/neovim    params:      CMAKE_BUILD_TYPE: Release - name: Install nvim release version  become: true  community.general.make:    chdir: deps/neovim    target: install

Installing efm-langserver

efm-langserver is a Language Server to make any LSP client understand the output of an arbitrary linter or formatter.

Let’s install it using Ansible too! It’s a Golang app, so we could use the Go toolchain to install it, but I had too much trouble going down this path, so now I just download the tarball from GitHub, unpack (with ansible.builtin.archive module) it and put the binary into my PATH (in $HOME/.local/bin):

- name: Create directory ~/.local/bin/  file:    path: "{{ ansible_env.HOME }}/.local/bin"    state: directory - name: Install efm-langserver in Linux  unarchive:    src: https://github.com/mattn/efm-langserver/releases/download/v0.0.37/efm-langserver_v0.0.37_linux_amd64.tar.gz    dest: "{{ ansible_env.HOME }}/.local/bin"    remote_src: true    extra_opts:      - "--strip-components=1"      - "efm-langserver_v0.0.37_linux_amd64/efm-langserver"  when: ansible_distribution == "Ubuntu" or ansible_distribution == "Fedora"

It isn’t as easy to do this in macOS though, just because it’s a zip file instead, which lacks the convenience of the extra_opts parameter… But I might be missing something.

Installing linters and formatters

I also install the linters and formatters I use with Ansible. Here’s a snippet of it:

- name: Install linters and formatters with dnf  become: true  dnf:    name:      - ShellCheck # this is shellcheck in apt and brew    state: present  when: ansible_distribution == "Fedora" # ... - name: Install linters and formatters with pip  ansible.builtin.pip:    name:      - flake8      - black      - yamllint      - git+https://github.com/Vimjas/vint@master    state: present    executable: pip3 - name: Install linters and formatters with npm  loop:    - eslint_d    - prettier  community.general.npm:    name: "{{ item }}"    global: true    state: present

If you’re interested, you can look at the file on GitHub.

Stow

Stow is a huge part of dotfiles management: it’s responsible to put the contents of my dofiles into my $HOME directory, as symlinks, while preserving the same folder structure.

I also install and run Stow in my playbook. There isn’t an Ansible module for Stow though, so I have to resort to ansible.builtin.shell, which isn’t great because it’s dumb. But not all is lost, we can make it smarter by tweaking the task parameters. We can determine if something changed between runs by running Stow in verbose mode (--verbose=2), to make it describe everything it did, and then analyze its output.

stow dir is /home/phelipe/dotfilesstow dir path relative to target /home/phelipe is dotfilesPlanning stow of package ....--- Skipping .profile as it already points to dotfiles/.profilePlanning stow of package .... doneProcessing tasks...

By looking at the output, we see that Stow says it skipped a file if there is already a symlink for it in $HOME. But in the case there isn’t, it’ll say it linked that file, like so:

LINK: .profile => dotfiles/.profile

So if the string LINK is in the stderr, it means a symlink was created, something changed:

- name: Run stow  shell: "stow . --target {{ ansible_env.HOME }} --verbose=2"  register: result  changed_when: 'result.stderr is search("LINK: ")'

This is a small thing, but it’s just nice to run a playbook and have everything labelled as “ok”.

GNOME

We can’t customize GNOME with configuration files. We normally configure it with GUIs, which is not a great thing if we’re interested in automating the customization step.

Fortunately, there is a command line interface to make this possible called dconf (there’s also gsettings but we won’t use it). The collection community.general has a module for dconf, so let’s use it.

For now, I just made two tweaks to my GNOME DE:

• Click by tapping on the touchpad.
• Show battery percentage next to the battery icon.

Here are the equivalent tasks:

- name: Enable tap to click on touchpad  community.general.dconf:    key: "/org/gnome/desktop/peripherals/touchpad/tap-to-click"    value: "true" - name: Show battery percentage  community.general.dconf:    key: "/org/gnome/desktop/interface/show-battery-percentage"    value: "true"

But how did I find that I needed this specific key /org/gnome/desktop/peripherals/touchpad/tap-to-click? That’s the hard part, but it’s easier if we use dconf-editor, a GUI to explore GNOME applications’ internal settings. Navigate through it until you find the desired setting that seems responsible for what you want, grab its path and pass the value you want to it.

Conclusion

You should now be able to automate downloading your favorite programs with your preferred configuration across operating systems, with tasks/roles written by you or by the community (which is the nicest part).

The next step should be to test if it really works by using a virtual machine — the Vagrant and Virtual Box combo seems to be a popular choice.

It’s not wise to assume that your playbook will work on a fresh machine because you might be relying on a dependency your playbook does not ensure it’s true, even though it is in your current host.