aalsuwaidi

Diving into Infrastructure as Code - Part 2 (Ansible)

Posted on Apr 14, 2022
tl;dr: Using Ansible to configure infrastructure

Our Goals

Picking up where we left off last time, we now have a newly provisioned DigitalOcean Droplet that we can use. The only problem is that other than the preloaded SSH Key, the droplet is not configured. This is where Ansible comes in, it is a tool developed by Redhat for configuration management. Referring back to my original set of goals, what we want to accomplish as a start is:

  1. Updating and installing packages.
  2. Creating and setting up my user.
  3. Configuring SSH access for that user (from my Desktop, Phone and iPad).
  4. Updating the SSH config to disable root login, and password authentication.
  5. Updating the hostname of the machine and configuring other small things.
  6. Installing and setting up Tailscale.

This would be the baseline configuration that I would want to have on any new server / system I want to use.

Ansible - The Configuration

Keeping the same approach as the previous post, we will start with the basic Ansible concepts that we will need to get started.

Getting Started

Ansible is included in the majority of OS package managers and can be installed directly from there. However, there are multiple other ways to install it as we can see from the documentation.

We will go through 7 basic concepts:

  1. Inventory
  2. Modules
  3. Playbooks
  4. Become
  5. Roles
  6. Ansible Vault
  7. Ansible Galaxy

Inventory

Like the name suggests, our Ansible Inventory is where we list the hosts that we want to manage with Ansible. An example inventory would be something like this:

192.168.1.6

[chatbot]
192.168.1.1

[webservers]
192.168.1.3 ansible_user=webuser
192.168.1.2 ansible_user=webuser

[appservers]
prod ansible_host=192.168.1.4
dev ansible_host=localhost

[appservers:vars]
ansible_user=myuser

At the core, each line defines an Ansible host along with any host specific variables that we might need. These variables may be overridden later on in one of our playbooks or roles.

We can define a host as an IP/hostname, or we can define an alias along with the ansible_host variable. Hosts can be grouped together, be a part of multiple groups or remained ungrouped. Adding variables at a group level is possible as we can see in the last entry, which avoids repeating the same variable on each host line.

Since we are only managing one host, our inventory file will be the one automatically generated from Terraform:

[bot_servers]
braize ansible_host=192.168.1.1 ansible_user=worldhopper

Inventory files can either be in a default location, or anywhere else on our system and be referenced with the -i flag when running an Ansible command.

Modules - Our Building Blocks

Before we can start building playbooks and executing plays with Ansible, we need to understand one of the building blocks. Modules are what get executed when we run a play.

Ansible comes with a set of built-in modules that can be used directly, but you can also install a collection of modules from Ansible Galaxy which we will go over later.

An example module we will use regularly is the package module that adds / removes packages from a host.

Each block is considered a play/task in Ansible, the name is used in logging and is also used to reference / jump to a task in a playbook.

- name: Install Git
  ansible.builtin.package:
    name: git
    state: present

- name: Remove Git
  ansible.builtin.package:
    name: git
    state: absent

We are calling the Generic Package module in Ansible. We call it with ansible.builtin.package and pass in the parameters that the module expects, in our case the name of the package and the state.

Let’s see how modules can be combined to create an Ansible Playbook.

Playbooks - Creating Plays out of our Modules

Ansible Playbooks is where we would define a set of repeated tasks / plays that we want to run on our hosts. They are a collection of YAML formatted modules that are executed in order.

At minimum, playbooks require two things: 1- The targeted host(s) 2- At least one task to execute

The targeted hosts can be selected using the group names that we defined earlier in our inventory or one of the patterns that Ansible supports.

Sample Playbook - Setting Up the Server

These are the collection of tasks that I have defined to set up my servers. Technically I am using them in the form of an Ansible Role (which we will go into later), but I have reformatted it as a playbook for reference.

- hosts: all
  become: true
  tasks:
    - name: Create Worldhopper User (Debian)
      ansible.builtin.user:
        name: worldhopper
        groups: sudo
        password: 'HASHED'
      when: ansible_distribution == "Debian" or ansible_distribution == "Ubuntu"

    - name: Create Worldhopper User (Centos)
      ansible.builtin.user:
        name: worldhopper
        groups: wheel
        password: 'HASHED'
      when: ansible_distribution == "CentOS" or ansible_distribution == "Red Hat Enterprise Linux"

    - name: Add SSH Key to user
      ansible.posix.authorized_key:
        user: worldhopper
        state: present
        key: '{{ item }}'
      with_file:
        - public_keys/roshar
        - public_keys/sel
        - public_keys/ashyn

    - name: Disable Password Login on SSH
      ansible.builtin.lineinfile:
        dest: /etc/ssh/sshd_config
        regexp: "^PasswordAuthentication"
        line: "PasswordAuthentication no"
        state: present
      notify:
      - Restart SSH

    - name: Disable root Login on SSH
      ansible.builtin.lineinfile:
        dest: /etc/ssh/sshd_config
        regexp: "^PermitRootLogin"
        line: "PermitRootLogin no"
        state: present
      notify:
      - Restart SSH

    - name: Update hostname
      ansible.builtin.hostname:
        name: '{{ cosmere_name }}'

    - name: Setup Tailscale
      include_role: 
        name: artis3n.tailscale
      vars: 
        tailscale_authkey: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          123456712345671234567123456712345671234567123456712345671234567123456712345671234567
          123456712345671234567123456712345671234567123456712345671234567123456712345671234567
          123456712345671234567123456712345671234567123456712345671234567123456712345671234567
          1234567
  handlers:
    - name: Restart SSH
      become: true
      ansible.builtin.service:
        name: sshd
        state: restarted

Dissecting this from top to bottom, we start the playbook by creating a user, here we are using a conditional with the when statement. The reason we do this is to add the user to the relevant group, either sudo or wheel depending on the Linux distribution. The ansible_distribution fact is checked, and the task is executed / skipped.

Afterwards, we add our SSH public keys to the server so that we can login with a certificate instead of a password based login. In this task we are using loops. Ansible has a with_file statement that will loop over the contents of each of the files and use it in the module where the {{item}} is used.

The next two modules update our SSH configuration, one to disable password based login and the other to prevent root login. The module checks for a single line in the file and replaces the existing one if it does not match our supplied value. Both of these modules notify a handler once they are changed to restart the SSH Daemon.

Finally, we update the hostname and setup Tailscale using the artis3n.tailscale role that we will install from Ansible Galaxy. Both of which we will go over in this post.

Become - Elevated Privileges

Some modules will require elevated privileges on the remote system, Ansible provides a way to execute certain tasks or entire playbooks as an elevated user. This is done through the become keyword which will use one of the built-in systems in the background (sudo, su, runas, etc.). The Ansible documentation goes into additional details on the become keyword.

Roles - Assigning Roles

The simplest way to describe Roles in Ansible is as an automatic way to group together a set of variables, plays, handlers, files, etc. encapsulating that to be reused anywhere else.

A role can be defined by creating a specific file structure that Ansible will expect. Here is an example adapted from the documentation:

Role Structure

# playbooks
site.yml
webservers.yml
fooservers.yml
roles/
    common/
        tasks/
        handlers/
        files/
        vars/
        defaults/
    webservers/
        tasks/
        defaults/

In the above, two roles are defined: common and webservers, and each of the subfolders (except for the files folder) would have a main.yml that would be loaded for that role.

A quick overview of what each folder will do:

  • tasks/main.yml - would have the plays that would be executed for this role
  • handlers/main.yml - would have the handlers that are associated with the role.
  • files - would have the files that are used in the role.
  • vars/main.yml would have the variables defined for this role.
  • defaults/main.yml would have the default values for this role.

A role does not necessarily need to have all the subfolders. For more details on this structure, you can visit the documentation.

Using Roles inside Playbooks

Once we create a role, think of using it as the process of assigning the host a specific role. For example, if we have a role called webserver, it would define a set of tasks that will perform all the necessary setups to run a web server from that host.

We have already seen one way to use a role inside a playbook with our Tailscale setup task, this was through the include_role statement. Another way would be similar to the example below where it is defined at a playbook level.

- hosts: all
  roles:
    - role: ../../../../ansible/roles/cosmere_server
      vars:
        cosmere_name: braize
        ansible_user: root
    - role: ../../../../ansible/roles/git
    - role: ../../../../ansible/roles/podman

We are including the roles (shown in the next section) prior to any task being executed from the playbook. As we can see, you are able to pass variables at this stage, and they would override the ones that are set in defaults/main.yml.

Sample Role - My Base Server Setup

Now to a more practical example of how roles can be used, I have included the current roles I have set up for my projects, for example I have a role that would install Podman or one to install git.

worldhopper@roshar ~/P/c/ansible (main)> tree roles
roles
├── caddy
│   ├── defaults
│   │   └── main.yml
│   ├── files
│   │   ├── caddy-api.service
│   │   ├── Caddyfile
│   │   └── caddy.service
│   ├── tasks
│   │   └── main.yml
│   └── vars
│       └── main.yml
├── cosmere_server
│   ├── defaults
│   │   └── main.yml
│   ├── files
│   │   └── public_keys
│   │       ├── ashyn
│   │       ├── roshar
│   │       └── sel
│   ├── handlers
│   │   └── main.yml
│   ├── tasks
│   │   └── main.yml
│   └── vars
│       └── main.yml
├── git
│   └── tasks
│       └── main.yml
└── podman
    └── tasks
        └── main.yml

16 directories, 15 files

The cosmere_server role is simply the same setup that was defined as a playbook above, but the handlers were split off to the handlers subfolder, the public keys into the files folder, and finally the tasks in their folder as well.

Once roles are defined, they can be reused in multiple playbooks. In our case, whenever we spin up new infrastructure with Terraform, our first Ansible playbook will include the cosmere_server role before running any other tasks. The only variable that we will update is the cosmere_name which will set up the hostname for that server.

Ansible Vault

Ansible vault provides us with the ability to encrypt variables and files so that they are not hardcoded as plaintext in our playbooks. There are many ways to use Ansible Vault and they are detailed in the documentation page. For us, we will examine two ways:

  • Encrypting Strings (Keys, Passwords, etc.)
  • Encrypting Files

Encrypting Strings - Tailscale setup task

We have used Ansible Vault in the Tailscale setup task, the value for the tailscale_authkey is decrypted once the playbook is executed.

- name: Setup Tailscale
  include_role: 
    name: artis3n.tailscale
  vars: 
    tailscale_authkey: !vault |
      $ANSIBLE_VAULT;1.1;AES256
      123456712345671234567123456712345671234567123456712345671234567123456712345671234567
      123456712345671234567123456712345671234567123456712345671234567123456712345671234567
      123456712345671234567123456712345671234567123456712345671234567123456712345671234567
      1234567

In order to encrypt our string, we simply execute the ansible-vault encrypt_string command and supply our vault password.

worldhopper@roshar ~/P/c/a/r/p/tasks (main)> ansible-vault encrypt_string 'SuperSecretPassword123' --name 'tailscale_authkey'
New Vault password: 
Confirm New Vault password: 
tailscale_authkey: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          38616665643838653537333462636138373463316634383135336639303165346532373933353063
          3666356661623766643663626161346338316335376461340a393332336162343666303231653963
          37356331616537616534303965396536396164333338346239616135616634316330653165633965
          6638356133346231300a323430316434613461666137636534366436653131623931333734333131
          33366332306664343933643532336633346162663961373833626437653730633030
Encryption successful

This value can then be used directly in the playbook and Ansible will decrypt it using a user-supplied vault password when the playbook is executed.

Encrypting Files - Chat Bot Configuration File

For my chatbot, I have to encrypt a configuration file that is not committed into version control, it contains some API keys and other sensitive information.

It is straightforward to use Ansible Vault on a file, we simply have to execute

ansible-vault encrypt filename

An example of this can be seen below:

worldhopper@roshar ~/Desktop> echo "This is a test config file" >> config
worldhopper@roshar ~/Desktop> cat config
This is a test config file
worldhopper@roshar ~/Desktop> ansible-vault encrypt config
New Vault password: 
Confirm New Vault password: 
Encryption successful
worldhopper@roshar ~/Desktop> cat config
$ANSIBLE_VAULT;1.1;AES256
63663133323532306664323761343364353063353036386239373033353134306536653566666134
6633353031376531313565303566653539343339346135360a306565333133343132653439386230
65663366373238373935643335373961646263656165393139613534396563633239623938336266
6663373762393631620a393565306332303735363834316331643633396639393833373330626335
32333765316232323039623665643237333432343637323131303766613164366235
worldhopper@roshar ~/Desktop> ansible-vault view config
Vault password: 
This is a test config file
worldhopper@roshar ~/Desktop>

The file is encrypted in-place, and can be used directly in our playbooks as if we were using a plaintext file. Ansible will detect that it is encrypted and decrypt it at runtime with the user supplied vault password.

Ansible Galaxy - Community Collections and Roles

Ansible Galaxy is a great resource when using Ansible. It is essentially a repository of community maintained collections and roles that can be installed and reused.

We can browse Ansible Galaxy to see if there are any collections / roles that we can use in our project instead of building it out ourselves. For example, in order to install and setup Tailscale we relied on a role from Ansible Galaxy. The command to install this role was:

ansible-galaxy install artis3n.tailscale

After we install a collection / role from Ansible Galaxy we can reuse it in one of our playbooks or roles by following the package specific instructions.

Putting it All Together - Configuring and Running the Bot on a new Server

Now that we have covered the basic Ansible building blocks, we can go through an example setup that I have been using successfully for the past few weeks to deploy and update my chatbot.

Initially, once the server is set up, the following playbook is executed:

Deploy Keeper Playbook

# deploy_keeper.yml
- hosts: all
  roles:
    - role: ../../../../ansible/roles/cosmere_server
      vars:
        cosmere_name: braize
        ansible_user: root
    - role: ../../../../ansible/roles/git
    - role: ../../../../ansible/roles/podman
- hosts: all
  roles:
    - keeper_code
    - keeper_podman

As we can see, initially we execute the cosmere_server role along with git and podman. Passing along the hostname we want for the server, in our case it is braize.

The reason we have split our playbook into two distinct plays is that initially we haven’t configured our user, so only the root user is present. Once our cosmere_server role executes we would have created the worldhopper that will be used for both the keeper_code and keeper_podman roles.

By separating the plays, we have made Ansible reconnect with the remote server and this time it would be with the worldhopper user as it is set in the default values for those roles.

Here is a breakdown of what those roles do:

  • keeper_code - Pulls the latest code for the bot and setups up the config files.
  • keeper_podman - Sets up the Podman configuration for the bot if it doesn’t exist and updates to the latest images (if present).

Whenever the deploy_keeper playbook is executed it will pull the latest version of the code, rebuild the containers and service files (if required)

  1. Pull the latest version of the code.
  2. Create the podman network and volumes (if required).
  3. Rebuild the containers (if required)
  4. Create the service files (if not created before)
  5. Ensure that the service is enabled and running
  6. Run Podman Auto Update which will ensure that the latest images are running.

To execute the playbook we would call it with the following command:

ansible-playbook -i inventory ./deploy_keeper.yml --ask-become-pass --ask-vault-pass
  • –ask-become-pass - Asks the user to supply the password used when elevating privilege.
  • –ask-vault-pass - Asks the user to supply the password used to decrypt the vault.

I have included the roles below for reference, but at this stage it could be anything that we want to set up for that server.

Keeper Code Role

Directory Structure

worldhopper@roshar ~/P/c/p/k/a/roles (main)> tree keeper_code/
keeper_code/
├── defaults
│   └── main.yml
├── files
│   └── config.js
├── tasks
│   └── main.yml
└── vars
    └── main.yml

4 directories, 4 files

Tasks

## tasks/main.yml
- name: Clone / Pull Keeper Code
  ansible.builtin.git:
    repo: "[email protected]:aalsuwaidi/keeper.git"
    dest: "~/keeper"
    version: '{{ branch_name }}'
    accept_hostkey: yes
- name: Copy Config File
  copy:
    src: config.js
    dest: ~/keeper/wabot/config.js

Keeper Podman Role

Directory Structure

worldhopper@roshar ~/P/c/p/k/a/roles (main)> tree keeper_podman
keeper_podman
├── tasks
│   └── main.yml
└── vars
    └── main.yml

2 directories, 2 files

Tasks

## tasks/main.yml
- name: Ensure Keeper Network is created
  containers.podman.podman_network:
    name: keeper_network

- name: Create Chrome Profile Volume
  containers.podman.podman_volume:
    name: chrome_profile
    state: present
    
- name: Create Ftbl Volume
  containers.podman.podman_volume:
    name: ftbl
    state: present

- name: Create Ftbl_DLD Volume
  containers.podman.podman_volume:
    name: ftbl_dld
    state: present

- name: Copy ftbl.db
  copy:
    remote_src: yes
    force: no 
    src: ~/keeper/ftbl/mount/ftbl.db
    dest: ~/.local/share/containers/storage/volumes/ftbl/_data/ftbl.db

- name: Build Srv Image
  containers.podman.podman_image:
    name: keeper_srv
    path: ~/keeper/srv
    state: present
    force: yes

- name: Build Ftbl Image
  containers.podman.podman_image:
    name: keeper_ftbl
    path: ~/keeper/ftbl
    state: present
    force: yes

- name: Build Ftbl_DLD Image
  containers.podman.podman_image:
    name: keeper_ftbl_dld
    path: ~/keeper/ftbl_dld
    state: present
    force: yes

- name: Build wabot Image
  containers.podman.podman_image:
    name: keeper_wabot
    path: ~/keeper/wabot
    state: present
    force: yes

- name: Check SRV Service File
  ansible.builtin.stat:
    path: ~/.config/systemd/user/container-keeper_srv.service
  register: keeper_srv

- name: Check FTBL_DLD Service File
  ansible.builtin.stat:
    path: ~/.config/systemd/user/container-keeper_ftbl_dld.service
  register: keeper_ftbl_dld

- name: Check FTBL Service File
  ansible.builtin.stat:
    path: ~/.config/systemd/user/container-keeper_ftbl.service
  register: keeper_ftbl
- name: Check NOVNC Service File
  ansible.builtin.stat:
    path: ~/.config/systemd/user/container-keeper_novnc.service
  register: keeper_novnc

- name: Check Wabot Service File
  ansible.builtin.stat:
    path: ~/.config/systemd/user/container-keeper_wabot.service
  register: keeper_wabot

- name: Create noVNC container
  containers.podman.podman_container:
    name: keeper_novnc
    image: docker.io/theasp/novnc
    network: keeper_network
    state: present
    ports:
      - 127.0.0.1:8080:8080
    env:
      DISPLAY_WIDTH: "1920"
      DISPLAY_HEIGHT: "1080"
      RUN_XTERM: "no"
    generate_systemd:
      new: yes
      path: /home/worldhopper/.config/systemd/user/
      time: 20
  when: not keeper_novnc.stat.exists

- name: Create SRV Container
  containers.podman.podman_container:
    name: keeper_srv
    image: keeper_srv:latest
    state: present
    network: keeper_network
    labels: {
      "io.containers.autoupdate": "local"
    }
    generate_systemd:
      new: yes
      path: /home/worldhopper/.config/systemd/user/
  when: not keeper_srv.stat.exists

- name: Create Ftbl_DLD Container
  containers.podman.podman_container:
    name: keeper_ftbl_dld
    image: keeper_ftbl_dld:latest
    network: keeper_network
    state: present
    volume:
      - ftbl_dld:/app/goals
    labels: {
      "io.containers.autoupdate": "local"
    }
    generate_systemd:
      new: yes
      path: /home/worldhopper/.config/systemd/user/
  when: not keeper_ftbl_dld.stat.exists

- name: Create Ftbl Container
  containers.podman.podman_container:
    name: keeper_ftbl
    image: keeper_ftbl:latest
    network: keeper_network
    state: present
    volume:
      - ftbl:/app/mount
    labels: {
      "io.containers.autoupdate": "local"
    }
    generate_systemd:
      new: yes
      path: /home/worldhopper/.config/systemd/user/
  when: not keeper_ftbl.stat.exists

- name: Create Wabot Container
  containers.podman.podman_container:
    name: keeper_wabot
    image: keeper_wabot:latest
    network: keeper_network
    state: present
    cmd_args: ["--requires", "keeper_novnc"]
    volume:
      - chrome_profile:/app/profile
    labels: {
      "io.containers.autoupdate": "local"
    }
    generate_systemd:
      new: yes
      path: /home/worldhopper/.config/systemd/user/
  when: not keeper_wabot.stat.exists

- name: Run Keeper
  ansible.builtin.systemd:
    daemon_reload: yes
    enabled: yes
    name: "{{item}}"
    scope: user    
    state:  started
  loop:
    - container-keeper_srv
    - container-keeper_ftbl_dld
    - container-keeper_ftbl
    - container-keeper_novnc
    - container-keeper_wabot

- name: check if worldhoopper lingers
  stat: path=/var/lib/systemd/linger/worldhopper
  register: linger

- name: enable linger for worldhopper
  command: loginctl enable-linger worldhopper
  when: not linger.stat.exists 

- name: Podman Auto-Update
  ansible.builtin.command: podman auto-update

What is next?

Ansible is quite a powerful tool that can be used to effectively manage and configure our infrastructure. What we have covered in this post is just the surface of what is possible.

For the next post, we will look at how we can use both Terraform and Ansible together. What directory structure we can use, how to dynamically build an inventory, what possible automations can be performed.