Ansible, Jinja2 and a nested list

Rocky Linux General
1

Hi,

I’m slowly but steadily adopting Ansible for real work. It’s a slippery fish, but once you get the hang of it, you won’t miss it.

Right, here goes. I’ve written a playbook to manage my (local and public) backup servers using Rocky Linux and Rsnapshot. To write the rsnapshot.conf configuration file, I used a Jinja2 template file. Here’s a snippet from the file:

#######################
# BACKUP TARGET HOSTS #
#######################

{% for backup_target in backup_targets %}
# {{ backup_target }}
{% for backup_dir in backup_dirs %}
backup  root@{{ backup_target }}:{{ backup_dir}}  {{ backup_target }}
{% endfor %}

{% endfor %}

The corresponding variables are in the host_vars section and look like this:

backup_targets:
  - alphamule.microlinux.lan
  - gustave.microlinux.lan
  - proxy.microlinux.lan
backup_dirs:
  - /etc
  - /home

With all this, here’s what the resulting configuration file on the target looks like:

#######################
# BACKUP TARGET HOSTS #
#######################

# alphamule.microlinux.lan
backup  root@alphamule.microlinux.lan:/etc  alphamule.microlinux.lan
backup  root@alphamule.microlinux.lan:/home alphamule.microlinux.lan

# gustave.microlinux.lan
backup  root@gustave.microlinux.lan:/etc    gustave.microlinux.lan
backup  root@gustave.microlinux.lan:/home   gustave.microlinux.lan

# proxy.microlinux.lan
backup  root@proxy.microlinux.lan:/etc  proxy.microlinux.lan
backup  root@proxy.microlinux.lan:/home proxy.microlinux.lan

Now here comes the tricky part. For now I’m assuming that Rsnapshot does backups of the same directory trees on every host, e. g. /etc and /home in the example above.

I’m now encountering a situation where I have different backup_dirs for different backup_targets. Something like this:

- alphamule.microlinux.lan:
    - /etc
    - /home
- gustave.microlinux.lan:
    - /home
- proxy.microlinux.lan:
    - /etc
    - /var
    - /srv

How can I make Jinja2 loop over this list of lists? It looks like I’m missing some IQ points to translate this into correct Jinja2 syntax. So I thought I’d ask the usual suspects among the gurus in this forum for a little help.

Cheers,

Niki

2

Make it a list of dictionaries first and then loop over it.

backup_targets:
  - hostname1:
    - /etc
    - /home
  - hostname2:
    - /etc

## example template
{% for item in backup_targets %}
{% for data in item|dict2items %}
target: {{ data.key }}
# join the list
dirs: {{ data.value|join(', ') }}
# or alternatively, loop the list
{% for dir in data.value %}
{{ dir }}
{% endfor %}
{% endfor %}
{% endfor %}

## generates the below
target: hostname1
# join the list
dirs: /etc, /home
# or alternatively, loop the list
/etc
/home
target: hostname2
# join the list
dirs: /etc
# or alternatively, loop the list
/etc

## another example template
{% for item in backup_targets %}
{% for k,v in item.items() %}
{{ k }} {{ v|join(', ') }}
{% endfor %}
{% endfor %}

## generates the below
hostname1 /etc, /home
hostname2 /etc
1 Like
3

The backup_targets is a list of dictionaries. Took me a while to figure that out. An alternative is to have bit different dictionaries in the list:

  vars:
  - backup_targets:
    - alphamule.microlinux.lan:
      - /etc
      - /home
    - gustave.microlinux.lan:
      - /home

  - other:
    - host: alphamule.microlinux.lan
      path: /etc
    - host: alphamule.microlinux.lan
      path: /home
    - host: gustave.microlinux.lan
      path: /home

{% for backup_target in backup_targets %}
{% for key, value in backup_target.items() %}
{% for dir in value %}
# {{ key }}:{{ dir }} {{ dir }}
{% endfor %}
{% endfor %}
{% endfor %}


{% for backup_target in other %}
# {{ backup_target.host }}:{{ backup_target.path }} {{ backup_target.path }}
{% endfor %}
4

Thank you very much for your detailed reply. I’m experimenting with it and trying to adapt it to my play, and I’m slowly getting somewhere.

I have a little follow-up question to it. My initial variable definition looked like this:

backup_targets:
  - alphamule.microlinux.lan
  - gustave.microlinux.lan
  - proxy.microlinux.lan

Now the simple backup_targets list was used in other parts of the playbook, here for example:

- name: Gather SSH public keys from backup target hosts
  ansible.builtin.known_hosts:
    name: "{{ item }}"
    state: present
    key: "{{ lookup('pipe', 'ssh-keyscan -t rsa {{ item }}') }}"
  with_items:
    - "{{ backup_targets }}"

The new variable definition looks like this:

backup_targets:
  - sandbox.microlinux.lan:
    - /etc
    - /home
  - squidbox.microlinux.lan:
    - /etc/
    - /var
  - sd-155842.dedibox.fr:
    - /etc
    - /var
    - /root
    - /srv

Now this part doesn’t work anymore as it did before:

  with_items:
    - "{{ backup_targets }}"

Question: how can I rewrite this so I can “extract” (sort of) the simple list of my backup target hosts and the play works again?

5

One option is to use dict2items and a bit of creativity. Another option is to use .keys()|first.

- name: Gather SSH public keys from backup target hosts
  ansible.builtin.known_hosts:
    name: "{{ (item|dict2items).0.key }}"
    state: present
    key: "{{ lookup('pipe', 'ssh-keyscan -t rsa {{ (item|dict2items).0.key }}') }}"
  with_items:
    - "{{ backup_targets }}"

- name: Gather SSH public keys from backup target hosts
  ansible.builtin.known_hosts:
    name: "{{ item.keys()|first }}"
    state: present
    key: "{{ lookup('pipe', 'ssh-keyscan -t rsa {{ item.keys()|first }}') }}"
  with_items:
    - "{{ backup_targets }}"

Debug example of the second option.

backup_targets:
  - hostname1:
    - /etc
    - /home
  - hostname2:
    - /etc

## example task
- name: "items"
  ansible.builtin.debug:
    msg: "{{ item.keys()|first }}"
  loop: "{{ backup_targets }}"

## produces...
ok: [localhost] => (item={'hostname1': ['/etc', '/home']}) => {
    "msg": "hostname1"
}
ok: [localhost] => (item={'hostname2': ['/etc']}) => {
    "msg": "hostname2"
}
1 Like
6

Here’s to everybody in this thread (with a special mention to Louis): a big warm thank you for your detailed answers. I did some experimenting using your various suggestions, and now my configuration works perfectly as expected.

You guys are the greatest.

On a side note, I’d really like to get a firm handle on these things. I wonder if this is the time to (finally) start to learn Python. I’m hesitating since I don’t plan on becoming a developer. Would this be overkill, a bit like studying food chemistry when all you want to do is cook a few good dishes with a selection of good recipes? Or is it worth it?

Cheers,

Niki

7

Ah, that does allow:

{% for backup_target in backup_targets %}
{% for dir in backup_target.values()|first %}
# {{ backup_target.keys()|first }}:{{ dir }}
{% endfor %}
{% endfor %}

That was what I was stumbling with too.

backup_targets:
  - alphamule
  - gustave

is a list of values, while

backup_targets:
  - sandbox: something
  - squidbox: other

is a list of dictionaries, just like the

backup_targets:
  - host: sandbox
    value: something
  - host: squidbox
    value: other

I don’t know much of Python. Yes, it would probably save time to know some elementary things about it, like that dict objects have keys() and values() … and to know a dict is a dict when one sees it.