, ,

Hack: Podman volumes and Ansible template idempotency

Anyone mildly familiar with Ansible will attest, maintaining idempotency is a key secret-sauce to stable automation.  Without idempotency, it’s all but impossible to detect drift and/or predictably manage state changes.  Similarly, anyone beyond a complete-beginning Podman user, will know that defining and using volumes are essential operations.

Now for the problem: In Ansible-land, the template module is an important tool for both deployment and ongoing management of remote file contents.  However, if you try to automate podman volume contents management with templates, you’ll quickly realize a major shortcoming:

Ansible gets completely choked up over user-namespaces.

Note: If user-namespaces are not something you’re familiar with yet, I recommend reading Dan’s “Using files and devices in Podman rootless containers“.  It will either enlighten you, or confuse you more.  In the latter case, I recommend my “Rootless Podman user-namespaces in plain English” article before continuing.

For instance, consider the playbook ./local_1.yml below. It’s intended to deploy a web server container, without a mapping of the host user to the root user (userns: nomap).  This lack of user -> root map is intended to protect the host. Should a process escape from the container, it won’t have any access to the host user’s files or processes. Please look past the targeting of localhost, that’s simply for demonstration purposes.  But do take note of the tasks explicitly managing volume and configuration file ownership:

./local_1.yml

---

- hosts: localhost
  gather_facts: true
  gather_subset: user
  become: false
  tasks:
    - name: Config volume for nginx container
      containers.podman.podman_volume:
        name: nginx_confd
        options:
          - o=uid=0,gid=0
        state: present
      register: vresult

    - name: Deploy nginx configuration
      template:
        src: default.conf.j2
        dest: >-
         {{ vresult.volume.Mountpoint }}/default.conf
        mode: u=rw,g=r,o=r
        owner: 0
        group: 0

    - name: Run nginx container
      containers.podman.podman_container:
        name: webserver
        image: nginx
        publish: ["8080:80"]
        userns: nomap
        volume:
          - >-
            {{ vresult.volume.Name }}:/etc/nginx/conf.d:Z,U
        recreate: true
        state: started

In case you don’t already have it, be sure to install the podman collection:

$ ansible-galaxy collection install containers.podman

Also before you run the playbook, make sure you grab a copy of the built-in default configuration.  The example can use this as a stub for the configuration template, so there’s no need to actually stick any jinja2 elements in it:

$ mkdir templates
$ podman run -it --rm nginx \
    cat /etc/nginx/conf.d/default.conf \
    > ./templates/default.conf.j2

Finally, when the playbook is applied as ansible-playbook -i localhost, -l localhost -c local local_1.yml, the result should be a failure message similar to:

FAILED! => {"changed": false, "checksum": "...", "mode": "0644",
"msg": "chown failed: [Errno 1] Operation not permitted:
.../nginx_confd/_data/default.conf" ...}

One possibly obvious workaround, might be to simply give the user sudo access, utilize become: true in the task, and manually offset the ID’s according to some hard-coded calculation.  But think very carefully before you do this.  It might be fine for a quick/dirty playbook, but it definitely will not scale.

This is entirely caused because of the global namespace execution context.  Ansible is executing the tasks on the “remote” host as a regular user, without sudo access (become: false).  Further, the template module is attempting to write to a file (Ansible assumes) owned by UID/GID zero.  Whereas in reality it should be owned by some UID/GID offset from $UID/$GID (determined by /etc/subuid and /etc/subgid).  Worse, Ansible will never be able to tell the operator about any default.conf changes, nor trigger related notify-tasks such as restarting the container.  So, this is bad in almost every way possible.  Darn!

Management of auto-generated, low-level system details like IDs completely breaks the “general-purpose” nature of Ansible (at scale).  It demands it manipulate of aspects outside of its intended purview.  Consider this not too far-fetched, but perfectly valid /etc/subuid content:

...cut...
fred:100000:65536
fred:165536:1024
fred:166560:131072
wilma:100000:999999
...cut...

In case it’s not obvious: Ansbile should never tangle directly with subuid/subgid mappings for the exact same reason it shouldn’t directly manipulate /etc/passwd or /etc/group: These are OS and system implementation details that can/will vary unexpectedly. Think of the case where user-namespaces are migrated to an identity provider, like FreeIPA or Active Directory, what then, throw away all your playbooks?

Fortunately there is a way out, albeit slightly “hacky”: Abstract the away the user-namespace machinations in a way that’s nearly transparent to Ansible.  The key is to insert a podman unshare command into the remote process chain created by Ansible, starting with python.  This can be done quite simply, with a trivial wrapper script:

./files/podman_unshare_wrapper.sh

#!/bin/sh

exec /usr/bin/podman unshare $(type -p python3) "$@"

As in the ./local_2.yml example playbook below, deploying the wrapper is straight-forward, make sure to create it in the local ./files sub-directory.  Then the Ansible copy module’s default behavior of creating any missing remote destination directories may be employed. Putting the wrapper to use is similarly simple. Temporarily override the “remote” python interpreter location.  Then the template module can properly handle user-namespace translations via the wrapper, to set the intended file ownership:

./local_2.yml

---

- hosts: localhost
  gather_facts: true
  gather_subset: user
  become: false
  tasks:
    - name: Config volume for nginx container
      containers.podman.podman_volume:
        name: nginx_confd
        options:
          - o=uid=0,gid=0
        state: present
      register: vresult

    - name: Deploy podman unshare wrapper
      copy:
        src: podman_unshare_wrapper.sh
        dest: >-
          {{ ansible_user_dir }}/.local/bin/
        mode: u=rxw,g=rx,o=rx

    - name: Deploy nginx configuration
      vars:
        ansible_python_interpreter: >-
          {{ ansible_user_dir }}/.local/bin/podman_unshare_wrapper.sh
      template:
        src: default.conf.j2
        dest: >-
         {{ vresult.volume.Mountpoint }}/default.conf
        mode: u=rw,g=r,o=r
        owner: 0
        group: 0

    - name: Run nginx container
      containers.podman.podman_container:
        name: webserver
        image: nginx
        publish: ["8080:80"]
        userns: nomap
        volume:
          - >-
            {{ vresult.volume.Name }}:/etc/nginx/conf.d:Z,U
        recreate: true
        state: started

Try running that playbook exactly as before, ansible-playbook -i localhost, -l localhost -c local local_2.yml.  You should find it completes successfully and fires up a working nginx container (listening on http://localhost:8080).  Further, you may verify the correct file ownership from the host, with commands like:

$ vol=$(podman volume inspect -f '{{.Mountpoint}}' nginx_confd)
$ podman unshare ls -lan "$vol"
total 4
drwxr-xr-x. 2 0 0   26 Feb  6 15:23 .
drwx------. 3 0 0   19 Feb  6 15:23 ..
-rw-r--r--. 1 0 0 1093 Feb  6 15:23 default.conf

Assuming that all worked, the Ansible template module now has a new super-power!  Perhaps the best part is, this version of the template task will provide full change details if/when the content drifts or a handler needs to run.  Go ahead and make some modification to the local stub template file.  Then run the playbook again, note the changed status, and use this command to see it updated the file cleanly:

$ podman exec -it webserver cat /etc/nginx/conf.d/default.conf

Granted, this wrapper-script is definitely a hack, so it may not work forever, nor for every Ansible module needing it. But at least for now, hopefully the examples have highlighted the involved components and key operations. So maintaining the hack should be mostly easy. In all cases, with this new knowledge in hand, I hope you’re able to carry it forward into whatever future playbooks may require/benefit from it.

2 responses to “Hack: Podman volumes and Ansible template idempotency”

  1. David Avatar

    Why not use the podman_unshare become method from containers.podman? https://docs.ansible.com/ansible/latest/collections/containers/podman/podman_unshare_become.html

  2. Chris Evich Avatar

    Neat! I wasn’t aware of that plugin (looks like it was added in 1.9.0). Yes, if available I’d use that vs my hack. Thanks for sharing 😀

Leave a Reply

Subscribe

Sign up with your email address to receive updates by email from this website.


Search