Podman 6 is about to be released soon, so I would like to talk about the biggest change we made: a major rework of how we handle and parse our configuration files. This does not just affect Podman but also Buildah, Skopeo, and many other tools building on top of our underlying go.podman.io/image and go.podman.io/storage libraries.

The following files or locations are affected by this rework: 

• containers.conf
• registries.conf
• storage.conf
• policy.json
• registries.d
• certs.d

History

Before diving into the details on what changed, let us have a look at why we felt the need to do this. While Podman is a large application, we do also maintain our other tools such as Buildah and Skopeo.  They do similar things, for example, pull an image; so there is a strong need for code sharing across our tools. To facilitate that, we use the go.podman.io/storage (used to be known as containers/storage), go.podman.io/image (used to be known as containers/image) and go.podman.io/common (used to be known as containers/common) go libraries, which are now part of container-libs project.

Now I wasn’t around in the early days of the project but the fact is that Podman is our newest application given that both Skopeo and Buildah were created before it. The storage and registry libraries were created before Podman as well. Digging a bit into our git history I found the first storage.conf addition was in May 2017 and the first registries.conf addition in August 2017.

The first commit in the Podman repo is from November that same year.

Based on that and the fact that these projects all were in its early days, I can assume the focus was just to have something working. In the early days there was not even yet rootless support which means the config files did not account for that initially either. These things were added on top later and so started a long time adding specific features to specific files.

One problem was that the config file lived in different repositories using different code to parse the files, so over time they diverged more and more and got different rules with weird special cases.

For example because storage.conf contains the storage paths, graphroot and runroot, which had to be set there they point to a root owned place such as /var/lib/containers. That then made it impossible for a rootless user to use these paths so storage.conf opted for a rule to not read these fields in the /etc/containers/storage.conf location and just rely on user configuration in ~/.config/containers/storage.conf. But then administrators also wanted to set a global default path for all users so a rootless_storage_path was added which was read in the /etc location so we ended up with complicated logic to parse only certain fields in certain contexts.

Over the years we also got other demands, support for having the config file in /usr/share/containers. This support was added to some but not all files, simply because adding across many repos would have been a lot of work so often things were only fixed in places where we had user requests but not consistently for other files.

Ultimately we ended up with a lot of code scattered across three repositories and many differences. This resulted not just in problems for our users but also us maintainers as we needed to learn 3+ different parsing rules. Every time someone asked me how/where to set option X, I had to check what special cases exist for this file.

One thing that helped here is that we integrated our three libraries (storage, image and common) into a single monorepository last year which allowed us to make changes across all of them more efficiently.

With Podman 6 on the horizon and many user issues about various small things in regard to our different config files, we decided it is time to solve this and unify the behaviors and make them share code to reduce tech debt.

Given that some config files just have different and conflicting behaviors it means that there was really no way to avoid some breaking changes so waiting for the major release was the logical choice.

Goals

The main aim of the changes is to unify the config file parsing experience. Instead of using a different set of config file locations, .i.e. some supported /usr/share/containers others did not, we like to have the same set of search locations across all files. In particular, adding support for /usr/share/containers means packagers can deliver the config file there and administrators can use /etc/containers without having to deal with package conflicts on upgrades.

There are also various new use cases we wanted to solve. We often have asks from admins that they want to configure certain config options only for all users but not root or the other way around. If /etc/containers/registries.conf is read for all users that is of course not suitable.

Another thing is the inconsistent use of XDG_CONFIG_HOME for the user config files. Some files did honor it while others did not. We should use it consistently for all files.

We also have some config files that use environment variables overwrites for the file locations. This is very useful for local testing of certain options so the goal is to ensure other files can support an overwrite as well.

Changes

All files will now be looked up under /usr/share/containers as well if there were not previously. In addition we use XDG_CONFIG_HOME if set for the per user config file lookup. This is now also used for the root user so a root only config could be put under /root/.config/containers.

The biggest change is how we search and order the config files, the new logic is inspired by the UAPI Configuration Files Specification. That means that as a first step we look for the main file and only read one file. So if both ~/.config/containers/containers.conf and /etc/containers/containers.conf exist the code only reads the one with the higher precedence which is the one in the home directory. Then it loads the drop-in directory files (filenames must end in .conf) from all search hierarchies. All drop-in files will be sorted in the lexicographic order of the file name, if there are two or more drop-in files with the same file name only the one with the highest precedence will be used and thus replaces the prior one.

Note I am using the linux paths in the examples here, please see the containers-config(5) man page for more info about the other paths.

The following locations are searched:

• /usr/share/containers
• /etc/containers
• $XDG_CONFIG_HOME/containers or $HOME/.config/containers if XDG_CONFIG_HOME is not set

The locations contain the main file and its drop-in directory, .i.e. containers.conf and containers.conf.d/. In addition for the /usr and /etc locations there will be the additional containers.rootful.conf.d/ directory when running as root (uid 0) or when running as rootless then containers.rootless.conf.d/ and containers.rootless.conf.d/$UID/. These locations are there to allow setting system wide defaults for just root or the rootless users. The per UID subdirectory is there to allow an administrator to set a setting for just one user without having to edit their home directory content.

So now let us have a look at what this means in practice for the various files.

containers.conf

The containers.conf file logic changes because previously all files were read in order of their precedence. So previously /usr/share/containers/containers.conf,  /etc/containers/containers.conf, and ~/.config/containers/containers.conf got read all in order. Now only the file with the highest precedence is read. Similarly for the drop-in files, now ~/.config/containers/containers.conf will be read before a drop-in file from /etc/containers/containers.conf.d/.

The support for reading /etc/containers/containers.rootless.conf was dropped in favour of the drop-in directory /etc/containers/containers.rootless.conf.d/.

storage.conf

With the new parsing logic storage.conf now gains support for drop-in configuration files which was an often requested feature. In addition the new storage.rootless.conf.d/ storage.rootful.conf.d/ drop-in directory locations allow configuring options system wide for all users, not just some special options like rootless_storage_path.

Because of this the rootless_storage_path field has been deprecated, instead set graphroot to the same value in a drop-in file under storage.rootless.conf.d/.

Now previously many distributions shipped a default storage.conf file that contained the graphroot=/var/lib/containers and runroot=/run/containers options. If that file stays in the default location it means rootless users will parse these and try to use the root owned paths causing runtime failures. If these options were set to these default values they just should be removed or commented out. If a specific root only graphroot was set then one must move these options to a new drop-in under storage.rootful.conf.d/ to avoid problems for the rootless users on the system.

registries.conf

The registries.conf file no longer supports the old v1 syntax and using it will result in an error. The v2 syntax is already around for many years so most users should have already moved to the new syntax, see the registry.conf man page for more details on the format.

With the new parsing logic, registries.conf now gains support for reading the default file under /usr/share/containers so packagers should ship the default file there. It also correctly uses XDG_CONFIG_HOME now for the per user file lookup.

registries.d

Note registries.d is not the drop-in format of registries.conf, it is a separate yaml file format to set registry specific signature options, see containers-registries.d(5).

Because this file only supports drop-in file locations it is a bit special in the parsing as there is no main file location. However the parsing and ordering of the drop-in files still happens as described with the other files.

This means it now gains support for reading files under /usr/share/containers/registries.d/. So any default packaged configs should be moved there.

policy.json

This is another special case, because we did not define any config option merging logic we do not support drop-in files for it. However we did add the /usr/share/containers/policy.json location as a search location. So packagers should move the default policy file there.

certs.d

This is also not a normal config file. Instead it is a directory location where users can provide custom TLS certificates for registries. So again the only change here is the addition of the /usr/share/containers/certs.d/ lookup location.

Note for Packagers

Due to the various (partially breaking) changes and the use of these config files from different tools it is important to update the various affected applications in sync. I also recommend moving all packaged config files to /usr/share/containers, which leaves /etc/containers empty for system administrators to use.

These config changes were released in the container-libs common/v0.68.0 release and then in our end user tools Podman 6.0, Buildah 1.44 and Skopeo 1.23. An example config change for the fedora package can be found here.

Also with Podman 6 and Buildah 1.44 we now also require the use of netavark and aardvark-dns v2. So these version updates must be added as well.

If there are any remaining packaging questions or problems please reach out to us by creating an issue.