One longstanding problem with Podman is that your rootful containers may loose network connectivity after a while. For many users it is not be obvious what is causing such problems. Netavark configures NAT and port forwarding firewall rules. When another process is deleting our firewall rules the containers will loose connectivity. One such process is firewalld, when firewall-cmd –reload is executed it will delete all firewall rules and then only add rules known to firewalld back leaving podman containers without connection. The interesting part here is that this command may not even have to be executed by users. Some applications run this command on update, for example it is possible that dnf update results in firewalld being reloaded making it harder for users to know what happened. NOTE: Starting with firewalld 2.0 it may no longer flushes all rules and only the firewalld specific ones.
One solution to restore the rules is to simply restart the containers as all the necessary rules will be added on container start. However this is undesirable as some container application may need a long time to stop and/or start. To address this issue Podman added the podman network reload command a long time ago. This command removes and then adds the network configuration back. It worked recently well but there is still one big problem with it: The command needs to be explicitly executed after the rules are lost. Therefore it will not help users that are not aware when rules are lost.
With netavark v1.9.0 we added a new systemd service called netavark-firewalld-reload.service which should fix the problem once and for all. This service runs a small netavark command which listens on dbus for the firewalld reload event. This event is emitted every time firewalld was reloaded and flushed out the netavark firewall rules. If the event is received the service will then add back all the netavark firewall rules.
The new systemd service is designed in such a way that it only runs when firewalld itself is also running to not consume resources unless needed. The following shows an example on how to enable the service.
First we enable the service, notice that enable does not necessarily mean start on boot. For our unit we add a wants to firewalld which means the service is only started when firewalld is started. This ensures that on systems were firewalld is not used or disabled it won’t run unnecessarily. The service will also stop when firewalld is stopped.
$ sudo systemctl enable netavark-firewalld-reload.service
Created symlink /etc/systemd/system/firewalld.service.wants/netavark-firewalld-reload.service → /usr/lib/systemd/system/netavark-firewalld-reload.service.Now restart firewalld to see that our service will get started with it as well. Alternatively directly start our service or reboot assuming firewalld is enabled.
$ sudo systemctl restart firewalld.service
$ sudo systemctl status netavark-firewalld-reload.service
● netavark-firewalld-reload.service - Listen for the firewalld reload event and reapply all netavark firewall rules.
Loaded: loaded (/usr/lib/systemd/system/netavark-firewalld-reload.service; enabled; preset: disabled)
Drop-In: /usr/lib/systemd/system/service.d
└─10-timeout-abort.conf
Active: active (running) since Mon 2023-11-13 18:00:17 CET; 22s ago
Main PID: 72763 (netavark)
Tasks: 3 (limit: 38149)
Memory: 15.4M
CPU: 9ms
CGroup: /system.slice/netavark-firewalld-reload.service
└─72763 /usr/libexec/podman/netavark firewalld-reload
Nov 13 18:00:17 pholzing-fedora systemd[1]: Started netavark-firewalld-reload.service - Listen for the firewalld reload event and reapply all netavark firewall rules..Lets start a simple webserver and see if it works.
$ sudo podman run -dt -p 8080:80 docker.io/library/nginx
52f2c11810d78f69287874e2469d4caec99176b583e378732cee4623cbd09476
$ curl -s 127.0.0.1:8080 | head -n4
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
$ sudo firewall-cmd --reload
success
$ curl -s 127.0.0.1:8080 | head -n4
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>Without the service the curl command after the firewalld reload command would just hang. This shows that the netavark-firewalld-reload.service is actually working as expected.
Notice that this is a netavark specific feature and if you are still using the old CNI backend it will not work.
Try the new service and let us know if you encounter any problems with it report issues on the netavark github repo.
Leave a Reply