In this post, I am going to share my journey as an LFX Mentee. I worked with the Podman Container Tools organization to solve a specific networking issue in Netavark using Rust.

What is LFX Mentorship?

It is a remote program organized by the Linux Foundation. It is designed to help people start contributing to open-source projects. You are paired with mentors who guide you as you work on a real project. It is a great way to learn and gain experience.

How It Started

One day, while recovering from a humerus fracture I sustained while arm wrestling, I was randomly looking at the CNCF Mentoring LFX Mentorship GitHub repository, and a Pull Request caught my eye. It was a PR from Podman Container Tools!

I was surprised that software I use frequently was going to participate in the LFX Mentorship.

When I looked at what language was the prerequisite, I was even more excited; it was Rust, the language I had the most experience in and also my favorite language!

Going Down the Rabbit Hole

I then told myself, “Alright, you can try.” I tried to understand the project description. 

The project aimed to solve a specific networking issue: In case of UDP, Conntrack creates a stale record of a failed connection attempt because the data was being sent before the container was alive. As a result, the sender on that port is blocked until the stale entry in the conntrack table times out.

The goal was to implement conntrack entry clearing into Netavark, Podman’s network management tool. To resolve this, Netavark should delete any invalid connection tracking entries associated with a port immediately after that port is mapped to a container.

However, since Netavark is written in Rust and the existing Rust netlink bindings presently do not include support for Conntrack, the plan was twofold: first, expand the bindings to cover Conntrack, and second, use those bindings in Netavark to flush Conntrack entries on network change.

There was a lot of new jargon, so I started by Googling the new words one by one.

I started reading blogs, and because of some very good articles, I started understanding what the conntrack subsystem is, why it is needed, what the netlink protocol is, and what it is used for.

I also, in parallel, started making notes of what I was understanding in my own words to later include in my project proposal/cover letter. I would recommend anyone trying to get selected for the LFX Mentorship to do the same.

Once I had a big picture of what needed to be done, I decided to dig deeper. I started looking at some existing conntrack libraries which I could refer to for understanding the structure of a conntrack netlink message.

A library that stood out to me was ti-mo/conntrack because of its tagline “Pure-Go Conntrack implementation; for humans”. I found its code to be understandable. I started reading its codebase and again making notes. I also used strace with the conntrack-tools CLI to understand the structure of a conntrack netlink message. After this, I had a good understanding of the conntrack subsystem, the netlink protocol, and the conntrack netlink message structure.

Also, because I made notes, I started to prepare my project proposal/cover letter using those notes. They turned out to be very useful when preparing the cover letter. I was able to submit a very detailed cover letter because of them.

Getting My Hands Dirty

After this, I realized I also needed a way to prove to my (future) mentors that I could work on the Netavark codebase. The easiest way to do that was to contribute and fix some GitHub issues. I was not alone in this; I remember that there were 2-3 people (who were also trying for the LFX Mentorship) also contributing to Netavark during this time.

I realized that if I wanted to stand out, I had to pick an issue with the most relevance to netlink, and I certainly did that.

I picked an issue where:

The problem was that in unmanaged mode, aardvark-dns was hardcoded to bind to the gateway IP. In the case when the gateway IP was not on the host, aardvark-dns simply failed.

The solution was to just check the existing bridge for IP addresses (using netlink, yay) and bind aardvark-dns to them instead of the gateway IP.

PR: https://github.com/containers/netavark/pull/1304

So, I had to use the netlink-packet-route library to get the IP addresses of the existing bridge. I created a pull request. Soon after, Paul Holzinger (one of my future mentors) reviewed my pull request. I followed his comments, and then Matt Heon (one of my future mentors as well) reviewed it, and it got merged soon after!

Beating the Clock

I soon realized that the deadline for the submission of the cover letter and resume was near and I should probably finalize my application quickly. I looked for some low-hanging fruit and found the perfect candidate:

The issue was to add a firewalld StrictForwardPorts test to check if the detection of Firewalld’s StrictForwardPorts property was working properly in netavark. The test could not be added previously because firewalld StrictForwardPorts is a new feature with a new version of firewalld, but that version of firewalld was not on the previous CI/CD environment (Fedora 41).

I checked the current CI/CD environment and found out that it had been updated to Fedora 42, which has the required firewalld version (yay). However, I realized that my own local dev environment did not have the required firewalld version, so I built firewalld from source, and it worked. So, I wrote some BATS integration tests to check if StrictForwardPorts was working properly and created a PR. The PR was soon reviewed by Paul Holzinger and Matt Heon, receiving an LGTM from both, and was merged.

While doing all this stuff, I was simultaneously reading the code of the ti-mo/conntrack library and making notes for my own reference to understand the structure of a conntrack netlink message. Using all my prepared notes from other code, blogs, and scarce documentation, I quickly finished my detailed cover letter to submit it before the deadline, and I was successful.

The Interview

After waiting for some days, I received an email from Matt and Paul, and I was surprised! I got shortlisted for the interview!

After receiving the email, I asked my friend Meso to conduct a mock interview the day before the actual one. He grilled me, making the session very intense. However, that preparation helped me gain confidence, which was crucial since this was going to be my first professional interview.

When the big day arrived, the actual interview went very well. Thanks to the mock session, I was much less nervous. Matt and Paul asked me to explain my pull requests to Netavark to make sure that I was not copying and pasting code from LLMs. Other questions were from my resume and questions just to get to know me. My proposal was very detailed, so they did not ask me many questions related to the netlink protocol.

I soon received another email confirming that I had been selected for the mentorship.

I was extremely happy and overjoyed.

Wrestling with Netlink

Soon after, the LFX mentorship contribution period started.

I divided my final goal into two smaller goals:

1. Add conntrack support to netlink-packet-netfilter.

2. Use netlink-packet-netfilter to flush conntrack entries on port forwarding setup and teardown.

I locked in on the first goal. I realized that to add conntrack support to netlink-packet-netfilter, I had to understand the netlink-packet-core library and how it is used for serialization and deserialization of netlink messages.

I learned that the communication relies on three key concepts:

• Socket Communication: Open a Netlink socket (socket()), bind it to a process ID for replies (bind()), and use sendto() and recvfrom() for communication.
• Message Structure: Every Netlink message must be carefully constructed with a specific header (nlmsghdr) and a payload. For this task, the message will also include a Netfilter-specific header (nfgenmsg).

• Payload Attributes: The payload itself is made of many individual attributes in a Length-Type-Value (LTV) format. These attributes will specify the exact conntrack entry we want to delete based on details like protocol, source/destination ports, and IP addresses.

I broke this down into the following tasks:

• Understand netlink-packet-core

• Study the existing codebase (nflog) of netlink-packet-netfilter to grasp the library’s structure

• Add conntrack support to netlink-packet-netfilter by referring to the kernel code and the ti-mo/conntrack go library

The documentation for netlink-packet-core was sparse, so I often had to read the source code directly to understand the implementation.

There was a toy protocol example in the netlink-packet-core docs. I referred to that and started to gain an understanding of netlink-packet-core. The toy protocol example in the docs did not cover all the features of netlink-packet-core, so I extended those examples: https://github.com/shivkr6/serialization-deserialization-netlink-rs

I’m planning to add my examples to improve the docs of netlink-packet-core.

After this, I implemented netlink conntrack messages using netlink-packet-core. I used Wireshark to get hex dumps of the messages and wrote tests to ensure my code was correct.

I then added those types to netlink-packet-netfilter with conntrack get, delete, and new message types.

After this, the first goal of adding conntrack types to netlink-packet-netfilter was completed. You can see the implementation in these Pull Requests:
Use netlink-packet-core 0.8.1
Conntrack get netlink netfilter message types
Conntrack delete netlink netfilter message types
Conntrack new netlink netfilter message types

Green Tests & Victory

So, I shifted my focus to the second goal of using the conntrack library (netlink-packet-netfilter) to flush conntrack entries on port forwarding setup and teardown.

But I was confused about which ports or IP addresses to match in the conntrack entry. So, I reached out to my mentors, Paul and Matt. They told me to refer to the CNI (Container Network Interface) and Moby (Docker) codebases.

After studying their code, I realized we needed to match and flush entries based on two specific rules:
1) Match the container’s IP address against the Source or Destination IP of the reply tuple.
2) Match the HostPort against the destination port of the original tuple.

I soon implemented that in Netavark and wrote unit tests and BATS integration tests to verify it was working correctly.

But one test with the Firewalld backend was failing, and I had no idea why. So, I consulted my mentors again, and it turned out the failure was due to a known issue with Firewalld port forwarding ranges > 1 in the test environment, rather than a bug in my code. We decided to skip that specific test case, resulting in green tests again.

For a deep dive into the implementation details, check out the full Pull Request here:
firewall: flush stale UDP conntrack entries on port_forward setup/teardown

Wrapping It Up

And with this, I had successfully completed my LFX Mentorship project.

It was challenging because I had to prepare for my exams, and I also got sick in between. There were also festivals going on. But I was able to manage it eventually. My mentors, Paul and Matt, were also very happy with me.

Working with Shivang was a pleasure throughout. He achieved all our project objectives in a timely fashion, and we’re very satisfied with the outcome. He effectively broke down the complex project into smaller, achievable steps to work efficiently towards the end goal. Most importantly for us, he communicates well – he’s not afraid to ask for help or clarification when he needs it, and requests feedback on his progress early and often.

Matt Heon and Paul Holzinger, Podman Maintainers

Overall, it was a great experience. Beyond the technical skills, I truly appreciated how welcoming and helpful the Podman team was.

Paul and Matt were incredibly helpful and responsive. We had weekly meetings, and I really appreciate the time they dedicated to me. Honestly, they are some of the best mentors anyone could ask for; their support kept my motivation high.

I also want to express my gratitude to Gris Ge, the maintainer of the netlink-packet-netfilter and rust-netlink crates. He reviewed my PRs for the conntrack support.

I also want to extend a special thanks to my friend Meso, for conducting my mock interview. I am equally grateful to my friend Firstyear (William) for allowing me to list him as a reference. Finally, a big shoutout to my friends rv178, uncomfyhalomacro, and theabbie for being incredibly supportive since the start of my programming journey and for introducing me to cool stuff.

Connect with me

If you have any questions or just want to connect, feel free to reach out:

• GitHub: https://github.com/shivkr6
• LinkedIn: http://linkedin.com/in/shivkr6
• Email: shivangraghuvanshi2005@gmail.com