Docker Security Advisory for runc, BuildKit, and Moby (is this affecting the current release?)

[GuyWithAQuestion]

Apologies if this is a duplicate post to what ncohafmuta put up here, but I think we are discussing separate topics.

 

CVE-2024-21626 was published a few days ago.

I'm currently running 6.12.6 and running 'docker version' from command line reveals docker engine version 20.10.24 and runc version 1.1.5.

 

Is the best course of action is to disable docker updates until docker is updated? Or am I misunderstanding and this is this not affecting unRAID?
Thank you.

[T0rqueWr3nch]

I evaluated this yesterday on my own system and came to the following conclusion: We are currently affected on Unraid- including the runc and BuildKit vulnerabilities as well as with the separate Docker engine vulnerability ("All users on versions older than 23.0 could be impacted.").

 

The runc and BuildKit vulnerabilities are the most severe vulnerabilities. They are especially bad considering that Unraid runs Docker as root; container escape means that the attacker would have full root filesystem access on the host. Even worse, it's possible that an attacker could further obtain full root command abilities on the Unraid host.

 

As for disabling container updates, that depends on your risk profile, but yes, that could be a potential risk mitigation, especially given that this vulnerability is now public. Depending on where you source your images, it's possible (albeit unlikely) that a malicious image is pulled. Given the layered nature of building Docker images (where it's common to build images on top of other images), there's also the possibility of a supply chain attack, where an image maintainer unknowingly pulls a layer that has itself been compromised.

[T0rqueWr3nch]

4 hours ago, GuyWithAQuestion said:

Apologies if this is a duplicate post to what ncohafmuta put up here, but I think we are discussing separate topics.

 

 

Nope, separate vulnerability in glibc which is also severe given its ubiquity (and that of the affected function therein).

[SpencerJ]

Hey there,

thanks for posting. We are aware of this vulnerability and are working on releasing a patch as soon as possible.
We vet all of our community apps images, but we cannot guarantee the security of third-party docker images.

 

more to come… 

[GuyWithAQuestion]

Awesome. Thanks for keeping us updated and taking the time to post a reply, SpencerJ! And thank you, T0rqueWr3nch, for confirming and your insight as well. I appreciate both responses.

[sir_storealot]

"One of the best practices while running Docker Container is to run processes with a non-root user." (dockerlabs)

 

I think this incident raises the following questions.

 

Are there any specific reasons why root user is required to run docker in Unraid?

Are there any plans to change to a non-root user in the future?

Is it possible to change the configuration by hand to run as non-root user? Any recommendations on how to best achieve this?

 

Would appreciate an answer @limetech @SpencerJ

 

Thank you!

[T0rqueWr3nch]

Hi @sir_storealot, I know your question wasn't directed at me, but I imagine the biggest problem is because Unraid is built on Slackware. Great for making a lightweight server, not as much fun from a sysadmin/distro maintainer perspective as it's a barren Linux distribution. Almost everything has to be built by hand.

 

Quote

Is it possible to change the configuration by hand to run as non-root user? Any recommendations on how to best achieve this?

 

There is the --user option which can be passed in the extra parameters section of the container template. That option can be used to force the container processes to run as a non-root user (some containers actually do this in their Dockerfile- I believe many/most Linuxserver images do this; hence their special PUID/PGID options which make them unique). The problem with the --user option is that it forces the process inside the container, to also run under this UID (i.e. this isn't user namespace mapping), which doesn't work with all Docker images (think things like postgresql which requires root inside the container IIRC).

 

I did a very cursory overview to scope out what it would take to pull this request off (rootless Docker) and I think it's achievable; especially given that Slackware has more features than I thought it would (i.e. it looks like we do have uidmapping, which was quite a pleasant surprise). If Unraid is looking for a part-time dev/DevOps/security admin, just let me know!😆

[ljm42]

Everything in Unraid runs as root, including the Docker Daemon. The better containers then reduce their privs to user 99 and group 100.

 

Moving away from root is a much larger discussion with many consequences. Actually forcing all containers to run as a non-privileged user will reduce the functionality of Docker. For instance, you wouldn't be able to have a backup container that has access to all your files, or give an editor like Code Commit access to edit files on the flash drive. So this is not a change that should be made lightly.

 

This is the first ever Docker escape flaw, and we will have an rc release out soon that resolves it. Note that this flaw is only a problem if the Docker containers you choose to install are malicious. If you are concerned, you may choose to disable any risky containers you have installed, and avoid updating/installing containers until you have updated to the forthcoming rc release.

[sir_storealot]

9 hours ago, ljm42 said:

Moving away from root is a much larger discussion with many consequences. Actually forcing all containers to run as a non-privileged user will reduce the functionality of Docker. For instance, you wouldn't be able to have a backup container that has access to all your files, or give an editor like Code Commit access to edit files on the flash drive. So this is not a change that should be made lightly.

 

I don't get your example. For my backup container, I map the directories I need backed up as a read-only volume volume into the docker. I definitely would NOT want my backup container to have unrestricted r/w access to everything.

Edit: on second thought, I think I understand what you mean, the backup user would have to have the right permissions to read the files that have been mapped into the container. Makes sense.

 

9 hours ago, ljm42 said:

 

Note that this flaw is only a problem if the Docker containers you choose to install are malicious.

 

According to the description, this attack also allows for someone exploiting a vulnerability in a non-malicious container to escape the container, which is the much bigger concern for me (and probably other users running various services processing external data) - see:

https://github.com/opencontainers/runc/security/advisories/GHSA-xr7r-f8xq-vfvv

 

"Strictly speaking, while attack 3a is the most severe from a CVSS perspective, attacks 2 and 3b are arguably more dangerous in practice because they allow for a breakout from inside a container as opposed to requiring a user execute a malicious image."

 

This opens up the attack surface SIGNIFICANTLY in comparison to just malicious containers. I will try to manually reduce my container privileges as suggested, seems to be a good idea anyway.

 

Thanks a lot for explaining the bigger implications!

Edited February 6 by sir_storealot

[sir_storealot]

12 hours ago, T0rqueWr3nch said:

There is the --user option which can be passed in the extra parameters section of the container template. That option can be used to force the container processes to run as a non-root user (some containers actually do this in their Dockerfile- I believe many/most Linuxserver images do this; hence their special PUID/PGID options which make them unique). The problem with the --user option is that it forces the process inside the container, to also run under this UID (i.e. this isn't user namespace mapping), which doesn't work with all Docker images (think things like postgresql which requires root inside the container IIRC).

 

Thank you for your response! So if I understand you correctly, as a mitigation, or even a future best practice implemented on the user side, we could try to run the containers with the --user=nobody option, and check if they still work? This way, attack surface could be reduced significantly, I would be quite happy with this approach.

[jlh]

Hey guys, thank you for the information and feedback. T0rqueWr3nch, I appreciate the analysis on the viability of Docker-rootless on Slackware, that could be an interesting option to look into.

I can give a little input on the situation, as the container security specialist on the Unraid team.
 

In general, containers are designed to be just as secure as virtual machines, with no chance of break-out when used properly, even when running as root. This is because the Docker runtime takes advantage of cgroups and namespace features built into the Linux kernel, allowing full process isolation on a shared kernel, at the cost of a larger attack surface.

In the rare event that a vulnerability is found in Docker or the Linux kernel that does allow a container to break-out, it's a big deal. This is why this vulnerability, that allows attackers to break out of the filesystem namespace, has been hitting the headlines under the name "Leaky Vessels".
https://snyk.io/blog/leaky-vessels-docker-runc-container-breakout-vulnerabilities/

 

Of course, when security is important, you don't want to just have one line of defense against attackers, as we've just seen with this vulnerability. That's where security features like uid namespaces, non-root container processes, non-root docker daemons, and isolated kernels come into play.

We already use these security hardening features for our cloud infrastructure here at Unraid, which is why our websites and Unraid Connect were not directly affected by this vulnerability.

However, these security features have tradeoffs when it comes to usability, which is why they are not used/enforced in the OS. The most obvious one is being able to access files on your array, but many third party containers are also just not compatible with security hardening and will not work with "--user 1000". There are also many kinds of workloads, such as DHCP servers or device passthrough, that aren't possible without root or privileged access to the Unraid server host.

I have dealt with these incompatibilities a lot as a Red Hat consultant for Red Hat Openshift Kubernetes, where containers are always blocked from running as root. It breaks a lot of container images, and it can be very frustrating!

It's worth noting that the impact of "Leaky Vessels" CVE-2024-21626 does not change whether your container runs as root or not. The vulnerability cannot be mitigated by using the the "--user" option to force processes to spawn as a non-root user. The vulnerability allows attackers to steal the root permissions of runc, which is always root in all but the most hardened container runtimes.

Quote

By default, access privileges will be the same as that of the in-use containerization solution, such as Docker Engine or Kubernetes. Generally, this is the root user, and it is therefore, possible to escalate from disk access to achieve full host root command execution.

https://snyk.io/blog/cve-2024-21626-runc-process-cwd-container-breakout/
(Counter examples of runtimes that don't run as root would be Podman and the aforementioned docker-rootless, but these are not officially supported by any major Linux distro and have major incompatibilities with standard (rootful) Docker)

In any case, we're working hard to get this security update out to users and close this rare security gap. If you want to increase your security level beyond the Docker default going forward, I would make sure that you're running reputable docker images that take security seriously, and that use the "USER" directive in their dockerfiles to run as a non-root user in a consistent way.

In cases where secure docker images aren't available, you could try writing some of your own dockerfiles. It's a very good skill to have as a sysadmin with the growing Cloud Native IT landscape, and you can ensure that your container is secure and compatible with security hardening. Most of the containers that we use for Unraid's cloud infrastructure are written and maintained by our own team. Tools like Github Actions and Github Container Repository (GHCR) make it very easy to publish Docker images from a git repo for free.
 

Here are good tutorials for writing your own dockerfiles and publishing your own Docker images from Github using Github Actions and GHCR:
https://linuxize.com/post/how-to-build-docker-images-with-dockerfile/
https://docs.github.com/en/actions/publishing-packages/publishing-docker-images

[T0rqueWr3nch]

Another option, albeit a clunky one, is to not use Docker in Unraid and instead run a VM where you can run rootless docker with user namespace mapping; just let Unraid be a NAS. This option would probably be overkill for most containers (and users) and you'd really be losing out on some of the advantages of running Docker locally on Unraid. (I still think there should be further consideration of rootless Docker in Unraid- as previously stated, sure there are disadvantages, but there's a reason this is an industry standard).

 

@sir_storealot Consider your threat model. Are you most concerned about ransomware or data exfiltration? What are potential attack vectors an attacker could use to get at you? Do you have SMBs with guest access and no passwords? Do you have your Unraid server exposed to the internet? Are you running containers that have an elevated risk of an attack? (In which case, don't run sketchy images). I only mention these things because I think you should always be concerned about security, but I don't think you necessarily need to worry. The higher likelihood risks to your data on Unraid come more from things external to Unraid (such as encryption of network shares or people exposing their Unraid webGUI to the internet) than locally on Unraid itself.

 

That being said, I'm glad this vulnerability is being fixed. Thanks everyone.

[sir_storealot]

3 hours ago, T0rqueWr3nch said:

@sir_storealot Consider your threat model. Are you most concerned about ransomware or data exfiltration? What are potential attack vectors an attacker could use to get at you? Do you have SMBs with guest access and no passwords? Do you have your Unraid server exposed to the internet? Are you running containers that have an elevated risk of an attack? (In which case, don't run sketchy images). I only mention these things because I think you should always be concerned about security, but I don't think you necessarily need to worry. The higher likelihood risks to your data on Unraid come more from things external to Unraid (such as encryption of network shares or people exposing their Unraid webGUI to the internet) than locally on Unraid itself.

 

You make some excellent points, and I agree 100 % with your risk assesment. However, I believe I have already tighened all the "other screws" that you mention a long time ago, with docker containers from non-official sources (hard to judge what really is "reputable", it's always a grey area) now being the 2nd biggest exposure I can see, which is why I am so nervous about it. The biggest exposure in my eyes are Unraid plugins... But it's so hard to live without them. As always, a trade-off needs to be made between security and convenience...

 

@jlh Thanks for your additional in-depth explanation. Really great to see you guys are on top of this!

[T0rqueWr3nch]

@jlh, you know, it's funny, I use podman all the time at work (hence the request for rootless Docker) but I never considered that there may be a preexisting podman package for Slackware. It turns out there is:

I've attached the feature request for reference. What's nice is that by using podman, we could actually sidestep having to deal with migrating over to rootless Docker. Those who wish to run Docker as root by default can continue to do so, those who want a bit more security can switch to podman. It even supports namespace mapping.

 

[jlh]

Yeah, I was actually thinking to myself about how good Podman is when I wrote my reply. There's a few disadvantages when it comes to development tools like docker-compose and Tilt when running it on a workstation, and you often can't use containers that talk directly to the Docker socket, but it strikes a really good balance between security and usability for server workloads. That's a really good feature request, I'm not responsible for the Docker subsystem in Unraid, but I'll bring it up in our internal discussions.

[ljm42]

Regarding the Docker security advisory, Unraid 6.12.7-rc2 is now available with Docker 24.0.9 and runc 1.1.12: