A container is, in essence, just a process that runs on the host system. The permissions this container/process receives from the host system depends on how the container was launched.
To make containers more secure, we should provide the containers with the least amount of privileges it needs to run. The container runtime assigns a set of default privileges (capabilities) to the container. In contrast to what you might expect, this default set of privileges is, or can be, harmful. Therefore the best practice is to drop all privileges and only add the ones you need.
In this lesson, we will show how you can secure your containers by giving them only the privileges they need. First, we will show you that a default configured container system gives you far more privileges than most processes need, unnecessarily opening them up to exploits. We will explain how Linux capabilities work and how they can be used to make your containers more secure. Finally, we will teach you how to configure your Kubernetes code to only allow the least amount of capabilities.
Always thought what happens in a container, stays in a container? Let's see how we can make this a reality by reading on!
This mindset of giving only the privileges needed to complete a task is called the Principle of Least Privilege
After some careful crafting of a URL request, we manage to get access to a root shell on a remote system (how that's done, I'll leave that for another lesson). We get really excited and start to explore what we can do.
First, we check what processes are running on the machine. We run
ps -aux to see the list of processes. Hmmm, that is a bit disappointing! We only see a couple of processes, where's the rest? Maybe we are in a container? That's easy to check : we run
cat /proc/1/cgroup to find out. We see it has the word docker in there, so yes it's a container. Ok, not the full access we hope for, but hey ... as if that is going to stop us. Next let's find out more about the network.
In the example attack we've shown how we got access to network traffic between other containers all from getting root access inside another container. Over the years this technique has been used to cause all sorts of havoc from sniffing the traffic (as we did), faking DNS requests in Kubernetes and even rerouting complete networks over ipv6.
As we'll explain in the next section, the out-of-thebox security of running Docker containers still allows for serious security issues and requires you to harden configuration to improve security.
Here's some references of this in the wild: Want to see the database password? sniff the network ✅ , Want to rewrite the DNS replies in a Kubernetes network? ✅, Want to inject html inside http replies? ✅ , Want to route your ipv6 traffic through this container? ✅
A container is in essence just a process that runs on the host system. The permissions the container/process receives from the host system depend on how the container was launched. In Linux there are many ways to give processes permissions. In the explanation we'll first go through the basics of file permissions and how they were extended with the concept of Linux Capabilities. From there we'll learn how we can read/set capabilities to improve security. Finally we explain how containers get launched with a default set of capabilities to give us permission to do things inside of a container.
In security a big part of the job is allowing the right people access certain things while denying others.
Chances are that you've come across using the command chmod to set the permission of files and directories. For example
chmod 775 somedir gives the owner and the group permission to read/write/enter the directory somedir and others only read and enter permissions. A bit more human friendly equivalent of the command would be
chmod u+rwx,g+rwx,o+rx somedir basically translating these numbers into user friendly mnemonics.
mkdir somedirchmod 755 somedirls -ldrwxrwxr-x 2 patrick staff 64 Sep 7 13:11 somedir/
As part of file permission options there is also the setUID (+s) or setGUID flag (+S) which allows users to become a certain user/group when they run a file. This comes in handy for a command like ping : on Linux , to send special ping packets (ICMP) it needs the root privilege (to open a NET_RAW socket) Older Linux systems therefore used the setUID flag to make the ping command run as root.
ls -l /sbin/ping-r-sr-xr-x 1 root wheel 55344 Sep 22 2020 /sbin/ping
But ... that's a whole lot of privileges for just sending a ping! There must be a better way. That's why Linux capabilities were invented.
Starting with kernel 2.2, Linux divides the privileges traditionally associated with superuser into distinct units, known as capabilities, which can be independently enabled and disabled. To control the capabilities of processes and files we have a couple of useful commands. You get them as part of the
libcap2-bin package (or similar to your OS).
capsh: cli to manage capabilities in a human friendly way
getcap: get the capabilities of a file
setcap: set the capabilities of a file
To see the capabilities of your current shell process we can access the status of it.
We can find the capabilities of the current process using
grep Cap /proc/$BASHPID/status. This will give us a list of capabilities, but they are in an easy readable form.
To relate this to containers, we must first understand how process permission delegation works.
On Linux, process ID 1 (INIT) has all the privileges after booting. After that, privileges are delegated and possibly restricted. The container system is no different, a container is nothing but a process with a set of given privileges delegated by the container system. Obviously, we want containers to only have a set of restricted permissions and in the following snippet of the Docker daemon source code you can see that it starts a container with a default set of capabilities.
This is already a very limited set of capabilities, but still it allows for example for cap_mknod, cap_net_raw. In our exploit we've shown that just by having the cap_net_raw we were able to hijack network traffic. It turns out that in most cases you can actually drop all privileges/capabilities, which results in a better security posture.
In case you do need to give your container a certain capability, it's good practice to first drop all capabilities and then allow the needed one after that. You might hesitate about the cap_net_bind_service though, isn't my app going to have to bind to the network? Well the permission only applies for binding to ports lower than 1024. So when you bind your server to for example 8080 you don't need that privilege.
A diagram which shows the difference between running with default capabilities and with restricted capabilities
If we follow the principles of least privilege, the best practice from a security perspective would be to only provide the capabilities which our container actually needs. In Kubernetes, you can manage capabilities assigned to your container through the
You can drop all capabilities by adding the following configuration to your .yaml file:
In the case where you do have to allow capabilities it is good practice to first drop all default capabilities and only then add only the ones you need. In the example below you can see how we first drop all capabilities and then add the
As we've explained, capabilities are delegated from one process to another, but processes can also gain new privileges if they are allowed to. We can disable this behavior by:
--useroption, if we are regular user, the kernel will not pass us any special capabilities
allowPrivilegeEscalationflag in the Kubernetes config, (equivalent to the Docker command line option
--security-opt=no-new-privileges), we avoid that the privileges given to a process are extended
These additional measures help in creating a defense in depth by providing different layers of protection. In addition, Kubernetes and containers have other mechanisms to limit permissions: cgroups, seccomp and AppArmor. They can be used in combination with the capabilities limiting we've presented. The explanation of these concepts is beyond the scope of this lesson.
As a Kubernetes admin, you can configure the cluster to enforce securityContext settings. Kubernetes has the PodSecurityPolicy controller built in which allows you to enforce securityContext settings. However, please note that this has been deprecated in the v1.21 release and is scheduled for removal in v1.25.
It will eventually be replaced by the new Pod Security Admission feature; however, as of v1.22, it is still in alpha status. Good alternatives are the externally maintained projects Open Policy Agent or Kyverno.
You’ve learned what Linux capabilities are and how to apply them to improve container security. We hope you will apply your new knowledge wisely and make your code much safer. Feel free to rate how valuable this lesson was for you and provide feedback to make it even better! Also, make sure to check out our lessons on other common vulnerabilities.