How to Give Kubernetes Immunity from Privilege Escalation

This is part of a series of posts leading up to KubeCon + CloudNativeCon on October 24-28.

Security in virtualized and physical infrastructure environments has been refined over the past 20 years into a sophisticated craft that enables granular control over access and availability of application workloads. The advent of Kubernetes, the distributed container runtime platform built on top of this infrastructure, brought new challenges that required innovative methods to ensure the same level of security across the digital estate.

One of the pillars of cybersecurity is application isolation through various forms of confinement, either through logical or physical separation. Kubernetes is able to leverage this practice along with the ability to dynamically control the placement of application workloads to enable new types of security patterns not previously thought possible. One such common pattern that is being adopted more and more within Kubernetes is multi-tenant architecture.

Traditional segmentation

Historically, this has been addressed by isolating physical components such as switches, compute, and storage devices. With logical isolation boundaries and strict role-based access control per workload, it is now possible to allow Kubernetes constructs such as namespaces to help create a shared environment that enables privacy and security. When complemented with additional components such as advanced container network interfaces and container storage interface providers, network transport and data-at-rest can also be confined to the scope of their workload applications.

The advantages of a segmented architecture are twofold: higher utilization per physical unit due to increasing workload density and homogeneous configuration management. The latter is very important from a security perspective, because when the system is constrained to a multi-tenant Kubernetes cluster, role-based access control policies (RBAC), permissions (ACL), and security tools can converge ubiquitously. Additionally, constraining an environment by default benefits the security posture of Kubernetes itself. A practical example is running the Kubernetes control plane on heavily locked down and regulated nodes, disabling port access and privileged operations.

As these patterns become more commonplace thanks to improvements in the security posture and general awareness of Kubernetes, attackers are increasingly looking for weak entry points within the system. The weak link is increasingly becoming the Linux host operating system, which many operators neglect due to complexity and lack of domain-specific knowledge.

The Frightening Reality of the Container Escape

Containerized applications have given people the wrong impression of total workflow isolation. However, the reality is very different. From the foundation in 2019 Felix Wilhelm tweet Outlining how the abuse of cgroups release_agent feature could lead to an escape from a privileged Docker container or a Kubernetes pod, container escape has become one of the most widely abused vulnerabilities in attacks against DevOps tools.

The container flight could be extremely dangerous. Once an attacker gains access to the host system, they can escalate their privilege to access other containers running on the machine or run malicious code on the host. Depending on how vulnerable the host is, the actor could also access other hosts on the network.

There are many ways attackers can choose to successfully perform container escape attacks. However, they all boil down to two main causes: misconfiguration and excessive privileges.

While traditional countermeasures such as implementing better authentication practices, permissions management, Tiger network policies, and patching can help make life harder for attackers, it is impossible to completely eradicate the threat. There will always be “that one application” that requires elevation, and new CVEs will always be discovered (e.g. in the Linux kernel).

It is therefore very important to view Kubernetes and the host operating system as a single, integrated system and to implement security controls at multiple levels of the stack.

OS restriction to the rescue

This means that the base operating system must have a way to prevent Kubernetes or other running applications from accessing files, networks, processes, or other system resources without requesting that specific access.

This practice is known as confinement, and while the concept originated with the Unix chroot system call and command in 1979, modern operating systems have taken the concept to new heights by explicitly targeting access privileges to the administrator and using Linux kernel security features. including AppArmor, seccomp, and namespaces to prevent applications and services from accessing the broader system.

Restricted host operating system environments have emerged in response to this vulnerability. While still nascent, they demonstrate that they can provide increased cybersecurity resilience across the stack. Combined with traditional security practices at the physical, network, storage, and virtualization layers, they provide a higher level of confidence when running application workloads on Kubernetes. Should they escape their Kubernetes constraint, the host operating system running the container runtime has logical boundaries to block forms of takeover, access escalation, or backdooring from within the container itself.

To learn more about cloud-native topics, join the Cloud Native Computing Foundation and the cloud-native community at KubeCon + CloudNativeCon North America 2022 in Detroit (and virtually), October 24-28.