Introduction to Kubernetes

Kubernetes is an open-source platform used for automating the deployment and management of containerised applications. This containerisation solution is widely used across both internal systems and in Cloud Solutions, with over 5.3 million users actively developing via Kubernetes. This substantially large community of developers relies on the security of Kubernetes for keeping their containerised applications safe from attack, however many Kubernetes attacks do not come from vulnerabilities found within the software, but with common misconfigurations of the Kubernetes environment.

This whitepaper is set to discuss the most common Kubernetes vulnerabilities found by researchers at OWASP who have set out a “Top Ten” Kubernetes vulnerability list, similar to their Web Application and API list

Basic Kubernetes Information

Before we get into the OWASP top ten, there are some basic keywords that will be used within this paper which could be confusing if not defined. They are as follows:

Node – The servers that make up the Kubernetes cluster, there are two types of nodes, Control-Nodes that handle the running of processes and communication with the rest of the cluster, and Worker-Nodes that are in control of running the Pods within the cluster.

  • Pod – A containerised service ran within the Kubernetes Cluster, runs on worker-nodes.
  • Volume – A way of mounting part of the node’s file system into a pod for shared usage.
  • Cluster – The Kubernetes system as a whole, including all nodes.
  • K8S – Shorthand for Kubernetes.

Common Vulnerabilities: What and Why?

As previously mentioned, these vulnerabilities are split into 10 categories, with each category covering a range of vulnerabilities with the same root cause.

K01: Insecure Workload Configuration

The first of the common vulnerabilities is “Insecure Workload Configurations”, which applies to the configurations of pods within worker nodes. This category encompasses many issues such as pods running as root, which gives the pod unnecessary levels of access to the system and, if compromised, would allow an attacker to gain full root level privileges over the underlying system. Also included in this category would be insecure volumes used, where a volume is mounted in an insecure location such as the root directory. This is a large security issue as a pod with a volume set as the root of the file system has access to all files in the host system, including sensitive files such as /etc/passwd and /etc/shadow.

K02: Supply Chain Vulnerabilities

The second common vulnerability is supply chain attacks, where the pod itself may not be vulnerable, but a part of the pod’s third-party components could be vulnerable. This can come in many different forms, for example issues within container images themselves, such as images using old versions of software with known vulnerabilities within them. An example of this could be that an image uses a vulnerable version of log4j to log errors, which in turn makes the pod vulnerable to remote code execution.

K03: Overly Permissive RBAC

Role-Based Access Control (RBAC) is used within Kubernetes to give users different roles within the cluster. These roles should be given out using the “Least-Privilege” model, where a user only has the roles that they need and nothing else, however this is often not the case. Overly permissive RBAC can allow users to access parts of the cluster that they should not be able to access, with roles such as “cluster-admin” giving the user essentially super-admin permissions across the whole cluster.

K04: Policy Enforcement

Policy enforcement is a key issue to look out for, as if your policies are not enforced by a centralised party, certain pods may be overlooked, allowing for possible vulnerabilities. As an example, you could have a policy to only allow trusted registries for image distribution. If this is only specified on a few pods and not all, some pods could get images from insecure registries, possibly leading to vulnerable images being downloaded and ran within your pod.

K05: Inadequate Logging

Logging and monitoring should be used within the cluster in order to keep track of any suspicious activity. Kubernetes has many logging functions, such as Audit logging which logs all actions taken by the API. Along with this, Kubernetes Events indicate whenever any Kubernetes resource has changed state or thrown an error, along with information on what happened.

K06: Broken Authentication

Similar to the OWASP Web Application Top Ten, broken authentication is an misconfiguration found in Kubernetes quite often. This issue is very broad, ranging from Service tokens with excessive lifetimes, to no multi-factor authentication when connecting to the cluster, to no authentication whatsoever.

K07: Network Segmentation

Network segmentation is a necessary part of security, as a lack of segmentation allows for any pod to communicate with any other pod within the cluster. This can cause issues such as allowing an attacker to move laterally across the cluster, gaining more access to unauthorised data.

K08: Secrets Management

Secrets Management is another large issue as, with all configurations of pods being done within YAML files, the storage of secrets such as usernames and passwords is vital to keeping your pod secure. Having the username and password in plaintext within your pod’s YAML file is very dangerous as any attacker that gains access to the YAML file will be able to steal these credentials very easily. A good solution to this is to use Kubernetes’ function to encrypt secrets at REST, which encrypts all secrets within etcd.

K09: Misconfigured Cluster Components

Another Misconfiguration to lookout for is the misconfiguration of the components of the cluster. This includes the services that run the cluster, such as kubelet, which handles all pod deployment and management. Kubelet has many possible very dangerous configurations, such as anonymous authentication, allowing any user to make requests to kubelet without authentication. Along with this, the “Always allow” authentication setting will automatically accept any request, no matter what user it came from, this is the default authorisation method and should be turned off. Other misconfigurations to lookout for are misconfigurations within the etcd service, as this is the secure keystore for Kubernetes, so it is essential to keep it secure.

K10: Vulnerable Components

Lastly, Vulnerable Components are a large issue to look out for. As Kubernetes is a large system, it is key to make sure that there are no vulnerable Kubernetes components in use such as an outdated version of kubectl in use, which could have known CVE’s within it.

Real World Examples

Issues within this top ten have a lot of real-world impact, take one of the biggest car companies in the world for example, Tesla, which got breached due to an insecure Kubernetes layout back in 2018. Hackers found one of Tesla’s K8S admin consoles, which was not secured with a password. This allowed hackers access to all of the pods running within the cluster, and within one of these pods plaintext AWS credentials were found, giving hackers access to Tesla’s S3 buckets. These buckets were then used to mine cryptocurrency for the attackers in a crypto-jacking attack. This issue would fall under Broken Authentication (OWASP K06) and secrets management (OWASP K08), as the admin portal was not secured with a password, and secrets such as the AWS credentials were not stored securely, instead being stored within plain text.

As for more recent attacks, in January of this year, a misconfiguration was found within over 1000 public-facing Kubernetes clusters running within Google’s cloud Kubernetes solution, the Google Kubernetes Engine (GKE). A group named “system:authenticated” exists within the GKE, which was assumed by many to be a group including all authenticated google accounts linked to the organisation, however it was discovered that this group gives roles to anyone with a google account, no matter what organisation they belong to. This becomes incredibly dangerous when overly permissive RBAC is used within this group, for example if this group is given cluster admin roles, it could allow any malicious user with a Google account full access to their whole cluster.

Mitigations

As these issues are broad, many different mitigations are possible, but here is a brief description of basic mitigations for each category:

  • K01: Insecure Workload Configurations – Ensure no pods are running as the root / administrator user.
  • K02: Supply Chain Vulnerabilities – Ensure all components of the K8S cluster are up to date with the latest patches.
  • K03: Overly Permissive RBAC – Ensure that the RBAC system in place users the “Least Privilege” model, allowing the user to access only what is necessary.
  • K04: Policy Enforcement – Ensure that policies are enforced cluster-wide via a centralized policy enforcer such as the Admission Controller.
  • K05: Inadequate Logging – Ensure that all forms of logging are enabled to allow for proper incident response and identification.
  • K06: Broken Authentication – Ensure certificates are not used for user authentication, make sure MFA is used when possible and ensure that authentication methods are in use and implemented securely.
  • K07: Network Segmentation – Ensure that all pods are separated network-wise unless strictly necessary to block lateral movement. If networking is necessary, use NetworkPolicies to restrict networking to only allow for what is absolutely necessary.
  • K08: Secrets Management – Ensure all secrets are Encrypted at rest via etcd.
  • K09: Misconfigured Cluster Components – Perform regular audits on your K8S cluster to ensure there are no security misconfigurations.
  • K10: Vulnerable Components – Ensure all components of your K8S cluster are up to date on the latest security patches.

References

https://owasp.org/www-project-kubernetes-top-ten/

https://www.trendmicro.com/vinfo/gb/security/news/cybercrime-and-digital-threats/tesla-and-jenkins-servers-fall-victim-to-cryptominers

https://www.darkreading.com/cloud-security/anyone-with-google-account-can-hack-misconfigured-kubernetes-clusters