Hacking Alibaba Cloud's Kubernetes Cluster

Hacking Alibaba Cloud's Kubernetes Cluster with Hillai Ben-Sasson &Ronen Shustin, Security Researchers at Wiz and Bart Farrell, KubeFM Host

Securing Kubernetes clusters is one of the toughest challenges in cloud security, but for Ronen Shustin and Hillai Ben-Sasson at Wiz, it's just another day at work. These top-tier researchers are fearless in diving into the deep end. Their latest exploit? Cracking Alibaba Cloud's Kubernetes clusters through clever PostgreSQL vulnerabilities.

Join Bart Farell as he dives into how their innovative approach identifies vulnerabilities and enhances the overall security of cloud ecosystems.

You can watch (or listen to) this interview here.

Bart: What are three emerging Kubernetes or other tools that you're keeping an eye on?

Hillai: Ronen and I have extensive knowledge of Kubernetes, but our expertise only originates from working directly with Kubernetes. We're hackers who transitioned into Kubernetes hacking, not Kubernetes experts who started hacking. So, we need to familiarize ourselves with many Kubernetes tools. Most of the tools we know are those we've encountered and exploited during our engagements. Therefore, we might not be the best sources for the latest Kubernetes tools, but we are excited about ongoing Kubernetes research.

Bart: Are there any specific tools or infrastructure that you particularly like?

Ronen: Instead of specific tools, we're more interested in infrastructure elements like service meshes. From an attacker's perspective, engaging with these is quite fascinating. Currently, we need to mention standout tools.

Bart: For those unfamiliar, can you tell us more about your roles and what you do at Wiz?

Hillai: Ronen and I work at Wiz, a cloud security company, as part of the vulnerability research team. We focus on researching primary cloud services and providers like Azure, GCP, AWS, and more. We utilize their open bug bounty programs to find and report vulnerabilities. By sharing our findings, we aim to enhance the security of the cloud community, not just for our clients but for everyone.

Bart: Is hacking cloud environments your primary focus, or is this a specialized area within security research?

Hillai: It's unique. We didn't start with cloud environments. We began as general security researchers, focusing on hacking techniques. Over time, we transitioned into specializing in cloud security. Our research aims to discover innovative ways attackers might exploit cloud systems, ultimately leading to more secure cloud environments for everyone.

Bart: How has your hacking experience influenced your approach to Kubernetes security? Did you discover any exciting findings during this research?

Hillai: Many cloud providers rely on Kubernetes and container technology to manage their services efficiently. Traditionally, setting up individual virtual or physical machines for each customer would only be scalable for some companies. Containers offer a more efficient way to manage large infrastructures. Focusing on cloud environments, we discovered Kubernetes as the go-to tool for Alibaba Cloud and companies like IBM. Our journey started with cloud security research and ultimately led us to specialize in Kubernetes security within that domain.

Ronen: Our initial focus was on container security. We researched container escapes and other vulnerabilities that might impact containers. This research naturally led us to Kubernetes, as many infrastructures we encountered used it. We had to learn Kubernetes and develop specific techniques to achieve our goals.

Bart: If you could go back in time and share one career tip with your younger self, what would it be?

Hillai: Always follow your curiosity. Research is all about pursuing leads and hunches. We were curious about cloud security, even though we didn't start in that field. It became popular, and we wanted to explore this new area.

Bart: What resources do you use to stay updated on Kubernetes?

Ronen: I rely on technical documents the most. I also follow blogs from cloud providers, mainly the CNCF blog, because they have valuable information. I use The Kubernetes community on Twitter to learn about new features and technologies; they are highly active there.

Hillai: Additionally, I recommend Reddit. Many communities focused on security, Kubernetes, and cloud computing offer great content.

Bart: We came across an article about how you hacked Alibaba Cloud's Kubernetes cluster and a talk you gave at KubeCon. What motivated you to do this research, and did your company support you?

Hillai: Our company supports security research. At Wiz, we focus on cloud security research, often utilizing offensive security methodologies. We act like attackers to find vulnerabilities and then report them to the vendors. By identifying vulnerabilities, we can report them to the cloud providers and prevent actual attacks. Alibaba Cloud is just one example of this engagement.

Ronen: Our research often leads us to discover new hacking techniques we need to learn about. We share these discoveries with everyone so they can protect themselves.

Bart: One of our previous guests talked about Kubernetes secrets management and threat modelling. How do you approach exploiting vulnerabilities from a hacker's perspective?

Ronen:Our best security insights come from working with different applications, frameworks, and cloud systems. When we engage with one, our primary goal is to find critical security mistakes in its setup. To do this, we must fully understand how the system works and where attackers might discover weaknesses.

Hillai: There's an interesting difference between traditional and cloud security research. In traditional research, the goal is often to achieve "Remote Code Execution" (RCE) on a specific application, which means taking control of a machine and running unauthorized code. However, in the cloud, things are different. Since you often have access to a virtual machine yourself, RCE becomes less attractive.

The real challenge in cloud security lies in breaching the barriers between different customers. Unlike traditional environments, the cloud is a shared space with hundreds of thousands of users. Our focus is to demonstrate the possibility of attackers moving between these customers, even without data access. This risk highlights a unique cloud security risk - the potential for attackers to "jump" from one user to another and compromise their information. This type of research, proving a breach of trust without actually stealing data, is a crucial aspect of cloud security and something rarely seen in traditional security research.

Bart: When starting this research, why did you choose Alibaba Cloud?

Ronen: Our initial study focused on PostgreSQL. Since many cloud providers offer managed PostgreSQL instances, we were interested in how they handle the infrastructure. We discovered vulnerabilities that allowed us to execute code on these instances. We tested several providers, including Alibaba, and presented our findings at the Black Hat talk.

Hillai: We began with PostgreSQL and expanded to Alibaba and other cloud providers. Our blog post provides more details about PostgreSQL and our Black Hat talk.

Bart: Why did you choose to focus on PostgreSQL for your research?

Ronen: PostgreSQL is a robust database with many features, including the ability to execute code within the database. While this capability can benefit certain users, it poses a potential security risk in cloud environments.

Cloud providers typically modify PostgreSQL to prevent users from executing code on their managed instances. However, our research identified vulnerabilities in these modifications, not in the core PostgreSQL code itself. We were able to exploit these vulnerabilities to bypass the restrictions and still execute code on the managed databases.

Bart: How does PostgreSQL relate to Kubernetes in this context? Did you find a way to access a Kubernetes cluster by exploiting the PostgreSQL vulnerabilities?

Hillai: Cloud providers often use containers and orchestration tools like Kubernetes to manage large-scale services, including PostgreSQL. This approach allows them to offer these services to many customers efficiently. While exploiting the PostgreSQL vulnerabilities, we discovered that we were actually in a Kubernetes environment. The user interface typically abstracts away the underlying infrastructure from the user, but our research methods disclosed it.

Ronen: We've seen various infrastructures, but Alibaba and IBM used Kubernetes for their managed services. Other providers might use different implementations.

Bart: Security experts often talk about avoiding vulnerabilities caused by misconfigurations, which can be human errors. What were the biggest misconfigurations you found that created security risks?

Hillai: The biggest misconfiguration we found is treating containers as the only security barrier. It's important to remember that containers can be a security layer within a more extensive security system, but they should be relied on only partially. Containers alone wouldn't be strong enough to isolate each company's data from each other entirely because any security flaw in the core Linux system (the kernel) could bypass container security. We were able to exploit such misconfigurations during our research.

Another problem is poorly managed secrets within the Kubernetes environment. These secrets could read information across the system and write and change it, which meant we could overwrite software packages used by many cloud services and customer accounts within Alibaba. Essentially, these powerful secrets allowed someone to access different environments, services, and customer data—all with a single key. That's a significant security risk we wouldn't recommend taking.

Ronen: The specific secret we found was the image pull secret. In Kubernetes, when you want to download images from a private registry, you need this secret to configure network access. If you misconfigure it, you might accidentally include a secret key with push permissions instead of pull permissions. This key should only allow downloading images, not uploading them. If an attacker gains access to a key with push permissions (like what we achieved in Alibaba), it could have devastating consequences for your entire environment.

Bart: To those without a strong background in security, it may seem that security experts click a button, scan your system, and find vulnerabilities. However, security research, like many other fields, is a blend of art and science. Can you elaborate on this further?

Hillai: Security research requires a lot of creativity. When you hear about a new attack vector, it boils down to creative thinking - coming up with something no one else has considered. In this research, we started by looking for patterns we already knew were risky, like overly permissive settings and shared volumes. We had to think outside the box. Returning to the Alibaba Cloud control panel, we began experimenting. This exploration led us to a breakthrough when we discovered a button enabling SSL encryption for the PostgreSQL instance. Clicking it triggered new activity in the container, which we followed to escape the container.

Bart: To help our audience understand, could you explain SCP, its role in the attack, and how you exploited it?

Hillai: SCP stands for Secure Copy. It's a standard tool on Linux systems that transfers files between machines using secure SSH connections. In our case, the SSL encryption feature we triggered used a new Alibaba management container. This container ran the SCP command on our container to move the SSL certificate.

SCP reads its configuration from a directory we control within our container by default. We placed a malicious SSH configuration file there. When the SCP command loaded this configuration, it ran a command we placed within the file. This trick let us escape our limited container and jump to the Alibaba Management Container because it unknowingly executed our command.

Ronen: A crucial factor in this exploit was the shared volume. This volume acted like a shared home directory for our container and the management container since the same user existed in both containers. We could exploit this shared space because SCP reads its configuration from the user's home directory by default. By replacing the default configuration with ours containing a malicious command, we tricked the management container into running it when it used SCP.

Bart: What does successfully creating a privileged container using the Docker API tell us about cloud security in general?

Ronen: Many cloud environments rely on Docker to manage their containers. You can create a new container through an HTTP request if you gain access to the Docker API socket. This container could be privileged, meaning it shares resources like namespaces and possibly even volumes with the underlying host machine, the Kubernetes node. Spawning a privileged container grants you access to almost everything the node has access to.

Hillai: You transition from being a guest in the container to gaining complete control of the host machine.

Bart: Gainin access to the node would only give you control of some of the Kubernetes clusters, would it?

Ronen: With code execution on the node, we could use Kubelet credentials to explore further, looking for commands, codes, secrets, and other information. In our case, Alibaba had misconfigured its Kubelet credentials: it was too powerful. We could list all pods, see all the code in the cluster, potentially containing customer data, and even retrieve all the secrets using the "kubectl get secret" command. This misconfiguration was the key that unlocked broader access for us.

Bart: Did you achieve the entire exploit on a single node within the cluster?

Ronen: Yes, we were on a single node. Using the compromised Kubelet credentials, we could see all the other nodes and resources in the cluster.

Hillai: While the specific node we compromised was isolated and didn't contain data from other customers, the service account associated with Kubelet had excessive permissions. Even though the node itself was secure, this service account allowed us to access sensitive information across the entire cluster, including pods, nodes, and secrets belonging to other customers.

Bart: What was the next step after taking over Alibaba's managed PostgreSQL offering? Did you contact Alibaba to report your findings?

Hillai: Once we discovered the ability to access data belonging to other customers, our research stopped immediately. We wouldn't risk even accidentally accessing someone else's data. At that point, we documented everything we found and sent a detailed report to Alibaba Cloud, and they responded quickly and professionally. They kept us updated on the fixes they deployed throughout the research process. We immediately report any critical issues to prevent others from exploiting them.

Bart: Can you tell us about any specific fixes they implemented based on your findings?

Ronen: The first issue was a misconfiguration that falsely indicated increased resource consumption. We exploited it to execute unauthorized code on the operating system. We collaborated with Alibaba Cloud to fix this problem. They also resolved the SCP vulnerability problem that allowed unauthorized access to their management container. Finally, they restricted the Kubelet permissions to a narrower scope, granting only specific permissions.

Hillai: Following our research, Alibaba took several steps to address the vulnerabilities we discovered. They limited image pull secret permissions to read-only access, preventing unauthorized uploads. Additionally, they implemented a secure container technology similar to Google's gVisor project. This technology hardens containers and makes them more difficult to escape from, adding another layer of security.

Bart: Throughout this process, what key lessons did you learn?

Hillai: There are two main lessons learned. First, containers shouldn't be relied on as the sole security barrier. While they can be a layer of security, they can be bypassed in various ways. Additional precautions are crucial to ensure proper isolation between customers. We recommend building a layered defense so that a single vulnerability doesn't allow unauthorized access to a competitor company's data.

Second, strong credentials require careful management. As Ronen mentioned, Alibaba originally had a powerful secret that could be read and written across the cluster. This secret also had push access to the central Docker image registry. Following our report, they limited the scope of these credentials. It's essential to be very cautious with such powerful secrets. Ideally, you should scope the secrets to specific actions and minimize them whenever possible. A powerful secret can allow attackers to move across different environments, including production, development, testing, and even development workstations.

Another lesson learned relates to the container itself. The SCP vulnerability we exploited highlights the risk of shared namespaces between containers. In the Alibaba incident, the shared namespace and home directory allowed us to exploit the SCP vulnerability. Always be very careful when sharing namespaces between trusted and untrusted containers. The lesson learned is to minimize what you share and never grant unnecessary permissions. Attackers may exploit even seemingly minor misconfigurations.

Bart: Can you recommend any specific tools that people might need to be aware of if they want to discuss implementing some of these mitigation tactics with their managers?

Hillai: There's one framework I highly recommend: Peach. It's an open-source project developed by our research team and contributions from fantastic people at many companies.

Peach is a framework that outlines how to build secure and isolated environments, whether in the cloud or not. Like a white paper, it's a valuable resource that guides you on properly isolating tenants or customers in a multi-tenant environment. It covers common mistakes to avoid, what to look out for, and how to implement the necessary precautions.

If you manage a multi-tenant environment or need to isolate resources within your environment, Peach is a valuable resource worth exploring. It covers the common mistakes to avoid and offers best practices for implementing protection. It's completely open-source and available on GitHub. We also welcome contributions from anyone with additional tips or tricks we might need to know.

Ronen: I also recommend using secret scanning tools. These tools are essential in our research; we use them to identify potential secrets-related vulnerabilities.

Bart: Do you have any recommendations for securing multi-tenant Kubernetes clusters?

Ronen: Securing multi-tenant Kubernetes clusters involves a few key areas. First, prioritize network security. By default, Kubernetes doesn't restrict node communication, so strong network isolation is essential.

Second, separating namespaces between customers is a good practice when dealing with multi-tenancy.

Additionally, consider implementing container security technologies like gVisor or Kata Containers. Don't solely rely on Docker's security features to prevent container escapes.

Bart: What advice would you give for hardening containers to make them more secure?

Ronen: Our case study with Alibaba revealed they were using shared Linux namespaces between containers, such as their management container and our container. Sharing Linux namespaces can be dangerous. When designing a system that shares namespaces or resources between management and regular user containers, constantly carefully assess and be aware of the risks involved. Container technologies like GVisor and Kata Containers can mitigate the risk of attackers exploiting Linux kernel vulnerabilities in your environment to achieve kernel-level code execution and jump to the Kubernetes node.

Bart: What advice would you give to Kubernetes engineers needing more security experience?

Hillai: Security is crucial. Companies of all sizes, from startups to large corporations, are constantly targeted by malicious actors, not just ethical hackers like us. Anyone managing a service on the internet must understand that they are a potential target for cyberattacks. These attacks range from data breaches to ransomware attacks that turn off your entire operation. Even small projects need to pay more attention to security.

The good news is that many tools can help you achieve security without being a security expert. Tools like gVisor are relatively easy to implement because you don't need to write them from scratch. By using security hardening tools, you gain significant protection benefits.

Ronen: Besides the tools, many online resources are available to learn about security. These resources can help you understand security risks and how to mitigate them. Kubernetes itself has built-in security features, including default security policies. Be security-conscious and take steps to secure your environment.

Bart: You discover a vulnerability and report it to the vendor. What prevents you from exploiting the vulnerability for malicious purposes instead? Wouldn't Alibaba eventually find the problem on its own?

Ronen: We started seeing signs that Alibaba was taking steps to address the issue while we were still in the research phase. They were transparent with us about their efforts. Cloud providers all have security teams that constantly monitor their environments. They likely knew we were there.

Hillai: Cloud providers are doing a great job with security. We're ethical hackers; our goal is to improve security for the cloud community. Penetration testing, or offensive research, is a tool to achieve that goal. We want to fix the vulnerabilities, and it's rewarding to hear that our reports lead to security updates that benefit many customers. We do this to make cloud products more secure and help users learn how to secure their deployments.

We publish blogs and give talks so that security professionals and developers can learn from our research and identify potential problems in their environments.

Bart: What's next on the agenda for you both?

Hillai: We're always working on new research projects. Sagi from our team recently published a blog about a vulnerability in Hugging Face, an AI provider. We have several ongoing projects under disclosure, meaning we can only reveal them once we fix the vulnerabilities.

Follow our blog; it's the first place we announce new findings.

Ronen: Our research will benefit the Kubernetes security community as well.

Bart: How can people contact you if they have questions?

Hillai: We're both on Twitter. My handle is @hillai, and Ronen's is @RonenSHH. You can also email us at research@wiz.io, but Twitter is the best way. Make sure to spell the names correctly.

Wrap up

If you enjoyed this interview and want more Kubernetes stories and opinions, visit KubeFM and subscribe to the podcast.

• If you want to keep up-to-date with Kubernetes, subscribe to Learn Kubernetes Weekly.
• If you're going to become an expert in Kubernetes, look at courses on Learnk8s.
• If you want to keep in touch, follow me on Linkedin.