Improving Security for Kubernetes Deployments at Scale

by Carlo GutierrezSeptember 10, 2018

Over 21,000 systems were found open on the Internet. Out of those, more than 300 required no authentication whatsoever to access.

Table of Contents

Security lags behind speed and scalability

Since its initial launch in June of 2014, Kubernetes, a container orchestration system, has gradually risen in popularity over the years. One of the reasons for this growth in adoption is the increased rate of application development and ease of infrastructure scaling it provides.

Another factor that has contributed to the rise of Kubernetes is the ability to deploy Kubernetes in the cloud. While it can still be deployed on premises, cloud-based installations enable organizations to forego the costs associated with having and maintaining their own infrastructure.

The problem with cloud-based Kubernetes deployments, however, is that these require a different approach when it comes to security.

At a recent Kubernetes meetup in Silicon Valley, Dan Hubbard, Chief Product Officer at Lacework, revealed some of the vulnerabilities with Kubernetes workloads in the cloud and also shared some recommendations to improve security.

“Kubernetes, like a lot of these new technologies, have this speed and scale, as well as secure and safe balancing act. Speed and scale win.” —Dan Hubbard, Lacework

Dan Hubbard at the meetup in Silicon Valley (Photo credit: Altoros)

How bad is it?

According to the research conducted by Lacework in June 2018, over 21,000 container orchestration and API management systems were discoverable on the Internet. While being visible on the Internet isn’t always a vulnerability, it is certainly an unnecessary security risk as this becomes a “window of opportunity” that malicious attackers can take advantage of.

Kubernetes accounted for 76% of the discovered systems (Image credit)

Lacework conducted their research using Shodan—an engine that encapsulates SSL data mining, web crawling, and other functionality to identify potentially vulnerable systems.

“With simple tools and a relatively simple query, I can find all kinds of vulnerable things.”
—Dan Hubbard, Lacework

While Kubernetes does include built-in security features, such as SSL and default authentication, the Lacework team encountered glaring issues in some of the Kubernetes systems they found, including:

open dashboards in the middle of configuration
open dashboards that required no authentication
open dashboards that could be brute-forced
information disclosure on nodes with invalid certificates

In cases where the dashboard was completely exposed, malicious attackers can gain full remote administration of the corresponding systems. When this happens, attackers have full access to perform such operations as adding applications, modifying infrastructure, changing credentials, extracting data, etc.

Some examples of the exposed dashboards (Image credit)

The Lacework team also found 38 instances of healthz, a Kubernetes container health check service, which required no authentication whatsoever.

“We found roughly 21,000 systems open on the Internet. That doesn’t mean 21,000 are insecure. Many of them could have full hardware tokens built-in or other forms of security. Of those, we found a little over 300 with no username or password.” —Dan Hubbard, Lacework

During Lacework’s search, Dan made it clear that they did not attempt to access any of the systems that were open. Instead, they reached out to the nodes that were potentially exposing critical data.

How can security be improved?

After exploring the vulnerabilities uncovered, Lacework came up with some general recommendations on improving security:

Regardless of the network policy in place, use multi-factor authentication for all access
Apply strict controls to network access, especially for UI and API ports
Use SSL for all servers and use valid certificates with proper expiration and enforcement policies
Deploy VPNs or reverse proxy to enable connection to sensitive servers

Dan also shared a few security tips specific for Kubernetes:

Build a pod security policy by preventing pods from running as root, as well as from accessing host ports and certain volume types
Configure your Kubernetes pods to run read-only file systems
Restrict privilege escalation in Kubernetes with role-based access control

Example pod security policy (Image credit)

For organizations without a security team, Dan suggested the use of existing products and services, such as Lacework’s Polygraph—which constitutes a set of behavioral maps—in order to manage container security. Polygraph is a security platform for cloud deployments, which is capable of capturing relevant activities, organizing data logically, establishing a baseline of container behavior, identifying deviations in container behavior, and delivering actionable insights.

“We really believe that security needs to change with modern deployment methods. It’s not about bringing an appliance into the cloud, virtualizing something, and pushing it out, or taking a normal piece of software that’s designed for on premises and putting it in the cloud. It’s about building and designing purpose-fit technologies for your public cloud.”
—Dan Hubbard, Lacework

Lacework Polygraph dashboard (Image credit)

With so many companies looking to speed up and scale their development life cycle, it’s not a surprise to see the increased adoption rate of container orchestration systems like Kubernetes. That said, security shouldn’t be an afterthought and should always be part of the process.

Want details? Watch the video!

Table of contents

Who are we? (1’18”)
What is Kubernetes? (2’48”)
Why are monocultures hard to secure? (9’47”)
Demo: Shodan (14’40”)
Lacework research statistics (26’20”)
What are the risks and threats? (29’20”)
What are the security recommendations? (31’00”)
Questions and answers (36’03”)

About the expert

Dan Hubbard is Chief Product Officer at Lacework, where he is responsible for driving the company’s product and security strategy for public and private clouds and security research. Dan’s expertise spans from reputation and advanced classification systems to large-scale security data mining and cloud security. Prior to Lacework, he served as CTO at OpenDNS, helping to deliver the world’s largest cloud security network that led to the $600M acquisition by Cisco. Before that, Dan was also CTO at Websense, where he led R&D, launched the Websense Security Labs, and was instrumental in the company’s success from early days through successful IPO. Dan owns several patents in the areas of data classification and cloud security and is a frequent speaker at security conferences globally.