Kubernetes Security: The Complete Guide

What is Multi Tenancy in Kubernetes?

Kubernetes is the de facto standard in container orchestration, and is used by organizations large and small to manage container workloads. However, while Kubernetes is extremely powerful, it also raises new security concerns that traditional approaches and tools cannot solve.

According to a 2020 StackRox survey, 86% of organizations running containers manage them with Kubernetes, but over 50% feel they do not invest enough in container security, and are concerned about security affecting their production schedule or resulting in damaging security breaches.

Kubernetes workloads are vulnerable to several types of security threats, including:

• Compromise of the Kubernetes control plane—critical components like etcd and the API Server are not sufficiently secured by default. An attacker gaining access to a Kubernetes master node can gain control of an entire cluster.
• Compromise of nodes and pods—an attacker can gain access to a physical host running several Kubernetes pods, or to an individual pod, enabling exfiltration of data within the pods, and lateral movement to other parts of the cluster.
• Compromise of network connections—unless Kubernetes is configured securely, containers and pods may be able to freely connect to each other, and may be exposed to the public Internet. Any such open connections are an entry point for attackers.
• Compromise of containers—individual containers can become vulnerable, due to a vulnerability or backdoor in a container image, or a misconfiguration. Containers with excess privileges can allow attackers access to the physical host.

Read on to better understand these threats and learn key considerations on how to secure your Kubernetes clusters.

This is part of our series of articles on Kubernetes storage.

In this article, you will learn:

• Kubernetes Security Concepts
  • The Four Cs of Cloud Native Security
    • Pod Security Policies
    • Pod Security Context
    • Secrets
  • Kubernetes Security: Key Considerations
    • Hardening and Compliance
    • Balancing Security and Agility
    • Helm Charts for Day 1, Operators for Day 2
  • Kubernetes Security Best Practices
    • Isolate Kubernetes Nodes
    • Leverage Process Whitelisting
    • Compare Runtime Activity in Similar Pods
    • Enable Role-Based Access Control (RBAC)
  • Kubernetes Security Q&A
    • Why is a Zero Trust Approach Important to Securing Kubernetes?
    • Common Open-Source Kubernetes Security Tools?
  • Kubernetes Storage with Cloudian

Kubernetes Security Concepts

Here are a few key concepts you need to know as you plan your Kubernetes security strategy.

The Four Cs of Cloud Native Security

In the official Kubernetes documentation, the makers of the platform suggest an approach called 4C for cloud-native security:

• Code
• Container
• Cluster
• Cloud

At each level, security best practices should be followed. There may be different security considerations at different levels. For example:

• At the code level, you can ensure all data being transferred is encrypted
• At the container level, you can restrict access to network ports
• At the cluster level, you can harden cluster components like etcd
• At the cloud level, you can set up robust identity and access management (IAM)

These are just examples of security concerns that may exist at each level of the 4C model

Pod Security Policies

There are three default security policies for Kubernetes pods. To properly protect your environment, you need to understand the tradeoff between availability and security.

• Privileged—the highest level of authority. If the pod is compromised, this may allow an attacker to escalate privileges. Baseline—prevents escalation of known privileges.
• Restricted—the recommended setting according to Kubernetes pod hardening best practices.

Pod Security Context

The pod security context configures security settings at runtime. This includes aspects such as access control, whether to run in privileged mode or not, and mounting read-only file systems.

The difference between security policies and the security context is that policies define security parameters, and the context is what executes policies at runtime.

Secrets

You should never put an authentication token or password in a YAML configuration file. Kubernetes provides a mechanism called secrets, for storing this type of sensitive information.

Keep in mind that Kubernetes secrets are not secured by default—you can and should enable encryption for secrets, and set RBAC rules defining who can read or write the secret.

Kubernetes Security: Key Considerations

Here are a few considerations for building a secure Kubernetes environment.

Hardening and Compliance

The Kubernetes platform is extremely rich in features, some activated by default and some not. You could use security benchmarks, such as those published by the Center for Internet Security (CIS) however, these checklists, while extensive, do not take into account the performance impact of some settings, or the relative importance of different security controls.

Hardening Kubernetes is an exacting process. Clusters must be configured to ensure API server access over HTTPS, the use of X.509 certificates for communication and authentication, etcd datastore encryption, and more. Every configuration change should be tested to determine its production impact.

Balancing Security and Agility

Security automation from day one almost always reaps dividends. CI/CD pipelines should employ automated unit and functional tests, and should integrate vulnerability scanners and other automated security gates. To facilitate this, DevOps teams should enable AppDev teams to uniformly integrate monitoring, alerting, and logging services, while deploying microservice-based applications.

In addition, while adding a service mesh to a Kubernetes cluster may seem complex, it actually improves the visibility of a product’s business logic. Previously, developers needed to build the logic into their code. Now, these capabilities can be made available as part of the Kubernetes platform, paring down development activities and accelerating the delivery of microservices.

Helm Charts for Day 1, Operators for Day 2

Helm Charts is an open-source tool used to automate the provisioning and configuration of Kubernetes applications. They are primarily useful for Day 1 operations. For Day 2 operations, Kubernetes Operators are the preferred option.

Operators assist in packaging, deploying, and managing complex Kubernetes applications, by encoding practical human expertise into dynamic software instructions. This lets application developers deliver Kubernetes-run services with automated lifecycle management. If required, a Kubernetes operator can contain a Helm Chart for synchronic deployment.

Operators also help maintain supported configurations for Kubernetes services, improving security by resetting an unsupported service configuration to its initial, declared configuration.

Kubernetes Security Best Practices

Here are some practices that improve the security of Kubernetes clusters.

Isolate Kubernetes Nodes

Kubernetes nodes must be on a separate network and should not be exposed directly to public networks. Preferably, they should not be exposed to the broader corporate network either.

To achieve this isolation, you must separate Kubernetes control and data traffic. Otherwise, both types of data flow through the same pipe, and open access to the data plane implies open access to the control plane. Ideally, nodes should be configured to only allow connections from the master node on the specified port, managed through the network access control list (ACL).

Related content: read our guide to Kubernetes multi tenancy

Leverage Process Whitelisting

Whitelisting starts by observing an application’s normal course of behavior, and drawing up a list of known good processes. This becomes a whitelist that defines which processes are allowed to run, excluding other, unexpected processes. This is challenging to do at large scale; commercial container security products can help operationalize runtime process analysis.

Compare Runtime Activity in Similar Pods

When replicating containerized applications—whether to increase availability, fault tolerance, or scaling—replicas should behave identically. Any significant deviation warrants examination.

When integrating Kubernetes security tools with security systems, such as Security Information and Event Management (SIEM), or collaboration tools like Slack, use deployment labels or annotations to alert the relevant teams to potential threats.

Enable Role-Based Access Control (RBAC)

Role-Based Access Control (RBAC) regulates access and permissions to the Kubernetes API. It is enabled by default in Kubernetes as of version 1.6. Configuration settings, however, should be checked after upgrading. Legacy Attribute-Based Access Control (ABAC) should be disabled when enabling RBAC.

After enforcing RBAC, namespace-specific permissions should be favored over cluster-wide permissions. Cluster-admin privileges should only be granted on a case-by-case basis, even while debugging.

Kubernetes Security Q&A

Why is a Zero Trust Approach Important to Securing Kubernetes?

Zero trust is a new security model, which assumes there is no security perimeter—all entities both inside and outside the corporate network are considered hostile. And any user or device must be securely authenticated, authorized, and monitored.

By default, all Pods in a Kubernetes cluster can communicate with each other. Taking a zero trust approach is important to preventing a range of attacks on Kubernetes infrastructure. To ensure the security of your cluster, set network policies to prevent containers from communicating with each other, unless explicitly allowed.

Common Open-Source Kubernetes Security Tools?

Kubernetes security tools commonly used include:

• Kubernetes Network Policy—provides automated IP address-based segmentation.
• Istio—a service mesh that includes routing, authentication, and encryption. Istio is not in itself a tool for detecting attacks or threats, but can assist with visibility.
• Grafeas—a tool to audit and manage the software supply chain.
• Clair—enables image vulnerability scanning but does not provide workflow support or registry integration.
• Kubernetes CIS Benchmark—a tool to examine the auditing and compliance of Kubernetes security as defined by the CIS benchmark.

Related Articles: Kubernetes Multi Tenancy and Kubernetes Storage Solutions

Kubernetes Storage with Cloudian

Containerized applications require storage that’s agile and scalable. The Cloudian Kubernetes S3 Operator lets you access exabyte-scalable Cloudian storage from your Kubernetes-based applications. Built on the S3 API, Cloudian lets you dynamically or statically provision object storage with this lightweight Operator using S3 APIs. You get cloud-like storage access in your own data center.

Cloudian’s key features for Kubernetes storage include:

• S3 API for Application Portability—eliminates lock-in and enhances application portability. Provides fast, self-serve storage access using the standard Kubernetes Persistent Volume (PV) and Persistent Volume Claim (PVC) methodology to provision assets.
• Multi-tenancy for Shared Storage—lets you create separate namespaces and self-serve management environments for development and production users. Each tenant’s environment is isolated, with data invisible to other tenants. Performance can be managed with integrated quality of services (QoS) controls.
• Hybrid Cloud-Enabled—males it easy to replicate or migrate data to AWS, GCP, or Azure. Data stored to the cloud is always stored in that cloud’s native format, meaning it’s directly accessible to cloud-based applications, with no lock-in.

Learn more about the Cloudian Kubernetes solution