.png)
As more businesses are moving from the traditional development pipeline to a more dynamic DevOps model, security is quickly becoming a forgotten element. This is leaving modern software delivery approaches with a dilemma - do you fit security into your DevOps setup, or do you keep your security team as a separate entity?
What is DevSecOps?
That’s where DevSecOps comes in. DevSecOps is, quite simply, the integration of security into your DevOps. This covers both ensuring that security is an ongoing consideration during the continuous development cycle and that security engineers are included in that cycle to ensure that security is built-in to every project and setup.
Thomas Fricke, Cloud Security Architect and founder of Endocode, joined us in September 2021 to discuss why security needs to be part of your DevOps setup, the security flaws within Kubernetes, and how Istio can help you to keep your pipeline secure.
Kubernetes security
As any cybersecurity expert will attest to, every new asset added to a businesses’ IT infrastructure weakens the overall security structure of a business. When these new assets are not properly secured, they offer attackers a way into the network.
The problem with Kubernetes is that using it introduces a range of insecure vulnerabilities that can easily be exploited if they’re not handled. Given that Kubernetes pods need to communicate with each other, one compromised pod, cluster, or even container can result in a catastrophic network breach.
Unfortunately, the default configuration of Kubernetes pods, clusters, and containers doesn’t take this insecurity into account. By default, every pod can talk to all other pods within the network until it’s manually assigned a network policy. So, without careful management of every pod on the network, it’s easy for security to fall by the wayside.
Security and cloud-based Kubernetes
Thomas illustrates this with an overview of how a Google Kubernetes Engine - or GKE - can be set up with pod security in mind.
“What you can do is set up a custom controller to expose transport layer security services, which are exposed through this controller in the cloud. This can give you a local answer over how secure this out-of-the-box cloud service is”, explains Thomas.
Because most out-of-the-box Kubernetes solutions will usually run on default settings, using a custom controller in this way can help engineers to fix security issues as they arise.
This will also expose whether engineers need to implement specific ingress rules and network policies like Calico, Weave, and Istio to control the traffic flow between pods, clusters, and containers.
With this in place, engineers can run simple scripts to understand what namespaces are running, which ports are active, and what privileges exist on the network. Because privileged ports will expose the container network interface, or CNI, security teams need to know which ports can expose the entire network.
Here, Thomas demonstrates a simple one-line script for exposing insecurities within the service account:
“This doesn’t cover the underlying layers of namespaces or privileges, but this script checks for the security of Kubernetes that it can’t orchestrate on its own,” Thomas states.
For pod security, Thomas offers another template that can expose each namespace being used by each pod, including what ports those namespaces are using and the security context for each of those ports.
Using Istio for Kubernetes network security
Many cybersecurity experts are becoming increasingly wary of cloud-based services because of the inherent security risks associated with outsourcing computing to third-party providers. The trust nothing philosophy - where every connection between services is manually approved, implemented, and centralized - is becoming a significant part of DevSecOps where cloud-based computing is a key part of the development pipeline.
Service meshes, like Istio, provide a firewall-like layer of security between a cloud-based Kubernetes provider and a business’s network. This service, in particular, is popular because it’s a one-click GKE security solution. But is it secure?
The way Istio works is that it adds a “sidecar” to every pod, which acts as a proxy for all incoming and outgoing traffic.
“This means that Istio has to change the network rules, so that all of the traffic inside the cluster is directed through the Istio envoy proxy. Because all of the traffic goes through the Istio envoy proxies, you can completely control all of the traffic”, Thomas adds.
Unfortunately, this also adds additional security issues. To allow engineers full control of traffic on the network, each Istio container needs to have network admin privileges. While these containers can be terminated quickly if a breach is detected, having more containers on a network with these privileges does add significant security risk to the network.
Not only that, but Istio security settings can be changed from within a pod. So, once again, it only takes a single compromised pod to potentially lead to a hostile network breach, and a compromised Istio envoy container gives an attacker a direct way into the network admin settings.
The best way to mitigate this security risk is by installing Istio to the container network interface (CNI) rather than to pods directly.
The CNI doesn’t exist in regular user space, and it can only be accessed by privileged users. By installing Istio so traffic is controlled through the CNI, rather than through pods directly, engineers can mitigate some of the security risks associated with allowing all users the same level of access.
“It’s not hard to implement,” Thomas explains, “but it takes a lot of time. It only takes one mistake to take over the entire cluster from inside an application”.
Given the scope of this security flaw, Kubernetes users must understand and advocate for security audits to be performed on any out-of-the-box Kubernetes service.
It’s worth noting that Istio is aware of this security flaw in its service and has since released v. 1.2 to fix the issue. However, this version isn’t available everywhere, so if you want to use Istio, you need to ensure you’re installing v. 1.2 or above.
Implementing Kubernetes security in DevOps
All of this leads to a strong argument that security needs to be integrated within DevOps workflows. Given the nature of Kubernetes and that new containers, clusters, and pods can be spun up at a moment’s notice, security needs to be an integral part of this dynamic development environment.
“DevSecOps means you’re adding security engineers to your DevOps team and not just security tasks to your existing DevOps team and overloading them. Sometimes you might think developers can also do security, but this will overload them. You need to be wary of adding too much burden to your existing teams”, explains Thomas.
By adding security engineers to your DevOps team, businesses ensure that the current DevOps engineers aren’t overburdened by a new set of requirements that they may not have available cognitive load to handle.
DevSecOps on Kubernetes: In summary
Unfortunately, you can’t rely on out-of-the-box security with Kubernetes cloud services, regardless of how much money you spend on having that service provided for you. While Kubernetes is a powerful foundation to have in your cloud infrastructure setup, there are several security flaws that can weaken your network security.
Not only do Kubernetes users need to advocate for security protocols to be taken seriously with these cloud services, but organisations also need to begin integrating security aspects into their DevOps workflows. As we move towards a continuous development process, security checklists and compliance protocols also need to be made dynamic to keep the network and end product secure.
Thanks again to Thomas Fricke for sharing his insights into DevSecOps and Kubernetes. If you haven’t watched the full meetup, you can find the recording here.
