Over the past few years, we have watched AI move from completing lines of code to taking on work we might once have handed to another engineer. Coding agents can inspect a repository, propose an implementation, modify several files, write tests, and return something that looks remarkably close to finished. What has not become equally easy is deciding whether that work belongs in a production system.

Someone still has to understand what the change does, whether it solves the right problem, how it can fail, what it exposes, how it behaves under load, and whether the surrounding system can absorb it. As of today, the code may arrive faster, but the confidence required to move it into production still has to be established separately.

This changes where engineering value is concentrated: less in producing each line of code, and more in understanding systems, verifying outcomes, and operating software safely. It also means AI will amplify the strengths and weaknesses of the engineering environment around it.

The other day, I was talking with some former teammates who now work as software engineers at different companies, and the conversation turned to their relationships with their security teams. A familiar theme quickly emerged around the disconnect between security recommendations and the realities of their day-to-day engineering work. Sometimes it feels like we have come a long way as an industry, but many security teams still operate at a considerable distance from the engineers building and maintaining the systems they are trying to protect.

In this tutorial, we are going to spin up a CockroachDB secure cluster running in Kubernetes with certificates managed by HashiCorp Vault and issued by cert-manager. Before we get to the final state, we are going to evolve the installation step by step to understand how each component is contributing to the setup and what we are gaining.

We’ll focus on the execution, assuming you are minimally familiar with these components. Links will be provided for additional context when necessary.

This tutorial can be executed from your local machine, no cloud resources are necessary.

We all know the benefits of using managed Kubernetes services like GKE, EKS, AKS, etc. Given the complexity of managing the cluster infrastructure and its core components (control plane, auto-scaling, monitoring, networking, storage, etc.), using a managed Kubernetes service is generally the first choice when running workloads in production.

However, in some situations, provisioning and managing the Kubernetes cluster from scratch might be necessary. Specific product features, security & compliance, costs, vendor independency, etc. are some factors that usually justify the decision of running Kubernetes by yourself. Of course, many challenges come with managing a Kubernetes cluster and the discussion around which route - self-managed vs. managed - to take is far from simple. I want to keep this discussion out of the scope of this tutorial since it would require special attention.

Currently, the tool of choice when considering provisioning and managing a Kubernetes cluster is Cluster API. From the docs:

Cluster API is a Kubernetes sub-project focused on providing declarative APIs and tooling to simplify provisioning, upgrading, and operating multiple Kubernetes clusters.

[…] The supporting infrastructure, like virtual machines, networks, load balancers, and VPCs, as well as the Kubernetes cluster configuration are all defined in the same way that application developers operate deploying and managing their workloads. This enables consistent and repeatable cluster deployments across a wide variety of infrastructure environments.

In this post, I want to share the approach I have been using to scope and make decisions when defending a Kubernetes environment. The approach presented here defines ten high-level security concerns and is used with other frameworks and benchmarks to contextualize and guide the security efforts. For each security concern, I present an informal and opinionated description based on my experience.

Infrastructure as Code (IaC) is an essential piece of the modern software development landscape. IaC tools allow teams to define their infrastructure in a reproducible-automated fashion, increasing speed and consistency while preventing errors, misconfiguration, and configuration drifts.

IaC also plays an important role in security. It gives visibility over the whole infrastructure, including its security aspects and components, even before they have been deployed. Development and/or security and operations teams can define sensible security defaults and a security baseline enforcing best practices that are automated and applied via the CICD pipeline, where changes can be reviewed and approved following the separation of duties principle.

In this tutorial, we will deploy Cilium to an Amazon EKS cluster and limit traffic to Pods using Kubernetes Network Policies.

Network Policies are not available in Kubernetes out-of-the-box. To leverage Network Policies, you must use a network plugin that implements such a feature - here is where Cilium comes into play.

It’s worth noting that while this is the scope of this tutorial, Cilium is not limited to Network Policies and not even limited to Kubernetes. Check the Introduction to Cilium to learn more about it and the Kubernetes Integration doc about how Cilium works with Kubernetes.

In this tutorial, we are going to play with the Google Kubernetes Engine Dataplane V2 and check how we can use it along with Kubernetes Network Policies to limit traffic to Pods and to obtain real-time visibility on cluster network activity.

Dataplane V2 is a recent feature in GKE, with GA starting on version 1.20.6-gke.700 as of May 10, 2021. It uses Cilium to process network packets in-kernel using Kubernetes-specific metadata without relying on the kube-proxy and iptables for service routing, resulting in performance improvements. Dataplane V2 brings some exciting features for cluster operations and security, such as:

• Built-in Network Policies enforcement without the need of Calico and;
• Real-time visibility, enabling cluster networking troubleshooting, auditing, and alerting.

Let’s talk about something that most developers and engineering teams don’t pay the necessary attention to. Do you have a process to update your application dependencies? How likely is it that your application could be compromised because of a vulnerable dependency?

In this tutorial, we are going to get a static website up and running with Hugo. We will deploy the site using GitHub Pages and make sure new changes are automatically published using GitHub Actions.

In this tutorial, we are going to configure and explore the HashiCorp Vault AWS Auth method with Amazon EKS. We will start performing the Vault authentication using the EC2 instances (Kubernetes nodes) identity and later we will use a Kubernetes service account to impersonate an AWS IAM Role and have more fine-grained control at the Pod level.

In the source code for this tutorial, we extend the Getting started with Cloud Endpoints for GKE with ESP documentation guide to provide an example of how to configure HTTPS between the LB and the ESP.

In this tutorial, we’re going to go through the Workload Identity feature and see how it helps to improve the way we manage access to Google Services and APIs from applications running in Google Kubernetes Engine (GKE).

Workload Identity is the recommended way to access Google Cloud APIs from within GKE due to its improved security properties and manageability. With Workload Identity you can control access to APIs using Google service accounts and IAM roles without deploying static service account JSON keys to Pods and without relying on the node’s service account.