Using Network Policies in EKS with Cilium
In this tutorial, we will deploy Cilium to an Amazon EKS cluster and limit traffic to Pods using Kubernetes Network Policies.
Network Policies are not available in Kubernetes out-of-the-box. To leverage Network Policies, you must use a network plugin that implements such a feature - here is where Cilium comes into play.
It’s worth noting that while this is the scope of this tutorial, Cilium is not limited to Network Policies and not even limited to Kubernetes. Check the Introduction to Cilium to learn more about it and the Kubernetes Integration doc about how Cilium works with Kubernetes.
Prerequisites
To follow this tutorial, you need an AWS Account and credentials to create and manage an EKS cluster properly provisioned and installed in your local environment. You also need the eksctl and kubectl CLIs installed.
Check the Getting started with Amazon EKS – eksctl Prerequisites section if you need help.
Create the EKS cluster
Quick note about minimal configuration
Note that in this tutorial (and all other tutorials in this blog, unless said the contrary), we will use minimal configuration to get things running quickly and focus on the feature we are evaluating. Never use this configuration to set up a production environment. Refer to the official installation guides linked in the tutorial for more thorough instructions.
Declare some variables that we’ll use throughout the tutorial.
cluster_name=le-cluster
cluster_region=us-east-1
Create the cluster.
eksctl create cluster --name $cluster_name --region $cluster_region
Check the eksctl docs for more details about creating and managing EKS clusters.
Verify that your kube-config is currently pointing to the created cluster.
kubectl config current-context
You can also check the nodes for more details about your installation.
kubectl get nodes -o wide
Install Cilium
Download the Cilium CLI.
curl -L https://github.com/cilium/cilium-cli/releases/latest/download/cilium-darwin-amd64.tar.gz \
-o cilium-darwin-amd64.tar.gz ; tar xzvf $_
Run the install command.
./cilium install
[...]
Auto-detected IPAM mode: eni
Auto-detected datapath mode: aws-eni
Custom datapath mode: aws-eni
[...]
The cilium install command uses your current kube context to detect the best configuration for your Kubernetes distribution and perform the installation. As you can see in the partial output listed above, Cilium is being installed on AWS ENI mode. It’s also possible to use Cilium with the AWS VPC CNI plugin - refer to the docs for more details.
When the installation finishes, you can check that Cilium is running with the CLI using:
./cilium status --wait
and validate the installation with:
./cilium connectivity test
To manually check that Cilium is running you can also check that the Cilium Pods are in running status.
kubectl get po -n kube-system
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system cilium-9vnf8 1/1 Running 0 3m43s
kube-system cilium-bjdb4 1/1 Running 0 3m43s
kube-system cilium-operator-7c9b465f5c-blk4w 1/1 Running 0 3m43s
kube-system coredns-7d74b564bd-2d2t9 1/1 Running 0 3m34s
kube-system coredns-7d74b564bd-tkxgp 1/1 Running 0 3m19s
kube-system kube-proxy-cnvph 1/1 Running 0 23m
kube-system kube-proxy-hzkt9 1/1 Running 0 23m
Note that when running Cilium in an autoscaling cluster, you’ll want to add the node.cilium.io/agent-not-ready=true:NoSchedule taint to the nodes to prevent that Pods are scheduled in the node before Cilium is ready. Check the Cilium installation guide for more details.
Deploy a Demo Application
Let’s deploy a demo application to perform some tests.
git clone https://github.com/microservices-demo/microservices-demo
kubectl create namespace sock-shop
kubectl apply -f microservices-demo/deploy/kubernetes/complete-demo.yaml
Sock Shop is an application that implements an online socks shop and is usually used to demonstrate and test microservices and cloud-native technologies.
Wait until all the Pods are in running status.
watch 'kubectl get po -n sock-shop'
NAME READY STATUS RESTARTS AGE
carts-b4d4ffb5c-t8w2r 1/1 Running 0 3m45s
carts-db-6c6c68b747-5wdm7 1/1 Running 0 3m45s
catalogue-5bd97b4988-dsg6b 1/1 Running 0 3m45s
catalogue-db-96f6f6b4c-dbwts 1/1 Running 0 3m45s
front-end-6649c54d45-p7c66 1/1 Running 0 3m44s
orders-7664c64d75-ffw4d 1/1 Running 0 3m44s
orders-db-659949975f-x7cqq 1/1 Running 0 3m44s
payment-dd67fc96f-rvrx4 1/1 Running 0 3m44s
queue-master-5f6d6d4796-vx4f5 1/1 Running 0 3m44s
rabbitmq-5bcbb547d7-z7bxf 2/2 Running 0 3m43s
session-db-7cf97f8d4f-w8mhg 1/1 Running 0 3m43s
shipping-7f7999ffb7-4bnhj 1/1 Running 0 3m43s
user-888469b9f-lcv4n 1/1 Running 0 3m43s
user-db-6df7444fc-77ht7 1/1 Running 0 3m43s
When the Pods are running, you can optionally test it locally by port-fowarding the frond-end service.
kubectl port-forward service/front-end -n sock-shop 8080:80
You can now access it on the browser at localhost:8080. Use the username and password user / password to login.
Lateral Movement
As you can see in the Architecture Diagram, this application has a few components. Among them is the user component that interacts with the user-db, which is a MongoDB instance used to store users’ information.
We will play the role of a malicious actor that has gained access to create a Pod in our cluster and use this access to connect to the user-db.
List the services and observe the users-db service responding on port 27017.
kubectl get svc -n sock-shop
Let’s launch a temporary Pod in interactive mode using the mongo image to execute commands against the user-db.
kubectl run rogue-1 --image=mongo:3.2 --restart=Never --rm -it -- /bin/bash
From inside the container run the following command to connect to the MongoDB instance:
root@rogue-1:/# mongo --host user-db.sock-shop.svc.cluster.local:27017
Now, from the mongo session, run:
> show dbs
local 0.000GB
users 0.000GB
> use users
switched to db users
> show collections
addresses
cards
customers
> db.cards.find()
{ "_id" : ObjectId("57a98d98e4b00679b4a830ae"), "longNum" : "5953580604169678", "expires" : "08/19", "ccv" : "678" }
{ "_id" : ObjectId("57a98d98e4b00679b4a830b1"), "longNum" : "5544154011345918", "expires" : "08/19", "ccv" : "958" }
{ "_id" : ObjectId("57a98d98e4b00679b4a830b4"), "longNum" : "0908415193175205", "expires" : "08/19", "ccv" : "280" }
{ "_id" : ObjectId("57a98ddce4b00679b4a830d2"), "longNum" : "5429804235432", "expires" : "04/16", "ccv" : "432" }
Kubernetes Network Policies
Let’s use a Kubernetes Network Policy to restrict ingress traffic to the users-db Pod to only traffic from the users Pod.
To create this Network Policy, we need to know the labels used for each of these Pods. We can check this by running:
kubectl describe deploy user-db -n sock-shop
Name: user-db
Namespace: sock-shop
[...]
Pod Template:
Labels: name=user-db
Containers:
[...]
kubectl describe deploy user -n sock-shop
Name: user
Namespace: sock-shop
[...]
Pod Template:
Labels: name=user
Containers:
[...]
Note the Pod labels name=user-db and name=user.
Create the Network Policy.
kubectl apply -f- << EOF
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: user-db-allow-from-user
namespace: sock-shop
spec:
policyTypes:
- Ingress
PodSelector:
matchLabels:
name: user-db
ingress:
- from:
- PodSelector:
matchLabels:
name: user
EOF
Create a temporary Pod again and try to connect to the MongoDB instance.
kubectl run rogue-1 --image=mongo:3.2 --restart=Never --rm -it -- /bin/bash
root@rogue-1:/# mongo --host user-db.sock-shop.svc.cluster.local:27017
[...]
It should fail now.
A note on Kubernetes Security
The goal of this tutorial is to check how Cilium works with EKS allowing us to experiment with Network Policies and other features. The focus is not to provide best practices on how to secure your Kubernetes cluster. For instance, notice that a malicious actor, depending on the level of access that has been granted, can use different approaches to bypass this policy and get access to PII data or lead to service disruption. To properly secure your cluster, you should consider a combination of techniques such as using an Admission Controller (e.g. Gatekeeper or Kyverno); tuned RBAC permissions; proper monitoring and alerting on suspicious behavior (e.g. Falco); among others.
Access the application on the browser at localhost:8080 and verify that everything is still working fine.
Besides being able to use standard Kubernetes Network Policies, Cilium also allows you to use CiliumNetworkPolicy and CiliumClusterwideNetworkPolicy that extend the standard policies, providing more capabilities. See the Network Policies tutorials for examples. It’s also worth taking a look at the functionality overview doc for a list of Cilium features that go beyond Network Policies.
Clean up
eksctl delete cluster --name $cluster_name --region $cluster_region