Module 15 ยท Lesson 15.1
Networking: VPC CNI
Every lab so far has run on Calico, an overlay network: pod IPs come from a
virtual address space (192.168.0.0/16 on your cluster) that Calico tunnels
across your 3 nodes, with no relationship to your Mac's real network. EKS
does something fundamentally different, and it's the first thing that
surprises people moving from kind/minikube to production AWS.
VPC CNI: pods get real VPC IPs
The Amazon VPC CNI plugin โ EKS's default โ gives every pod a real, routable IP address from your VPC subnet, allocated from the ENI (Elastic Network Interface) attached to the pod's node. No overlay, no tunnel: a pod's IP is a first-class citizen of your VPC, visible to anything else in it exactly like an EC2 instance's IP would be. That's genuinely useful โ pods can talk directly to RDS, other VPC resources, and on-prem networks over Direct Connect/VPN without any extra NAT layer.
Your kind cluster vs. a real EKS cluster
| Your kind cluster | Real EKS | |
|---|---|---|
| Networking | Calico overlay โ pod IPs from an arbitrary virtual range (192.168.0.0/16), tunneled between nodes | VPC CNI โ pod IPs from your real VPC subnet via ENIs, routable like any EC2 instance |
| Ingress / LB | Traefik, Service type ClusterIP โ no real external address | AWS Load Balancer Controller provisions a real NLB/ALB from Service/Ingress annotations |
| Storage | local-path-provisioner โ a directory on the node's own disk | EBS CSI driver โ a real, network-attached EBS volume (or EFS CSI for shared access) |
| Identity for pods | No cloud IAM โ nothing to authenticate to | IRSA or Pod Identity โ a ServiceAccount maps to a real IAM role |
The IP-exhaustion problem this creates
Because every pod consumes a real IP from the subnet, and every node
reserves IPs up front for the pods it might run, small subnets run out of
IPs surprisingly fast โ a problem that simply doesn't exist with an overlay
network like Calico, where the pod address space is independent of your
VPC's CIDR entirely. The standard mitigation is prefix delegation:
instead of attaching IPs to an ENI one at a time, the node reserves whole
/28 prefixes (16 IPs each), dramatically raising the max-pods-per-node
ceiling for a given instance type without needing more ENIs. It's a config
flag on the VPC CNI plugin, not something you write into your own
manifests โ but it's exactly the kind of setting you'll need to know about
the first time kubectl describe pod shows a scheduling failure that
traces back to "no IPs available," not CPU or memory.
What carries over, what doesn't
NetworkPolicies, Services, and DNS all work identically on top of VPC
CNI โ the Kubernetes API doesn't change. What's different is purely
how the data plane assigns and routes addresses underneath. Any
NetworkPolicy you write using CIDR blocks (ipBlock), though, needs to
reflect your VPC's real subnet ranges on EKS โ not an arbitrary overlay
range you picked, since there is no overlay range anymore. You'll fix
exactly this kind of CIDR mismatch in this lesson's lab, using the real
pod subnet your own cluster already runs on (192.168.0.0/16) as the
stand-in for "the subnet you'd actually need to know in production."
The infrastructure analogy
If you've worked with traditional VPC networking or ENI-per-instance quotas in EC2, VPC CNI is that exact same mental model, just applied per pod instead of per instance โ which is precisely why it inherits EC2's IP-planning headaches at a much higher density.
๐งช Lab: lab-30-vpc-cni
Preview onlyGoal
Fix a NetworkPolicy that was written with the wrong CIDR โ a direct consequence of VPC CNI meaning pod IPs come from a real subnet you have to actually know, not an arbitrary overlay range.
Tasks
- Apply the starting manifests:
kubectl apply -f manifests/ - Wait for both pods to be
Running:kubectl get pods -n lab-30-vpc-cni - Get the client pod's real IP:
kubectl get pod client -n lab-30-vpc-cni -o jsonpath='{.status.podIP}'โ notice it's in192.168.x.x, not10.0.x.x. - Try reaching the server from the client:
kubectl exec -n lab-30-vpc-cni client -- wget -qO- -T 3 http://<server-pod-ip>(get the server's IP withkubectl get pod server -n lab-30-vpc-cni -o jsonpath='{.status.podIP}'). It should hang/fail โ theallow-from-vpcNetworkPolicy'sipBlockCIDR doesn't match this cluster's real pod subnet. - Find the real pod subnet: check
scripts/kind-cluster.yamlat the repo root (networking.podSubnet) โ that's the "VPC CIDR" stand-in for this lab, exactly the kind of value a real EKS NetworkPolicy would need to get from the account's actual VPC. - Edit
manifests/pods-and-policy.yaml: fix theipBlock.cidrto the correct subnet, thenkubectl apply -f manifests/pods-and-policy.yaml. - Retry step 4 โ it should now succeed.
Check
Run the check once the client can reach the server.
This lab runs against a real local Kubernetes cluster with an automated grader โ clone the repo and run make start to do it for real.
๐ Quiz
1. On EKS with the default VPC CNI plugin, where does a pod's IP address come from?
2. On a small EKS subnet, pods start failing to schedule with no CPU or memory pressure anywhere. What's the most likely VPC-CNI-specific cause?scenario
3. What does prefix delegation actually change?
4. You copy a NetworkPolicy with an `ipBlock` CIDR from your kind cluster (192.168.0.0/16, Calico's overlay range) straight onto an EKS cluster. What's wrong with that?scenario
5. Which Kubernetes-level behaviors change when you move from Calico (overlay) to VPC CNI (EKS)?
Progress isn't saved in this preview โ run the course locally to track completion and grade labs for real.