Module 10 · Lesson 10.1

Taints & Tolerations

You've been letting the scheduler place pods wherever it likes. Now you'll start steering it — starting with the "repel" side of the equation.

Taints push pods away; tolerations let them land anyway

A taint on a node says: "don't schedule here unless you're specifically okay with this." A toleration on a pod says: "I'm okay with that." This is the opposite direction from a nodeSelector or nodeAffinity (next lesson), which is the pod requesting a node — a taint is the node repelling pods by default. Mixing these two up is the single most common confusion here, so it's worth saying twice: taints are a node-side deny-by-default; tolerations are a pod-side opt-in; neither one attracts anything by itself.

Taints repel, tolerations let a pod past the gate

node-a is tainted and the pod has no matching toleration — it's repelled, so it lands on node-b instead.

Three effects, three different severities

NoSchedule — new pods without a matching toleration won't be placed here. Pods already running stay put.
PreferNoSchedule — a soft version; the scheduler avoids it if it reasonably can, but will use it under pressure.
NoExecute — the strictest: pods without a matching toleration are actively evicted if they're already running there, not just blocked from landing. This is what you saw without naming it in Module 1 — nodes getting marked unreachable automatically tolerate NoExecute taints for a grace period (tolerationSeconds) before their pods are evicted.

Why this exists

Real clusters use taints to reserve nodes for a purpose: GPU nodes tainted so only GPU-requesting pods land there, nodes under maintenance tainted so nothing new schedules while you drain them, or a dedicated tenant's nodes tainted so only that tenant's pods (carrying the matching toleration) can use them. It's an opt-in gate, not a routing preference — a pod with the right toleration can land there, but a taint alone never forces it to.

The infrastructure analogy

Think of a taint as a firewall rule that defaults to deny for a specific tag, and a toleration as the one explicit allow rule that lets a particular workload through. It's the inverse of putting a server in a specific subnet so only things routed there can reach it (that's closer to nodeAffinity, which you'll meet next) — a taint actively pushes everyone away unless they present the right credential.

In the lab, you'll taint one of your two worker nodes, watch an ordinary pod avoid it, then add a toleration and watch it become eligible there too.

🧪 Lab: lab-20-taints-and-tolerations

Preview only

Goal

Taint a real worker node, prove a pod without a matching toleration can't land there, then fix it with a toleration.

Tasks

Taint the node first, then apply the starting manifests — order matters here: a NoSchedule taint only blocks new placements, it doesn't evict anything already running, so if you create the pod before the taint exists it'll schedule fine and the lab won't demonstrate anything. Taints are a node-level property, not a YAML resource you apply (see manifests/README.md):
```
kubectl taint nodes k8s-course-worker dedicated=gpu:NoSchedule
kubectl apply -f manifests/
```
manifests/pod.yaml requires that exact node via nodeAffinity, so it can't just drift to the other worker instead — that makes the taint's effect on it unambiguous.
Check the pod: kubectl get pod gpu-job -n lab-20-taints-and-tolerations — it's Pending.
Find out why: kubectl describe pod gpu-job -n lab-20-taints-and-tolerations — look at the Events at the bottom. It names the taint directly.
Pod specs are mostly immutable once created, so: delete it (kubectl delete pod gpu-job -n lab-20-taints-and-tolerations), add a tolerations entry to manifests/pod.yaml matching the taint (dedicated=gpu, effect NoSchedule), then re-apply: kubectl apply -f manifests/pod.yaml
Confirm it's now Running on k8s-course-worker: kubectl get pod gpu-job -n lab-20-taints-and-tolerations -o wide

Check

Run the check once gpu-job is Running on k8s-course-worker.

Cleanup before moving on

This lab leaves a real taint on k8s-course-worker — that's deliberate, so you can re-run the check any time without redoing the setup. But this cluster only has 2 worker nodes, and a couple of later labs (Module 7's StatefulSet lab, the Capstone) create persistent storage that can bind to whichever node happens to be free — if that's the tainted one, their pods can get stuck Pending for a reason that has nothing to do with those labs. Once you're satisfied with this lesson, clean up:

kubectl taint nodes k8s-course-worker dedicated=gpu:NoSchedule-

(If you later see a storage lab's pod stuck with both "untolerated taint(s)" and "didn't match PersistentVolume's node affinity" in its events, this is almost certainly why — untaint here and delete the stuck pod/PVC so it can reschedule.)

This lab runs against a real local Kubernetes cluster with an automated grader — clone the repo and run make start to do it for real.

📝 Quiz

1. Which statement correctly describes the relationship between taints and tolerations?

2. A node has a NoSchedule taint. A pod that's already been running there for days has no matching toleration. What happens to it?scenario

3. Which taint effect actively evicts already-running pods that lack a matching toleration?

4. You want a pod to run ONLY on GPU nodes, and you want ordinary pods to never accidentally land on those (expensive) GPU nodes. What's the typical combination?scenario

5. In the infrastructure analogy, a taint is closest to:

Progress isn't saved in this preview — run the course locally to track completion and grade labs for real.