The Reconciliation Loop

This is the single most important idea in Kubernetes. Almost everything else — Deployments, Services, autoscalers — is a specific instance of this one pattern.

Desired state vs. actual state

When you kubectl apply a manifest, you're not running a command that does something once. You're writing a row into etcd that says "this is what I want to exist." That's the desired state.

Separately, the cluster has an actual state: what's really running right now, as reported by kubelets.

A controller is a small, dumb, relentless loop:

for {
  observed := get_actual_state()
  desired  := get_desired_state()
  if observed != desired {
    take_action_to_close_the_gap()
  }
}

The reconciliation loop

There's a Deployment controller, a ReplicaSet controller, a Node controller, a Job controller — dozens of them, each watching one slice of the API and running this same loop, forever. None of them ever declare victory and stop.

Why this matters for troubleshooting

This reframes almost every production issue you'll debug: something isn't "broken," it's that actual state can't reach desired state, and your job is to find out why. A Pod stuck in Pending? The scheduler's loop can't find a node satisfying the desired constraints. A Deployment stuck mid-rollout? The ReplicaSet controller's loop is blocked on a readiness check. You'll come back to this framing constantly, especially in the observability module.

The traditional-infra contrast

Think about a config-management run (Ansible/Puppet/Chef): you execute it, it converges the box, then it stops until you run it again — drift can creep back in between runs. Kubernetes controllers never stop running. If something drifts — a pod gets OOM-killed, a node dies — the gap is noticed and closed automatically, usually within seconds, with no cron job or human involved.

In the lab, you'll kill something the control plane manages and watch it come back without you lifting a finger a second time.