The Kubernetes handbook

01 What Kubernetes actually does

One idea, repeated everywhere: you describe what you want, and a loop keeps reality matching it.

You never tell Kubernetes to do things. You tell it what should be true — "there should be three copies of this container running" — and write that down in a YAML file. Kubernetes stores that as the desired state. Separately, controllers watch the actual state of the cluster and, whenever the two differ, take a small step to close the gap. Forever.

Every object in this handbook plugs into step 3. A Deployment controller, a ReplicaSet controller, a Job controller — same loop, different definition of "correct".

Why this matters on day one

Two practical consequences that trip up nearly everyone:

02 Anatomy of a manifest

Every YAML file you'll ever write has the same four keys. Learn them once.

Kubernetes objects look wildly different until you notice they're all the same envelope with different contents. Whether it's a Pod, a Service, or something a vendor invented last week, the top level is identical.

apiVersion: apps/v1        # which API group + version owns this kind
kind: Deployment          # what type of object this is
metadata:                 # who it is: name, namespace, labels, annotations
  name: vote
  labels:
    app: vote
spec:                     # WHAT YOU WANT — the desired state. You write this.
  replicas: 3
  ...
# status:                # WHAT IS. The cluster writes this. Never put it in your file.

kubectl explain pod.spec.containers.resources
kubectl explain ingress.spec.rules --recursive

Three YAML rules that cause most beginner errors

$ kubectl apply -f vote.yaml --dry-run=server    ← validates against the real API, changes nothing
deployment.apps/vote configured (server dry run)

$ kubectl diff -f vote.yaml                     ← shows exactly what would change

03 The object map

How the pieces in Part 2 fit together, before you meet them one at a time.

There are only two kinds of relationship in this picture, and telling them apart makes the rest of Kubernetes readable. Solid lines are ownership: this object created that one and will delete it. Dashed lines are selection: this object found that one by matching labels, and neither owns the other.

Delete the Deployment and the ReplicaSet and Pods go with it. Delete the Service and the Pods carry on, unreachable — nothing owned them.

04 Nodes

The machines your Pods actually run on.

A Node is a worker machine — VM or bare metal — registered with the cluster's control plane. Every Node runs three things you rely on constantly: the kubelet (an agent that talks to the API server and starts/stops containers), kube-proxy (programs the networking rules that make Services work), and a container runtime (containerd, CRI-O). The control plane itself — API server, scheduler, controller-manager, etcd — usually isn't a "Node" you deploy workloads onto; it's the brain deciding where your Pods go.

The control plane decides placement; Nodes are just where Pods physically live.

What you'd actually run

NAME       STATUS   ROLES           AGE   VERSION   INTERNAL-IP
node-a     Ready    <none>          14d   v1.30.2   10.128.0.4
node-b     Ready    <none>          14d   v1.30.2   10.128.0.5

05 Pods

The smallest thing you can deploy — and the thing you almost never create directly.

A Pod wraps one or more containers that share a network namespace (same IP, containers reach each other over localhost) and can share storage volumes. Pods are disposable: when one dies, Kubernetes doesn't repair it — it throws it away and, if something is managing it, creates a brand-new Pod with a new IP. Nothing about a Pod's identity is meant to be stable.

Containers inside one Pod talk over localhost; the Pod as a whole talks out to other Pods via Services.

A bare Pod spec (don't do this in practice — see below)

apiVersion: v1
kind: Pod
metadata:
  name: worker
  labels:
    app: worker
spec:
  containers:
    - name: worker
      image: dockersamples/examplevotingapp_worker
      env:
        - name: REDIS_HOST
          value: "redis"
        - name: DB_HOST
          value: "db"

$ kubectl delete pod worker           (created directly, no controller)
pod "worker" deleted
$ kubectl get pods
No resources found                    ← gone. nothing brings it back.

$ kubectl delete pod vote-7d9f8b6c4-x2z1p   (owned by a ReplicaSet)
pod "vote-7d9f8b6c4-x2z1p" deleted
$ kubectl get pods
vote-7d9f8b6c4-m4q7k   1/1   Running   0   4s   ← replaced automatically

06 Labels & selectors

The glue. Almost nothing in Kubernetes connects to anything by name — it connects by label.

A label is a key/value pair you stick on an object. A selector is a query over labels. That's the entire mechanism, and it's how a Service finds its Pods, how a ReplicaSet knows which Pods are "its", and how you filter anything on the command line. Once this clicks, the next four sections stop being mysterious.

Matching is exact and case-sensitive. There is no warning when a selector matches nothing — you just get a Service that silently routes to no one.

metadata:
  labels:                 # what this object IS
    app: vote
    tier: frontend
    env: prod
  annotations:            # notes for humans and tools — never used for selection
    owner: "team-payments"
    kubernetes.io/change-cause: "bump to v2"
---
# A selector is the query side. Two forms:
selector:
  matchLabels:            # simple equality — 95% of what you write
    app: vote
  matchExpressions:       # set-based, for the other 5%
    - {key: env, operator: In, values: [prod, staging]}
    - {key: tier, operator: Exists}

$ kubectl get pods -l app=vote                 ← everything wearing the sticker
$ kubectl get pods -l app=vote,tier=frontend   ← comma means AND, never OR
$ kubectl get pods -l 'env in (prod,staging)'  ← set-based
$ kubectl get pods -l '!canary'                ← label absent
$ kubectl get pods --show-labels               ← what is actually on them

$ kubectl delete pods -l app=vote              ← same query, destructive. check with get first.

07 ReplicaSets

The controller that keeps "N copies of this Pod" true, always.

A ReplicaSet's job is narrow: watch for Pods matching its label selector, and reconcile reality toward a desired count. Too few → create more. Too many (someone manually made an extra) → delete one. It runs this reconcile loop continuously, not just once. You'll rarely author a ReplicaSet by hand — Deployments create and own them — but understanding this layer is what makes Deployment rollouts make sense.

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: redis
spec:
  replicas: 2
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
        - name: redis
          image: redis:alpine

$ kubectl delete pod redis-8f6d9c7-k2n1x
pod "redis-8f6d9c7-k2n1x" deleted
$ kubectl get pods -l app=redis --watch
redis-8f6d9c7-k2n1x   1/1   Terminating
redis-8f6d9c7-p9v3q   0/1   ContainerCreating   ← RS noticed within its next tick
redis-8f6d9c7-p9v3q   1/1   Running

08 Deployments

What you actually write. Manages ReplicaSets so you get rollouts and rollbacks for free.

A Deployment is a ReplicaSet manager. Change the Pod template — a new image tag, a new env var — and the Deployment doesn't edit Pods in place; it creates a new ReplicaSet with the new template and shifts replica count from the old ReplicaSet to the new one, gradually, per its strategy. This is literally the mechanism behind a "rolling update." The old ReplicaSet isn't deleted — it's scaled to zero and kept around, which is what makes instant rollback possible.

The Service's selector doesn't change — it just starts finding new-version Pods as the old ReplicaSet scales down.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vote
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: vote
  template:
    metadata:
      labels:
        app: vote
    spec:
      containers:
        - name: vote
          image: dockersamples/examplevotingapp_vote:v1
          ports:
            - containerPort: 80
          readinessProbe:
            httpGet: { path: /, port: 80 }
            initialDelaySeconds: 3

$ kubectl set image deployment/vote vote=dockersamples/examplevotingapp_vote:v2
$ kubectl rollout status deployment/vote
Waiting for deployment "vote" rollout to finish: 1 old replicas pending termination...
Waiting for deployment "vote" rollout to finish: 2 of 3 updated replicas available...
deployment "vote" successfully rolled out

$ kubectl rollout undo deployment/vote     ← instant, because old RS never went away

09 Services

A stable address in front of Pods that keep coming and going.

Pods are disposable and their IPs change constantly; a Service gives you one thing that doesn't change — a virtual IP and a DNS name — and load-balances traffic across whichever Pods currently match its label selector and are passing their readiness probe. Under the hood, an Endpoints object tracks that live Pod list, and kube-proxy programs the actual routing rules on every Node.

Same selector-based routing underneath all three — they differ only in where the entry point is exposed.

apiVersion: v1
kind: Service
metadata:
  name: vote
spec:
  type: NodePort
  selector:
    app: vote
  ports:
    - port: 80
      targetPort: 80
      nodePort: 31000
---
# redis needs no type — ClusterIP is the default,
# and it's all a same-cluster worker needs to reach it
apiVersion: v1
kind: Service
metadata:
  name: redis
spec:
  selector:
    app: redis
  ports:
    - port: 6379

$ kubectl get endpoints vote
NAME   ENDPOINTS
vote   10.244.1.7:80,10.244.2.9:80,10.244.2.11:80

$ kubectl delete pod vote-7d9f8b6c4-x2z1p
$ kubectl get endpoints vote
vote   10.244.2.9:80,10.244.2.11:80   ← 3rd IP dropped instantly, requests never hit it

10 ConfigMaps & Secrets

Getting config and credentials out of your image.

Same API shape, different intent: a ConfigMap holds non-sensitive key/value config (which two options the vote app shows), a Secret holds sensitive values (the Postgres password). Both can be injected into a Pod two ways — as environment variables, fixed at container start, or mounted as files, which can update live. That distinction matters more than it sounds like.

apiVersion: v1
kind: Secret
metadata:
  name: db-creds
stringData:                    # kubectl encodes this to base64 for you
  POSTGRES_USER: postgres
  POSTGRES_PASSWORD: postgres
---
# inside the db container spec:
env:
  - name: POSTGRES_PASSWORD
    valueFrom:
      secretKeyRef:
        name: db-creds
        key: POSTGRES_PASSWORD

11 Storage — PersistentVolume & PersistentVolumeClaim

Making sure votes survive a Pod restart.

A container's own filesystem dies with the container. For db, that would mean every vote vanishes on a routine restart. A PersistentVolumeClaim (PVC) is a request — "5Gi, mountable by one Node at a time" — and a PersistentVolume (PV) is the actual disk that satisfies it, usually provisioned automatically by a StorageClass rather than hand-written. The Pod just mounts the claim; it doesn't care what's backing it.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: db-data
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 5Gi
---
# inside the db Pod spec:
volumes:
  - name: pgdata
    persistentVolumeClaim:
      claimName: db-data
containers:
  - name: db
    volumeMounts:
      - mountPath: /var/lib/postgresql/data
        name: pgdata

12 Namespaces

Folders for your cluster — and the reason your kubectl get pods looks empty.

A namespace is a naming scope. Two teams can each have a Service called api as long as they're in different namespaces. It's also the boundary that RBAC permissions and resource quotas attach to, which is what makes it more than cosmetic organisation.

$ kubectl get namespaces
NAME              STATUS   AGE
default           Active   14d       # where you land if you never say otherwise
kube-system       Active   14d       # the cluster's own components. don't deploy here.
kube-public       Active   14d
kube-node-lease   Active   14d       # node heartbeats live here

$ kubectl get pods -n voting         # one namespace
$ kubectl get pods -A                # all of them — the "where did it go" command

$ kubectl config set-context --current --namespace=voting
                                     # stop typing -n on every command

What is and isn't namespaced

Unsure? kubectl api-resources --namespaced=false lists the cluster-wide ones on your cluster.

Talking across namespaces

DNS is how you cross the boundary. Inside one namespace, the short name works. From another, use the longer form:

redis                              # same namespace — what your app config usually holds
redis.voting                       # from another namespace
redis.voting.svc                   # fully explicit-ish
redis.voting.svc.cluster.local     # the real, complete name

13 Ingress

One entry point, many Services — routed by hostname and URL path.

A LoadBalancer Service gives you a public IP per Service, which gets expensive and unmanageable fast. An Ingress is a single front door that reads the HTTP request and decides which Service should handle it. It works at layer 7, so it understands hostnames, paths, and TLS in a way a Service never can.

Three public hostnames, one cloud load balancer, three ordinary ClusterIP Services behind it.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: voting
spec:
  ingressClassName: nginx      # WHICH controller handles this. omit it and nothing may.
  tls:
    - hosts: [vote.example.com]
      secretName: vote-tls      # a Secret of type kubernetes.io/tls
  rules:
    - host: vote.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: vote      # a normal ClusterIP Service
                port: { number: 80 }
    - host: result.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: result
                port: { number: 80 }

14 StatefulSets

For Pods that need a name and a disk that follow them around.

A Deployment treats its Pods as interchangeable — random name suffixes, any order, all sharing whatever volume you gave them. That's exactly wrong for databases, queues and anything that clusters. A StatefulSet gives each Pod a stable ordinal identity (db-0, db-1), its own PersistentVolumeClaim created from a template, and ordered startup, update and shutdown.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: db
spec:
  serviceName: db            # must be a HEADLESS service (clusterIP: None)
  replicas: 3
  selector:
    matchLabels: { app: db }
  template:
    metadata:
      labels: { app: db }
    spec:
      containers:
        - name: postgres
          image: postgres:16
          volumeMounts:
            - { name: data, mountPath: /var/lib/postgresql/data }
  volumeClaimTemplates:      # each Pod gets its OWN 5Gi disk from this
    - metadata: { name: data }
      spec:
        accessModes: ["ReadWriteOnce"]
        resources: { requests: { storage: 5Gi } }

15 DaemonSets

Exactly one copy on every Node — including Nodes that don't exist yet.

Some things are per-machine rather than per-application: shipping logs, exporting metrics, running a CNI agent. A DaemonSet has no replicas field because the count isn't yours to choose — it's however many Nodes there are. Add a Node to the cluster and a Pod appears on it automatically; drain the Node and it goes away.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: log-agent
spec:
  selector:
    matchLabels: { app: log-agent }
  template:
    metadata:
      labels: { app: log-agent }
    spec:
      tolerations:                    # run on control-plane nodes too
        - { key: node-role.kubernetes.io/control-plane, operator: Exists, effect: NoSchedule }
      containers:
        - name: agent
          image: fluent/fluent-bit:3.0
          volumeMounts:
            - { name: varlog, mountPath: /var/log, readOnly: true }
      volumes:
        - name: varlog
          hostPath: { path: /var/log }   # reads the Node's own filesystem

16 Jobs & CronJobs

For work that finishes — migrations, batch imports, nightly reports.

Every controller so far assumes the Pod should run forever, and restarts it when it exits. A Job inverts that: it runs a Pod until it exits successfully, then stops. A CronJob creates Jobs on a schedule. Use these for anything with an end, and never try to model a migration as a Deployment — it'll loop forever in CrashLoopBackOff for succeeding.

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
spec:
  backoffLimit: 4              # retries before the Job is marked Failed
  ttlSecondsAfterFinished: 3600 # auto-delete an hour after finishing
  template:
    spec:
      restartPolicy: OnFailure   # Never or OnFailure — "Always" is rejected here
      containers:
        - name: migrate
          image: myapp/migrate:v2
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-report
spec:
  schedule: "0 2 * * *"        # standard cron, in the CLUSTER's timezone (UTC by default)
  concurrencyPolicy: Forbid    # skip a run if the previous one is still going
  successfulJobsHistoryLimit: 3
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - { name: report, image: myapp/report:v2 }

$ kubectl get jobs
NAME         COMPLETIONS   DURATION   AGE
db-migrate   1/1           12s        3m      ← done. Pod stays around so you can read logs.

$ kubectl logs job/db-migrate                 ← works even after it finished

17 Resources, probes & autoscaling

The three settings that decide whether your app survives a busy Tuesday.

Requests vs limits

A request is what the scheduler reserves for you — it's how a Node is chosen, and it's guaranteed. A limit is the ceiling the runtime enforces. They do completely different jobs and are constantly confused.

containers:
  - name: vote
    image: dockersamples/examplevotingapp_vote:v2
    resources:
      requests: { cpu: "100m", memory: "128Mi" }   # 100m = 0.1 of a core
      limits:   { memory: "256Mi" }

    startupProbe:        # "has it booted yet?" — the other two wait for this
      httpGet: { path: /healthz, port: 80 }
      failureThreshold: 30
      periodSeconds: 2    # allows up to 60s to start, without a slow liveness probe

    readinessProbe:      # "should it get traffic?" — failing removes it from the Service
      httpGet: { path: /ready, port: 80 }
      periodSeconds: 5

    livenessProbe:       # "is it wedged?" — failing RESTARTS the container
      httpGet: { path: /healthz, port: 80 }
      periodSeconds: 10
      failureThreshold: 3

Horizontal Pod Autoscaler

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: vote
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: vote }
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target: { type: Utilization, averageUtilization: 70 }

18 RBAC & ServiceAccounts

Who is allowed to do what — to the cluster, and from inside it.

Two questions, four objects. Who is a User (you, from your kubeconfig) or a ServiceAccount (a Pod). What is a Role — a list of verbs on resources, scoped to one namespace — or a ClusterRole, the same thing cluster-wide. A RoleBinding connects the two. That's the whole model.

apiVersion: v1
kind: ServiceAccount
metadata: { name: vote-reader }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: pod-reader }
rules:
  - apiGroups: [""]              # "" is the core group
    resources: ["pods", "pods/log"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: vote-reader-can-read-pods }
subjects:
  - { kind: ServiceAccount, name: vote-reader }
roleRef:
  { kind: Role, name: pod-reader, apiGroup: rbac.authorization.k8s.io }
---
# then, in the Pod spec:
spec:
  serviceAccountName: vote-reader
  automountServiceAccountToken: true

$ kubectl auth can-i delete pods
no

$ kubectl auth can-i get pods --as=system:serviceaccount:default:vote-reader
yes                                ← test someone else's permissions without being them

$ kubectl auth can-i --list         ← everything you're allowed to do here

kubectl, the useful half

There are hundreds of flags. These are the ones you'll use every day.

Looking around

Debugging a running thing

Changing things

alias k=kubectl
source <(kubectl completion bash)   # or zsh — tab-completes Pod names, saves hours
complete -o default -F __start_kubectl k

Troubleshooting by symptom

Find the status you're staring at. Nearly all of these end at the same place: the Events section of kubectl describe pod.

Glossary

Words this page and every Kubernetes doc assume you already know.

Cluster

The whole thing: a control plane plus the Nodes it manages.

Control plane

The brain — API server, scheduler, controller-manager, etcd. Decides what should happen; doesn't run your app.

Node

A machine that runs Pods. Runs a kubelet and a container runtime.

kubelet

The agent on each Node. Takes instructions from the API server and makes containers exist.

etcd

The database holding all cluster state. If you lose etcd with no backup, you lose the cluster's memory.

Controller

A loop that watches one kind of object and nudges reality toward its spec. Kubernetes is a pile of these.

Reconciliation

The comparing-and-correcting that controllers do. The core idea on this page.

Declarative

You describe the end state, not the steps. apply is declarative; create is not.

Manifest

A YAML file describing one or more objects.

Object / resource

Anything with an apiVersion and kind that the API server can store.

Label

A key/value tag used for selection. How Services find Pods.

Annotation

A key/value note for humans and tools. Never used for selection.

Selector

A query over labels.

Endpoints / EndpointSlice

The live list of Pod IPs currently behind a Service. Your first stop when a Service seems dead.

Ephemeral

Thrown away rather than repaired. Pods are ephemeral; that's a design choice, not a limitation.

Taint / toleration

A Node repels Pods (taint) unless a Pod says it can cope (toleration). How control-plane Nodes stay free of your app.

Affinity

Rules pulling Pods toward or away from Nodes or each other.

Drain / cordon

Cordon stops new Pods landing on a Node; drain also evicts what's there. What you run before maintenance.

QoS class

Guaranteed, Burstable or BestEffort, derived from your requests and limits. Decides who gets killed first under pressure.

Finalizer

A "wait, clean up first" marker that blocks deletion until a controller clears it. The usual cause of stuck Terminating.

CRD

CustomResourceDefinition — a new kind added by an add-on, treated exactly like a built-in one.

Operator

A custom controller plus a CRD, encoding how to run a specific piece of software.

Headless Service

A Service with clusterIP: None. No load balancing — DNS returns the Pod IPs directly. Required by StatefulSets.

CNI

The plugin giving every Pod an IP and making Pod-to-Pod networking work. Calico, Cilium, Flannel.

Helm

A package manager for Kubernetes — templated manifests bundled as a "chart".