rootnode-academy/content/root/root-10-kubernetes-troubleshooting-observability/m01-health-probes/01-lesson-health-check-concepts.md

# Why Health Checks Exist & Probe Types

Kubernetes cannot see inside your application. It doesn't know if your process is deadlocked,
stuck in an infinite loop, or quietly failing every request while the process itself stays
alive. Without a way to ask "are you actually working?", the only signal Kubernetes has is
whether the container's process is still running — and a hung process still counts as running.

**A health probe is how your application tells Kubernetes the truth.** It's a small check —
an HTTP request, a TCP connection attempt, or a command run inside the container — that
Kubernetes runs on a schedule and uses to make real decisions: restart this container, or stop
sending it traffic.

## The three probe types

Kubernetes supports three distinct probes, and they answer three different questions. Mixing
them up is the single most common probe mistake in production.

**Liveness probe — "Is this container still alive, or should it be restarted?"**
If a liveness probe fails repeatedly, the kubelet kills the container and it gets restarted per
its restart policy. Use this for detecting a genuinely stuck process — a deadlock, a crashed
event loop. Don't use liveness to mean "not ready for traffic yet" — that's what readiness is
for, and confusing the two causes the exact incident you're about to fix.

**Readiness probe — "Is this container ready to receive traffic right now?"**
If a readiness probe fails, the Pod is removed from the Service's list of endpoints —
Kubernetes stops routing traffic to it, but does **not** restart the container. This is the
right tool for "still starting up," "warming a cache," or "temporarily overloaded, back off for
a bit." A Pod can be Running and healthy but still correctly Not Ready.

**Startup probe — "Has this slow-starting container finished starting up yet?"**
Some applications take a long time to become ready (loading a large dataset, running
migrations). A startup probe disables liveness/readiness checks until it succeeds once, so a
slow-but-healthy startup doesn't get killed by an impatient liveness probe. Once it succeeds,
liveness and readiness take over as normal.

## How a probe is actually defined

Every probe has a **mechanism** (how the check runs) and a **schedule** (how often, how long to
wait, how many failures before acting):

- `httpGet` — a GET request to a path and **port** on the container. A 2xx or 3xx response
  counts as success.
- `tcpSocket` — a TCP connection attempt to a **port**. A successful connection counts as
  success — this doesn't check application-level health, just "is something listening."
- `exec` — runs a command inside the container. Exit code `0` counts as success.

Timing fields (`initialDelaySeconds`, `periodSeconds`, `timeoutSeconds`, `failureThreshold`,
`successThreshold`) control how patient Kubernetes is before it acts — but no amount of correct
timing fixes a probe that's pointed at the wrong port or path in the first place.

## Common production mistakes

1. **Wrong port.** The probe's `httpGet.port` (or `tcpSocket.port`) doesn't match the port the
   application actually listens on. The container is completely healthy — Kubernetes is just
   asking the wrong door to open. This is the exact failure you'll diagnose in this module's lab.
2. **Liveness probe too aggressive on a slow-starting app**, with no startup probe — the app
   gets killed for "not being ready yet," restarts, and never gets far enough to pass.
3. **Readiness probe that always passes** (e.g. checking a static file instead of real
   dependencies) — traffic gets routed to a Pod that can't actually serve it.
4. **No probes configured at all** — Kubernetes has no way to know the container is stuck, so a
   deadlocked process just sits there serving nothing, forever, looking "Running" the whole time.

## Instructor Notes

**Approximate timing:** 12-15 minutes for a live read-through with pauses for questions;
10 minutes for a self-paced read.

**Demo plan:** Before moving to the guided exercise, it's worth live-editing one Deployment's
`readinessProbe.httpGet.port` in front of the class to an obviously wrong value and running
`kubectl get pods` / `kubectl describe pod` together, so students see the Ready column and the
Events section change in real time before they do it themselves.

**Likely misconceptions to watch for:**
- Students often assume a failing liveness probe and a failing readiness probe look the same in
  `kubectl get pods` — they don't (liveness failures show restart counts climbing; readiness
  failures show `0/1` Ready with no restarts). Worth calling out explicitly.
- Students sometimes think `tcpSocket` checks are "less safe" than `httpGet` — reframe as
  "checks a different, shallower thing," not "worse."

**Discussion question:** "Why might a team deliberately choose a readiness probe that fails
under high load, even though it takes healthy capacity out of rotation?" (Answer: shedding load
from an overloaded Pod is often better than serving every request slowly or with errors.)

**Optional extension:** Ask students to sketch what would happen if liveness and readiness
probes were swapped (readiness probe restarts on failure, liveness probe just removes from
Service) — this clarifies the distinction by making the wrong version visibly worse.