rootnode-academy/content/root/root-10-kubernetes-troubleshooting-observability/m01-health-probes/01-lesson-health-check-concepts.md

Why Health Checks Exist & Probe Types

Kubernetes cannot see inside your application. It doesn't know if your process is deadlocked, stuck in an infinite loop, or quietly failing every request while the process itself stays alive. Without a way to ask "are you actually working?", the only signal Kubernetes has is whether the container's process is still running — and a hung process still counts as running.

A health probe is how your application tells Kubernetes the truth. It's a small check — an HTTP request, a TCP connection attempt, or a command run inside the container — that Kubernetes runs on a schedule and uses to make real decisions: restart this container, or stop sending it traffic.

The three probe types

Kubernetes supports three distinct probes, and they answer three different questions. Mixing them up is the single most common probe mistake in production.

Liveness probe — "Is this container still alive, or should it be restarted?" If a liveness probe fails repeatedly, the kubelet kills the container and it gets restarted per its restart policy. Use this for detecting a genuinely stuck process — a deadlock, a crashed event loop. Don't use liveness to mean "not ready for traffic yet" — that's what readiness is for, and confusing the two causes the exact incident you're about to fix.

Readiness probe — "Is this container ready to receive traffic right now?" If a readiness probe fails, the Pod is removed from the Service's list of endpoints — Kubernetes stops routing traffic to it, but does not restart the container. This is the right tool for "still starting up," "warming a cache," or "temporarily overloaded, back off for a bit." A Pod can be Running and healthy but still correctly Not Ready.

Startup probe — "Has this slow-starting container finished starting up yet?" Some applications take a long time to become ready (loading a large dataset, running migrations). A startup probe disables liveness/readiness checks until it succeeds once, so a slow-but-healthy startup doesn't get killed by an impatient liveness probe. Once it succeeds, liveness and readiness take over as normal.

How a probe is actually defined

Every probe has a mechanism (how the check runs) and a schedule (how often, how long to wait, how many failures before acting):

• httpGet — a GET request to a path and port on the container. A 2xx or 3xx response counts as success.
• tcpSocket — a TCP connection attempt to a port. A successful connection counts as success — this doesn't check application-level health, just "is something listening."
• exec — runs a command inside the container. Exit code 0 counts as success.

Timing fields (initialDelaySeconds, periodSeconds, timeoutSeconds, failureThreshold, successThreshold) control how patient Kubernetes is before it acts — but no amount of correct timing fixes a probe that's pointed at the wrong port or path in the first place.

Common production mistakes

1. Wrong port. The probe's httpGet.port (or tcpSocket.port) doesn't match the port the application actually listens on. The container is completely healthy — Kubernetes is just asking the wrong door to open. This is the exact failure you'll diagnose in this module's lab.
2. Liveness probe too aggressive on a slow-starting app, with no startup probe — the app gets killed for "not being ready yet," restarts, and never gets far enough to pass.
3. Readiness probe that always passes (e.g. checking a static file instead of real dependencies) — traffic gets routed to a Pod that can't actually serve it.
4. No probes configured at all — Kubernetes has no way to know the container is stuck, so a deadlocked process just sits there serving nothing, forever, looking "Running" the whole time.

Instructor Notes

Approximate timing: 12-15 minutes for a live read-through with pauses for questions; 10 minutes for a self-paced read.

Demo plan: Before moving to the guided exercise, it's worth live-editing one Deployment's readinessProbe.httpGet.port in front of the class to an obviously wrong value and running kubectl get pods / kubectl describe pod together, so students see the Ready column and the Events section change in real time before they do it themselves.

Likely misconceptions to watch for:

• Students often assume a failing liveness probe and a failing readiness probe look the same in kubectl get pods — they don't (liveness failures show restart counts climbing; readiness failures show 0/1 Ready with no restarts). Worth calling out explicitly.
• Students sometimes think tcpSocket checks are "less safe" than httpGet — reframe as "checks a different, shallower thing," not "worse."

Discussion question: "Why might a team deliberately choose a readiness probe that fails under high load, even though it takes healthy capacity out of rotation?" (Answer: shedding load from an overloaded Pod is often better than serving every request slowly or with errors.)

Optional extension: Ask students to sketch what would happen if liveness and readiness probes were swapped (readiness probe restarts on failure, liveness probe just removes from Service) — this clarifies the distinction by making the wrong version visibly worse.