From 28c925069b52d689dd94feff23a0cf4bfd82fcdf Mon Sep 17 00:00:00 2001 From: ced4568 Date: Tue, 25 Aug 2026 23:29:32 -0500 Subject: [PATCH] feat(root): author the health-probes gold-standard module content/root/root-10-kubernetes-troubleshooting-observability/m01-health-probes/: lesson (probe concepts) -> demonstration (reading kubectl describe pod Events for a probe failure) -> knowledge check -> guided exercise (write a readiness probe on paper) -> Launch Lab intro (rootnode-probe-port-mismatch) -> reflection (root cause + Real Ops prevention takeaway). 5 of 6 activities carry instructor notes (demo plan, misconceptions, timing, discussion questions, extensions). This is the reference implementation docs/COURSE_AUTHORING_STANDARD.md documents for all future RootNode lesson authoring. Co-Authored-By: Claude Sonnet 5 --- .../01-lesson-health-check-concepts.md | 86 +++++++++++++++++++ .../02-demonstration-probes-in-action.md | 71 +++++++++++++++ .../m01-health-probes/03-knowledge-check.md | 56 ++++++++++++ .../04-guided-exercise-readiness-probe.md | 61 +++++++++++++ .../m01-health-probes/05-lab-intro.md | 47 ++++++++++ .../06-reflection-real-ops-takeaway.md | 61 +++++++++++++ 6 files changed, 382 insertions(+) create mode 100644 content/root/root-10-kubernetes-troubleshooting-observability/m01-health-probes/01-lesson-health-check-concepts.md create mode 100644 content/root/root-10-kubernetes-troubleshooting-observability/m01-health-probes/02-demonstration-probes-in-action.md create mode 100644 content/root/root-10-kubernetes-troubleshooting-observability/m01-health-probes/03-knowledge-check.md create mode 100644 content/root/root-10-kubernetes-troubleshooting-observability/m01-health-probes/04-guided-exercise-readiness-probe.md create mode 100644 content/root/root-10-kubernetes-troubleshooting-observability/m01-health-probes/05-lab-intro.md create mode 100644 content/root/root-10-kubernetes-troubleshooting-observability/m01-health-probes/06-reflection-real-ops-takeaway.md diff --git a/content/root/root-10-kubernetes-troubleshooting-observability/m01-health-probes/01-lesson-health-check-concepts.md b/content/root/root-10-kubernetes-troubleshooting-observability/m01-health-probes/01-lesson-health-check-concepts.md new file mode 100644 index 0000000..5d63a46 --- /dev/null +++ b/content/root/root-10-kubernetes-troubleshooting-observability/m01-health-probes/01-lesson-health-check-concepts.md @@ -0,0 +1,86 @@ +# Why Health Checks Exist & Probe Types + +Kubernetes cannot see inside your application. It doesn't know if your process is deadlocked, +stuck in an infinite loop, or quietly failing every request while the process itself stays +alive. Without a way to ask "are you actually working?", the only signal Kubernetes has is +whether the container's process is still running — and a hung process still counts as running. + +**A health probe is how your application tells Kubernetes the truth.** It's a small check — +an HTTP request, a TCP connection attempt, or a command run inside the container — that +Kubernetes runs on a schedule and uses to make real decisions: restart this container, or stop +sending it traffic. + +## The three probe types + +Kubernetes supports three distinct probes, and they answer three different questions. Mixing +them up is the single most common probe mistake in production. + +**Liveness probe — "Is this container still alive, or should it be restarted?"** +If a liveness probe fails repeatedly, the kubelet kills the container and it gets restarted per +its restart policy. Use this for detecting a genuinely stuck process — a deadlock, a crashed +event loop. Don't use liveness to mean "not ready for traffic yet" — that's what readiness is +for, and confusing the two causes the exact incident you're about to fix. + +**Readiness probe — "Is this container ready to receive traffic right now?"** +If a readiness probe fails, the Pod is removed from the Service's list of endpoints — +Kubernetes stops routing traffic to it, but does **not** restart the container. This is the +right tool for "still starting up," "warming a cache," or "temporarily overloaded, back off for +a bit." A Pod can be Running and healthy but still correctly Not Ready. + +**Startup probe — "Has this slow-starting container finished starting up yet?"** +Some applications take a long time to become ready (loading a large dataset, running +migrations). A startup probe disables liveness/readiness checks until it succeeds once, so a +slow-but-healthy startup doesn't get killed by an impatient liveness probe. Once it succeeds, +liveness and readiness take over as normal. + +## How a probe is actually defined + +Every probe has a **mechanism** (how the check runs) and a **schedule** (how often, how long to +wait, how many failures before acting): + +- `httpGet` — a GET request to a path and **port** on the container. A 2xx or 3xx response + counts as success. +- `tcpSocket` — a TCP connection attempt to a **port**. A successful connection counts as + success — this doesn't check application-level health, just "is something listening." +- `exec` — runs a command inside the container. Exit code `0` counts as success. + +Timing fields (`initialDelaySeconds`, `periodSeconds`, `timeoutSeconds`, `failureThreshold`, +`successThreshold`) control how patient Kubernetes is before it acts — but no amount of correct +timing fixes a probe that's pointed at the wrong port or path in the first place. + +## Common production mistakes + +1. **Wrong port.** The probe's `httpGet.port` (or `tcpSocket.port`) doesn't match the port the + application actually listens on. The container is completely healthy — Kubernetes is just + asking the wrong door to open. This is the exact failure you'll diagnose in this module's lab. +2. **Liveness probe too aggressive on a slow-starting app**, with no startup probe — the app + gets killed for "not being ready yet," restarts, and never gets far enough to pass. +3. **Readiness probe that always passes** (e.g. checking a static file instead of real + dependencies) — traffic gets routed to a Pod that can't actually serve it. +4. **No probes configured at all** — Kubernetes has no way to know the container is stuck, so a + deadlocked process just sits there serving nothing, forever, looking "Running" the whole time. + +## Instructor Notes + +**Approximate timing:** 12-15 minutes for a live read-through with pauses for questions; +10 minutes for a self-paced read. + +**Demo plan:** Before moving to the guided exercise, it's worth live-editing one Deployment's +`readinessProbe.httpGet.port` in front of the class to an obviously wrong value and running +`kubectl get pods` / `kubectl describe pod` together, so students see the Ready column and the +Events section change in real time before they do it themselves. + +**Likely misconceptions to watch for:** +- Students often assume a failing liveness probe and a failing readiness probe look the same in + `kubectl get pods` — they don't (liveness failures show restart counts climbing; readiness + failures show `0/1` Ready with no restarts). Worth calling out explicitly. +- Students sometimes think `tcpSocket` checks are "less safe" than `httpGet` — reframe as + "checks a different, shallower thing," not "worse." + +**Discussion question:** "Why might a team deliberately choose a readiness probe that fails +under high load, even though it takes healthy capacity out of rotation?" (Answer: shedding load +from an overloaded Pod is often better than serving every request slowly or with errors.) + +**Optional extension:** Ask students to sketch what would happen if liveness and readiness +probes were swapped (readiness probe restarts on failure, liveness probe just removes from +Service) — this clarifies the distinction by making the wrong version visibly worse. diff --git a/content/root/root-10-kubernetes-troubleshooting-observability/m01-health-probes/02-demonstration-probes-in-action.md b/content/root/root-10-kubernetes-troubleshooting-observability/m01-health-probes/02-demonstration-probes-in-action.md new file mode 100644 index 0000000..309228b --- /dev/null +++ b/content/root/root-10-kubernetes-troubleshooting-observability/m01-health-probes/02-demonstration-probes-in-action.md @@ -0,0 +1,71 @@ +# Demonstration: Probes in Action + +Watch what a probe failure actually looks like from the outside, before you're the one +diagnosing it. + +Picture a Deployment called `web` running `nginx`, which listens on port 80. Someone adds a +readiness probe but mistypes the port as `8080`: + +```yaml +readinessProbe: + httpGet: + path: / + port: 8080 + initialDelaySeconds: 3 + periodSeconds: 5 +``` + +**What you'd see immediately after this rolls out:** + +``` +$ kubectl get pods +NAME READY STATUS RESTARTS AGE +web-7d8f9c6b5d-abcde 0/1 Running 0 45s +``` + +`STATUS` says `Running` — the container started fine, nginx is up. But `READY` says `0/1` — the +readiness probe is failing, so this Pod has never been added to its Service's endpoints. No +traffic reaches it. `RESTARTS` stays at `0` because this is a readiness failure, not a liveness +failure — nothing is being killed, it's just being excluded. + +**The trail that explains why**, from `kubectl describe pod`: + +``` +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Scheduled 50s default-scheduler Successfully assigned default/web-... to node-1 + Normal Pulled 49s kubelet Container image "nginx:1.25" already present on machine + Normal Created 49s kubelet Created container web + Normal Started 49s kubelet Started container web + Warning Unhealthy 15s (x6 over 40s) kubelet Readiness probe failed: Get "http://10.244.0.7:8080/": dial tcp 10.244.0.7:8080: connect: connection refused +``` + +That last line is the whole story: kubelet tried to connect to port `8080` and nothing was +listening there — because nginx is listening on `80`. The container was never broken. The probe +was asking the wrong question. + +**The general diagnostic shape** for any probe issue: `kubectl get pods` tells you *that* +something is wrong (Ready count, restart count). `kubectl describe pod` and its Events section +tell you *what* is wrong, usually in plain English, including the exact port/path that was +tried and the exact error. Reach for Events before logs — logs show you what the application +did; Events show you what Kubernetes tried to do and how it failed. + +## Instructor Notes + +**Demo script:** Live-run this exact sequence against a throwaway Deployment (or the classroom +projector) rather than only showing static output — `kubectl get pods`, then +`kubectl describe pod `, scrolling straight to Events. Ask the class to predict the +`READY` column and `RESTARTS` count *before* you show it. + +**Approximate timing:** 8-10 minutes live; ~5 minutes as a self-paced read. + +**Common misconception to preempt:** students often reach for `kubectl logs` first when +something isn't Ready. For a probe misconfiguration, logs are frequently silent or unhelpful — +the application is healthy and has nothing to say. This is a good moment to reinforce +"Events explain what Kubernetes tried; logs explain what the app did," and that reading Events +first is usually faster for this class of failure. + +**Optional extension:** show the same failure signature for a `tcpSocket` probe (a bare +"connection refused" with no HTTP verb in the Events message) so students can tell the two +mechanisms apart from Events alone. diff --git a/content/root/root-10-kubernetes-troubleshooting-observability/m01-health-probes/03-knowledge-check.md b/content/root/root-10-kubernetes-troubleshooting-observability/m01-health-probes/03-knowledge-check.md new file mode 100644 index 0000000..f0650c5 --- /dev/null +++ b/content/root/root-10-kubernetes-troubleshooting-observability/m01-health-probes/03-knowledge-check.md @@ -0,0 +1,56 @@ +# Knowledge Check: Health Probes + +Answer each question yourself before reading the answer that follows it. This isn't graded — +it's a checkpoint to catch a misunderstanding before you're troubleshooting a live Pod in the +next activity. + +--- + +**1. A Pod shows `RESTARTS: 12` and `READY: 1/1`. What kind of probe failure, if any, is most +consistent with this?** + +*Answer:* A liveness probe that was failing and has now recovered (or is intermittently +failing) — the climbing restart count is the signature of liveness failures, since a failed +liveness check causes the kubelet to kill and restart the container. `READY: 1/1` means it's +currently passing its readiness check right now. + +--- + +**2. A Pod shows `RESTARTS: 0` and `READY: 0/1`, and has looked this way for several minutes. +What kind of probe failure is most consistent with this?** + +*Answer:* A readiness probe that is failing. The container is running fine (no restarts), but +Kubernetes has decided it isn't ready to receive traffic and has excluded it from the Service's +endpoints. This is exactly the symptom you'll investigate in this module's lab. + +--- + +**3. Why does Kubernetes offer a separate startup probe instead of just making +`initialDelaySeconds` on the liveness probe longer?** + +*Answer:* `initialDelaySeconds` is a fixed, one-size-fits-all wait before checks even begin — +set it too short and a normally-slow startup gets killed; set it too long and every fast +startup pays that same delay before liveness/readiness protection is even active. A startup +probe instead keeps checking until the application actually reports ready (bounded by its own +failure threshold), so liveness/readiness only take over once startup is confirmed done — +correct for both a consistently slow app and one whose startup time varies. + +--- + +**4. True or false: a readiness probe failure will cause Kubernetes to restart the container.** + +*Answer:* False. A readiness probe failure only removes the Pod from Service endpoints — it +does not restart anything. Only a liveness (or startup, before it succeeds once) probe failure +causes a restart. + +## Instructor Notes + +**Facilitation:** In a live classroom, ask these out loud and take verbal answers before +revealing each one, rather than having students read this silently — the goal is to surface a +wrong mental model while it's still cheap to correct. + +**Approximate timing:** 5 minutes. + +**Likely misconception this catches:** question 2 specifically catches students who default to +"something is broken, kubectl logs will show an error" — for a pure readiness-probe +misconfiguration, logs are usually clean, which is disorienting the first time you hit it. diff --git a/content/root/root-10-kubernetes-troubleshooting-observability/m01-health-probes/04-guided-exercise-readiness-probe.md b/content/root/root-10-kubernetes-troubleshooting-observability/m01-health-probes/04-guided-exercise-readiness-probe.md new file mode 100644 index 0000000..4a04cdb --- /dev/null +++ b/content/root/root-10-kubernetes-troubleshooting-observability/m01-health-probes/04-guided-exercise-readiness-probe.md @@ -0,0 +1,61 @@ +# Guided Exercise: Configure a Readiness Probe + +This is a paper-and-thinking exercise, not a Lab task — you won't run these commands against a +real cluster here (that comes next). The goal is to practice reading and writing a probe +definition correctly before you're diagnosing someone else's broken one. + +## The scenario + +You're deploying a small HTTP API that listens on port `3000` and exposes a health endpoint at +`/healthz`. It takes a couple of seconds to finish connecting to its database on startup — if a +request arrives before that finishes, it correctly returns a `503` at `/healthz` until it's +ready. + +## Your task + +Work out, on paper (or in a scratch file), what you'd write for this container's readiness +probe. Think through: + +1. Which mechanism fits — `httpGet`, `tcpSocket`, or `exec`? (Hint: you have a real health + endpoint that returns a meaningful status code — use it rather than a shallower check.) +2. What `port` and `path` should the probe target? +3. Roughly what `initialDelaySeconds` makes sense, given the database connection takes a couple + of seconds? +4. Should this be a `readinessProbe`, a `livenessProbe`, or both? Justify your answer using the + distinction from the lesson — what actually goes wrong if you get this specific choice wrong + here? + +## A worked shape to check yourself against + +Your answer should end up structurally similar to this (exact field names matter more than the +precise numbers you chose): + +```yaml +readinessProbe: + httpGet: + path: /healthz + port: 3000 + initialDelaySeconds: 3 + periodSeconds: 5 +``` + +If you reached for `livenessProbe` instead of `readinessProbe` for this specific scenario, +re-read the lesson's distinction before continuing — "returns 503 while warming up" is +precisely the readiness case, not the liveness case. A liveness probe here would restart a +perfectly healthy, still-starting container over and over, which never lets it finish starting. + +## Instructor Notes + +**Facilitation:** Have students write their probe block on paper or a shared doc before showing +the worked shape — resist showing the answer early, since the value here is in students +committing to an answer and comparing, not passively reading a correct one. + +**Approximate timing:** 10-15 minutes live with discussion; 10 minutes self-paced. + +**Common mistake to watch for:** choosing `livenessProbe` because "the app isn't healthy yet" +— this is the single most common real-world probe mistake and exactly what question 4 is +designed to surface before the next activity's real diagnostic task. + +**Optional extension:** ask what would need to change in this same YAML if the health endpoint +took up to 60 seconds to become available on cold start (answer: a `startupProbe` with a +generous `failureThreshold` × `periodSeconds`, so liveness/readiness don't engage too early). diff --git a/content/root/root-10-kubernetes-troubleshooting-observability/m01-health-probes/05-lab-intro.md b/content/root/root-10-kubernetes-troubleshooting-observability/m01-health-probes/05-lab-intro.md new file mode 100644 index 0000000..d65ccd0 --- /dev/null +++ b/content/root/root-10-kubernetes-troubleshooting-observability/m01-health-probes/05-lab-intro.md @@ -0,0 +1,47 @@ +# Launch Lab: Stuck Not-Ready: The Probe Port Mismatch + +Time to apply everything from this module against a real, running incident — not a diagram, an +actual Deployment in your own disposable Lab environment. + +## What you're walking into + +A deployment called `api-gateway` has been rolled out. The pods start, the container runs — but +the deployment never reaches Ready, and traffic isn't being routed to it. Somewhere on-call, an +alert is firing. + +You now have everything you learned in this module to work with: + +- **The lesson** told you what "stuck at 0/2 Ready with 0 restarts" actually means (a readiness + problem, not a crash). +- **The demonstration** showed you exactly where to look for the explanation — `kubectl get + pods`, then `kubectl describe pod` and its Events section — and what a port-mismatch failure + looks like in that output. +- **The guided exercise** had you reason through what a correct probe definition looks like, + which is exactly what you'll need to compare the broken one against. + +## Your task in this Lab + +Diagnose why the pods are failing their readiness check, correct the probe configuration, and +restore the deployment to fully Ready. Start with pod status and events, the same way you just +practiced — read before you patch. + +This Lab has its own in-context hints if you get stuck, revealed one at a time — use the +concepts from this module first; reach for a hint only if you're genuinely stuck, not as the +first move. + +When the Lab confirms your fix, come back to this module to close it out with a short +reflection on what actually caused this. + +## Instructor Notes + +**Classroom flow:** if teaching this live, this is the natural point to have students launch +their own Lab environment and work independently or in pairs, while you circulate. The +Real-Time Classroom dashboard's help-request and progress views (see the instructor Curriculum +view) show who's stuck without needing to walk around asking. + +**Approximate timing:** 15-20 minutes for most students; some will resolve it in under 5 once +the module's concepts have landed. + +**Common mistake to watch for:** students jumping straight to `kubectl logs` here, which will +be unhelpful (nginx has nothing to say about a probe pointed at the wrong port) — this is a good +moment to reinforce the demonstration's point about reading Events first for this failure class. diff --git a/content/root/root-10-kubernetes-troubleshooting-observability/m01-health-probes/06-reflection-real-ops-takeaway.md b/content/root/root-10-kubernetes-troubleshooting-observability/m01-health-probes/06-reflection-real-ops-takeaway.md new file mode 100644 index 0000000..6f6c3c3 --- /dev/null +++ b/content/root/root-10-kubernetes-troubleshooting-observability/m01-health-probes/06-reflection-real-ops-takeaway.md @@ -0,0 +1,61 @@ +# Root-Cause Reflection & Real Ops Takeaway + +You just fixed a real incident pattern, not a puzzle. Before moving on, take a few minutes to +think through what actually happened and why it matters beyond this one Lab. + +## What actually went wrong + +The container was never broken. `nginx` started fine and was serving requests correctly the +entire time. The only thing wrong was that the readiness probe was configured to check a port +nginx wasn't listening on — so Kubernetes, working exactly as designed, correctly reported the +Pod as Not Ready based on the information it had. The system didn't fail; it was given wrong +information and acted rationally on it. + +This is worth sitting with: most "Kubernetes is broken" incidents are actually "Kubernetes is +doing exactly what its configuration tells it to do, and the configuration is wrong." + +## Reflection questions + +Think through your own answers to these — there's no single correct wording, but you should be +able to explain each one out loud: + +1. If you had jumped straight to `kubectl logs` instead of `kubectl describe pod`, how long do + you think it would have taken you to find this? What would the logs have actually shown you? +2. This Lab's port mismatch was introduced deliberately. What real-world change could introduce + the exact same failure by accident? (Consider: someone changes the container's listening + port in a new image version, but forgets the probe definition was hand-written separately.) +3. Why does a rolling update stall — not fail outright, just hang — when new Pods can't pass + their readiness probe? What is Kubernetes protecting you from by refusing to proceed? + +## Real Ops takeaway: prevention, not just fixes + +In a real production system, you'd want to prevent this class of incident from recurring, not +just fix it once: + +- **Keep the probe port derived from the same source as the container's actual port**, where + your tooling allows it (a shared value in a Helm chart or Kustomize overlay), instead of two + independently hand-typed numbers that can drift apart silently. +- **Treat a stalled rollout as a first-class alert**, not just "the deploy is taking a while" — + a Deployment sitting at partial readiness for an unusual length of time is exactly this + failure mode, and catching it via rollout status is faster than waiting for a user-facing + outage report. +- **Practice reading Events before logs** for anything in the "Running but not Ready" state — + you just did this under low stakes; the habit is what pays off during a real incident at 2am. + +This module is complete. Return to the course to continue with the next module. + +## Instructor Notes + +**Discussion questions for a live classroom:** use the three reflection questions above as +discussion prompts rather than a silent worksheet — question 2 in particular tends to produce +good real-world examples from students who've touched CI/CD pipelines before. + +**Approximate timing:** 10 minutes for a live discussion; 5-8 minutes self-paced writing. + +**Optional extension:** ask students to write the one-sentence postmortem summary they'd post +in a team channel for this incident (real on-call teams write these) — a good answer names the +root cause (probe port mismatch) and the fix (corrected port), not just "it's fixed now." + +**Why this activity is a reflection, not another knowledge check:** the goal here is +consolidation and transfer to real-world judgment, not testing recall — resist turning this +into another right/wrong quiz.