feat(root): author the health-probes gold-standard module

content/root/root-10-kubernetes-troubleshooting-observability/m01-health-probes/: lesson (probe concepts) -> demonstration (reading kubectl describe pod Events for a probe failure) -> knowledge check -> guided exercise (write a readiness probe on paper) -> Launch Lab intro (rootnode-probe-port-mismatch) -> reflection (root cause + Real Ops prevention takeaway). 5 of 6 activities carry instructor notes (demo plan, misconceptions, timing, discussion questions, extensions). This is the reference implementation docs/COURSE_AUTHORING_STANDARD.md documents for all future RootNode lesson authoring. Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-25 23:29:32 -05:00
parent e2298e95de
commit 28c925069b
6 changed files with 382 additions and 0 deletions
@@ -0,0 +1,86 @@
 # Why Health Checks Exist & Probe Types
 Kubernetes cannot see inside your application. It doesn't know if your process is deadlocked,
 stuck in an infinite loop, or quietly failing every request while the process itself stays
 alive. Without a way to ask "are you actually working?", the only signal Kubernetes has is
 whether the container's process is still running — and a hung process still counts as running.
 **A health probe is how your application tells Kubernetes the truth.** It's a small check —
 an HTTP request, a TCP connection attempt, or a command run inside the container — that
 Kubernetes runs on a schedule and uses to make real decisions: restart this container, or stop
 sending it traffic.
 ## The three probe types
 Kubernetes supports three distinct probes, and they answer three different questions. Mixing
 them up is the single most common probe mistake in production.
 **Liveness probe — "Is this container still alive, or should it be restarted?"**
 If a liveness probe fails repeatedly, the kubelet kills the container and it gets restarted per
 its restart policy. Use this for detecting a genuinely stuck process — a deadlock, a crashed
 event loop. Don't use liveness to mean "not ready for traffic yet" — that's what readiness is
 for, and confusing the two causes the exact incident you're about to fix.
 **Readiness probe — "Is this container ready to receive traffic right now?"**
 If a readiness probe fails, the Pod is removed from the Service's list of endpoints —
 Kubernetes stops routing traffic to it, but does **not** restart the container. This is the
 right tool for "still starting up," "warming a cache," or "temporarily overloaded, back off for
 a bit." A Pod can be Running and healthy but still correctly Not Ready.
 **Startup probe — "Has this slow-starting container finished starting up yet?"**
 Some applications take a long time to become ready (loading a large dataset, running
 migrations). A startup probe disables liveness/readiness checks until it succeeds once, so a
 slow-but-healthy startup doesn't get killed by an impatient liveness probe. Once it succeeds,
 liveness and readiness take over as normal.
 ## How a probe is actually defined
 Every probe has a **mechanism** (how the check runs) and a **schedule** (how often, how long to
 wait, how many failures before acting):
 - `httpGet` — a GET request to a path and **port** on the container. A 2xx or 3xx response
  counts as success.
 - `tcpSocket` — a TCP connection attempt to a **port**. A successful connection counts as
  success — this doesn't check application-level health, just "is something listening."
 - `exec` — runs a command inside the container. Exit code `0` counts as success.
 Timing fields (`initialDelaySeconds`, `periodSeconds`, `timeoutSeconds`, `failureThreshold`,
 `successThreshold`) control how patient Kubernetes is before it acts — but no amount of correct
 timing fixes a probe that's pointed at the wrong port or path in the first place.
 ## Common production mistakes
 1. **Wrong port.** The probe's `httpGet.port` (or `tcpSocket.port`) doesn't match the port the
   application actually listens on. The container is completely healthy — Kubernetes is just
   asking the wrong door to open. This is the exact failure you'll diagnose in this module's lab.
 2. **Liveness probe too aggressive on a slow-starting app**, with no startup probe — the app
   gets killed for "not being ready yet," restarts, and never gets far enough to pass.
 3. **Readiness probe that always passes** (e.g. checking a static file instead of real
   dependencies) — traffic gets routed to a Pod that can't actually serve it.
 4. **No probes configured at all** — Kubernetes has no way to know the container is stuck, so a
   deadlocked process just sits there serving nothing, forever, looking "Running" the whole time.
 ## Instructor Notes
 **Approximate timing:** 12-15 minutes for a live read-through with pauses for questions;
 10 minutes for a self-paced read.
 **Demo plan:** Before moving to the guided exercise, it's worth live-editing one Deployment's
 `readinessProbe.httpGet.port` in front of the class to an obviously wrong value and running
 `kubectl get pods` / `kubectl describe pod` together, so students see the Ready column and the
 Events section change in real time before they do it themselves.
 **Likely misconceptions to watch for:**
 - Students often assume a failing liveness probe and a failing readiness probe look the same in
  `kubectl get pods` — they don't (liveness failures show restart counts climbing; readiness
  failures show `0/1` Ready with no restarts). Worth calling out explicitly.
 - Students sometimes think `tcpSocket` checks are "less safe" than `httpGet` — reframe as
  "checks a different, shallower thing," not "worse."
 **Discussion question:** "Why might a team deliberately choose a readiness probe that fails
 under high load, even though it takes healthy capacity out of rotation?" (Answer: shedding load
 from an overloaded Pod is often better than serving every request slowly or with errors.)
 **Optional extension:** Ask students to sketch what would happen if liveness and readiness
 probes were swapped (readiness probe restarts on failure, liveness probe just removes from
 Service) — this clarifies the distinction by making the wrong version visibly worse.
@@ -0,0 +1,71 @@
 # Demonstration: Probes in Action
 Watch what a probe failure actually looks like from the outside, before you're the one
 diagnosing it.
 Picture a Deployment called `web` running `nginx`, which listens on port 80. Someone adds a
 readiness probe but mistypes the port as `8080`:
 ```yaml
 readinessProbe:
  httpGet:
    path: /
    port: 8080
  initialDelaySeconds: 3
  periodSeconds: 5
 ```
 **What you'd see immediately after this rolls out:**
 ```
 $ kubectl get pods
 NAME                   READY   STATUS    RESTARTS   AGE
 web-7d8f9c6b5d-abcde   0/1     Running   0          45s
 ```
 `STATUS` says `Running` — the container started fine, nginx is up. But `READY` says `0/1` — the
 readiness probe is failing, so this Pod has never been added to its Service's endpoints. No
 traffic reaches it. `RESTARTS` stays at `0` because this is a readiness failure, not a liveness
 failure — nothing is being killed, it's just being excluded.
 **The trail that explains why**, from `kubectl describe pod`:
 ```
 Events:
  Type     Reason     Age   From     Message
  ----     ------     ----  ----     -------
  Normal   Scheduled  50s   default-scheduler  Successfully assigned default/web-... to node-1
  Normal   Pulled     49s   kubelet   Container image "nginx:1.25" already present on machine
  Normal   Created    49s   kubelet   Created container web
  Normal   Started    49s   kubelet   Started container web
  Warning  Unhealthy  15s (x6 over 40s)  kubelet   Readiness probe failed: Get "http://10.244.0.7:8080/": dial tcp 10.244.0.7:8080: connect: connection refused
 ```
 That last line is the whole story: kubelet tried to connect to port `8080` and nothing was
 listening there — because nginx is listening on `80`. The container was never broken. The probe
 was asking the wrong question.
 **The general diagnostic shape** for any probe issue: `kubectl get pods` tells you *that*
 something is wrong (Ready count, restart count). `kubectl describe pod` and its Events section
 tell you *what* is wrong, usually in plain English, including the exact port/path that was
 tried and the exact error. Reach for Events before logs — logs show you what the application
 did; Events show you what Kubernetes tried to do and how it failed.
 ## Instructor Notes
 **Demo script:** Live-run this exact sequence against a throwaway Deployment (or the classroom
 projector) rather than only showing static output — `kubectl get pods`, then
 `kubectl describe pod <name>`, scrolling straight to Events. Ask the class to predict the
 `READY` column and `RESTARTS` count *before* you show it.
 **Approximate timing:** 8-10 minutes live; ~5 minutes as a self-paced read.
 **Common misconception to preempt:** students often reach for `kubectl logs` first when
 something isn't Ready. For a probe misconfiguration, logs are frequently silent or unhelpful —
 the application is healthy and has nothing to say. This is a good moment to reinforce
 "Events explain what Kubernetes tried; logs explain what the app did," and that reading Events
 first is usually faster for this class of failure.
 **Optional extension:** show the same failure signature for a `tcpSocket` probe (a bare
 "connection refused" with no HTTP verb in the Events message) so students can tell the two
 mechanisms apart from Events alone.
@@ -0,0 +1,56 @@
 # Knowledge Check: Health Probes
 Answer each question yourself before reading the answer that follows it. This isn't graded —
 it's a checkpoint to catch a misunderstanding before you're troubleshooting a live Pod in the
 next activity.
 ---
 **1. A Pod shows `RESTARTS: 12` and `READY: 1/1`. What kind of probe failure, if any, is most
 consistent with this?**
 *Answer:* A liveness probe that was failing and has now recovered (or is intermittently
 failing) — the climbing restart count is the signature of liveness failures, since a failed
 liveness check causes the kubelet to kill and restart the container. `READY: 1/1` means it's
 currently passing its readiness check right now.
 ---
 **2. A Pod shows `RESTARTS: 0` and `READY: 0/1`, and has looked this way for several minutes.
 What kind of probe failure is most consistent with this?**
 *Answer:* A readiness probe that is failing. The container is running fine (no restarts), but
 Kubernetes has decided it isn't ready to receive traffic and has excluded it from the Service's
 endpoints. This is exactly the symptom you'll investigate in this module's lab.
 ---
 **3. Why does Kubernetes offer a separate startup probe instead of just making
 `initialDelaySeconds` on the liveness probe longer?**
 *Answer:* `initialDelaySeconds` is a fixed, one-size-fits-all wait before checks even begin —
 set it too short and a normally-slow startup gets killed; set it too long and every fast
 startup pays that same delay before liveness/readiness protection is even active. A startup
 probe instead keeps checking until the application actually reports ready (bounded by its own
 failure threshold), so liveness/readiness only take over once startup is confirmed done —
 correct for both a consistently slow app and one whose startup time varies.
 ---
 **4. True or false: a readiness probe failure will cause Kubernetes to restart the container.**
 *Answer:* False. A readiness probe failure only removes the Pod from Service endpoints — it
 does not restart anything. Only a liveness (or startup, before it succeeds once) probe failure
 causes a restart.
 ## Instructor Notes
 **Facilitation:** In a live classroom, ask these out loud and take verbal answers before
 revealing each one, rather than having students read this silently — the goal is to surface a
 wrong mental model while it's still cheap to correct.
 **Approximate timing:** 5 minutes.
 **Likely misconception this catches:** question 2 specifically catches students who default to
 "something is broken, kubectl logs will show an error" — for a pure readiness-probe
 misconfiguration, logs are usually clean, which is disorienting the first time you hit it.
@@ -0,0 +1,61 @@
 # Guided Exercise: Configure a Readiness Probe
 This is a paper-and-thinking exercise, not a Lab task — you won't run these commands against a
 real cluster here (that comes next). The goal is to practice reading and writing a probe
 definition correctly before you're diagnosing someone else's broken one.
 ## The scenario
 You're deploying a small HTTP API that listens on port `3000` and exposes a health endpoint at
 `/healthz`. It takes a couple of seconds to finish connecting to its database on startup — if a
 request arrives before that finishes, it correctly returns a `503` at `/healthz` until it's
 ready.
 ## Your task
 Work out, on paper (or in a scratch file), what you'd write for this container's readiness
 probe. Think through:
 1. Which mechanism fits — `httpGet`, `tcpSocket`, or `exec`? (Hint: you have a real health
   endpoint that returns a meaningful status code — use it rather than a shallower check.)
 2. What `port` and `path` should the probe target?
 3. Roughly what `initialDelaySeconds` makes sense, given the database connection takes a couple
   of seconds?
 4. Should this be a `readinessProbe`, a `livenessProbe`, or both? Justify your answer using the
   distinction from the lesson — what actually goes wrong if you get this specific choice wrong
   here?
 ## A worked shape to check yourself against
 Your answer should end up structurally similar to this (exact field names matter more than the
 precise numbers you chose):
 ```yaml
 readinessProbe:
  httpGet:
    path: /healthz
    port: 3000
  initialDelaySeconds: 3
  periodSeconds: 5
 ```
 If you reached for `livenessProbe` instead of `readinessProbe` for this specific scenario,
 re-read the lesson's distinction before continuing — "returns 503 while warming up" is
 precisely the readiness case, not the liveness case. A liveness probe here would restart a
 perfectly healthy, still-starting container over and over, which never lets it finish starting.
 ## Instructor Notes
 **Facilitation:** Have students write their probe block on paper or a shared doc before showing
 the worked shape — resist showing the answer early, since the value here is in students
 committing to an answer and comparing, not passively reading a correct one.
 **Approximate timing:** 10-15 minutes live with discussion; 10 minutes self-paced.
 **Common mistake to watch for:** choosing `livenessProbe` because "the app isn't healthy yet"
 — this is the single most common real-world probe mistake and exactly what question 4 is
 designed to surface before the next activity's real diagnostic task.
 **Optional extension:** ask what would need to change in this same YAML if the health endpoint
 took up to 60 seconds to become available on cold start (answer: a `startupProbe` with a
 generous `failureThreshold` × `periodSeconds`, so liveness/readiness don't engage too early).
@@ -0,0 +1,47 @@
 # Launch Lab: Stuck Not-Ready: The Probe Port Mismatch
 Time to apply everything from this module against a real, running incident — not a diagram, an
 actual Deployment in your own disposable Lab environment.
 ## What you're walking into
 A deployment called `api-gateway` has been rolled out. The pods start, the container runs — but
 the deployment never reaches Ready, and traffic isn't being routed to it. Somewhere on-call, an
 alert is firing.
 You now have everything you learned in this module to work with:
 - **The lesson** told you what "stuck at 0/2 Ready with 0 restarts" actually means (a readiness
  problem, not a crash).
 - **The demonstration** showed you exactly where to look for the explanation — `kubectl get
  pods`, then `kubectl describe pod` and its Events section — and what a port-mismatch failure
  looks like in that output.
 - **The guided exercise** had you reason through what a correct probe definition looks like,
  which is exactly what you'll need to compare the broken one against.
 ## Your task in this Lab
 Diagnose why the pods are failing their readiness check, correct the probe configuration, and
 restore the deployment to fully Ready. Start with pod status and events, the same way you just
 practiced — read before you patch.
 This Lab has its own in-context hints if you get stuck, revealed one at a time — use the
 concepts from this module first; reach for a hint only if you're genuinely stuck, not as the
 first move.
 When the Lab confirms your fix, come back to this module to close it out with a short
 reflection on what actually caused this.
 ## Instructor Notes
 **Classroom flow:** if teaching this live, this is the natural point to have students launch
 their own Lab environment and work independently or in pairs, while you circulate. The
 Real-Time Classroom dashboard's help-request and progress views (see the instructor Curriculum
 view) show who's stuck without needing to walk around asking.
 **Approximate timing:** 15-20 minutes for most students; some will resolve it in under 5 once
 the module's concepts have landed.
 **Common mistake to watch for:** students jumping straight to `kubectl logs` here, which will
 be unhelpful (nginx has nothing to say about a probe pointed at the wrong port) — this is a good
 moment to reinforce the demonstration's point about reading Events first for this failure class.
@@ -0,0 +1,61 @@
 # Root-Cause Reflection & Real Ops Takeaway
 You just fixed a real incident pattern, not a puzzle. Before moving on, take a few minutes to
 think through what actually happened and why it matters beyond this one Lab.
 ## What actually went wrong
 The container was never broken. `nginx` started fine and was serving requests correctly the
 entire time. The only thing wrong was that the readiness probe was configured to check a port
 nginx wasn't listening on — so Kubernetes, working exactly as designed, correctly reported the
 Pod as Not Ready based on the information it had. The system didn't fail; it was given wrong
 information and acted rationally on it.
 This is worth sitting with: most "Kubernetes is broken" incidents are actually "Kubernetes is
 doing exactly what its configuration tells it to do, and the configuration is wrong."
 ## Reflection questions
 Think through your own answers to these — there's no single correct wording, but you should be
 able to explain each one out loud:
 1. If you had jumped straight to `kubectl logs` instead of `kubectl describe pod`, how long do
   you think it would have taken you to find this? What would the logs have actually shown you?
 2. This Lab's port mismatch was introduced deliberately. What real-world change could introduce
   the exact same failure by accident? (Consider: someone changes the container's listening
   port in a new image version, but forgets the probe definition was hand-written separately.)
 3. Why does a rolling update stall — not fail outright, just hang — when new Pods can't pass
   their readiness probe? What is Kubernetes protecting you from by refusing to proceed?
 ## Real Ops takeaway: prevention, not just fixes
 In a real production system, you'd want to prevent this class of incident from recurring, not
 just fix it once:
 - **Keep the probe port derived from the same source as the container's actual port**, where
  your tooling allows it (a shared value in a Helm chart or Kustomize overlay), instead of two
  independently hand-typed numbers that can drift apart silently.
 - **Treat a stalled rollout as a first-class alert**, not just "the deploy is taking a while" —
  a Deployment sitting at partial readiness for an unusual length of time is exactly this
  failure mode, and catching it via rollout status is faster than waiting for a user-facing
  outage report.
 - **Practice reading Events before logs** for anything in the "Running but not Ready" state —
  you just did this under low stakes; the habit is what pays off during a real incident at 2am.
 This module is complete. Return to the course to continue with the next module.
 ## Instructor Notes
 **Discussion questions for a live classroom:** use the three reflection questions above as
 discussion prompts rather than a silent worksheet — question 2 in particular tends to produce
 good real-world examples from students who've touched CI/CD pipelines before.
 **Approximate timing:** 10 minutes for a live discussion; 5-8 minutes self-paced writing.
 **Optional extension:** ask students to write the one-sentence postmortem summary they'd post
 in a team channel for this incident (real on-call teams write these) — a good answer names the
 root cause (probe port mismatch) and the fix (corrected port), not just "it's fixed now."
 **Why this activity is a reflection, not another knowledge check:** the goal here is
 consolidation and transfer to real-world judgment, not testing recall — resist turning this
 into another right/wrong quiz.