ced4568andClaude Sonnet 5
28c925069b
content/root/root-10-kubernetes-troubleshooting-observability/m01-health-probes/: lesson (probe concepts) -> demonstration (reading kubectl describe pod Events for a probe failure) -> knowledge check -> guided exercise (write a readiness probe on paper) -> Launch Lab intro (rootnode-probe-port-mismatch) -> reflection (root cause + Real Ops prevention takeaway). 5 of 6 activities carry instructor notes (demo plan, misconceptions, timing, discussion questions, extensions). This is the reference implementation docs/COURSE_AUTHORING_STANDARD.md documents for all future RootNode lesson authoring. Co-Authored-By: Claude Sonnet 5 <[email protected]>
62 lines
3.4 KiB
Markdown
62 lines
3.4 KiB
Markdown
# Root-Cause Reflection & Real Ops Takeaway
|
|
|
|
You just fixed a real incident pattern, not a puzzle. Before moving on, take a few minutes to
|
|
think through what actually happened and why it matters beyond this one Lab.
|
|
|
|
## What actually went wrong
|
|
|
|
The container was never broken. `nginx` started fine and was serving requests correctly the
|
|
entire time. The only thing wrong was that the readiness probe was configured to check a port
|
|
nginx wasn't listening on — so Kubernetes, working exactly as designed, correctly reported the
|
|
Pod as Not Ready based on the information it had. The system didn't fail; it was given wrong
|
|
information and acted rationally on it.
|
|
|
|
This is worth sitting with: most "Kubernetes is broken" incidents are actually "Kubernetes is
|
|
doing exactly what its configuration tells it to do, and the configuration is wrong."
|
|
|
|
## Reflection questions
|
|
|
|
Think through your own answers to these — there's no single correct wording, but you should be
|
|
able to explain each one out loud:
|
|
|
|
1. If you had jumped straight to `kubectl logs` instead of `kubectl describe pod`, how long do
|
|
you think it would have taken you to find this? What would the logs have actually shown you?
|
|
2. This Lab's port mismatch was introduced deliberately. What real-world change could introduce
|
|
the exact same failure by accident? (Consider: someone changes the container's listening
|
|
port in a new image version, but forgets the probe definition was hand-written separately.)
|
|
3. Why does a rolling update stall — not fail outright, just hang — when new Pods can't pass
|
|
their readiness probe? What is Kubernetes protecting you from by refusing to proceed?
|
|
|
|
## Real Ops takeaway: prevention, not just fixes
|
|
|
|
In a real production system, you'd want to prevent this class of incident from recurring, not
|
|
just fix it once:
|
|
|
|
- **Keep the probe port derived from the same source as the container's actual port**, where
|
|
your tooling allows it (a shared value in a Helm chart or Kustomize overlay), instead of two
|
|
independently hand-typed numbers that can drift apart silently.
|
|
- **Treat a stalled rollout as a first-class alert**, not just "the deploy is taking a while" —
|
|
a Deployment sitting at partial readiness for an unusual length of time is exactly this
|
|
failure mode, and catching it via rollout status is faster than waiting for a user-facing
|
|
outage report.
|
|
- **Practice reading Events before logs** for anything in the "Running but not Ready" state —
|
|
you just did this under low stakes; the habit is what pays off during a real incident at 2am.
|
|
|
|
This module is complete. Return to the course to continue with the next module.
|
|
|
|
## Instructor Notes
|
|
|
|
**Discussion questions for a live classroom:** use the three reflection questions above as
|
|
discussion prompts rather than a silent worksheet — question 2 in particular tends to produce
|
|
good real-world examples from students who've touched CI/CD pipelines before.
|
|
|
|
**Approximate timing:** 10 minutes for a live discussion; 5-8 minutes self-paced writing.
|
|
|
|
**Optional extension:** ask students to write the one-sentence postmortem summary they'd post
|
|
in a team channel for this incident (real on-call teams write these) — a good answer names the
|
|
root cause (probe port mismatch) and the fix (corrected port), not just "it's fixed now."
|
|
|
|
**Why this activity is a reflection, not another knowledge check:** the goal here is
|
|
consolidation and transfer to real-world judgment, not testing recall — resist turning this
|
|
into another right/wrong quiz.
|