feat(root): expand troubleshooting and observability curriculum

The Phase 6A gold-standard Health Probes module is preserved byte-for- byte unchanged — it's the template, not a page to rewrite for consistency. root-10-m03-troubleshooting-framework — a new PARTIAL module introducing a repeatable Observe -> Gather Evidence -> Form Hypothesis -> Test -> Repair -> Validate -> Prevent Recurrence framework, applied to a CrashLoopBackOff and a broken Service selector. Lesson, demonstration (the full framework walked through a real CrashLoopBackOff, command by command), knowledge check, and real Labs (crashloop-fix, endpoint-fix-task). Sequenced between the gold module and the existing bare-lab bank, which now sits behind it. Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-26 09:07:34 -05:00
parent 960a638ec9
commit 831486d664
3 changed files with 233 additions and 0 deletions
@@ -0,0 +1,92 @@
+# The RootNode Troubleshooting Framework
+
+## Why a framework, not just "get good at kubectl"
+
+The gold-standard Health Probes module taught you a lot about ONE specific failure mode
+(readiness probe misconfiguration) and the specific commands to diagnose it. This module steps
+back and asks: what's the repeatable PROCESS that works for almost any Kubernetes failure, not
+just the ones you've already seen before?
+
+Guessing under pressure — trying random fixes and seeing what sticks — is slow, often makes
+things worse, and doesn't build a transferable skill. A framework does.
+
+## Observe → Gather Evidence → Form Hypothesis → Test → Repair → Validate → Prevent Recurrence
+
+**Observe** — what's the actual, specific symptom? Not "it's broken," but "the Deployment shows
+0/3 Ready" or "requests are timing out" or "the Pod restarts every 30 seconds." Precision here
+saves time later.
+
+**Gather Evidence** — before touching anything, collect what the system itself is telling you:
+`kubectl get pods`, `kubectl describe pod`, `kubectl logs`, `kubectl get events`. Resist the urge
+to start changing things before you've looked.
+
+**Form Hypothesis** — based on the evidence, state a SPECIFIC guess about the cause. "The
+container is crashing because the command it's told to run doesn't exist" is testable. "Something's
+wrong with networking" is not specific enough to act on yet.
+
+**Test** — check whether your hypothesis is actually true, ideally without yet making the fix
+(read the exact command being run, compare against what the image expects) — confirm before you
+commit to a repair.
+
+**Repair** — make the minimal change that addresses the confirmed cause. Not a bigger change
+"while you're in there" — exactly the fix the evidence supports.
+
+**Validate** — confirm the fix actually worked, using the same kind of evidence you gathered
+initially (not just "I made a change, it should be fine now").
+
+**Prevent Recurrence** — the step most often skipped under time pressure. Could this class of
+mistake happen again? Is there a smaller, structural change (a default, a validation, a check)
+that would catch it earlier next time?
+
+## Applying it: a CrashLoopBackOff
+
+- **Observe:** `crash-app`'s Pods show `STATUS: CrashLoopBackOff`, restart count climbing.
+- **Gather Evidence:** `kubectl describe deployment crash-app` (check the configured command),
+  `kubectl logs -l app=crash-app` (what did the container actually print before dying, if
+  anything).
+- **Form Hypothesis:** "The command override is invalid/wrong for this image, so the container
+  exits immediately every time it starts."
+- **Test:** compare the command actually configured against what the image's default/expected
+  entrypoint should be.
+- **Repair:** fix the command to what the container image actually expects.
+- **Validate:** `kubectl get pods` shows `Running`, restart count stops climbing.
+- **Prevent Recurrence:** a code-review habit of never overriding a container's command without
+  verifying it against the image's documentation first.
+
+## Applying it: a broken Service selector
+
+- **Observe:** traffic isn't reaching `my-svc`'s backend at all.
+- **Gather Evidence:** `kubectl get endpoints my-svc` (empty?), `kubectl get pods
+  --show-labels` (what labels do the actual Pods have?).
+- **Form Hypothesis:** "The Service's selector doesn't match the Pods' actual labels."
+- **Test:** compare `kubectl describe service my-svc`'s selector against the Pods' real labels
+  directly, side by side.
+- **Repair:** correct the selector to match.
+- **Validate:** `kubectl get endpoints my-svc` now shows the Pod IPs.
+- **Prevent Recurrence:** keeping label keys/values consistent (or generated from one source)
+  between a Deployment's Pod template and any Service that targets it.
+
+## Instructor Notes
+
+**Approximate timing:** 15 minutes read; 10-12 minutes live walking through both worked
+examples.
+
+**Demo plan:** Use the demonstration activity right after this lesson to show the CrashLoopBackOff
+example live, narrating each framework step explicitly as you go rather than just running
+commands.
+
+**Likely misconceptions to watch for:**
+- Students under time pressure often skip straight to "Repair" without confirming a hypothesis
+  first. Reinforce that a wrong guess implemented as a "fix" can make evidence harder to read
+  afterward, not easier.
+- "Prevent Recurrence" is the step most likely to be treated as optional/skippable. Frame it as
+  what separates a team that keeps hitting the same incident from one that doesn't.
+
+**Discussion question:** "What's the cost of skipping 'Gather Evidence' and jumping straight to
+a guessed fix, even if the guess happens to be right?" (Answer: even a correct guess doesn't
+teach you whether your reasoning was right — you got lucky, and the same class of mistake could
+still recur since you never confirmed the actual cause.)
+
+**Optional extension:** ask students to apply this exact framework, from memory, to the gold
+module's `rootnode-probe-port-mismatch` incident they already solved — a good retroactive
+exercise connecting a lab they already completed to the generalized framework.
@@ -0,0 +1,91 @@
+# Demonstration: Applying the Framework to a CrashLoopBackOff
+
+Watch the full Observe → Evidence → Hypothesis → Test → Repair → Validate → Prevent sequence
+against a real symptom, narrated step by step.
+
+## Observe
+
+```
+$ kubectl get pods
+NAME                        READY   STATUS             RESTARTS   AGE
+crash-app-6d9f8b7c5d-abcde  0/1     CrashLoopBackOff    5          4m
+```
+
+Specific symptom: `CrashLoopBackOff`, restart count climbing (5 and counting) — the container
+is starting, exiting, and being restarted repeatedly, not just failing to start once.
+
+## Gather Evidence
+
+```
+$ kubectl logs crash-app-6d9f8b7c5d-abcde
+nginx: [emerg] invalid command "daemon-off": expected valid command
+```
+
+```
+$ kubectl describe deployment crash-app
+...
+    Command:
+      nginx
+      -g
+      daemon-off;
+...
+```
+
+The logs show nginx itself rejecting the command it was given. The Deployment's configured
+command shows exactly why: `daemon-off;` (a typo — missing space, should be `daemon off;`).
+
+## Form Hypothesis
+
+"The container's command override has a syntax error that nginx can't parse, so nginx exits
+immediately every time it starts — which is why it's crash-looping rather than just failing to
+start once."
+
+## Test
+
+Compare the broken command (`daemon-off;`) against nginx's actual expected syntax
+(`daemon off;`, two separate words) — confirmed: this is exactly the kind of typo that would
+produce exactly this error.
+
+## Repair
+
+```
+$ kubectl patch deployment crash-app --type='json' \
+  -p='[{"op":"replace","path":"/spec/template/spec/containers/0/command","value":["nginx","-g","daemon off;"]}]'
+```
+
+## Validate
+
+```
+$ kubectl get pods
+NAME                        READY   STATUS    RESTARTS   AGE
+crash-app-7f8c9d6b5d-fghij  1/1     Running   0          15s
+```
+
+New Pod, `Running`, `RESTARTS: 0` — the fix is confirmed, not just assumed.
+
+## Prevent Recurrence
+
+A command override like this is a hand-typed string with no validation until the container
+actually tries to run it. A team that hit this once might decide: don't override a well-known
+image's default command unless there's a specific reason to, and if you must, test the exact
+command string locally (`docker run` or similar) before deploying it — catching a typo in
+seconds locally beats debugging a CrashLoopBackOff in the cluster.
+
+## Instructor Notes
+
+**Demo script:** Run this live if time allows, deliberately narrating which framework step
+you're on at each command — students retain the framework far better tied to real commands than
+as an abstract list.
+
+**Approximate timing:** 8-10 minutes live; 5-7 minutes self-paced.
+
+**Common misconception to preempt:** students sometimes skip straight from "Observe" to
+"Repair" once they spot the typo, without stating a hypothesis or testing it. Reinforce that
+even an "obvious" fix benefits from the explicit hypothesis step — it's what makes the eventual
+Validate step meaningful (you're confirming a SPECIFIC claim, not just "trying something").
+
+**Optional extension:** ask what evidence you'd gather differently if the logs had been
+completely empty instead of showing a clear nginx error (a good bridge to cases where the
+container dies before it can log anything at all — `kubectl describe pod`'s Events section
+becomes the primary evidence source in that case, same as the gold module's probe-mismatch
+lesson).
@@ -0,0 +1,50 @@
+# Knowledge Check: The Troubleshooting Framework
+
+---
+
+**1. You spot what looks like an obvious typo causing a failure and fix it immediately without
+writing down a hypothesis first. What's the actual cost of skipping that step, even if the fix
+works?**
+
+*Answer:* You don't get a confirmed, testable claim about the cause — if the fix happens to
+"work" for an unrelated reason (a coincidental restart, a different underlying issue also
+resolving), you'd have no way to distinguish that from your fix actually being correct. The
+hypothesis step is what makes Validate meaningful.
+
+---
+
+**2. During the "Gather Evidence" step for a CrashLoopBackOff, which two commands give you the
+most direct evidence, and what does each one tell you that the other doesn't?**
+
+*Answer:* `kubectl logs` (what the container itself printed, if anything, before exiting) and
+`kubectl describe pod`/`describe deployment` (the container's actual configuration, plus recent
+Events). Logs show application-level output; describe/Events show what Kubernetes itself
+observed and configured — a crash with no logs at all often means the failure happened before
+the application could log anything, which describe/Events can still reveal.
+
+---
+
+**3. True or false: "Prevent Recurrence" only applies to major incidents, not small
+one-off mistakes like a typo.**
+
+*Answer:* False. A small mistake that recurs repeatedly across a team becomes a real, ongoing
+cost even if no single instance is severe — the framework applies the same reasoning regardless
+of incident size: could a small, structural change catch this earlier next time?
+
+---
+
+**4. A broken Service selector (endpoint-fix-task's scenario) and a CrashLoopBackOff
+(crashloop-fix's scenario) are very different failures. What's the SAME about how you'd
+approach both?**
+
+*Answer:* The same seven-step framework — Observe the specific symptom, Gather Evidence before
+guessing, form a specific testable Hypothesis, Test it, make the minimal Repair the evidence
+supports, Validate with the same kind of evidence you started with, and consider how to Prevent
+Recurrence. The framework doesn't change; only the specific commands and evidence do.
+
+## Instructor Notes
+
+**Facilitation:** Question 4 is the whole point of this module — make sure it's discussed even
+if answered correctly, since it's the transferability claim the entire framework rests on.
+
+**Approximate timing:** 5-7 minutes.