feat(root): expand troubleshooting and observability curriculum

The Phase 6A gold-standard Health Probes module is preserved byte-for- byte unchanged — it's the template, not a page to rewrite for consistency. root-10-m03-troubleshooting-framework — a new PARTIAL module introducing a repeatable Observe -> Gather Evidence -> Form Hypothesis -> Test -> Repair -> Validate -> Prevent Recurrence framework, applied to a CrashLoopBackOff and a broken Service selector. Lesson, demonstration (the full framework walked through a real CrashLoopBackOff, command by command), knowledge check, and real Labs (crashloop-fix, endpoint-fix-task). Sequenced between the gold module and the existing bare-lab bank, which now sits behind it. Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-26 09:07:34 -05:00
parent 960a638ec9
commit 831486d664
3 changed files with 233 additions and 0 deletions
@@ -0,0 +1,92 @@
 # The RootNode Troubleshooting Framework
 ## Why a framework, not just "get good at kubectl"
 The gold-standard Health Probes module taught you a lot about ONE specific failure mode
 (readiness probe misconfiguration) and the specific commands to diagnose it. This module steps
 back and asks: what's the repeatable PROCESS that works for almost any Kubernetes failure, not
 just the ones you've already seen before?
 Guessing under pressure — trying random fixes and seeing what sticks — is slow, often makes
 things worse, and doesn't build a transferable skill. A framework does.
 ## Observe → Gather Evidence → Form Hypothesis → Test → Repair → Validate → Prevent Recurrence
 **Observe** — what's the actual, specific symptom? Not "it's broken," but "the Deployment shows
 0/3 Ready" or "requests are timing out" or "the Pod restarts every 30 seconds." Precision here
 saves time later.
 **Gather Evidence** — before touching anything, collect what the system itself is telling you:
 `kubectl get pods`, `kubectl describe pod`, `kubectl logs`, `kubectl get events`. Resist the urge
 to start changing things before you've looked.
 **Form Hypothesis** — based on the evidence, state a SPECIFIC guess about the cause. "The
 container is crashing because the command it's told to run doesn't exist" is testable. "Something's
 wrong with networking" is not specific enough to act on yet.
 **Test** — check whether your hypothesis is actually true, ideally without yet making the fix
 (read the exact command being run, compare against what the image expects) — confirm before you
 commit to a repair.
 **Repair** — make the minimal change that addresses the confirmed cause. Not a bigger change
 "while you're in there" — exactly the fix the evidence supports.
 **Validate** — confirm the fix actually worked, using the same kind of evidence you gathered
 initially (not just "I made a change, it should be fine now").
 **Prevent Recurrence** — the step most often skipped under time pressure. Could this class of
 mistake happen again? Is there a smaller, structural change (a default, a validation, a check)
 that would catch it earlier next time?
 ## Applying it: a CrashLoopBackOff
 - **Observe:** `crash-app`'s Pods show `STATUS: CrashLoopBackOff`, restart count climbing.
 - **Gather Evidence:** `kubectl describe deployment crash-app` (check the configured command),
  `kubectl logs -l app=crash-app` (what did the container actually print before dying, if
  anything).
 - **Form Hypothesis:** "The command override is invalid/wrong for this image, so the container
  exits immediately every time it starts."
 - **Test:** compare the command actually configured against what the image's default/expected
  entrypoint should be.
 - **Repair:** fix the command to what the container image actually expects.
 - **Validate:** `kubectl get pods` shows `Running`, restart count stops climbing.
 - **Prevent Recurrence:** a code-review habit of never overriding a container's command without
  verifying it against the image's documentation first.
 ## Applying it: a broken Service selector
 - **Observe:** traffic isn't reaching `my-svc`'s backend at all.
 - **Gather Evidence:** `kubectl get endpoints my-svc` (empty?), `kubectl get pods
  --show-labels` (what labels do the actual Pods have?).
 - **Form Hypothesis:** "The Service's selector doesn't match the Pods' actual labels."
 - **Test:** compare `kubectl describe service my-svc`'s selector against the Pods' real labels
  directly, side by side.
 - **Repair:** correct the selector to match.
 - **Validate:** `kubectl get endpoints my-svc` now shows the Pod IPs.
 - **Prevent Recurrence:** keeping label keys/values consistent (or generated from one source)
  between a Deployment's Pod template and any Service that targets it.
 ## Instructor Notes
 **Approximate timing:** 15 minutes read; 10-12 minutes live walking through both worked
 examples.
 **Demo plan:** Use the demonstration activity right after this lesson to show the CrashLoopBackOff
 example live, narrating each framework step explicitly as you go rather than just running
 commands.
 **Likely misconceptions to watch for:**
 - Students under time pressure often skip straight to "Repair" without confirming a hypothesis
  first. Reinforce that a wrong guess implemented as a "fix" can make evidence harder to read
  afterward, not easier.
 - "Prevent Recurrence" is the step most likely to be treated as optional/skippable. Frame it as
  what separates a team that keeps hitting the same incident from one that doesn't.
 **Discussion question:** "What's the cost of skipping 'Gather Evidence' and jumping straight to
 a guessed fix, even if the guess happens to be right?" (Answer: even a correct guess doesn't
 teach you whether your reasoning was right — you got lucky, and the same class of mistake could
 still recur since you never confirmed the actual cause.)
 **Optional extension:** ask students to apply this exact framework, from memory, to the gold
 module's `rootnode-probe-port-mismatch` incident they already solved — a good retroactive
 exercise connecting a lab they already completed to the generalized framework.
@@ -0,0 +1,91 @@
 # Demonstration: Applying the Framework to a CrashLoopBackOff
 Watch the full Observe → Evidence → Hypothesis → Test → Repair → Validate → Prevent sequence
 against a real symptom, narrated step by step.
 ## Observe
 ```
 $ kubectl get pods
 NAME                        READY   STATUS             RESTARTS   AGE
 crash-app-6d9f8b7c5d-abcde  0/1     CrashLoopBackOff    5          4m
 ```
 Specific symptom: `CrashLoopBackOff`, restart count climbing (5 and counting) — the container
 is starting, exiting, and being restarted repeatedly, not just failing to start once.
 ## Gather Evidence
 ```
 $ kubectl logs crash-app-6d9f8b7c5d-abcde
 nginx: [emerg] invalid command "daemon-off": expected valid command
 ```
 ```
 $ kubectl describe deployment crash-app
 ...
    Command:
      nginx
      -g
      daemon-off;
 ...
 ```
 The logs show nginx itself rejecting the command it was given. The Deployment's configured
 command shows exactly why: `daemon-off;` (a typo — missing space, should be `daemon off;`).
 ## Form Hypothesis
 "The container's command override has a syntax error that nginx can't parse, so nginx exits
 immediately every time it starts — which is why it's crash-looping rather than just failing to
 start once."
 ## Test
 Compare the broken command (`daemon-off;`) against nginx's actual expected syntax
 (`daemon off;`, two separate words) — confirmed: this is exactly the kind of typo that would
 produce exactly this error.
 ## Repair
 ```
 $ kubectl patch deployment crash-app --type='json' \
  -p='[{"op":"replace","path":"/spec/template/spec/containers/0/command","value":["nginx","-g","daemon off;"]}]'
 ```
 ## Validate
 ```
 $ kubectl get pods
 NAME                        READY   STATUS    RESTARTS   AGE
 crash-app-7f8c9d6b5d-fghij  1/1     Running   0          15s
 ```
 New Pod, `Running`, `RESTARTS: 0` — the fix is confirmed, not just assumed.
 ## Prevent Recurrence
 A command override like this is a hand-typed string with no validation until the container
 actually tries to run it. A team that hit this once might decide: don't override a well-known
 image's default command unless there's a specific reason to, and if you must, test the exact
 command string locally (`docker run` or similar) before deploying it — catching a typo in
 seconds locally beats debugging a CrashLoopBackOff in the cluster.
 ## Instructor Notes
 **Demo script:** Run this live if time allows, deliberately narrating which framework step
 you're on at each command — students retain the framework far better tied to real commands than
 as an abstract list.
 **Approximate timing:** 8-10 minutes live; 5-7 minutes self-paced.
 **Common misconception to preempt:** students sometimes skip straight from "Observe" to
 "Repair" once they spot the typo, without stating a hypothesis or testing it. Reinforce that
 even an "obvious" fix benefits from the explicit hypothesis step — it's what makes the eventual
 Validate step meaningful (you're confirming a SPECIFIC claim, not just "trying something").
 **Optional extension:** ask what evidence you'd gather differently if the logs had been
 completely empty instead of showing a clear nginx error (a good bridge to cases where the
 container dies before it can log anything at all — `kubectl describe pod`'s Events section
 becomes the primary evidence source in that case, same as the gold module's probe-mismatch
 lesson).
@@ -0,0 +1,50 @@
 # Knowledge Check: The Troubleshooting Framework
 ---
 **1. You spot what looks like an obvious typo causing a failure and fix it immediately without
 writing down a hypothesis first. What's the actual cost of skipping that step, even if the fix
 works?**
 *Answer:* You don't get a confirmed, testable claim about the cause — if the fix happens to
 "work" for an unrelated reason (a coincidental restart, a different underlying issue also
 resolving), you'd have no way to distinguish that from your fix actually being correct. The
 hypothesis step is what makes Validate meaningful.
 ---
 **2. During the "Gather Evidence" step for a CrashLoopBackOff, which two commands give you the
 most direct evidence, and what does each one tell you that the other doesn't?**
 *Answer:* `kubectl logs` (what the container itself printed, if anything, before exiting) and
 `kubectl describe pod`/`describe deployment` (the container's actual configuration, plus recent
 Events). Logs show application-level output; describe/Events show what Kubernetes itself
 observed and configured — a crash with no logs at all often means the failure happened before
 the application could log anything, which describe/Events can still reveal.
 ---
 **3. True or false: "Prevent Recurrence" only applies to major incidents, not small
 one-off mistakes like a typo.**
 *Answer:* False. A small mistake that recurs repeatedly across a team becomes a real, ongoing
 cost even if no single instance is severe — the framework applies the same reasoning regardless
 of incident size: could a small, structural change catch this earlier next time?
 ---
 **4. A broken Service selector (endpoint-fix-task's scenario) and a CrashLoopBackOff
 (crashloop-fix's scenario) are very different failures. What's the SAME about how you'd
 approach both?**
 *Answer:* The same seven-step framework — Observe the specific symptom, Gather Evidence before
 guessing, form a specific testable Hypothesis, Test it, make the minimal Repair the evidence
 supports, Validate with the same kind of evidence you started with, and consider how to Prevent
 Recurrence. The framework doesn't change; only the specific commands and evidence do.
 ## Instructor Notes
 **Facilitation:** Question 4 is the whole point of this module — make sure it's discussed even
 if answered correctly, since it's the transferability claim the entire framework rests on.
 **Approximate timing:** 5-7 minutes.