diff --git a/content/root/root-10-kubernetes-troubleshooting-observability/m03-troubleshooting-framework/01-lesson-framework.md b/content/root/root-10-kubernetes-troubleshooting-observability/m03-troubleshooting-framework/01-lesson-framework.md new file mode 100644 index 0000000..d41ac28 --- /dev/null +++ b/content/root/root-10-kubernetes-troubleshooting-observability/m03-troubleshooting-framework/01-lesson-framework.md @@ -0,0 +1,92 @@ +# The RootNode Troubleshooting Framework + +## Why a framework, not just "get good at kubectl" + +The gold-standard Health Probes module taught you a lot about ONE specific failure mode +(readiness probe misconfiguration) and the specific commands to diagnose it. This module steps +back and asks: what's the repeatable PROCESS that works for almost any Kubernetes failure, not +just the ones you've already seen before? + +Guessing under pressure — trying random fixes and seeing what sticks — is slow, often makes +things worse, and doesn't build a transferable skill. A framework does. + +## Observe → Gather Evidence → Form Hypothesis → Test → Repair → Validate → Prevent Recurrence + +**Observe** — what's the actual, specific symptom? Not "it's broken," but "the Deployment shows +0/3 Ready" or "requests are timing out" or "the Pod restarts every 30 seconds." Precision here +saves time later. + +**Gather Evidence** — before touching anything, collect what the system itself is telling you: +`kubectl get pods`, `kubectl describe pod`, `kubectl logs`, `kubectl get events`. Resist the urge +to start changing things before you've looked. + +**Form Hypothesis** — based on the evidence, state a SPECIFIC guess about the cause. "The +container is crashing because the command it's told to run doesn't exist" is testable. "Something's +wrong with networking" is not specific enough to act on yet. + +**Test** — check whether your hypothesis is actually true, ideally without yet making the fix +(read the exact command being run, compare against what the image expects) — confirm before you +commit to a repair. + +**Repair** — make the minimal change that addresses the confirmed cause. Not a bigger change +"while you're in there" — exactly the fix the evidence supports. + +**Validate** — confirm the fix actually worked, using the same kind of evidence you gathered +initially (not just "I made a change, it should be fine now"). + +**Prevent Recurrence** — the step most often skipped under time pressure. Could this class of +mistake happen again? Is there a smaller, structural change (a default, a validation, a check) +that would catch it earlier next time? + +## Applying it: a CrashLoopBackOff + +- **Observe:** `crash-app`'s Pods show `STATUS: CrashLoopBackOff`, restart count climbing. +- **Gather Evidence:** `kubectl describe deployment crash-app` (check the configured command), + `kubectl logs -l app=crash-app` (what did the container actually print before dying, if + anything). +- **Form Hypothesis:** "The command override is invalid/wrong for this image, so the container + exits immediately every time it starts." +- **Test:** compare the command actually configured against what the image's default/expected + entrypoint should be. +- **Repair:** fix the command to what the container image actually expects. +- **Validate:** `kubectl get pods` shows `Running`, restart count stops climbing. +- **Prevent Recurrence:** a code-review habit of never overriding a container's command without + verifying it against the image's documentation first. + +## Applying it: a broken Service selector + +- **Observe:** traffic isn't reaching `my-svc`'s backend at all. +- **Gather Evidence:** `kubectl get endpoints my-svc` (empty?), `kubectl get pods + --show-labels` (what labels do the actual Pods have?). +- **Form Hypothesis:** "The Service's selector doesn't match the Pods' actual labels." +- **Test:** compare `kubectl describe service my-svc`'s selector against the Pods' real labels + directly, side by side. +- **Repair:** correct the selector to match. +- **Validate:** `kubectl get endpoints my-svc` now shows the Pod IPs. +- **Prevent Recurrence:** keeping label keys/values consistent (or generated from one source) + between a Deployment's Pod template and any Service that targets it. + +## Instructor Notes + +**Approximate timing:** 15 minutes read; 10-12 minutes live walking through both worked +examples. + +**Demo plan:** Use the demonstration activity right after this lesson to show the CrashLoopBackOff +example live, narrating each framework step explicitly as you go rather than just running +commands. + +**Likely misconceptions to watch for:** +- Students under time pressure often skip straight to "Repair" without confirming a hypothesis + first. Reinforce that a wrong guess implemented as a "fix" can make evidence harder to read + afterward, not easier. +- "Prevent Recurrence" is the step most likely to be treated as optional/skippable. Frame it as + what separates a team that keeps hitting the same incident from one that doesn't. + +**Discussion question:** "What's the cost of skipping 'Gather Evidence' and jumping straight to +a guessed fix, even if the guess happens to be right?" (Answer: even a correct guess doesn't +teach you whether your reasoning was right — you got lucky, and the same class of mistake could +still recur since you never confirmed the actual cause.) + +**Optional extension:** ask students to apply this exact framework, from memory, to the gold +module's `rootnode-probe-port-mismatch` incident they already solved — a good retroactive +exercise connecting a lab they already completed to the generalized framework. diff --git a/content/root/root-10-kubernetes-troubleshooting-observability/m03-troubleshooting-framework/02-demonstration-crashloop.md b/content/root/root-10-kubernetes-troubleshooting-observability/m03-troubleshooting-framework/02-demonstration-crashloop.md new file mode 100644 index 0000000..928678c --- /dev/null +++ b/content/root/root-10-kubernetes-troubleshooting-observability/m03-troubleshooting-framework/02-demonstration-crashloop.md @@ -0,0 +1,91 @@ +# Demonstration: Applying the Framework to a CrashLoopBackOff + +Watch the full Observe → Evidence → Hypothesis → Test → Repair → Validate → Prevent sequence +against a real symptom, narrated step by step. + +## Observe + +``` +$ kubectl get pods +NAME READY STATUS RESTARTS AGE +crash-app-6d9f8b7c5d-abcde 0/1 CrashLoopBackOff 5 4m +``` + +Specific symptom: `CrashLoopBackOff`, restart count climbing (5 and counting) — the container +is starting, exiting, and being restarted repeatedly, not just failing to start once. + +## Gather Evidence + +``` +$ kubectl logs crash-app-6d9f8b7c5d-abcde +nginx: [emerg] invalid command "daemon-off": expected valid command +``` + +``` +$ kubectl describe deployment crash-app +... + Command: + nginx + -g + daemon-off; +... +``` + +The logs show nginx itself rejecting the command it was given. The Deployment's configured +command shows exactly why: `daemon-off;` (a typo — missing space, should be `daemon off;`). + +## Form Hypothesis + +"The container's command override has a syntax error that nginx can't parse, so nginx exits +immediately every time it starts — which is why it's crash-looping rather than just failing to +start once." + +## Test + +Compare the broken command (`daemon-off;`) against nginx's actual expected syntax +(`daemon off;`, two separate words) — confirmed: this is exactly the kind of typo that would +produce exactly this error. + +## Repair + +``` +$ kubectl patch deployment crash-app --type='json' \ + -p='[{"op":"replace","path":"/spec/template/spec/containers/0/command","value":["nginx","-g","daemon off;"]}]' +``` + +## Validate + +``` +$ kubectl get pods +NAME READY STATUS RESTARTS AGE +crash-app-7f8c9d6b5d-fghij 1/1 Running 0 15s +``` + +New Pod, `Running`, `RESTARTS: 0` — the fix is confirmed, not just assumed. + +## Prevent Recurrence + +A command override like this is a hand-typed string with no validation until the container +actually tries to run it. A team that hit this once might decide: don't override a well-known +image's default command unless there's a specific reason to, and if you must, test the exact +command string locally (`docker run` or similar) before deploying it — catching a typo in +seconds locally beats debugging a CrashLoopBackOff in the cluster. + +## Instructor Notes + +**Demo script:** Run this live if time allows, deliberately narrating which framework step +you're on at each command — students retain the framework far better tied to real commands than +as an abstract list. + +**Approximate timing:** 8-10 minutes live; 5-7 minutes self-paced. + +**Common misconception to preempt:** students sometimes skip straight from "Observe" to +"Repair" once they spot the typo, without stating a hypothesis or testing it. Reinforce that +even an "obvious" fix benefits from the explicit hypothesis step — it's what makes the eventual +Validate step meaningful (you're confirming a SPECIFIC claim, not just "trying something"). + +**Optional extension:** ask what evidence you'd gather differently if the logs had been +completely empty instead of showing a clear nginx error (a good bridge to cases where the +container dies before it can log anything at all — `kubectl describe pod`'s Events section +becomes the primary evidence source in that case, same as the gold module's probe-mismatch +lesson). diff --git a/content/root/root-10-kubernetes-troubleshooting-observability/m03-troubleshooting-framework/03-knowledge-check.md b/content/root/root-10-kubernetes-troubleshooting-observability/m03-troubleshooting-framework/03-knowledge-check.md new file mode 100644 index 0000000..868e7f6 --- /dev/null +++ b/content/root/root-10-kubernetes-troubleshooting-observability/m03-troubleshooting-framework/03-knowledge-check.md @@ -0,0 +1,50 @@ +# Knowledge Check: The Troubleshooting Framework + +--- + +**1. You spot what looks like an obvious typo causing a failure and fix it immediately without +writing down a hypothesis first. What's the actual cost of skipping that step, even if the fix +works?** + +*Answer:* You don't get a confirmed, testable claim about the cause — if the fix happens to +"work" for an unrelated reason (a coincidental restart, a different underlying issue also +resolving), you'd have no way to distinguish that from your fix actually being correct. The +hypothesis step is what makes Validate meaningful. + +--- + +**2. During the "Gather Evidence" step for a CrashLoopBackOff, which two commands give you the +most direct evidence, and what does each one tell you that the other doesn't?** + +*Answer:* `kubectl logs` (what the container itself printed, if anything, before exiting) and +`kubectl describe pod`/`describe deployment` (the container's actual configuration, plus recent +Events). Logs show application-level output; describe/Events show what Kubernetes itself +observed and configured — a crash with no logs at all often means the failure happened before +the application could log anything, which describe/Events can still reveal. + +--- + +**3. True or false: "Prevent Recurrence" only applies to major incidents, not small +one-off mistakes like a typo.** + +*Answer:* False. A small mistake that recurs repeatedly across a team becomes a real, ongoing +cost even if no single instance is severe — the framework applies the same reasoning regardless +of incident size: could a small, structural change catch this earlier next time? + +--- + +**4. A broken Service selector (endpoint-fix-task's scenario) and a CrashLoopBackOff +(crashloop-fix's scenario) are very different failures. What's the SAME about how you'd +approach both?** + +*Answer:* The same seven-step framework — Observe the specific symptom, Gather Evidence before +guessing, form a specific testable Hypothesis, Test it, make the minimal Repair the evidence +supports, Validate with the same kind of evidence you started with, and consider how to Prevent +Recurrence. The framework doesn't change; only the specific commands and evidence do. + +## Instructor Notes + +**Facilitation:** Question 4 is the whole point of this module — make sure it's discussed even +if answered correctly, since it's the transferability claim the entire framework rests on. + +**Approximate timing:** 5-7 minutes.