rootnode-academy/content/root/root-10-kubernetes-troubleshooting-observability/m03-troubleshooting-framework/01-lesson-framework.md

The RootNode Troubleshooting Framework

Why a framework, not just "get good at kubectl"

The gold-standard Health Probes module taught you a lot about ONE specific failure mode (readiness probe misconfiguration) and the specific commands to diagnose it. This module steps back and asks: what's the repeatable PROCESS that works for almost any Kubernetes failure, not just the ones you've already seen before?

Guessing under pressure — trying random fixes and seeing what sticks — is slow, often makes things worse, and doesn't build a transferable skill. A framework does.

Observe → Gather Evidence → Form Hypothesis → Test → Repair → Validate → Prevent Recurrence

Observe — what's the actual, specific symptom? Not "it's broken," but "the Deployment shows 0/3 Ready" or "requests are timing out" or "the Pod restarts every 30 seconds." Precision here saves time later.

Gather Evidence — before touching anything, collect what the system itself is telling you: kubectl get pods, kubectl describe pod, kubectl logs, kubectl get events. Resist the urge to start changing things before you've looked.

Form Hypothesis — based on the evidence, state a SPECIFIC guess about the cause. "The container is crashing because the command it's told to run doesn't exist" is testable. "Something's wrong with networking" is not specific enough to act on yet.

Test — check whether your hypothesis is actually true, ideally without yet making the fix (read the exact command being run, compare against what the image expects) — confirm before you commit to a repair.

Repair — make the minimal change that addresses the confirmed cause. Not a bigger change "while you're in there" — exactly the fix the evidence supports.

Validate — confirm the fix actually worked, using the same kind of evidence you gathered initially (not just "I made a change, it should be fine now").

Prevent Recurrence — the step most often skipped under time pressure. Could this class of mistake happen again? Is there a smaller, structural change (a default, a validation, a check) that would catch it earlier next time?

Applying it: a CrashLoopBackOff

• Observe: crash-app's Pods show STATUS: CrashLoopBackOff, restart count climbing.
• Gather Evidence: kubectl describe deployment crash-app (check the configured command), kubectl logs -l app=crash-app (what did the container actually print before dying, if anything).
• Form Hypothesis: "The command override is invalid/wrong for this image, so the container exits immediately every time it starts."
• Test: compare the command actually configured against what the image's default/expected entrypoint should be.
• Repair: fix the command to what the container image actually expects.
• Validate: kubectl get pods shows Running, restart count stops climbing.
• Prevent Recurrence: a code-review habit of never overriding a container's command without verifying it against the image's documentation first.

Applying it: a broken Service selector

• Observe: traffic isn't reaching my-svc's backend at all.
• Gather Evidence: kubectl get endpoints my-svc (empty?), kubectl get pods --show-labels (what labels do the actual Pods have?).
• Form Hypothesis: "The Service's selector doesn't match the Pods' actual labels."
• Test: compare kubectl describe service my-svc's selector against the Pods' real labels directly, side by side.
• Repair: correct the selector to match.
• Validate: kubectl get endpoints my-svc now shows the Pod IPs.
• Prevent Recurrence: keeping label keys/values consistent (or generated from one source) between a Deployment's Pod template and any Service that targets it.

Instructor Notes

Approximate timing: 15 minutes read; 10-12 minutes live walking through both worked examples.

Demo plan: Use the demonstration activity right after this lesson to show the CrashLoopBackOff example live, narrating each framework step explicitly as you go rather than just running commands.

Likely misconceptions to watch for:

• Students under time pressure often skip straight to "Repair" without confirming a hypothesis first. Reinforce that a wrong guess implemented as a "fix" can make evidence harder to read afterward, not easier.
• "Prevent Recurrence" is the step most likely to be treated as optional/skippable. Frame it as what separates a team that keeps hitting the same incident from one that doesn't.

Discussion question: "What's the cost of skipping 'Gather Evidence' and jumping straight to a guessed fix, even if the guess happens to be right?" (Answer: even a correct guess doesn't teach you whether your reasoning was right — you got lucky, and the same class of mistake could still recur since you never confirmed the actual cause.)

Optional extension: ask students to apply this exact framework, from memory, to the gold module's rootnode-probe-port-mismatch incident they already solved — a good retroactive exercise connecting a lab they already completed to the generalized framework.