rootnode-academy/content/root/root-10-kubernetes-troubleshooting-observability/m03-troubleshooting-framework/01-lesson-framework.md

# The RootNode Troubleshooting Framework

## Why a framework, not just "get good at kubectl"

The gold-standard Health Probes module taught you a lot about ONE specific failure mode
(readiness probe misconfiguration) and the specific commands to diagnose it. This module steps
back and asks: what's the repeatable PROCESS that works for almost any Kubernetes failure, not
just the ones you've already seen before?

Guessing under pressure — trying random fixes and seeing what sticks — is slow, often makes
things worse, and doesn't build a transferable skill. A framework does.

## Observe → Gather Evidence → Form Hypothesis → Test → Repair → Validate → Prevent Recurrence

**Observe** — what's the actual, specific symptom? Not "it's broken," but "the Deployment shows
0/3 Ready" or "requests are timing out" or "the Pod restarts every 30 seconds." Precision here
saves time later.

**Gather Evidence** — before touching anything, collect what the system itself is telling you:
`kubectl get pods`, `kubectl describe pod`, `kubectl logs`, `kubectl get events`. Resist the urge
to start changing things before you've looked.

**Form Hypothesis** — based on the evidence, state a SPECIFIC guess about the cause. "The
container is crashing because the command it's told to run doesn't exist" is testable. "Something's
wrong with networking" is not specific enough to act on yet.

**Test** — check whether your hypothesis is actually true, ideally without yet making the fix
(read the exact command being run, compare against what the image expects) — confirm before you
commit to a repair.

**Repair** — make the minimal change that addresses the confirmed cause. Not a bigger change
"while you're in there" — exactly the fix the evidence supports.

**Validate** — confirm the fix actually worked, using the same kind of evidence you gathered
initially (not just "I made a change, it should be fine now").

**Prevent Recurrence** — the step most often skipped under time pressure. Could this class of
mistake happen again? Is there a smaller, structural change (a default, a validation, a check)
that would catch it earlier next time?

## Applying it: a CrashLoopBackOff

- **Observe:** `crash-app`'s Pods show `STATUS: CrashLoopBackOff`, restart count climbing.
- **Gather Evidence:** `kubectl describe deployment crash-app` (check the configured command),
  `kubectl logs -l app=crash-app` (what did the container actually print before dying, if
  anything).
- **Form Hypothesis:** "The command override is invalid/wrong for this image, so the container
  exits immediately every time it starts."
- **Test:** compare the command actually configured against what the image's default/expected
  entrypoint should be.
- **Repair:** fix the command to what the container image actually expects.
- **Validate:** `kubectl get pods` shows `Running`, restart count stops climbing.
- **Prevent Recurrence:** a code-review habit of never overriding a container's command without
  verifying it against the image's documentation first.

## Applying it: a broken Service selector

- **Observe:** traffic isn't reaching `my-svc`'s backend at all.
- **Gather Evidence:** `kubectl get endpoints my-svc` (empty?), `kubectl get pods
  --show-labels` (what labels do the actual Pods have?).
- **Form Hypothesis:** "The Service's selector doesn't match the Pods' actual labels."
- **Test:** compare `kubectl describe service my-svc`'s selector against the Pods' real labels
  directly, side by side.
- **Repair:** correct the selector to match.
- **Validate:** `kubectl get endpoints my-svc` now shows the Pod IPs.
- **Prevent Recurrence:** keeping label keys/values consistent (or generated from one source)
  between a Deployment's Pod template and any Service that targets it.

## Instructor Notes

**Approximate timing:** 15 minutes read; 10-12 minutes live walking through both worked
examples.

**Demo plan:** Use the demonstration activity right after this lesson to show the CrashLoopBackOff
example live, narrating each framework step explicitly as you go rather than just running
commands.

**Likely misconceptions to watch for:**
- Students under time pressure often skip straight to "Repair" without confirming a hypothesis
  first. Reinforce that a wrong guess implemented as a "fix" can make evidence harder to read
  afterward, not easier.
- "Prevent Recurrence" is the step most likely to be treated as optional/skippable. Frame it as
  what separates a team that keeps hitting the same incident from one that doesn't.

**Discussion question:** "What's the cost of skipping 'Gather Evidence' and jumping straight to
a guessed fix, even if the guess happens to be right?" (Answer: even a correct guess doesn't
teach you whether your reasoning was right — you got lucky, and the same class of mistake could
still recur since you never confirmed the actual cause.)

**Optional extension:** ask students to apply this exact framework, from memory, to the gold
module's `rootnode-probe-port-mismatch` incident they already solved — a good retroactive
exercise connecting a lab they already completed to the generalized framework.