ced4568andClaude Sonnet 5
831486d664
The Phase 6A gold-standard Health Probes module is preserved byte-for- byte unchanged — it's the template, not a page to rewrite for consistency. root-10-m03-troubleshooting-framework — a new PARTIAL module introducing a repeatable Observe -> Gather Evidence -> Form Hypothesis -> Test -> Repair -> Validate -> Prevent Recurrence framework, applied to a CrashLoopBackOff and a broken Service selector. Lesson, demonstration (the full framework walked through a real CrashLoopBackOff, command by command), knowledge check, and real Labs (crashloop-fix, endpoint-fix-task). Sequenced between the gold module and the existing bare-lab bank, which now sits behind it. Co-Authored-By: Claude Sonnet 5 <[email protected]>
3.3 KiB
Demonstration: Applying the Framework to a CrashLoopBackOff
Watch the full Observe → Evidence → Hypothesis → Test → Repair → Validate → Prevent sequence against a real symptom, narrated step by step.
Observe
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
crash-app-6d9f8b7c5d-abcde 0/1 CrashLoopBackOff 5 4m
Specific symptom: CrashLoopBackOff, restart count climbing (5 and counting) — the container
is starting, exiting, and being restarted repeatedly, not just failing to start once.
Gather Evidence
$ kubectl logs crash-app-6d9f8b7c5d-abcde
nginx: [emerg] invalid command "daemon-off": expected valid command
$ kubectl describe deployment crash-app
...
Command:
nginx
-g
daemon-off;
...
The logs show nginx itself rejecting the command it was given. The Deployment's configured
command shows exactly why: daemon-off; (a typo — missing space, should be daemon off;).
Form Hypothesis
"The container's command override has a syntax error that nginx can't parse, so nginx exits immediately every time it starts — which is why it's crash-looping rather than just failing to start once."
Test
Compare the broken command (daemon-off;) against nginx's actual expected syntax
(daemon off;, two separate words) — confirmed: this is exactly the kind of typo that would
produce exactly this error.
Repair
$ kubectl patch deployment crash-app --type='json' \
-p='[{"op":"replace","path":"/spec/template/spec/containers/0/command","value":["nginx","-g","daemon off;"]}]'
Validate
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
crash-app-7f8c9d6b5d-fghij 1/1 Running 0 15s
New Pod, Running, RESTARTS: 0 — the fix is confirmed, not just assumed.
Prevent Recurrence
A command override like this is a hand-typed string with no validation until the container
actually tries to run it. A team that hit this once might decide: don't override a well-known
image's default command unless there's a specific reason to, and if you must, test the exact
command string locally (docker run or similar) before deploying it — catching a typo in
seconds locally beats debugging a CrashLoopBackOff in the cluster.
Instructor Notes
Demo script: Run this live if time allows, deliberately narrating which framework step you're on at each command — students retain the framework far better tied to real commands than as an abstract list.
Approximate timing: 8-10 minutes live; 5-7 minutes self-paced.
Common misconception to preempt: students sometimes skip straight from "Observe" to "Repair" once they spot the typo, without stating a hypothesis or testing it. Reinforce that even an "obvious" fix benefits from the explicit hypothesis step — it's what makes the eventual Validate step meaningful (you're confirming a SPECIFIC claim, not just "trying something").
Optional extension: ask what evidence you'd gather differently if the logs had been
completely empty instead of showing a clear nginx error (a good bridge to cases where the
container dies before it can log anything at all — kubectl describe pod's Events section
becomes the primary evidence source in that case, same as the gold module's probe-mismatch
lesson).