ced4568andClaude Sonnet 5
831486d664
The Phase 6A gold-standard Health Probes module is preserved byte-for- byte unchanged — it's the template, not a page to rewrite for consistency. root-10-m03-troubleshooting-framework — a new PARTIAL module introducing a repeatable Observe -> Gather Evidence -> Form Hypothesis -> Test -> Repair -> Validate -> Prevent Recurrence framework, applied to a CrashLoopBackOff and a broken Service selector. Lesson, demonstration (the full framework walked through a real CrashLoopBackOff, command by command), knowledge check, and real Labs (crashloop-fix, endpoint-fix-task). Sequenced between the gold module and the existing bare-lab bank, which now sits behind it. Co-Authored-By: Claude Sonnet 5 <[email protected]>
92 lines
3.3 KiB
Markdown
92 lines
3.3 KiB
Markdown
# Demonstration: Applying the Framework to a CrashLoopBackOff
|
|
|
|
Watch the full Observe → Evidence → Hypothesis → Test → Repair → Validate → Prevent sequence
|
|
against a real symptom, narrated step by step.
|
|
|
|
## Observe
|
|
|
|
```
|
|
$ kubectl get pods
|
|
NAME READY STATUS RESTARTS AGE
|
|
crash-app-6d9f8b7c5d-abcde 0/1 CrashLoopBackOff 5 4m
|
|
```
|
|
|
|
Specific symptom: `CrashLoopBackOff`, restart count climbing (5 and counting) — the container
|
|
is starting, exiting, and being restarted repeatedly, not just failing to start once.
|
|
|
|
## Gather Evidence
|
|
|
|
```
|
|
$ kubectl logs crash-app-6d9f8b7c5d-abcde
|
|
nginx: [emerg] invalid command "daemon-off": expected valid command
|
|
```
|
|
|
|
```
|
|
$ kubectl describe deployment crash-app
|
|
...
|
|
Command:
|
|
nginx
|
|
-g
|
|
daemon-off;
|
|
...
|
|
```
|
|
|
|
The logs show nginx itself rejecting the command it was given. The Deployment's configured
|
|
command shows exactly why: `daemon-off;` (a typo — missing space, should be `daemon off;`).
|
|
|
|
## Form Hypothesis
|
|
|
|
"The container's command override has a syntax error that nginx can't parse, so nginx exits
|
|
immediately every time it starts — which is why it's crash-looping rather than just failing to
|
|
start once."
|
|
|
|
## Test
|
|
|
|
Compare the broken command (`daemon-off;`) against nginx's actual expected syntax
|
|
(`daemon off;`, two separate words) — confirmed: this is exactly the kind of typo that would
|
|
produce exactly this error.
|
|
|
|
## Repair
|
|
|
|
```
|
|
$ kubectl patch deployment crash-app --type='json' \
|
|
-p='[{"op":"replace","path":"/spec/template/spec/containers/0/command","value":["nginx","-g","daemon off;"]}]'
|
|
```
|
|
|
|
## Validate
|
|
|
|
```
|
|
$ kubectl get pods
|
|
NAME READY STATUS RESTARTS AGE
|
|
crash-app-7f8c9d6b5d-fghij 1/1 Running 0 15s
|
|
```
|
|
|
|
New Pod, `Running`, `RESTARTS: 0` — the fix is confirmed, not just assumed.
|
|
|
|
## Prevent Recurrence
|
|
|
|
A command override like this is a hand-typed string with no validation until the container
|
|
actually tries to run it. A team that hit this once might decide: don't override a well-known
|
|
image's default command unless there's a specific reason to, and if you must, test the exact
|
|
command string locally (`docker run` or similar) before deploying it — catching a typo in
|
|
seconds locally beats debugging a CrashLoopBackOff in the cluster.
|
|
|
|
## Instructor Notes
|
|
|
|
**Demo script:** Run this live if time allows, deliberately narrating which framework step
|
|
you're on at each command — students retain the framework far better tied to real commands than
|
|
as an abstract list.
|
|
|
|
**Approximate timing:** 8-10 minutes live; 5-7 minutes self-paced.
|
|
|
|
**Common misconception to preempt:** students sometimes skip straight from "Observe" to
|
|
"Repair" once they spot the typo, without stating a hypothesis or testing it. Reinforce that
|
|
even an "obvious" fix benefits from the explicit hypothesis step — it's what makes the eventual
|
|
Validate step meaningful (you're confirming a SPECIFIC claim, not just "trying something").
|
|
|
|
**Optional extension:** ask what evidence you'd gather differently if the logs had been
|
|
completely empty instead of showing a clear nginx error (a good bridge to cases where the
|
|
container dies before it can log anything at all — `kubectl describe pod`'s Events section
|
|
becomes the primary evidence source in that case, same as the gold module's probe-mismatch
|
|
lesson).
|