Files
rootnode-academy/content/root/root-12-real-ops/m01-real-ops-incidents/00-lesson-incident-response.md
T
ced4568andClaude Sonnet 5 de6c5b3286 feat(root): build Real Ops and capstone readiness content
root-12-real-ops gains a Detect -> Triage -> Diagnose -> Mitigate ->
Recover -> Verify -> Learn incident-response framing lesson at the
front of the existing 4-incident module, and a post-incident reflection
at the end — the incidents themselves are unchanged, still deliberately
un-guided ('you are responsible for this environment,' not 'complete
this exercise'). No fixes revealed; instructor notes carry the deeper
solution guidance the student-facing lesson intentionally withholds.

root-capstone-m02-capstone-readiness — a new READY orientation module:
what the Capstone measures, how the timed/randomized draw works, time
management and troubleshooting strategy under a clock, and a genuine
self-assessment checklist. Never publishes the current draw or answers.
See docs/ROOT_CAPSTONE_READINESS.md for why this module's completion is
deliberately independent of the real capstone's server-side unlock gate.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-26 09:08:01 -05:00

4.9 KiB

The RootNode Incident-Response Framework

This module feels different on purpose

Everything before ROOT 12 asked you to complete an exercise. This module asks something different: you are responsible for this environment. The incidents in this module are modeled on real operational failures — not textbook exercises designed to teach one clean concept, but the kind of messy, "why is this actually broken" situations a real on-call engineer gets paged for.

Detect → Triage → Diagnose → Mitigate → Recover → Verify → Learn

This is a lightweight incident-response framework — not enterprise incident management tooling, just a way of thinking that scales down to "one engineer, one broken thing" as well as it scales up to a real incident-response team.

Detect — something alerted you (a monitoring page, a user report, or in this Lab's case, the scenario telling you something is wrong). You know an incident exists before you know what it is.

Triage — how bad is this, right now, before you dive into root cause? Is anything actively getting worse? Is this affecting one thing or cascading into others? Triage decides how urgently you act, before you fully understand why.

Diagnose — this is where ROOT 10's troubleshooting framework (Observe, Gather Evidence, Form Hypothesis, Test) does its work — the specific mechanics of finding the actual cause.

Mitigate — the fastest safe action that stops things from getting worse, which is not always the same as the final fix. Sometimes mitigation and repair are the same step; sometimes mitigation buys time (e.g. temporarily reducing load) while a proper fix is prepared.

Recover — restore the system to its actual working state — not just "stopped getting worse" but genuinely operational again.

Verify — confirm recovery with real evidence, the same discipline as ROOT 10's Validate step. "I think it's fixed" is not the same as checking.

Learn — what would prevent this exact class of incident from happening again? This is where rootnode-enforce-limits picks up directly after rootnode-runaway-monitor — not a separate topic, but the deliberate "Learn" step of the SAME incident, made into its own scenario.

What this module will NOT tell you

The specific fixes for each incident are not in this lesson, and instructor notes on the individual incident activities go deeper than what's shown here — this module gives you the FRAMEWORK for approaching an unfamiliar incident, not a cheat sheet for these specific four. That's deliberate: a framework you had to apply yourself is one you'll actually remember under real pressure.

Instructor Notes

Lesson goal: Set the tone shift for this entire module — from "complete an exercise" to "you own this." This lesson is short by design; the real teaching happens through the incidents themselves.

Approximate timing: 15 minutes read; 10 minutes live framing discussion.

Preparation: Read each incident's own instructor notes (in the Lab activities below) before teaching this module live — several contain deeper solution guidance not repeated here, specifically so students still have to work for it.

Demonstration plan: Not applicable to this framing lesson — the four incidents themselves serve as the demonstrations.

Likely misconceptions to watch for:

  • Students sometimes treat "Mitigate" and "Recover" as the same step. They can be, but aren't always — worth naming a real-world example (rolling back a bad deploy mitigates immediately; fully recovering might still require redeploying the CORRECT version afterward).
  • "Learn" is the step most likely to get skipped once a Lab is marked complete. Explicitly connect it to why rootnode-enforce-limits exists as its own scenario.

Questions to ask students: "Which step of this framework do you think real teams are worst at, under real pressure, and why?" (There's no single right answer — the goal is getting students to reason about incident response as a discipline, not just a checklist.)

When to let a student struggle vs. provide a hint: These incidents are intentionally less guided than earlier modules. Let a student sit with "I don't know where to start" for a few minutes before nudging — the discomfort of a genuinely unfamiliar failure is part of what this module is teaching. Redirect toward the Diagnose step's ROOT 10 framework if truly stuck, rather than pointing at the specific fix.

Optional extension: ask students to write a two-sentence incident summary for rootnode-runaway-monitor as if posting it to a real team channel — Detect/Diagnose/Recover in plain language, no jargon.

Expected takeaway: A framework applied under pressure beats improvisation, even for engineers who are technically skilled enough to eventually stumble into the right answer anyway — the framework is what makes the RIGHT answer arrive faster and more reliably.