Author SHA1 Message Date
ced4568andClaude Sonnet 5 b6516a74db docs(phase6d): document the ROOT Foundations milestone
Adds PHASE6D_ROOT_FOUNDATIONS.md (the phase's own record: the generic-scenario-engine finding,
the tested container-runtime decision, what was authored, the bugs found and fixed, and honest
test/regression/rehearsal results) and ROOT_FOUNDATIONS_PILOT_READINESS.md (module-by-module
READY/DEFERRED classification for ROOT 01-04 against the unchanged Phase 6C readiness standard —
ROOT 01 READY, ROOT 02-04 DEFERRED by design).

Adds ROOT_PROGRAM_DURATION_MODEL.md — real per-course instructional-hour totals computed from
actual estimated_duration_minutes values (2,931 minutes / 48.9 hours across all authored
content), not invented, with an explicit note on why instructor-led timing isn't a separate
summable field yet.

Updates ROOT_PROGRAM_ARCHITECTURE.md (ROOT 01's new READY status and the real 01-04 prerequisite
chain), ROOT_PILOT_CURRICULUM.md (ROOT 01 added to the pilot, module-level detail), and
ROOT_CURRICULUM_COVERAGE_MATRIX.md (updated module/scenario totals) to reflect what Phase 6D
actually built, without restating Phase 6C's own Kubernetes-scenario tables.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-26 13:00:41 -05:00
ced4568andClaude Sonnet 5 f5379afd5c test(phase6d): acceptance suite, real-Lab rehearsal, and 6A/6B resume-point fixes
Adds platform/test/e2e-phase6d.js (20 checks): ROOT 01 hierarchy, ROOT 02-04 honestly empty
(not padded), activity ordering, lab-cannot-be-manually-completed, full course completion via
lesson/lab/assessment, the prerequisite chain proven on two students (one who completed ROOT 01,
one fresh), existing ROOT 05+/Capstone state unaffected, is_required unaffected, self-paced
no-instructor-deferral check, instructor-notes exclusion, RBAC, logout/login persistence,
cross-student isolation, version-bump stability, and content validation. 20/20 pass against both
a warm and a freshly-reset dev stack.

Adds infra/scripts/drills/{run,test}-phase6d-foundations-rehearsal — the real disposable-Lab
dress rehearsal: login -> enroll -> lesson/guided-exercise -> real Lab launch -> real bash
hardware inspection (nproc/free/df, written to a real file) -> real validation -> a genuine
permission denial (a second Linux user actually denied via `su -`) then a real chmod fix ->
remaining modules -> assessment -> course completion -> ROOT 02 unlock -> logout/login
persistence -> instructor visibility of the resulting learning state. 11/11 pass against a real
Lab container, no mocking. (First run caught a real triple-escaping bug in the hardware-report
step's inline heredoc — fixed by writing the script to a file and docker-cp-ing it in instead.)

Fixes two stale hardcoded assumptions in e2e-phase6a.js and e2e-phase6b.js that this phase's own
ROOT 01 addition broke: both suites asserted `resume` landed on ROOT 05's gold module as "the
first real content," which was true before this phase and is now ROOT 01. Both suites' original
intent (resume must never land on a bare-lab bank) is preserved, just against the current front
door. Both suites re-verified green (21/21 and 24/24) against a fresh stack.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-26 13:00:17 -05:00
ced4568andClaude Sonnet 5 15b616274b feat(root01): author Computing & Linux Foundations, and the real 01->05 prerequisite chain
Adds all 6 ROOT 01 modules (how computers run software, the shell, filesystem navigation,
files & text, processes, users & permissions foundations), each with a real lesson,
demonstration/guided exercise, knowledge check where warranted, a hands-on Lab, and a
reflection — 35 activities, 29 new content files.

Adds 6 new hands-on Linux scenarios (linux-inspect-hardware, linux-shell-basics,
linux-filesystem-navigation, linux-files-and-text, linux-process-management,
linux-permissions-foundations) plus 2 MCQ scenarios, using the existing generic scenario/
validation engine (docs/ROOT_FOUNDATIONS_LAB_STRATEGY.md §1) — every one tested end-to-end via
the real scenario setup/validate API, including negative tests confirming they correctly FAIL
when not actually solved (e.g. the process-management scenario fails while the target process
is still running; the permissions scenario is tested against a real `su - labuser` denial).

Wires rootnode-linux-foundations-assessment (2 required + 4 random-draw scenarios) using the
existing Phase 3B assessment engine.

Adds a real, server-side ROOT 01->02->03->04->05 course prerequisite chain
(courses.prerequisites) — verified live on two students: one who completed ROOT 01 sees ROOT 02
unlock; a fresh student sees ROOT 02 correctly locked while 03/04/05 stay vacuously satisfied
(their own prerequisite courses are still empty). ROOT 06+ and the Capstone are unaffected.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-26 12:59:52 -05:00
ced4568andClaude Sonnet 5 87526bcf7f feat(labs): add a nested Docker Engine to the Lab image for ROOT 04
Installs docker.io + a static docker-compose v2 binary (same "curl a static binary" pattern
already used for k3s) inside the same already-Privileged Lab container that runs k3s.
scripts/entrypoint.sh starts dockerd (vfs storage driver, to avoid nesting overlay-on-overlay)
non-blockingly right after k3s's own API server is confirmed up — a Docker daemon failure only
logs a warning and never breaks the Kubernetes-only Labs that are the vast majority of usage.

This does not weaken the security boundary: the container was already Privileged (required for
k3s, the existing accepted threat model), so a second runtime inside it grants no new host
capability, and the Lab Host's own real Docker socket is never touched — the nested dockerd
listens on its own /var/run/docker.sock entirely inside the guest's filesystem namespace.

Tested empirically against the real production resource caps (2 vCPU, 4096MB, 512 PIDs,
Privileged: true) before merging: k3s + dockerd + a running nginx:alpine container (with a
published port, curled successfully) + a real docker build all together used 884MB/4096MB and
160/512 PIDs — comfortable headroom in every dimension. See docs/ROOT_FOUNDATIONS_LAB_STRATEGY.md
§3 for the full test log.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-26 12:59:23 -05:00
ced4568andClaude Sonnet 5 3c815cfeea docs(foundations): define ROOT 01-04 architecture and Lab strategy
ROOT_FOUNDATIONS_ARCHITECTURE.md designs all 26 modules across ROOT 01 (Computing & Linux
Foundations), 02 (Linux Administration & Networking), 03 (Git & Collaborative Development), and
04 (Containers & Docker), using the existing course/module/activity model — outcomes,
prerequisites, activities, and Real Ops connections for each, per docs/COURSE_AUTHORING_STANDARD.md.

ROOT_FOUNDATIONS_LAB_STRATEGY.md documents the central finding this phase is built on: the
existing scenario engine (backend/server.js's runCommand()) executes an arbitrary shell command
string and checks its output — nothing about it is Kubernetes-specific. The existing `lab`
activity type and scenario JSON format already fully support Linux and Git exercises with real,
deterministic validation. No shell_lab/git_lab/container_lab activity type is built or needed.

Also documents the container-runtime decision for ROOT 04, evaluated and tested empirically
(not assumed) against the real Lab image and production resource caps — see the following
commit for the implementation.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-26 12:59:00 -05:00
ced4568andClaude Sonnet 5 bd0a10fbfb docs(phase6c): pilot readiness standard, pilot curriculum manifest, ROOT 11 scope
Adds the four required Phase 6C documents:

- ROOT_PILOT_READINESS_STANDARD.md — the explicit 8-point sufficiency bar
  used to judge every module this phase, replacing any fixed activity-count
  assumption.
- ROOT11_DEVOPS_PLATFORM_SCOPE.md — evaluates all 7 proposed ROOT 11 modules
  individually (READY TO AUTHOR NOW / PARTIAL-LESSON-ONLY / REQUIRES FUTURE
  LAB ENVIRONMENT), with what current infra supports vs. what GitOps/CI-CD/
  IaC/observability would each require.
- ROOT_PILOT_CURRICULUM.md — the authoritative pilot manifest: instructor-led
  as the first pilot's delivery mode, honest READY/PARTIAL/DEFERRED/NOT IN
  PILOT status per course and module, and what is intentionally excluded
  (ROOT 01-04 never authored, ROOT 08 non-capstone-gating and deferred, ROOT
  11 modules 2-7, three practice-bank modules instructor-led-only this pilot).
- PHASE6C_CURRICULUM_COMPLETION.md — the phase's own record of what was
  upgraded, why, the bug found and fixed, and the pilot readiness conclusion.

Updates ROOT_CURRICULUM_COVERAGE_MATRIX.md with the verified before/after
module counts (21->22 modules, 6->16 READY) and per-module upgrade rationale,
and ROOT_CAPSTONE_READINESS.md with a re-review confirming the unlock gate's
6 stages, and ROOT 08/09's non-gating status, are unchanged by this phase's
curriculum expansion.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-26 11:21:09 -05:00
ced4568andClaude Sonnet 5 01d3764138 feat(frontend): visibility note for standalone assessment-type activities
CourseView.jsx already special-cased 'capstone' activities with a pointer to
the Capstone tab (since only capstone-type activities existed before this
phase). Phase 6C's new rootnode-security-assessment introduces the first
non-capstone 'assessment'-type module activity — without a similar note, a
student would see it in the course tree with no way to discover it needs to
be started from the Assessments tab. Adds the same pattern for 'assessment'
type, naming the assessment by its own title.

No other frontend change was needed: CurriculumView.jsx (instructor view)
already renders activity_type labels, scenario_id/assessment_definition_id,
is_required, instructor_notes availability, and READY/PARTIAL/PLANNED status
pills generically — Phase 6C's new module/activity shapes render correctly
under Phase 6B's existing generic logic, verified by inspection.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-26 11:20:51 -05:00
ced4568andClaude Sonnet 5 903b0ead43 test(phase6c): acceptance suite, real-k3s rehearsal, and 6A/6B staleness fixes
Adds platform/test/e2e-phase6c.js (17 checks): upgraded-module activity
chains, documented-exception promotions, optional-lab non-blocking behavior,
no-fake-completion-from-viewing, idempotent completion, instructor-notes
exclusion and RBAC on new content, security-assessment pool/draw/cross-student
isolation/completion, the capstone gate's documented indifference to ROOT 09,
ROOT 11's honest partial/lesson-only rendering, and version-bump content
stability — plus running validate-course-content.js as part of the suite.
All 17 pass against both a warm and a freshly-reset dev stack.

Adds infra/scripts/drills/{run,test}-phase6c-security-config-rehearsal —
Section 22's real-k3s dress rehearsal: login -> program -> course -> lesson ->
demonstration -> knowledge-check -> guided-exercise -> real Lab -> real
kubectl (create Secret, apply Pod, wait for Ready) -> validation -> progress
persistence across logout/login -> instructor visibility of the resulting
learning state, walking root-07-m02-configuration-and-secrets (upgraded from
PARTIAL this phase). 10/10 pass against a real k3s Lab, no mocking.

Fixes two stale hardcoded assumptions in e2e-phase6a.js and e2e-phase6b.js
that Phase 6C's own legitimate content changes broke: the gold module
(root-10-m01) now has 7 activities (6 required + 1 new optional lab, not 6),
and root-05-m01 now has 2 required manual-marker activities the course-
completion path must also complete (its new bookending lesson/reflection).
Both suites are back to green (21/21 and 24/24) against a fresh stack.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-26 11:20:37 -05:00
ced4568andClaude Sonnet 5 7f6e1857c0 feat(curriculum): Phase 6C module upgrades, security assessment bank, ROOT 11 orientation
Upgrades six PARTIAL modules to the gold-module standard (lesson/demonstration/
knowledge-check/guided-exercise/lab(s)/reflection), preserving every existing lab
activity_id and scenario_id: root-06-m03 (jobs/cronjobs/daemonsets), root-07-m04
(ingress/networkpolicy), root-07-m02 (configuration/secrets, converted in place
from a bare 5-lab bank), root-09-m02 (serviceaccounts/rbac), root-09-m03
(network/supply-chain security), root-10-m03 (troubleshooting framework).

root-05-m01 (20-task kubectl-fluency bank) gets a minimal bookending lesson +
reflection rather than a full redesign, since each task already carries its own
explanation. root-12-m01, root-capstone-m01, and root-09-m04 are promoted to
READY via documented exceptions (no new activities) where adding one would
misrepresent their intentional design.

Wires a new rootnode-security-assessment (exam, 60min, hint_policy disabled)
using the existing Phase 3B assessment engine: 3 required hands-on CORE
scenarios + 4 random-draw pools covering SecurityContext, RBAC, network
policy, and the 7 previously-unwired CKS MCQs. Wires the previously-unwired
readiness-probe-task scenario as an OPTIONAL practice lab in root-10-m01,
using the existing is_required model.

Adds root-11-m01-devops-platform-thinking — an honest, lesson-only ROOT 11
orientation module naming DevOps/platform engineering practices already
exercised in ROOT 05-10/12, explicit that GitOps/CI-CD/IaC/observability need
lab infrastructure this platform doesn't yet provide. No fake labs.

Adds infra/scripts/validate-course-content.js: cross-checks the DB against
scenario/content files for duplicate ids, orphans, invalid references, cycles,
sequence collisions, instructor-notes leakage, and (new) self-paced content
deferring explanation to a live instructor. 0 errors against the resulting
22 courses/modules, 163 activities, 98 scenario files.

No application code changed — every gap was satisfiable with existing tables
(course_modules, module_activities, assessment_definitions) and the existing
content-delivery pipeline.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-26 11:20:17 -05:00
ced4568andClaude Sonnet 5 2db136875f docs(phase6b): document curriculum expansion
PHASE6B_ROOT_CURRICULUM_EXPANSION.md — the phase summary: what was
authored per course, the is_required extension, the capstone unlock
fix, verified scenario coverage before/after (44 -> 88 of 98, 89.8%,
by direct query not estimate), and honest known limitations/Phase 6C
recommendations.

ROOT_CAPSTONE_READINESS.md — the readiness module and the server-side
unlock fix in full detail, including why the module's completion and
the real assessment's unlock gate are deliberately independent systems.

Updated ROOT_CURRICULUM_INVENTORY.md (Phase 6B coverage numbers),
ROOT_PROGRAM_ARCHITECTURE.md (ROOT 08/09 PARTIAL, new module lists per
course), COURSE_CONTENT_MODEL.md (is_required completion rule, a new
content-versioning section per this phase's own requirement),
COURSE_AUTHORING_STANDARD.md (classifying near-duplicate/advanced
scenarios), ENGINEERING_SOP.md (new Phase 6B section),
REMOTE_AND_SELF_PACED_ARCHITECTURE.md (cross-reference), and
platform/README.md (e2e-phase6b.js, the Phase 6B rehearsal, isolation-
per-suite regression guidance).

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-26 09:09:47 -05:00
ced4568andClaude Sonnet 5 505783e5f7 test(phase6b): add ROOT curriculum acceptance coverage + real-Lab rehearsal
e2e-phase6b.js (24 checks): hierarchy for the 13 new modules across
ROOT 05-10/12/Capstone, prerequisite chains through the newly-inserted
front-door modules, lesson/knowledge-check completion in brand-new
modules, a newly-wired scenario (job-task) completing correctly, the
optional-vs-required (is_required) completion rule, persistence/
isolation/RBAC/draft-hiding proven against the new content, and the
capstone server-side unlock fix in both directions (a direct bypass
attempt rejected with blocking_stage_ids, an ordinary exam unaffected,
a legitimately unlocked attempt still accepted and its completion
reflected in the course content model).

test-phase6b-root-course-rehearsal.js + wrapper (11 checks, real k3s):
one continuous student narrative through a brand-new module (ROOT 07
Services & Service Discovery, not the Phase 6A gold module) — real Lab
launch, the Lab's own setup_commands, a real 'kubectl expose' against
the live cluster, real validate, logout/login/Resume Learning, Lab
destroy/recreate with academic progress intact — plus an instructor
narrative proving Curriculum view, Prepare Labs, and classroom/learning
tooling all still work against the expanded curriculum.

e2e-phase6a.js updated for two of this phase's own intentional,
documented behavior changes (not defects): resume now points at
root-05's new mental-model module instead of the old bare-lab bank, and
root-10's gold module now unlocks the new troubleshooting-framework
module rather than the bare-lab bank directly.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-26 09:09:20 -05:00
ced4568andClaude Sonnet 5 e0f954e714 feat(instructor): expand curriculum teaching visibility
CurriculumView.jsx's per-module activity table gains a Required column
(required/optional) so an instructor can see at a glance which of
Phase 6B's new CORE vs. GUIDED/INDEPENDENT/ADVANCED activities a
student needs to complete a module. Reuses the existing Classroom/
Prepare Labs tooling unchanged — no second classroom system.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-26 09:08:54 -05:00
ced4568andClaude Sonnet 5 54b35e8ad7 feat(student): polish structured ROOT learning flow
CourseView.jsx now shows an '(optional)' label on any activity marked
is_required=false, both in the module activity list and the activity
detail header — the only genuine UX need Phase 6B's new optional/
advanced practice content exposed. No other changes to the existing
Course/Module/Activity navigation, Launch Lab flow, or Practice tab.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-26 09:08:28 -05:00
ced4568andClaude Sonnet 5 de6c5b3286 feat(root): build Real Ops and capstone readiness content
root-12-real-ops gains a Detect -> Triage -> Diagnose -> Mitigate ->
Recover -> Verify -> Learn incident-response framing lesson at the
front of the existing 4-incident module, and a post-incident reflection
at the end — the incidents themselves are unchanged, still deliberately
un-guided ('you are responsible for this environment,' not 'complete
this exercise'). No fixes revealed; instructor notes carry the deeper
solution guidance the student-facing lesson intentionally withholds.

root-capstone-m02-capstone-readiness — a new READY orientation module:
what the Capstone measures, how the timed/randomized draw works, time
management and troubleshooting strategy under a clock, and a genuine
self-assessment checklist. Never publishes the current draw or answers.
See docs/ROOT_CAPSTONE_READINESS.md for why this module's completion is
deliberately independent of the real capstone's server-side unlock gate.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-26 09:08:01 -05:00
ced4568andClaude Sonnet 5 831486d664 feat(root): expand troubleshooting and observability curriculum
The Phase 6A gold-standard Health Probes module is preserved byte-for-
byte unchanged — it's the template, not a page to rewrite for
consistency.

root-10-m03-troubleshooting-framework — a new PARTIAL module
introducing a repeatable Observe -> Gather Evidence -> Form Hypothesis
-> Test -> Repair -> Validate -> Prevent Recurrence framework, applied
to a CrashLoopBackOff and a broken Service selector. Lesson,
demonstration (the full framework walked through a real
CrashLoopBackOff, command by command), knowledge check, and real Labs
(crashloop-fix, endpoint-fix-task). Sequenced between the gold module
and the existing bare-lab bank, which now sits behind it.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-26 09:07:34 -05:00
ced4568andClaude Sonnet 5 960a638ec9 feat(root): add Kubernetes security curriculum
Four new modules, moving ROOT 09 off PLANNED and building the
22-scenario CKS-style bank into a coherent course instead of
disconnected exam tasks — see docs/ROOT_CURRICULUM_COVERAGE_MATRIX.md
for the CORE/GUIDED/INDEPENDENT/ADVANCED/ASSESSMENT-BANK classification
behind every placement.

root-09-m01-security-model-and-securitycontext — READY, full
gold-standard pattern: defense in depth, least privilege, hardening a
Pod with SecurityContext. Lesson, demonstration (a Pod before/after
hardening), knowledge check, guided exercise (a hardening checklist),
real Labs (CORE: non-root/drop-capabilities/read-only-fs/
pod-security-context; GUIDED: seccomp; ADVANCED: AppArmor; INDEPENDENT:
the generic pod-security-context duplicate), and a reflection.

root-09-m02-serviceaccounts-and-rbac — PARTIAL: workload identity and
least-privilege permissions. Lesson + knowledge check + real Labs.

root-09-m03-network-and-supplychain-security — PARTIAL: NetworkPolicy
as a security control (default-deny), image/registry trust, TLS
ingress. Lesson + knowledge check + real Labs.

root-09-m04-cluster-hardening-and-runtime — PARTIAL, deliberately
survey-depth: Pod Security Admission and a lighter treatment of
audit-policy/runtime-sandboxing topics that don't belong in the core
daily-driver path.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-26 09:07:07 -05:00
ced4568andClaude Sonnet 5 73cea1ba1d feat(root): add storage and scheduling curriculum
Two new PARTIAL modules, moving ROOT 08 off PLANNED for the first time:

root-08-m01-storage-fundamentals — ephemeral vs. persistent storage,
PV/PVC/StorageClass. Lesson + knowledge check + real Labs (emptydir-pod,
pv-pvc-mount, pvc-dynamic-task, storageclass-mcq).

root-08-m02-scheduling-resource-governance — how the scheduler places
Pods and how quotas/limits/priority/disruption budgets shape that
placement. Lesson + knowledge check + real Labs (node-label-selector,
node-taint-toleration, pod-affinity-mcq, priority-class-task,
pod-disruption-budget-task, resource-quota-ns, limitrange-task).

cluster-upgrade-mcq and etcd-backup-mcq deliberately left unwired —
ROOT 11/DevOps territory, out of this phase's scope, not overlooked.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-26 09:06:43 -05:00
ced4568andClaude Sonnet 5 dbcc210889 feat(root): author networking and configuration curriculum
root-07-m03-services-and-service-discovery — a new, full gold-standard-
pattern module: the Service/Endpoints/DNS request path, choosing the
right Service type. Lesson, demonstration (a Service with zero
Endpoints — the single fastest diagnostic check for this failure
class), knowledge check, guided exercise (ClusterIP vs. NodePort vs.
LoadBalancer for three real components), a real Lab
(expose-service-basics), and a reflection.

root-07-m04-ingress-networkpolicy-foundations — a new PARTIAL module:
Ingress and NetworkPolicy at the architectural level (the security-
control framing of NetworkPolicy is deliberately deferred to ROOT 09).
Lesson + knowledge check + real Labs (ingress-task, network-policy).

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-26 09:06:19 -05:00
ced4568andClaude Sonnet 5 86ea035e9b feat(root): author workloads curriculum
root-06-m02-deployments-scaling-updates — a new, full gold-standard-
pattern module: why Deployments/ReplicaSets exist, how a rolling update
actually progresses, and what rollback restores. Lesson, demonstration
(a healthy rollout vs. one stalled on ImagePullBackOff), knowledge
check, guided exercise (planning a safe rollout strategy), real Labs
(rolling-update-task, deployment-rollback), and a reflection.

root-06-m03-jobs-cronjobs-daemonsets — a new PARTIAL module: Jobs,
CronJobs, and DaemonSets as distinct workload shapes from a
Deployment's always-on model. Lesson + knowledge check + real Labs
(job-task, cronjob-task).

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-26 09:06:00 -05:00
ced4568andClaude Sonnet 5 25409d3bd7 feat(root): author Kubernetes foundations curriculum
root-05-m00-kubernetes-mental-model — a new, full gold-standard-pattern
front-door module for ROOT 05: what Kubernetes solves, the desired-state
model, the control-plane/worker split, and Pods/Namespaces/labels
through kubectl. Lesson, demonstration (watching a Pod get replaced
after manual deletion), knowledge check, guided exercise (planning a
Namespace layout), a real Lab (namespaces-basics), and a reflection —
sequenced ahead of the existing bare kubectl-fluency lab bank.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-26 09:05:40 -05:00
ced4568andClaude Sonnet 5 046a0fba18 fix(capstone): enforce unlock rule server-side
POST /api/assessments/:definitionId/start only ever checked that the
definition existed and was active — it never checked unlock_rule. GET
/api/capstone correctly computed and displayed lock state, and the
student UI never exposed a start action while locked, but a direct POST
to this route bypassed the check entirely (documented as a known gap in
docs/PHASE6A_ROOT_COURSE_FOUNDATION.md §10). Student UI state is not the
enforcement boundary.

Fixed by binding to the exact same computeCurriculumState() gate
/api/capstone already relies on, rather than re-parsing
definition.unlock_rule independently — one computation of 'is this
unlocked,' not two that could drift. Only a definition the curriculum
path actually names as ITS capstone is gated this way; an ordinary exam
is completely unaffected. Returns a clean 403 { error: 'locked',
blocking_stage_ids } — the same shape /api/capstone already exposes.

Found while fixing this: e2e-phase5b.js's own capstone-completion test
had been passing only because of this exact gap — it never included the
real-ops stage in its completion loop before attempting to start the
capstone, and the missing unlock check let it succeed anyway. Fixed
alongside the platform code (REAL_OPS added to ALL_NON_CAPSTONE), not
routed around.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-26 09:05:18 -05:00
ced4568andClaude Sonnet 5 bee57d89fd feat(root): wire ROOT 05-10/12/Capstone curriculum expansion
023_phase6b_curriculum.sql — additive on top of 022_course_content.sql:
13 new teaching modules (ROOT 08/09 move from PLANNED to PARTIAL; ROOT
05-07/10 gain a new front-door module sequenced ahead of the existing
bare-lab bank; ROOT 12 gains an incident-response framing lesson +
reflection; Capstone gains a readiness module). No existing row is
deleted; a small number of existing rows have their sequence/
prerequisites updated so a new teaching module supersedes the bare-lab
bank as the front door — never their scenario_id/activity_type/
completion_behavior, so no student's derived completion changes meaning.

is_required (new module_activities column, defaults true — every Phase
6A activity unaffected) lets a module distinguish CORE teaching from
GUIDED/INDEPENDENT-practice duplicates and ADVANCED/optional extras.
course-progress.js's totals() now considers a module complete once
every REQUIRED activity is done, not literally every activity — an
incomplete optional one no longer blocks completion, though it still
counts toward the displayed percentage.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-26 09:04:55 -05:00
ced4568andClaude Sonnet 5 82f17a05e8 docs(curriculum): map Phase 6B ROOT coverage
Course -> module -> objective -> lesson/demo/knowledge-check/guided-
exercise -> scenario(s) -> Real Ops connection -> status, for ROOT 05-10,
12, and Capstone, grounded in the Phase 6A inventory. Every scenario
placement has an intentional reason, not a coverage-percentage target —
includes an explicit 'duplicate concepts identified' section for the
near-duplicate scenario pairs (pod-security-context vs.
cks-pod-security-context, rbac-clusterrole vs. cks-rbac-clusterrole,
network-policy's architectural vs. security-control framing) and which
scenarios are deliberately left out of structured curriculum entirely.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-26 09:04:30 -05:00
ced4568andClaude Sonnet 5 fd38d06055 docs(phase6a): document RootNode course foundation
COURSE_AUTHORING_STANDARD.md — the standard every future lesson/module
must follow (target learner, prerequisites, measurable objectives,
concept-before-lab, instructor notes, terminology, versioning, review
requirements), grounded in the gold-standard module as the reference
implementation.

PHASE6A_ROOT_COURSE_FOUNDATION.md — the phase summary: youth-ladder
context, the Google Workspace status already recorded in Phase 5C's own
docs, architecture/build summary, honest content states, test results,
explicit non-goals, and known limitations/Phase 6B recommendations.

Updated ENGINEERING_SOP.md (new Phase 6A section + a curriculum-content
cross-reference), REMOTE_AND_SELF_PACED_ARCHITECTURE.md (a 7th
closed gap: self-paced enrollment now resolves to a real course tree,
not just a scenario list), LAB_ARCHITECTURE.md (Phase 6A reuses the
disposable Lab unmodified, verified by a real rehearsal), and
platform/README.md (course model in the api/ summary, new e2e suite
descriptions for 5A/5B/5C/6A which were missing entirely, and the
real-Lab rehearsal script).

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-25 23:30:11 -05:00
ced4568andClaude Sonnet 5 8ffc923a84 test(phase6a): add course-delivery acceptance coverage + real-Lab rehearsal
e2e-phase6a.js (21 checks): hierarchy/ordering, honest READY/PARTIAL/
PLANNED states, all three completion behaviors (manual marker/
scenario-derived/assessment-derived), prerequisite locking (including
the zero-content-is-vacuously-satisfied rule), resume, persistence
across logout/login, cross-student isolation, RBAC for self-paced
no-cohort students, draft-content hiding, and content-version
stability.

test-phase6a-course-rehearsal.js + run-phase6a-course-rehearsal.sh: a
real-k3s dress rehearsal of the gold-standard module end to end — a
student reads the real lesson content, launches a real disposable Lab,
the Lab's own POST /api/scenarios/:id/setup (called through the
Gateway's existing /lab/:token/* proxy exactly as the browser UI does)
stands up the broken api-gateway deployment, a real kubectl patch fixes
the readiness probe, and the module completes and unlocks the next one.
No mocked KubeKosh anywhere in this suite.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-25 23:30:00 -05:00
ced4568andClaude Sonnet 5 db927b8081 feat(instructor): add curriculum teaching view
CurriculumView.jsx — a new 'Curriculum' tab in the admin portal:
program structure (courses -> modules -> activities), mapped scenarios/
assessments, estimated teaching time, and instructor notes rendered per
activity. A student_id lookup shows one student's real per-activity
progress alongside the same structure, for spotting where someone is
stuck. Read-only, no WYSIWYG editor — course/module authoring stays
Git/migration-based this phase per docs/COURSE_CONTENT_MODEL.md.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-25 23:29:50 -05:00
ced4568andClaude Sonnet 5 8c09ef8f3e feat(student): add structured course learning experience
CourseView.jsx — My Programs -> ROOT_ -> Course -> Module -> Activity ->
Launch Lab -> back -> continue, nested inside the existing self-paced
ProgramsTab rather than a second portal. Renders lesson/demonstration/
knowledge-check/guided-exercise/reflection content via the existing
react-markdown+remark-gfm stack (already used by ScenarioPanel.jsx — no
new dependency). Locked modules/courses are inert, not hidden — the
server is still the actual gate (course-routes.js).

ProgramsTab.jsx gains a 'View Courses' button alongside the existing
'Continue Learning' button — the original direct-to-Lab flow is
byte-for-byte unchanged; both paths call the identical launch sequence
underneath.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-25 23:29:41 -05:00
ced4568andClaude Sonnet 5 28c925069b feat(root): author the health-probes gold-standard module
content/root/root-10-kubernetes-troubleshooting-observability/m01-health-probes/:
lesson (probe concepts) -> demonstration (reading kubectl describe pod
Events for a probe failure) -> knowledge check -> guided exercise
(write a readiness probe on paper) -> Launch Lab intro
(rootnode-probe-port-mismatch) -> reflection (root cause + Real Ops
prevention takeaway). 5 of 6 activities carry instructor notes (demo
plan, misconceptions, timing, discussion questions, extensions).

This is the reference implementation docs/COURSE_AUTHORING_STANDARD.md
documents for all future RootNode lesson authoring.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-25 23:29:32 -05:00
ced4568andClaude Sonnet 5 e2298e95de feat(root): wire course/module/activity routes into the Platform API
course-routes.js — student-facing: the full course tree through an
enrollment (GET /api/student/programs/:id/courses) and the lesson
completion marker (POST /api/student/activities/:id/complete, refused
for anything but a manual_marker activity — a lab/assessment can never
be faked complete through this route).

admin-course-routes.js — instructor-facing, read-only (no WYSIWYG CMS
this phase): program structure with instructor notes
(GET /api/admin/programs/:id/courses) and per-student progress
(GET /api/admin/students/:id/courses), scoped via
requireStudentRelationshipAccess — the same self-paced-no-cohort rule
Phase 5C's notification routes already established, not
requireStudentCohortAccess (which would 403 a plain instructor out of
every self-paced student).

Dockerfile now also COPYs content/ alongside the existing scenarios/ —
same 'read-only, never executed' reasoning as that line.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-25 23:29:24 -05:00
ced4568andClaude Sonnet 5 fe702754cd feat(content): add course/module/activity content model
New tables (courses, course_modules, module_activities,
activity_completions) layered over the existing scenario/assessment/
achievement systems — a lab activity references an existing scenario_id,
an assessment/capstone activity references an existing
assessment_definition_id, neither duplicated. Long-form lesson prose
lives in git-tracked Markdown (content/root/), split into shared vs.
instructor-only sections by a '## Instructor Notes' heading convention
(content.js) rather than a second file or a CMS.

course-progress.js derives completion from EXISTING sources only:
auto_scenario reads the progress table, auto_assessment reads
achievements by assessment_definition_id, manual_marker is the one
genuinely new piece of state (for lesson-only activities with nothing
else to derive completion from). Prerequisite locking treats a
zero-content (PLANNED) prerequisite as vacuously satisfied, so an
honestly-unbuilt course never permanently locks everything after it.

Found by testing: computing a level's prerequisite lock and stripping
its working totals in the same map iteration made the result depend on
iteration order. Fixed with two explicit passes.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-25 23:29:13 -05:00
ced4568andClaude Sonnet 5 f152b9a489 docs(curriculum): define the ROOT_ program architecture
13 courses (ROOT 01-12 + Capstone) under the existing single
root-kubernetes-foundations program — root_ is one continuous ladder a
student enrolls in once, not thirteen separate purchases. Honest
READY/PARTIAL/PLANNED states grounded in the actual inventory: no course
reaches READY yet, 5 are fully PLANNED (no Linux/Git/Docker/DevOps
scenario content exists anywhere in this repo), 7 are PARTIAL (real
labs, no lessons yet). Documents the prerequisite graph and why it
deviates from a strictly linear 01->12 chain where the existing content
doesn't actually support one.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-25 23:29:03 -05:00
ced4568andClaude Sonnet 5 e4cbcf6c40 docs(curriculum): inventory existing RootNode learning assets
Full inventory of scenarios/data/ (98 scenarios), scenarios/bundles/,
rootnode-path.json's stage structure, assessment_definitions, and the
programs/achievements model — gathered by inspection before any Phase 6A
design work, nothing modified to produce this.

Only 3 scenarios are RootNode-authored; the other 95 (including all 44
wired into rootnode-path.json) are generic upstream Kubernetes/CKS
content. 54 scenarios exist with no rootnode-path.json stage at all —
a real, working content bank for future ROOT 07/08/09/10 modules,
inventoried and grouped by category rather than left undiscovered.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-25 23:28:55 -05:00
ced4568 4c0a950afb fix: retry Google's transient domain-wide-delegation propagation-delay error
Release to Docker Hub / Build & Push Docker Image (push) Canceled after 0s
Release to Docker Hub / Create GitHub Release (push) Canceled after 0s
Live Google Workspace delivery validation surfaced a real, self-resolving
Google-side condition: freshly-authorized domain-wide delegation can take
up to about a minute to propagate across Google's auth infrastructure,
producing intermittent unauthorized_client on the OAuth token exchange
even with correct configuration. gws-provider.js now classifies
unauthorized_client as a temporary (retryable) error instead of permanent,
so the existing bounded-backoff worker absorbs it automatically.

Also documents the alias-vs-delegated-user env passthrough in
docker-compose.dev.yml, and records both real findings (this one and the
GOOGLE_DELEGATED_USER alias distinction) plus the full live-delivery
validation outcome in the runbook and Phase 5C doc.

Validated live: isolated gws-email-test.js send (DKIM/SPF passing,
delivered) and a full real lifecycle test (create student, enrollment,
real activation email, activate, login, see program, real password-reset
email, reset, login with new password) via [email protected].
2026-08-25 21:36:12 -05:00
ced4568 7baf7074e5 fix(email): support a delegated-mailbox alias distinct from the delegated Workspace account
Real-world case found during setup: the sending address ([email protected]) is an
alias of an existing Workspace user, not its own paid seat (a deliberate cost-saving choice).
Domain-wide delegation authorizes a real Workspace ACCOUNT, not an arbitrary alias string, so
using the alias directly as the delegated 'sub' claim would have been wrong. Added
GOOGLE_DELEGATED_USER (the account that actually owns the alias and was granted delegation) as
an optional override, falling back to EMAIL_FROM_ADDRESS unchanged when the sending address is
itself a primary account. EMAIL_FROM_ADDRESS remains the visible From: header in both cases.
Documented in docs/RUNBOOK_GOOGLE_WORKSPACE_EMAIL.md with a concrete example.
2026-08-25 17:38:59 -05:00
ced4568 777efd168d docs(phase5c): enrollment communications documentation + Google Workspace runbook
- PHASE5C_ENROLLMENT_COMMUNICATIONS.md: the full design doc.
- RUNBOOK_GOOGLE_WORKSPACE_EMAIL.md: every REQUIRES GOOGLE WORKSPACE ADMIN ACTION step
  (mailbox, service account, domain-wide delegation scoped to gmail.send only, SPF/DKIM/DMARC),
  environment variables, testing procedure, production secret handling, rotation, revocation,
  troubleshooting, quotas.
- RUNBOOK_STUDENT_ACTIVATION.md: day-to-day operation of the pipeline from /admin.
- WEBSITE_ENROLLMENT_INTEGRATION.md: explicit website-vs-RootNode responsibility boundary,
  the new contact_email field, and closes the 'RootNode doesn't send activation email yet'
  known limitation this doc previously documented.
- RUNBOOK_SELF_PACED_ENROLLMENT.md, RUNBOOK_STUDENT_ACCOUNT_LIFECYCLE.md: point at the new
  automatic-email behavior and the Resend activation control.
- REMOTE_AND_SELF_PACED_ARCHITECTURE.md, ENGINEERING_SOP.md: close the enrollment-
  communication gap; new ENGINEERING_SOP §16.
- PRODUCTION_VALIDATION_CHECKLIST.md, README.md, platform/README.md: cross-references and a
  Phase 5C production-validation section.
2026-08-25 14:45:18 -05:00
ced4568 b38c46aaec test(phase5c): permanent regression suite + enrollment/activation dress rehearsal
- platform/test/e2e-phase5c.js (29 checks): activation-vs-confirmation routing,
  duplicate/concurrent webhook idempotency, temporary/permanent/exhausted retry
  classification, resend/regenerate token invalidation, activation/reset single-use and
  expiration, HTML-injection regression, production token-secrecy gating via a real container
  recreate with COOKIE_SECURE=1, minor/contact-email resolution, RBAC scoping, metrics/audit
  hygiene, and a real Platform API restart.
- infra/scripts/drills/test-phase5c-enrollment-activation.js (10 checks): one continuous
  story — website enrollment through a real rendered/extracted activation email, a real Lab
  launch and scenario completion, a second enrollment's confirmation email, a full
  password-reset round trip, and the minor/no-email contact-email path.
- infra/scripts/gws-email-test.js: the opt-in (RUN_GWS_EMAIL_TEST=true) live Google Workspace
  delivery test — does nothing without the explicit flag, sends exactly one message to one
  operator-configured recipient when enabled.
2026-08-25 14:44:25 -05:00
ced4568 be92be81cd feat(observability): notification delivery metrics, no PII/high-cardinality labels
- rootnode_notifications_total{notification_type,status} (computed fresh from the DB, like
  the existing enrollment_breakdown gauge — never drifts from what's actually stored),
  rootnode_notification_queue_depth, rootnode_notification_oldest_queued_seconds,
  rootnode_notification_delivery_attempts_total{provider,result} (a real in-process counter —
  per-attempt outcomes aren't reconstructable from the DB after the fact),
  rootnode_student_activations_{success,invalid}_total. Every label is a bounded enum —
  never student_id/email/token/notification_id/external_event_id.
2026-08-25 14:43:48 -05:00
ced4568 f25535027c feat(admin): notification operations — resend activation, delivery status, RBAC-scoped views
- notification-routes.js: global list (owner/admin only), per-student list (scoped via
  requireStudentRelationshipAccess), resend-activation (mechanically the same as
  regenerate — Phase 5A's token model already guarantees only the latest is ever valid, so one
  action covers both), a dev-only preview route (unconditionally refuses when
  COOKIE_SECURE=1, mirroring student-auth-routes.js's existing dev_reset_token gate).
- NotificationsView.jsx: owner/admin global notification list, paginated, matching
  AuditView.jsx's exact convention.
- StudentsView.jsx: student detail gained a Resend activation control and a Notifications
  history card; Credentials card now explains the automatic-email behavior.
- AdminPortal.jsx: new Notifications nav entry (owner/admin only).
2026-08-25 14:43:10 -05:00
ced4568 3c7f0ca59c feat(activation): wire notifications into enrollment/auth, real password-reset delivery
- integration-routes.js: accepts student.contact_email, calls notifyForEnrollment on a
  genuinely new enrollment only (never on an idempotent replay — inherits Phase 5B's proven
  enrollment-level idempotency, no separate notification dedup needed).
- admin-routes.js: create-student/convert/reset-password now enqueue the corresponding
  notification using the SAME token already minted synchronously (never mints twice, which
  would silently invalidate the token just handed back in the response). Fixed a real bug:
  request-password-reset would mint a reset token for a never-activated student, which makes
  no sense and would confuse them — now falls through to the generic anti-enumeration response.
- admin-programs-routes.js: manual enrollment triggers the same notification decision as the
  website path.
- student-auth-routes.js: /activate records a metric; /change-password and /reset-password
  both queue a password_changed security notice; /request-password-reset now actually delivers
  email (production still never returns the token — unchanged anti-enumeration/isProduction
  gating, just now with a real send behind it).
- rbac.js: requireStudentRelationshipAccess/checkStudentRelationshipAccess — the notification
  visibility rule (cohort-shared or an enrollment they created), reusable beyond just
  enrollments.
- server.js: wires notification-routes.js and starts the notification worker in main().
- docker-compose.dev.yml: PUBLIC_APP_ORIGIN, EMAIL_FROM_NAME, and a parametrized COOKIE_SECURE
  (was hard-commented-out; needed so e2e-phase5c.js can actually prove production gating).
2026-08-25 14:42:34 -05:00
ced4568 9d382c95c5 feat(email): provider-neutral email service — Google Workspace (Gmail API) + dev provider
- email/provider.js: the seam. EMAIL_PROVIDER unset always defaults to the dev provider,
  regardless of any other setting — misconfiguration means email silently doesn't send for
  real, never that a dev/test run accidentally reaches a real student.
- email/gws-provider.js: Gmail API users.messages.send via a service account with Workspace
  domain-wide delegation, impersonating one dedicated mailbox. No human password, no SMTP
  basic auth, no googleapis/google-auth-library dependency — a hand-rolled RS256 JWT-bearer
  OAuth exchange + one REST call (~120 lines), matching this codebase's established
  hand-roll-before-adding-a-dependency convention.
- email/dev-provider.js: never makes a network call, returns the rendered message back to the
  caller for dev/test inspection. Supports magic +forcetempfail/+forcepermfail recipient
  markers so the permanent test suite can deterministically exercise retry/failure
  classification without needing a real flaky provider.
- email/templates.js: activation/enrollment-confirmation/password-reset/password-changed,
  HTML + plain-text, no external images/tracking/JS. Every interpolated value is HTML-escaped
  in the HTML body (a real injection finding — display_name can come straight from a website
  integration event).
- notification-worker.js: a setInterval poller (same pattern as sessions.js's sweeper), claims
  via notifications.js, renders via templates.js, builds activation/reset URLs from
  PUBLIC_APP_ORIGIN, stores a dev-only preview (never in production) so tests can extract the
  real URL without a mailbox.
2026-08-25 14:42:01 -05:00
ced4568 5183a6af09 feat(notifications): durable notification lifecycle + minor-safe recipient resolution
- migrations/021: notification_events (queued/sending/sent/failed/cancelled, bounded
  exponential backoff via attempt_count/next_attempt_at, a classified last_error_code, never a
  raw stack trace or provider credential).
- notifications.js: enqueue (records a no-recipient case as already-failed rather than
  silently dropping it), claimNext (FOR UPDATE SKIP LOCKED — safe if two worker processes ever
  run), markSent/markFailedOrRetry (bounded retry, terminal-state token scrubbing),
  notifyForEnrollment (the shared activation-vs-confirmation decision every enrollment-creation
  path calls), maskEmail/toAdminView (the only shapes any admin API ever returns).
- enrollment.js: enrollmentContactEmail() — reads back the minor/no-email transitional
  contact-email convention (stored in the existing enrollments.metadata JSONB, never written to
  students.email) that createFromIntegrationEvent/createManualEnrollment now accept.
2026-08-25 14:41:29 -05:00
ced4568 1c08a30c7f docs(phase5b): self-paced enrollment documentation + website integration contract
- PHASE5B_SELF_PACED_ENROLLMENT.md: the full design doc.
- RUNBOOK_SELF_PACED_ENROLLMENT.md: operational runbook for /admin -> Programs.
- WEBSITE_ENROLLMENT_INTEGRATION.md: the complete contract for the separate website team —
  endpoint, auth, request/response schema, idempotency/retry rules, student matching, minor/
  no-email behavior, error responses, what not to send, local testing, production-secret
  handling. Written to be usable without reading the Platform API source.
- REMOTE_AND_SELF_PACED_ARCHITECTURE.md: closes the self-registration/curriculum-assignment/
  delivery-mode/enrollment-model gaps this doc previously flagged as Phase 5+ scope.
- ENGINEERING_SOP.md: new §15. LAB_ARCHITECTURE.md, PRODUCTION_VALIDATION_CHECKLIST.md,
  README.md, platform/README.md: cross-references and a Phase 5B production-validation section.
2026-08-25 12:10:13 -05:00
ced4568 23f881884d test(phase5b): permanent regression suite + self-paced dress rehearsal
- platform/test/e2e-phase5b.js (32 checks): program seed, entitlement gating
  (pending/future/expired/cancelled), cross-student isolation, real Lab launch/destroy/
  reaper-reconciliation/recreate persistence, server-derived idempotent completion via a full
  real capstone completion, the integration boundary (auth, idempotency, real concurrency,
  unknown/inactive program rejection, minor/no-email path, no card data), RBAC scoping, a
  regression test for the process-crash bug this suite found, Phase 5A smoke, and a real
  Platform API restart.
- infra/scripts/drills/test-phase5b-selfpaced-rehearsal.js (15 checks): one continuous story —
  activation, a simulated website enrollment event, real Lab work, Lab destroy, a new session,
  Continue Learning, a fresh Lab, full real capstone completion, Completed transition, and a
  second uninvolved student proven unable to reach any of it.
2026-08-25 12:09:34 -05:00
ced4568 f89a769540 feat(observability): enrollment/completion metrics, no PII/high-cardinality labels
- metrics.js: rootnode_enrollments_total{status,delivery_mode}, rootnode_selfpaced_active_
  enrollments, rootnode_program_completions_total{program_id}, rootnode_enrollment_events_
  {created,duplicate,rejected}_total. Every label is a bounded enum or an admin-curated slug —
  never student_id/email/enrollment_id/external_reference.
2026-08-25 12:09:17 -05:00
ced4568 4bc202e5f3 feat(integration): provider-neutral website enrollment boundary
- integration-routes.js: POST /api/integrations/enrollments — separate from student/instructor
  session auth entirely (no cookie read), pre-shared-secret header auth matching the existing
  Gateway<->Provisioner trust-boundary pattern. Idempotent via a DB unique constraint on the
  caller's own event_id (not app-level check-then-insert) — proven safe under real concurrent
  duplicate requests. Student resolution is create-or-match by email OR login_identifier
  (minor/no-email support), reusing Phase 5A's exact credential-issuing path. A small,
  provider-neutral status vocabulary (paid/active/confirmed/pending/awaiting_payment) — never
  a raw payment-provider object model.
- docker-compose.dev.yml: INTEGRATION_SHARED_SECRET, optional in dev (fails closed if unset).
2026-08-25 12:09:02 -05:00
ced4568 1b6da0a966 feat(selfpaced): student-facing My Programs dashboard
- ProgramsTab.jsx: Active/Completed programs, real progress, deterministic Continue Learning
  (reuses the same launch/poll pattern MyLearningTab already established), entitlement-blocked
  states shown with a real reason, never a silent dead end.
- StudentDashboard.jsx: new My Programs tab. api.js: programs/program/launchProgram.
2026-08-25 12:08:44 -05:00
ced4568 17160da89e feat(admin): Programs area, enrollment management, student enrolled-programs view
- admin-programs-routes.js: program CRUD (admin/owner authoring, all roles can view — matches
  assessment_definitions' convention), scoped enrollment listing, manual enroll, activate/
  cancel (via rbac.js's requireEnrollmentAccess).
- ProgramsView.jsx: cohort-picker-free program list/create, per-program detail with
  enrollment table and a manual-enroll form.
- StudentsView.jsx: student detail gained an 'Enrolled programs' card.
- AdminPortal.jsx: new Programs nav entry.
2026-08-25 12:08:29 -05:00
ced4568 611e1ac4ba feat(enrollment): enrollment lifecycle, entitlement gating, self-paced student API
- enrollment.js: server-only status transitions (pending->active->completed, cancel),
  deriveEnrollmentAccess (pure, derived-at-read-time like classroom.js's derivePresence —
  'expired' is never stored, computed from access_ends_at), maybeCompleteEnrollment
  (idempotent, server-derived from the existing capstone achievement).
- environment-view.js: publicEnvView/launchEnvironmentForStudent extracted out of server.js's
  original /api/environment/launch route body so the new entitlement-gated self-paced launch
  reuses the identical disposable-Lab sequence instead of a second copy — behavior for the
  existing cohort/pilot route is completely unchanged.
- program-status.js: combines an enrollment with real curriculum/progress state into one view
  (access, percent-complete, current stage, deterministic next-activity/resume target) — shared
  by the student and admin surfaces so both show identical information.
- selfpaced-routes.js: GET /api/student/programs(/:id), POST .../launch — entitlement enforced
  server-side before ever touching the Provisioner, never just hidden in the UI.
- rbac.js: requireEnrollmentAccess — a plain instructor may manage an enrollment they created
  or that belongs to a student sharing one of their cohorts; a self-paced enrollment can have
  no cohort at all, so the existing cohort-only checks can't cover it.
- server.js: found and fixed a real bug — POST /api/progress/record's handler predated the
  wrap() convention, so an unexpected internal error (a malformed environment_id) crashed the
  entire process instead of just 500ing that request. Now wrapped + input-validated.
2026-08-25 12:08:12 -05:00
ced4568 895e16fea4 feat(programs): first-class program model — init_/compile_/root_ ladder, seeded root-kubernetes-foundations
- migrations/019: programs table (slug PK, matching assessment_definitions/cohorts
  convention), family (init/compile/root) and delivery_mode (self_paced/cohort/hybrid) as real
  CHECK-constrained enums, versioned like assessment_definitions so an edit to
  curriculum_path/delivery_mode never silently changes what an existing enrollment means. Seeds
  exactly one program wrapping the existing RootNode/Kubernetes curriculum and capstone — no
  new curriculum content invented.
- programs.js: list/get/create/update, with automatic version bump only when a
  snapshot-relevant field actually changes.
2026-08-25 12:07:34 -05:00
206 changed files with 17044 additions and 91 deletions
+19
View File
@@ -42,6 +42,25 @@ RUN set -eux && \
ln -sf /usr/local/bin/k3s /usr/local/bin/kubectl && \
ln -sf /usr/local/bin/k3s /usr/local/bin/crictl
# ── Docker Engine (ROOT 04) + DNS tooling (ROOT 02) ──────────────────────────
# A second, independent container runtime nested inside this already-Privileged Lab container
# (see docs/ROOT_FOUNDATIONS_LAB_STRATEGY.md §3 for the security/resource evaluation and the
# real test results this decision is based on) — its own dockerd, its own
# /var/run/docker.sock, entirely separate from the Lab Host's real Docker socket, which is
# never mounted into this image. docker-compose v2 isn't bundled by Ubuntu's docker.io package,
# so fetch the static binary the same way k3s's own binary is fetched above, rather than adding
# a whole new apt repository for one plugin.
RUN apt-get update && apt-get install -y docker.io dnsutils && rm -rf /var/lib/apt/lists/*
RUN set -eux && \
case "$(uname -m)" in \
x86_64) COMPOSE_ARCH="x86_64" ;; \
aarch64) COMPOSE_ARCH="aarch64" ;; \
*) echo "Unsupported arch: $(uname -m)" && exit 1 ;; \
esac && \
mkdir -p /usr/local/lib/docker/cli-plugins && \
curl -fsSL "https://github.com/docker/compose/releases/latest/download/docker-compose-linux-${COMPOSE_ARCH}" \
-o /usr/local/lib/docker/cli-plugins/docker-compose && \
chmod +x /usr/local/lib/docker/cli-plugins/docker-compose
# ── App files ─────────────────────────────────────────────────────────────────
WORKDIR /app
+15 -1
View File
@@ -163,6 +163,19 @@ its core:
classes from `/admin` (presence, Lab/learning status, help-request queue, announcements,
guidance-only current-activity) — RootNode owns identity/coordination, an external service
(Meet/Zoom/Teams/etc.) owns the actual video/voice.
- Students can now be enrolled in a self-paced program (as of Phase 5B) — "My Programs" on the
student dashboard, entitlement enforced server-side (pending/future/expired/cancelled
enrollments cannot launch a Lab no matter what the browser sends), completion server-derived
from the existing capstone rules. `/admin` gained a Programs area (program authoring, manual
enrollment, activate/cancel). A separate, provider-neutral `POST /api/integrations/enrollments`
boundary — server-to-server only, no browser session involved — is what the future public
`rootnodeacademy.com` website/checkout will call to create these enrollments; see
`docs/PHASE5B_SELF_PACED_ENROLLMENT.md` and `docs/WEBSITE_ENROLLMENT_INTEGRATION.md`.
- As of Phase 5C, RootNode actually tells students what's happening — real activation,
enrollment-confirmation, and password-reset email, delivered asynchronously through a
provider-neutral notification worker (Google Workspace via the Gmail API in production, a
deterministic dev provider everywhere else — no test/dev run ever emails a real person). See
`docs/PHASE5C_ENROLLMENT_COMMUNICATIONS.md` and `docs/RUNBOOK_GOOGLE_WORKSPACE_EMAIL.md`.
**Never publish the raw privileged container** — front it with isolation (that's what the
Gateway is). The one-container `docker run --privileged -p 7554:80 ...` command above stays
@@ -200,7 +213,8 @@ Key properties of the platform (not this image):
Production topology and what's built vs. not yet: `docs/LAB_ARCHITECTURE.md`,
`docs/PHASE1_PLATFORM_FOUNDATION.md`, `docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md`,
`docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md`, `docs/PHASE3B_LEARNING_AND_ASSESSMENT.md`,
`docs/PHASE5A_STUDENT_IDENTITY_REMOTE_CLASSROOM.md`.
`docs/PHASE5A_STUDENT_IDENTITY_REMOTE_CLASSROOM.md`, `docs/PHASE5B_SELF_PACED_ENROLLMENT.md`,
`docs/PHASE5C_ENROLLMENT_COMMUNICATIONS.md`.
The production infrastructure package (Proxmox/Ansible/Terraform/firewall/monitoring/backup/CI)
that runs all of this on real hardware lives in `infra/` — see
@@ -0,0 +1,66 @@
# How Computers Run Software
## Hardware, OS, and application: three different layers
When you run a program, three distinct layers cooperate:
- **Hardware** — the physical CPU, RAM, and storage. It has no idea what a "file" or a "process"
is; it just executes instructions and stores bits.
- **The operating system (kernel)** — the layer that turns raw hardware into something usable:
it decides which program gets the CPU next, hands out memory, and organizes storage into
files and directories. Linux is a kernel plus a large set of standard tools around it.
- **Applications (userspace)** — the programs you actually run. They ask the kernel for
resources (memory, files, network) through a well-defined interface (system calls) instead of
touching hardware directly.
This module gives you just enough of this model to reason about what you're actually looking at
for the rest of ROOT_ — not a full computer architecture course.
## CPU: the thing that actually executes instructions
A CPU has some number of **cores** — each core can run one stream of instructions at a time.
`nproc` tells you how many cores/threads Linux believes it has available. When a system feels
"slow," it's very often because more processes want CPU time than there are cores to give them.
## RAM: fast, temporary, and finite
RAM holds the data and code a running program is actively using. It's fast but **volatile** —
its contents disappear when power is lost, which is exactly why persistent data lives on
storage instead. `free -h` shows you total memory, how much is in use, and how much is free.
## Storage: slower, but persistent
Storage (an SSD/disk) keeps data even when the machine is off. It's organized into a
**filesystem** — the tree of directories and files you already have some intuition for. `df -h`
shows how much space exists and how much is used on each mounted filesystem.
## Processes: a running program
A **process** is a program actually being executed — code plus its own private memory, tracked
by the kernel with a unique ID (a PID). One program can have many processes running at once
(open two terminals running the same shell, and you have two separate processes). ROOT 01
Module 5 goes much deeper into this; for now, just know: everything you run becomes a process
the kernel is responsible for scheduling onto a CPU core.
## Why this matters for everything after this module
Every concept the rest of ROOT_ builds on is a refinement of this same idea. A container is
(mostly) an isolated process with its own filesystem view. A Kubernetes Pod is a group of
containers the scheduler places onto a specific machine's CPU/RAM. None of that will make sense
as anything other than magic if you don't first have this base model of what a computer is
actually managing.
## Instructor Notes
**Lesson goal:** Give students just enough of a mental model to stop treating "the computer" as
an opaque box — without turning this into a full CS/architecture course.
**Approximate timing:** 15 minutes read; 10 minutes live with discussion.
**Common misconception to watch for:** students often conflate RAM and storage ("my computer
has 8GB" meaning disk space). Draw the volatile-vs-persistent distinction explicitly and early.
**Discussion question:** "Why can't a program just talk to the hardware directly instead of
going through the kernel?" (Answer: the kernel is what lets many programs safely share one CPU/
one block of RAM without stepping on each other — this is worth naming explicitly, as it's the
root of why containers and Kubernetes are even possible.)
@@ -0,0 +1,35 @@
# Guided Exercise: Predict Before You Look
Before you run the real inspection commands in the Lab, commit to a guess — this is what makes
the real numbers meaningful instead of just scrolling past them.
## Your task
On paper or in a scratch file, write down your best GUESS for:
1. How many CPU cores does this machine have?
2. How much total RAM, roughly, in GB?
3. How much free disk space is on the root filesystem?
Then think through: which of these three would you expect a "student Lab" environment to be
MOST constrained on, and why? (Hint: think about what happens if 50 students are each running
their own Lab on one shared physical machine at the same time.)
## A worked shape to check yourself against
There's no single right numeric answer here — the point is the habit of predicting first. A
reasonable answer to the "most constrained" question: memory and CPU are typically the tightest
shared resource on a multi-tenant host, since every additional Lab needs its own slice of both,
while disk is comparatively cheap to over-provision.
## Instructor Notes
**Facilitation:** Have students commit to a number verbally or on a shared doc BEFORE anyone
runs the real commands — the value is in noticing the gap between guess and reality, not in
getting it right.
**Approximate timing:** 5 minutes.
**Optional extension:** ask what would happen to their guess if this were a "real" laptop
they personally own vs. a shared cloud server — good primer for later resource-limit
discussions (ROOT 06's resource requests/limits, ROOT 09's resource governance).
@@ -0,0 +1,27 @@
# Reflection: You Just Looked Inside the Box
## Reflection questions
1. How close was your guess to the real numbers? What surprised you most?
2. Name one thing you now understand about "the computer" that you couldn't have explained
before this module.
3. When you eventually run `kubectl describe pod` and see `requests`/`limits` for CPU and
memory, what do you now expect those numbers to actually be constraining?
## Real Ops takeaway
**Every resource limit you'll ever configure — in Docker, in Kubernetes, in a systemd unit — is
a promise about exactly the three things this module named: CPU, memory, and storage.** You
can't reason well about "why did my container get killed" or "why is my Pod stuck Pending"
without this base model. This module isn't computer-science trivia — it's the vocabulary the
rest of ROOT_ assumes you have.
This module is complete. Continue to the next module.
## Instructor Notes
**Discussion questions for a live classroom:** question 3 is worth spending real time on — it's
the first moment students connect this abstract lesson to something they'll type as a real
command many courses from now. Naming that connection explicitly here pays off later.
**Approximate timing:** 5-8 minutes live discussion; 5 minutes self-paced writing.
@@ -0,0 +1,58 @@
# Linux and the Shell
## What "Linux" actually means
Linux is a **kernel** — the core piece of software from Module 1 that manages CPU, memory, and
storage. What most people call "a Linux system" (Ubuntu, Debian, Fedora, Alpine...) is a
**distribution**: the Linux kernel plus a chosen set of standard tools, package manager, and
configuration conventions layered on top. Different distributions, same kernel underneath —
which is why a command you learn on Ubuntu mostly still works on Alpine.
## Shell vs. terminal
These get used interchangeably but are different things:
- The **terminal** is the window/program that displays text and sends your keystrokes somewhere.
- The **shell** is the program actually interpreting what you type as commands — `bash` is the
shell this Lab uses. The terminal is just the display; the shell is the brain.
## Command structure
Every command follows roughly the same shape:
```
command [options] [arguments]
```
`ls -la /var/log` — `ls` is the command, `-la` are options (flags) that change its behavior,
`/var/log` is the argument (what to act on). Options can usually be combined (`-l -a` = `-la`).
## Exit codes: how a command reports success or failure
Every command returns a number when it finishes: `0` means success, anything else (1-255) means
some kind of failure. This isn't a Linux quirk — it's the exact mechanism scripts, CI pipelines,
and (much later) Kubernetes health checks all rely on to know whether something worked. You can
read the last command's exit code with the special variable `$?`.
## Getting help without leaving the terminal
- **Tab completion** — press Tab while typing a command or path; the shell completes it for you
or shows you the options. This is a genuine time-saver, not just a convenience.
- **Command history** — press the Up arrow to recall previous commands; `history` lists them.
- **`man <command>`** — the manual page for almost any command, with every option explained.
`man ls`, `man grep`, `man chmod` are all real, useful references — reaching for `man` before
a search engine is a habit worth building now.
## Instructor Notes
**Lesson goal:** Establish shell fluency vocabulary (shell vs. terminal, exit codes, options vs.
arguments) that every later module assumes without re-explaining.
**Approximate timing:** 15 minutes read; 10 minutes live with a short live demo of tab
completion and `man`.
**Common misconception to watch for:** students often think "Linux" and "the terminal" are the
same thing — clarify the kernel/distribution/shell/terminal layering explicitly.
**Demo idea:** live-type a command that fails (e.g. `ls /nonexistent`), then run `echo $?`
immediately after to show the exit code in real time — more convincing than describing it.
@@ -0,0 +1,49 @@
# Demonstration: Watching Exit Codes in Real Time
## OBSERVE
Run a command you know will succeed, then immediately check its exit code:
```
$ true
$ echo $?
0
```
## HYPOTHESIS
Now predict: what will the exit code be if the command fails instead?
## INSPECT
Run a command that fails, and check immediately:
```
$ ls /this-does-not-exist
ls: cannot access '/this-does-not-exist': No such file or directory
$ echo $?
2
```
## CHANGE
Try a few other failing commands and note their codes: `grep nomatch /etc/hostname` (exit 1 —
"no matches found," a different kind of failure than "file not found").
## VERIFY
Notice: exit codes aren't just "0 or 1" — different failures often get different non-zero
codes. `0` always means success; the exact non-zero value's MEANING varies by command, but ANY
non-zero value reliably means "something went wrong," which is the part every script actually
relies on (`if command; then ... else ...`).
## Instructor Notes
**Demo goal:** Make the abstract "exit code" concept concrete by actually watching `$?` change
in real time, immediately after each command — this is far more convincing live than described
in prose.
**Approximate timing:** 8 minutes live.
**Common mistake to watch for:** running `echo $?` after ANY other command in between (even
`ls`) silently checks the WRONG command's exit code — emphasize "check it immediately."
@@ -0,0 +1,37 @@
# Knowledge Check: Linux and the Shell
---
**1. What's the difference between a shell and a terminal?**
*Answer:* The terminal is the window/program that displays text and captures keystrokes; the
shell (e.g. `bash`) is the program actually interpreting what you type as commands. The terminal
is the display, the shell is the brain.
---
**2. A command exits with code `0`. What does that tell you?**
*Answer:* It succeeded. `0` is the universal "success" convention every script and pipeline
relies on; any non-zero value means some kind of failure occurred.
---
**3. You've typed half a long path and can't remember the rest. What should you press?**
*Answer:* Tab — tab completion finishes the path (or shows the options) without you needing to
type or remember it exactly.
---
**4. Where would you look to learn every option a command supports, without leaving the
terminal?**
*Answer:* Its manual page — `man <command>`.
## Instructor Notes
**Facilitation:** Question 1 is the one students most often blur together — worth confirming
verbally before moving on.
**Approximate timing:** 5 minutes.
@@ -0,0 +1,31 @@
# Guided Exercise: Predict the Exit Code Before You Run It
This is a paper exercise — commit to an answer before checking it in the Lab.
## Your task
For each command below, predict whether its exit code will be `0` (success) or non-zero
(failure), and why:
1. `cat /etc/hostname` (a file that exists)
2. `cat /etc/does-not-exist-at-all` (a file that doesn't)
3. `grep root /etc/passwd` (a pattern that's very likely present)
4. `grep zzz-not-a-real-user /etc/passwd` (a pattern that's very likely absent)
5. `mkdir /tmp/already-exists && mkdir /tmp/already-exists` (running the same `mkdir` twice)
## A worked shape to check yourself against
1. `0` — the file exists and is readable.
2. Non-zero — the file doesn't exist, `cat` fails.
3. `0` — `grep` succeeds (exit 0) when it finds at least one match.
4. Non-zero (specifically `1`) — `grep` "fails" when it finds NO matches, even though nothing
went technically wrong. This is a common surprise: `grep`'s exit code means "did I find a
match," not "did an error occur."
5. The SECOND `mkdir` fails (non-zero) — the directory already exists. The first one succeeds.
## Instructor Notes
**Facilitation:** Item 4 is the one worth pausing on — many students assume "no error message"
means "exit code 0," and `grep`'s no-match convention is a good early counter-example.
**Approximate timing:** 8-10 minutes live discussion; 5 minutes self-paced.
@@ -0,0 +1,25 @@
# Reflection: Exit Codes Are Everywhere After This
## Reflection questions
1. Before this module, did you know commands returned a machine-readable success/failure
signal at all? What did you assume happened instead?
2. Where do you think exit codes might matter later in ROOT_ — in a script, a CI pipeline, or
a container?
## Real Ops takeaway
**Every automated system you'll ever build or operate checks exit codes, not error messages.**
A script deciding whether to continue, a CI pipeline deciding whether to deploy, and (much
later) a container orchestrator deciding whether a process is healthy, all reduce to the exact
same question this module taught you to answer by hand: did this exit `0` or not? Getting
comfortable checking `$?` now means none of those later systems will feel like magic.
This module is complete. Continue to the next module.
## Instructor Notes
**Discussion questions for a live classroom:** question 2 previews Docker/Kubernetes health
checks well before students get there — useful to name explicitly without going deep yet.
**Approximate timing:** 5-8 minutes live discussion; 5 minutes self-paced writing.
@@ -0,0 +1,55 @@
# Navigating the Filesystem
## Everything is one tree, starting at `/`
Linux has exactly one filesystem tree, rooted at `/` (pronounced "root," unrelated to your
username). Every file and directory, no matter what physical disk it's actually on, appears
somewhere under this one tree — there's no separate "C:\" and "D:\" the way Windows shows
multiple drives.
## Absolute vs. relative paths
An **absolute path** starts with `/` and always means the same location no matter where you
currently are: `/home/student/notes.txt`. A **relative path** is interpreted relative to your
CURRENT directory: `notes.txt` means something different depending on where you're standing
when you type it. `.` means "this directory," `..` means "one directory up" — `cd ..` moves up
one level, `cd ../..` moves up two.
## Key commands
- `pwd` — print your current directory (where am I?)
- `ls` — list what's in a directory (`ls -la` shows hidden files and details)
- `cd <path>` — change directory (absolute or relative)
## Hidden files
A file or directory starting with `.` (like `.bashrc` or `.config`) is "hidden" — `ls` skips it
by default, `ls -a` shows it. This isn't a security feature, just a convention to keep
configuration/metadata files out of the way of normal directory listings.
## The important top-level directories (conceptually, not memorized)
You don't need to memorize the entire Filesystem Hierarchy Standard, but recognizing these by
purpose will save you real confusion later:
- **`/etc`** — system-wide configuration files (nginx's config lives at `/etc/nginx/nginx.conf`)
- **`/var`** — variable data that changes while the system runs: logs (`/var/log`), among others
- **`/home`** — personal directories for regular users
- **`/tmp`** — temporary files, safe to assume anything here can disappear on reboot
- **`/usr`** — most installed programs and their supporting files live somewhere under here
- **`/proc`** — not real files on disk at all — a live, kernel-generated view into running
processes and system state (you'll use this again in Module 5)
## Instructor Notes
**Lesson goal:** Give students a working mental model of "one tree, absolute vs. relative
paths" plus just enough top-level-directory recognition to not be lost later — not FHS
memorization.
**Approximate timing:** 15 minutes read; 10 minutes live.
**Common misconception to watch for:** students coming from Windows sometimes expect separate
drives/roots — emphasize the single unified tree explicitly.
**Demo idea:** `cd /` then `ls`, narrating what each top-level directory is for as you see it,
rather than listing them in the abstract.
@@ -0,0 +1,43 @@
# Demonstration: The Same Word, Two Different Destinations
## OBSERVE
Starting from `/home/student`, run `cd project` — you land at `/home/student/project`.
## HYPOTHESIS
Now, from `/home/student/project`, what do you predict `cd project` will do this time?
## INSPECT
```
$ pwd
/home/student/project
$ cd project
bash: cd: project: No such file or directory
```
It fails — there's no `project` directory INSIDE `project` (unless you happened to create one).
The exact same relative path meant something completely different depending on where you stood.
## CHANGE
From `/home/student/project`, run `cd ../project` instead — this correctly goes back up one
level, then into `project` again, landing you right back where you started.
## VERIFY
`pwd` after each `cd` to confirm exactly where you ended up — this habit (verify your location
before assuming it) prevents a huge share of "why did my command run somewhere unexpected"
confusion later.
## Instructor Notes
**Demo goal:** Make "relative paths depend on where you are" concrete and slightly surprising,
rather than an abstract rule to memorize.
**Approximate timing:** 8 minutes live.
**Common mistake to watch for:** students assume a relative path always means the SAME thing
once they've used it once — reinforce that it's re-evaluated fresh every time, from wherever
they currently are.
@@ -0,0 +1,36 @@
# Knowledge Check: Navigating the Filesystem
---
**1. What does `..` mean in a path?**
*Answer:* One directory up from your current location.
---
**2. You're at `/home/student/project`. Does `cd project` do the same thing here as it did at
`/home/student`?**
*Answer:* No — a relative path is re-evaluated from your CURRENT location every time. At
`/home/student`, `project` refers to `/home/student/project`; from inside that same directory,
`project` would refer to `/home/student/project/project`, which likely doesn't exist.
---
**3. A file is "hidden" from a normal `ls`. What actually makes it hidden?**
*Answer:* Its name starts with a `.` — it's purely a naming convention `ls` respects by
default, not a real permission or security feature. `ls -a` shows it.
---
**4. Where would you expect to find a running system's log files?**
*Answer:* `/var/log` — `/var` holds data that changes while the system runs.
## Instructor Notes
**Facilitation:** Question 2 is the one worth confirming out loud — it's the core "gotcha" of
relative paths.
**Approximate timing:** 5 minutes.
@@ -0,0 +1,39 @@
# Guided Exercise: Plan a Directory Structure on Paper First
Before you build anything in the Lab, plan it — this is the same discipline you'll use later
when planning a Kubernetes namespace layout or a Git repository structure.
## Your task
You're setting up a small project called `weather-app`. On paper, sketch a directory tree
containing:
- A place for source code
- A place for documentation
- A place for tests
- A single hidden configuration file at the project's root
Write out the exact `mkdir` and file-creation commands you'd use, including whether each path
you write is absolute or relative.
## A worked shape to check yourself against
```
mkdir -p weather-app/src weather-app/docs weather-app/tests
echo "ready" > weather-app/.config
```
(Or with an absolute path: `mkdir -p /srv/weather-app/src ...`.) The `-p` flag matters here —
without it, `mkdir` refuses to create `weather-app/src` if `weather-app` doesn't already exist.
## Instructor Notes
**Facilitation:** Have students commit to their command in writing before revealing the worked
answer — the value is in noticing what `-p` actually buys them, not just copying a correct
command.
**Approximate timing:** 8-10 minutes live discussion; 5 minutes self-paced.
**Common mistake to watch for:** forgetting `mkdir -p` and getting a "No such file or
directory" error when creating a nested path in one command — a good moment to connect back to
Module 2's exit-code lesson (that error IS a non-zero exit code).
@@ -0,0 +1,27 @@
# Reflection: Structure Isn't Bureaucracy, It's Communication
## Reflection questions
1. Why might a real engineering team insist on a consistent directory structure across every
project, instead of everyone organizing files however they like?
2. Which of `/etc`, `/var`, `/home`, `/tmp`, `/usr`, `/proc` do you feel most confident you could
explain to someone else right now? Which one still feels fuzzy?
## Real Ops takeaway
**A predictable filesystem layout is what lets tools (and other engineers) reason about a
system without asking you first.** `/etc` always means configuration, `/var/log` always means
logs — this convention is exactly why a monitoring tool can be pointed at "check `/var/log`" on
any Linux machine and just work. Container images later inherit this same convention (an
nginx container still keeps its config at `/etc/nginx/nginx.conf`) — you're not learning a
one-off Linux fact, you're learning a convention that outlives this specific module.
This module is complete. Continue to the next module.
## Instructor Notes
**Discussion questions for a live classroom:** question 1 connects directly to why Kubernetes
namespaces and Git repository conventions matter later — a good moment to name that connection
without going deep yet.
**Approximate timing:** 5-8 minutes live discussion; 5 minutes self-paced writing.
@@ -0,0 +1,52 @@
# Files and Text
## Basic file operations
- `touch <file>` — create an empty file (or update its timestamp if it exists)
- `mkdir <dir>` — create a directory (`-p` creates any missing parent directories too)
- `cp <src> <dst>` — copy a file
- `mv <src> <dst>` — move OR rename a file (Linux treats renaming as "moving to a new name in
the same place")
- `rm <file>` — delete a file (`-r` for a directory, `-f` to skip confirmation — dangerous
combined: `rm -rf` deletes without asking, permanently)
## Reading text
- `cat <file>` — print the whole file at once
- `less <file>` — page through a file interactively (press `q` to quit) — the right tool for
anything too long to read in one screen
- `head <file>` / `tail <file>` — the first/last 10 lines by default (`-n 20` for a different
count); `tail -f` follows a file live as new lines are appended — exactly how you'd watch a
log file update in real time
## Searching text: `grep`
`grep PATTERN file` prints only the lines matching PATTERN. This is one of the most-used
commands in real operations work — "find the lines that matter" beats "read everything."
## Pipes and redirection: composing commands
- **A pipe (`|`)** sends one command's output as the next command's input:
`cat app.log | grep ERROR` — read the file, then filter it.
- **Redirection (`>`)** sends a command's output into a file instead of the screen (overwriting
it): `grep ERROR app.log > errors.txt`. `>>` appends instead of overwriting.
## stdout vs. stderr
A command actually has TWO separate output streams: **stdout** (normal output) and **stderr**
(error messages), even though both print to your screen by default. This is why
`command 2>/dev/null` (redirecting stream `2`, stderr, to nowhere) can suppress error noise
while still showing normal output — you'll use exactly this pattern in later Labs.
## Instructor Notes
**Lesson goal:** Build real fluency composing small commands together — this is the single
highest-leverage Linux skill for troubleshooting work later in ROOT_.
**Approximate timing:** 15 minutes read; 10 minutes live.
**Common misconception to watch for:** students often don't realize stdout/stderr are separate
streams until they see `2>/dev/null` change what's suppressed — worth demonstrating live.
**Demo idea:** run a command that produces both real output and an error in the same call
(e.g. `ls /etc /nonexistent`) and show how `2>/dev/null` changes what appears.
@@ -0,0 +1,56 @@
# Demonstration: Filtering a Log Down to What Matters
## OBSERVE
A raw log file has five lines mixing `INFO` and `ERROR`:
```
$ cat app.log
INFO: boot ok
ERROR: disk full
INFO: retry
ERROR: timeout
INFO: done
```
## HYPOTHESIS
Predict what `grep ERROR app.log` will print, before running it.
## INSPECT
```
$ grep ERROR app.log
ERROR: disk full
ERROR: timeout
```
Only the matching lines — everything else is filtered out.
## CHANGE
Now redirect that filtered result into a new file instead of just viewing it:
```
$ grep ERROR app.log > errors.txt
$ cat errors.txt
ERROR: disk full
ERROR: timeout
```
## VERIFY
Count how many lines actually matched, without eyeballing them: `wc -l < errors.txt` (or
`grep -c ERROR app.log` directly). This composition — filter, then count, then act — is the
exact shape of a huge share of real troubleshooting work.
## Instructor Notes
**Demo goal:** Show the filter → redirect → verify chain as one continuous, composable
workflow, not three unrelated commands.
**Approximate timing:** 8 minutes live.
**Common mistake to watch for:** using `>` when they meant `>>` and accidentally overwriting a
file they wanted to append to — worth calling out explicitly since it's a real, common mistake
with no undo.
@@ -0,0 +1,37 @@
# Knowledge Check: Files and Text
---
**1. What's the difference between `>` and `>>`?**
*Answer:* `>` overwrites the target file's contents entirely; `>>` appends to the end of it,
preserving what was already there.
---
**2. A command prints both normal output and an error message. What are these two streams
called?**
*Answer:* stdout (normal output) and stderr (error messages) — two separate streams that both
happen to print to your screen by default.
---
**3. You want to watch a log file update live, as new lines are appended, without re-running a
command repeatedly.**
*Answer:* `tail -f <file>` — follows the file and prints new lines as they're written.
---
**4. `cat app.log | grep ERROR`. What does the `|` actually do?**
*Answer:* It pipes `cat`'s output directly into `grep`'s input, so `grep` filters the file's
contents without needing a separate temporary file.
## Instructor Notes
**Facilitation:** Question 2 is worth a live show-of-hands check — many students haven't
consciously registered that stdout/stderr are separate until this module.
**Approximate timing:** 5 minutes.
@@ -0,0 +1,31 @@
# Guided Exercise: Which Tool Actually Answers the Question?
This is a paper exercise — commit to an answer before checking it in the Lab.
## Your task
For each situation, name the ONE command (or short pipeline) you'd reach for first:
1. You want to see just the last 20 lines of a 50,000-line log file.
2. You want to know how many lines in a file contain the word "timeout."
3. You want to watch a log file update live while a process is running.
4. You want to see a file's full contents but it's too long for one screen and you want to
scroll through it interactively.
## A worked shape to check yourself against
1. `tail -n 20 file.log`
2. `grep -c timeout file.log`
3. `tail -f file.log`
4. `less file.log`
Notice: `cat` would technically "work" for #4 too, but it dumps everything at once with no way
to scroll back — `less` is the actual right tool once a file is longer than one screen.
## Instructor Notes
**Facilitation:** Item 4 is worth discussing explicitly — "it works" isn't the same bar as
"it's the right tool for this situation," and that distinction matters more as commands compose
into real troubleshooting workflows.
**Approximate timing:** 8-10 minutes live discussion; 5 minutes self-paced.
@@ -0,0 +1,26 @@
# Reflection: Filtering Is a Reasoning Skill, Not a Syntax Trick
## Reflection questions
1. Before this module, how would you have found "the important lines" in a huge log file?
How does that compare to `grep`?
2. Which combination (pipe, redirect, or a specific command like `tail -f`) do you expect to
use the most once you're doing real troubleshooting work?
## Real Ops takeaway
**"Read everything and eyeball it" does not scale — filtering does.** Every troubleshooting
framework you'll learn later in ROOT_ (starting with ROOT 02's own version, then again more
formally in ROOT 10) depends on being able to quickly narrow a huge amount of system output down
to the handful of lines that actually matter. `grep`, pipes, and redirection are the first real
tools you have for that — not a syntax exercise, a reasoning skill.
This module is complete. Continue to the next module.
## Instructor Notes
**Discussion questions for a live classroom:** question 1 often produces good answers ("I'd
just scroll and read") that make the value of filtering obvious by contrast — let students
articulate this themselves rather than stating it for them.
**Approximate timing:** 5-8 minutes live discussion; 5 minutes self-paced writing.
@@ -0,0 +1,64 @@
# Processes
## What a process actually is
A **process** is a running instance of a program — its own code, its own private memory, and a
unique identifier the kernel tracks called a **PID** (Process ID). Run the same program twice
(open two terminals both running `bash`), and you get two separate processes with two different
PIDs, even though they're running identical code.
## Foreground vs. background
A command you run normally occupies your terminal until it finishes (**foreground**) — you
can't type anything else until it's done. Append `&` to run it in the **background** instead,
freeing your terminal immediately while the process keeps running. `sleep 300 &` starts a
300-second sleep in the background and hands your prompt right back.
## Inspecting processes: `ps` and `top`
- `ps aux` — a snapshot of every process currently running, including its PID, how much CPU/
memory it's using, and the exact command that started it.
- `top` (or `htop`, if installed) — the same idea, but live-updating and sorted by resource
usage, useful for "what's actually eating CPU/memory right now."
## Stopping a process: `kill` and signals
`kill <PID>` doesn't instantly destroy a process — it sends it a **signal**, a message the
process can (usually) choose how to respond to. The default signal, `SIGTERM` (15), politely
asks a process to shut down, giving it a chance to clean up first. `kill -9 <PID>` sends
`SIGKILL`, which the kernel enforces immediately with no chance for the process to react —
appropriate only when a process is truly stuck and ignoring `SIGTERM`. `kill -0 <PID>` is a
special case: it sends NO real signal at all, just checks whether the PID still exists —
exactly how you confirm "did that process actually stop" without side effects.
## Process hierarchy
Every process (except the very first one at boot) has a **parent** — the process that started
it. Your shell is the parent of every command you run from it. This matters later: a parent
process's own lifecycle can affect its children (this is part of why a container, which is
often just one process tree, behaves the way it does when its main process exits).
## Why this matters for containers and Kubernetes
A container is, underneath, mostly an isolated process (or small process tree) with its own
view of the filesystem and network — when people say "a container exited," they mean its main
process exited, exactly the same event you'll cause on purpose in this module's Lab. Kubernetes'
entire job, at its core, is deciding which machine a process (inside a container) runs on and
restarting it if it dies unexpectedly — none of that will feel intuitive without this module's
model of what a process actually is.
## Instructor Notes
**Lesson goal:** Build a real process mental model (PID, foreground/background, signals,
parent/child) that ROOT 04 (containers) and ROOT 06 (Kubernetes workloads) both directly build
on without re-explaining.
**Approximate timing:** 15 minutes read; 10 minutes live.
**Common misconception to watch for:** students often think `kill` always means "forcefully
destroy" — clarify that plain `kill` sends a polite, ignorable request (SIGTERM), and `-9` is
the forceful escalation, not the default.
**Discussion question:** "What do you think happens to a container if its main process exits on
its own, versus if something outside kills it?" (Don't need a complete answer yet — just plant
the question for later.)
@@ -0,0 +1,53 @@
# Demonstration: Watching a Process Actually Terminate
## OBSERVE
Start a long-running background process and note its PID:
```
$ sleep 300 &
[1] 1421
```
`1421` is the PID (yours will differ every time).
## HYPOTHESIS
Predict: will `kill -0 1421` print anything if the process is still running?
## INSPECT
```
$ kill -0 1421 && echo "still running"
still running
```
`kill -0` sends no real signal — silence (or, here, our own `echo`) means the process exists.
## CHANGE
Now actually terminate it:
```
$ kill 1421
```
## VERIFY
```
$ kill -0 1421 2>/dev/null && echo "still running" || echo "gone"
gone
```
The exact same check now reports the opposite result — real, observable proof the process is
gone, not an assumption.
## Instructor Notes
**Demo goal:** Make "checking before and after" a visible habit, not just narrated — students
should see the SAME command return a different real answer once state actually changed.
**Approximate timing:** 8 minutes live.
**Common mistake to watch for:** reaching for `kill -9` by default "to be safe" — reinforce that
plain `kill` (SIGTERM) is the correct first attempt almost always.
@@ -0,0 +1,38 @@
# Knowledge Check: Processes
---
**1. What does a PID identify?**
*Answer:* One specific running instance of a program — the kernel's unique identifier for that
process, distinct from every other running process, even other copies of the same program.
---
**2. What's the actual difference between `kill <pid>` and `kill -9 <pid>`?**
*Answer:* Plain `kill` sends SIGTERM, a polite request the process can choose to handle (clean
up, then exit). `kill -9` sends SIGKILL, which the kernel enforces immediately with no chance
for the process to react — an escalation for when SIGTERM is ignored, not the default choice.
---
**3. You want to confirm a process is still running WITHOUT affecting it in any way. What do
you use?**
*Answer:* `kill -0 <pid>` — sends no real signal, only checks whether the PID still exists.
---
**4. Why might understanding processes matter for understanding containers later?**
*Answer:* A container is, underneath, mostly an isolated process (or small process tree) — a
container "exiting" is really its main process exiting, the exact same event this module's Lab
causes on purpose.
## Instructor Notes
**Facilitation:** Question 2 is worth confirming explicitly — "kill -9 by default" is a common,
worth-correcting habit.
**Approximate timing:** 5 minutes.
@@ -0,0 +1,28 @@
# Guided Exercise: Choosing the Right Signal
This is a paper exercise — commit to an answer before checking it in the Lab.
## Your task
For each situation, decide: plain `kill` (SIGTERM) first, or is `kill -9` (SIGKILL) actually
justified?
1. A normal web server process you want to shut down for a routine restart.
2. A process that received SIGTERM two minutes ago and is still running, showing no signs of
responding.
3. A script you just started by mistake, that hasn't done anything harmful yet.
## A worked shape to check yourself against
1. Plain `kill` — give it the chance to shut down cleanly (close connections, flush data, etc).
2. `kill -9` is now justified — you already gave it the polite chance, and it didn't respond.
3. Plain `kill` first, still — "just started" doesn't change the default; SIGKILL is a last
resort, not a shortcut for convenience.
## Instructor Notes
**Facilitation:** Item 3 sometimes trips students who reach for `-9` out of impatience — worth
reinforcing that SIGTERM-first is the discipline REGARDLESS of urgency, since a process that
ignores SIGTERM is the actual signal that escalation is warranted.
**Approximate timing:** 8-10 minutes live discussion; 5 minutes self-paced.
@@ -0,0 +1,27 @@
# Reflection: You Just Stopped a Real Process, Not a Simulation
## Reflection questions
1. Before checking `kill -0` after your `kill`, how confident were you the process was
actually gone? What would you have done if it hadn't stopped?
2. Why do you think SIGTERM-before-SIGKILL matters more for a real production service than for
a throwaway script in a Lab?
## Real Ops takeaway
**"I ran the kill command" is not the same as "I confirmed the process is gone."** This module's
Lab required you to verify state, not just take an action and assume it worked — that
verification step is exactly what separates a confident operator from someone hoping for the
best. You'll repeat this exact pattern (act, then verify) constantly for the rest of ROOT_,
starting as soon as ROOT 04's containers and continuing all the way through Kubernetes
troubleshooting.
This module is complete. Continue to the next module.
## Instructor Notes
**Discussion questions for a live classroom:** question 2 is a good bridge to ROOT 06's later
discussion of graceful shutdown/`terminationGracePeriodSeconds` — plant the idea now without
naming Kubernetes specifics yet.
**Approximate timing:** 5-8 minutes live discussion; 5 minutes self-paced writing.
@@ -0,0 +1,59 @@
# Users and Permissions Foundations
## Users, groups, and ownership
Every file on Linux has an **owner** (a user) and a **group**. `ls -la` shows both: the third
column is the owning user, the fourth is the owning group. Multiple users can belong to the same
group, which is how permissions get shared across a team without giving every individual user
direct ownership.
## Read, write, execute — for three different audiences
Every file has permissions for THREE separate audiences: the owner, the group, and everyone
else ("other"). Each audience gets some combination of:
- **r**ead — can view the file's contents (or list a directory's contents)
- **w**rite — can modify the file (or create/delete files inside a directory)
- e**x**ecute — can run the file as a program (or enter/traverse into a directory)
`ls -la` shows this as a 10-character string like `-rw-r--r--`: the first character is the file
type, then three groups of `rwx` for owner/group/other.
## Reading and setting permissions as octal numbers
Each `rwx` triplet can be written as one digit: read=4, write=2, execute=1, summed together.
`rw-` = 4+2 = `6`. `r--` = `4`. So `644` means: owner can read+write (6), group can read-only
(4), everyone else can read-only (4) — a very common, sensible default for a regular file.
`chmod 644 <file>` sets exactly that.
## `sudo`: temporary elevated privilege
Rather than logging in AS the root user (with permanent full access to everything), `sudo
<command>` runs just that one command with elevated privileges, after checking you're allowed
to. This is deliberate: the less time you spend with full privileges, the smaller the window for
a mistake (or a compromised session) to do serious damage — the exact same reasoning behind
"least privilege," which this module introduces by name for the first time.
## Least privilege: the principle, not just the command
**Least privilege** means: grant exactly the access something needs to do its job, and nothing
more. A user who only needs to read a file shouldn't also be able to write it. A process that
only needs to read one directory shouldn't have write access to the whole filesystem. This
principle isn't specific to Linux permissions — you'll meet it again, in a much more elaborate
form, in ROOT 09's Kubernetes RBAC and SecurityContext content. This module is where it starts.
## Instructor Notes
**Lesson goal:** Build real fluency with rwx/octal permissions AND plant the least-privilege
PRINCIPLE explicitly — not just command syntax, since ROOT 09 depends on students already
having this vocabulary.
**Approximate timing:** 15 minutes read; 10 minutes live.
**Common misconception to watch for:** students often memorize "644 = read/write" without being
able to derive it from 4+2+1 — walk through the arithmetic explicitly rather than treating the
numbers as magic.
**Discussion question:** "Why not just make everything readable and writable by everyone, to
avoid permission headaches?" (Good setup for the least-privilege principle — let students
generate the security argument themselves before you confirm it.)
@@ -0,0 +1,55 @@
# Demonstration: A Real Permission Denial, and the Real Fix
## OBSERVE
A file is owned by root with permissions `600` (owner read/write only):
```
$ stat -c '%a %U' data.txt
600 root
```
## HYPOTHESIS
Predict: can a different, non-root user read this file?
## INSPECT
```
$ su - labuser -c "cat data.txt"
cat: data.txt: Permission denied
```
Confirmed — `600` grants nothing to group or other, so `labuser` (neither the owner nor root)
is correctly denied.
## CHANGE
Grant read access to everyone, while keeping write access limited to the owner:
```
$ chmod 644 data.txt
$ stat -c '%a' data.txt
644
```
## VERIFY
```
$ su - labuser -c "cat data.txt"
sensitive
```
The exact same command that failed before now succeeds — the permission bits, not the file's
actual content, were the only thing that changed.
## Instructor Notes
**Demo goal:** Show a permission denial as a real, reproducible fact (not just a described
error message) and the exact minimal fix, not an overcorrection like `chmod 777`.
**Approximate timing:** 8 minutes live.
**Common mistake to watch for:** students reaching for `chmod 777` "to make the error go away"
— explicitly name why that's wrong (it also grants WRITE to everyone, not just the read access
actually needed) and connect it back to least privilege.
@@ -0,0 +1,38 @@
# Knowledge Check: Users and Permissions Foundations
---
**1. A colleague says "permission denied" and fixes it with `chmod 777`. What's wrong with
that fix?**
*Answer:* `777` grants read, write, AND execute to the owner, group, AND everyone else — almost
always far more access than the actual problem required. It "fixes" the symptom while violating
least privilege, usually granting write access nobody asked for.
---
**2. What does the permission `644` actually mean, broken down?**
*Answer:* Owner: read+write (4+2=6). Group: read-only (4). Other: read-only (4). Nobody but the
owner can modify the file; everyone can read it.
---
**3. Why does `sudo <command>` exist instead of just logging in as root permanently?**
*Answer:* It grants elevated privilege only for the one command that needs it, minimizing the
time spent with full access — reducing the chance a mistake (or a compromised session) causes
serious, wide damage.
---
**4. State the least-privilege principle in your own words.**
*Answer:* Grant exactly the access something needs to do its job, and nothing more.
## Instructor Notes
**Facilitation:** Question 1 is worth discussing as a group — most students have seen `chmod
777` somewhere online as "the fix," and unlearning that habit early matters.
**Approximate timing:** 5 minutes.
@@ -0,0 +1,32 @@
# Guided Exercise: Pick the Minimal Permission, Not the Easy One
This is a paper exercise — commit to an answer before checking it in the Lab.
## Your task
For each situation, write the exact `chmod` octal value that grants the MINIMUM permission
actually needed:
1. A private SSH key file — only the owner should ever read or write it, nobody else, no
execute for anyone.
2. A shared script every user on the system should be able to run, but only the owner should be
able to edit.
3. A log file every user should be able to read (for troubleshooting), but only a logging
service (the owner) should write to it.
## A worked shape to check yourself against
1. `600` — owner read+write (6), nothing for group or other (0, 0). This is the real, standard
permission for private SSH keys — SSH itself refuses to use a key file that's more open than
this.
2. `755` — owner read+write+execute (7), group and other read+execute (5, 5) so they can run it
but not modify it.
3. `644` — owner read+write (6), group and other read-only (4, 4).
## Instructor Notes
**Facilitation:** Item 1 is a good moment to mention that this isn't a hypothetical rule — SSH
genuinely refuses to use an overly-permissive private key file, a real-world consequence of
exactly this principle.
**Approximate timing:** 8-10 minutes live discussion; 5 minutes self-paced.
@@ -0,0 +1,33 @@
# Reflection: Least Privilege Is a Habit, Not a One-Time Fix
## Reflection questions
1. Before this module, what would you have done the first time you hit a "permission denied"
error? How does that compare to what you'd do now?
2. Name one place outside of file permissions where you'd expect "grant only what's needed" to
matter (a user account, an API key, a service's network access — anything).
## Real Ops takeaway
**Least privilege is the single idea from this entire module that reappears the most across
the rest of ROOT_.** File permissions are just the first, most concrete place you'll apply it —
ROOT 09 applies the exact same principle to Kubernetes RBAC (what can a user or workload
actually DO) and SecurityContext (what can a container actually access). Getting comfortable
asking "what's the minimum this actually needs" now, on a file you can see and touch, makes that
same question much easier to ask later about something more abstract, like a ServiceAccount's
permissions.
This module is complete. This is the end of ROOT 01 — continue to ROOT 02: Linux
Administration & Networking.
## Instructor Notes
**Discussion questions for a live classroom:** question 2 works well as a round-robin — the
variety of answers (API keys, database credentials, network firewall rules) is itself useful
evidence of how far this one principle travels.
**Approximate timing:** 5-8 minutes live discussion; 5 minutes self-paced writing.
**Milestone note:** this is the last module of ROOT 01 — consider a short recap of the whole
course's arc (hardware → shell → filesystem → files/text → processes → permissions) before
moving on, so students see it as one coherent foundation rather than six separate topics.
@@ -0,0 +1,108 @@
# What Kubernetes Solves & the Desired-State Model
## Before Kubernetes: the problem
Imagine you're running a web application on three servers by hand. One server crashes at 2am —
someone has to notice, log in, and restart the application. Traffic grows, so you need a fourth
server — someone has to provision it, install dependencies, and configure it to match the other
three. You ship a new version — someone has to update all four servers, one at a time, without
taking the whole application down.
None of this is hard in isolation. It becomes unmanageable at scale, and it depends entirely on
a human noticing and reacting fast enough. Kubernetes exists to make a computer do that
noticing and reacting instead — continuously, automatically, and the same way every time.
## The core idea: desired state
You don't tell Kubernetes *how* to run your application step by step. You tell it *what you
want to be true* — "3 replicas of this container should be running" — and Kubernetes
continuously works to make that true, correcting drift whenever reality doesn't match.
This is the single most important mental shift for someone new to Kubernetes: you're not
writing a script that runs once. You're declaring a **desired state**, and a background process
never stops comparing the real world against it.
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
spec:
replicas: 3 # <- the desired state: "3 of these, always"
...
```
If a Pod crashes, Kubernetes doesn't "restart the crashed one" as a special case — it just
notices reality (2 running) no longer matches desired state (3 wanted) and creates a
replacement. Same mechanism, no special-casing.
## The cluster mental model
A Kubernetes cluster has two conceptually different kinds of machines:
- **The control plane** — the "brain." It stores the desired state (in `etcd`), watches actual
state, and makes decisions (the API server, scheduler, and controllers all live here). You
talk to the control plane through `kubectl`; you never SSH into it to manage workloads.
- **Worker nodes** — where your containers actually run. Each node runs a `kubelet` (an agent
that starts/stops containers as the control plane instructs) and a container runtime.
You describe desired state to the control plane. The control plane decides which nodes should
run what. The nodes' kubelets make it actually happen. You never manually place a Pod on a
specific node in normal operation — that placement decision belongs to the scheduler.
## Declarative, not imperative
"Imperative" means giving step-by-step instructions ("start a container, then another, then
configure networking between them"). "Declarative" means describing an end state and letting
the system figure out the steps. Kubernetes' API objects (Pods, Deployments, Services, and
everything else you'll learn in ROOT_) are declarative — you write YAML describing what should
exist, `kubectl apply` submits it, and the control plane does the rest.
This is also why Kubernetes YAML looks the way it does: it's not a script, it's a specification.
## Pods, Namespaces, and labels — the vocabulary you need first
- **Pod** — the smallest deployable unit. Usually one container, sometimes a small group of
tightly-coupled containers that must run together on the same node and share networking.
You'll rarely create bare Pods directly in real use (a Deployment manages them for you, see
ROOT 06) but understanding the Pod itself is foundational to everything built on top of it.
- **Namespace** — a way to partition one cluster into logically separate areas (e.g. `dev`,
`staging`, teams, or projects), so names don't collide and access/quotas can be scoped.
- **Labels** — arbitrary key/value tags attached to objects (`app: web`, `env: prod`). Labels
are how almost everything in Kubernetes finds related objects — a Deployment finds "its" Pods
by label selector, not by name pattern matching.
## kubectl: your one tool for all of this
`kubectl` is the command-line client that talks to the control plane's API server. Nearly
everything you'll do in ROOT_ Labs goes through it: `kubectl get` (list/inspect),
`kubectl describe` (detailed status + recent events — your primary troubleshooting tool),
`kubectl apply -f` (submit a desired-state YAML file), `kubectl logs` (see what a container
printed). You'll build real fluency with these across this module and the ones that follow.
## Instructor Notes
**Approximate timing:** 15 minutes read, 10-12 minutes for a live walkthrough with discussion.
**Demo plan:** Before the demonstration activity, it's worth drawing the control-plane/worker
split on a whiteboard (or sharing a simple diagram) and tracing one request: `kubectl apply` ->
API server -> etcd -> scheduler -> kubelet -> container runtime. Students retain the mental
model much better when they've seen the path, not just read about it.
**Likely misconceptions to watch for:**
- Students often assume `kubectl` talks directly to nodes/containers. It doesn't — it only ever
talks to the API server. Worth stating explicitly.
- "Declarative" is sometimes read as "automatic" in a vague sense. Be concrete: it means "you
describe the end state; a controller loop makes it true," not "Kubernetes guesses what you
want."
**Discussion question:** "If desired state says 3 replicas and you manually run `kubectl delete
pod` on one of them, what happens next, and why?" (Answer: a new one is created almost
immediately — deleting a Pod doesn't change the Deployment's desired replica count, so the
control loop restores it.)
**Optional extension:** Ask students to predict what would happen if they edited a running Pod's
container image directly with `kubectl edit pod` instead of updating the Deployment that manages
it (answer: it works until that specific Pod is replaced for any reason, at which point the new
Pod reverts to whatever the Deployment's template says — a good early illustration of "the
Deployment is the source of truth, not the running Pod").
@@ -0,0 +1,76 @@
# Demonstration: Desired State Reconciliation
Watch what actually happens when reality drifts from desired state — this is the mechanism the
whole lesson describes, made concrete.
## Setup
A Namespace called `demo` exists with a Deployment `web` declaring `replicas: 2`. Right now,
reality matches desired state:
```
$ kubectl get pods -n demo
NAME READY STATUS RESTARTS AGE
web-7d8f9c6b5d-abcde 1/1 Running 0 3m
web-7d8f9c6b5d-fghij 1/1 Running 0 3m
```
## Someone (or something) deletes one Pod
```
$ kubectl delete pod web-7d8f9c6b5d-abcde -n demo
pod "web-7d8f9c6b5d-abcde" deleted
```
## What happens in the next second or two, without anyone doing anything else
```
$ kubectl get pods -n demo
NAME READY STATUS RESTARTS AGE
web-7d8f9c6b5d-fghij 1/1 Running 0 3m
web-7d8f9c6b5d-klmno 0/1 ContainerCreating 0 1s
```
A brand new Pod (`...-klmno`, a new random suffix — this is NOT the deleted Pod coming back)
appeared on its own. Nobody ran `kubectl create` again. The Deployment's controller noticed
actual state (1 running Pod matching its label selector) no longer matched desired state
(`replicas: 2`), and created exactly one replacement to close the gap — not two, not zero.
A few seconds later:
```
$ kubectl get pods -n demo
NAME READY STATUS RESTARTS AGE
web-7d8f9c6b5d-fghij 1/1 Running 0 3m
web-7d8f9c6b5d-klmno 1/1 Running 0 8s
```
Back to 2/2 — desired state restored, entirely without human intervention.
## Why this matters beyond the demo
This exact mechanism is what makes a node crash, a Pod eviction, or a container being
OOM-killed something Kubernetes recovers from automatically instead of something that pages a
human at 2am. The controller loop doesn't know or care WHY a Pod disappeared — crash, manual
delete, node failure — it only ever asks "does reality match desired state right now?" and acts
if not. Understanding this one loop explains a huge amount of Kubernetes' behavior that would
otherwise look like unrelated special cases.
## Instructor Notes
**Demo script:** Run this live rather than only showing the captured output above — have
students watch `kubectl get pods -n demo --watch` in one terminal while you delete a Pod from
another. Watching the new Pod name/age appear in real time lands the "this is a NEW Pod, not a
restart" point far better than static text.
**Approximate timing:** 8-10 minutes live; 5 minutes as a self-paced read.
**Common misconception to correct:** students frequently describe this as "Kubernetes restarted
the Pod." Correct this explicitly — a deleted Pod is gone permanently; a *new* Pod object was
created to satisfy the replica count. This distinction matters later when debugging (the new
Pod has zero history from the old one — no old logs, no old restart count).
**Optional extension:** ask what would happen if you deleted the Deployment itself instead of
one Pod (answer: all of its Pods are deleted and NOT replaced, because the thing declaring
desired state is now gone — a good way to reinforce that the Deployment, not any individual
Pod, is the actual source of truth).
@@ -0,0 +1,48 @@
# Knowledge Check: The Kubernetes Mental Model
Answer each question yourself before reading the answer that follows it.
---
**1. You run `kubectl delete pod` on one Pod belonging to a Deployment with `replicas: 3`. A
few seconds later, `kubectl get pods` shows 3 Pods again, but one has a different name than any
of the original three. What actually happened?**
*Answer:* The deleted Pod is gone permanently. The Deployment's controller noticed actual state
(2 Pods) no longer matched desired state (3 Pods) and created ONE new Pod to close the gap. It
is not the old Pod restarting — it's a brand new object.
---
**2. A Deployment has `replicas: 3`, and there are 5 nodes in the cluster. Which of these
decides WHICH node each of the 3 Pods actually lands on?**
*Answer:* The scheduler (part of the control plane), not the person who wrote the Deployment
YAML and not the nodes themselves. You declare desired state; the scheduler makes the placement
decision based on available resources and any placement rules configured.
---
**3. True or false: `kubectl` communicates directly with the kubelet on whichever node your
Pod happens to be running on.**
*Answer:* False. `kubectl` only ever talks to the API server (part of the control plane). The
API server is the single entry point for every read and write — it never bypasses the control
plane to reach a node directly for a normal `get`/`apply`/`describe` call.
---
**4. Why is Kubernetes YAML described as "declarative" rather than a script?**
*Answer:* A script describes a sequence of steps to execute once. Kubernetes YAML describes an
END STATE you want to exist ("3 replicas of this container should be running") — a controller
loop then continuously works to keep reality matching that description, correcting drift
whenever it appears, indefinitely, not just once at apply time.
## Instructor Notes
**Facilitation:** Ask these aloud in a live classroom before revealing answers — question 1 in
particular tends to surface the "it restarted" misconception the demonstration activity should
have already addressed; if it hasn't landed, this is the moment to circle back.
**Approximate timing:** 5 minutes.
@@ -0,0 +1,57 @@
# Guided Exercise: Planning a Namespace Layout
This is a paper-and-thinking exercise — work through it before the Lab, not during it.
## The scenario
You're setting up a small Kubernetes cluster for a class project with three teams (`frontend`,
`backend`, `data`), each running their own set of Pods, and you also need a shared area for
tools everyone uses (a shared logging dashboard, for example).
## Your task
Sketch out, on paper or in a scratch file:
1. What Namespaces would you create, and why? (Hint: think about who needs to see what, and
whether any team's mistake — e.g. deleting the wrong Pod by accident — should be able to
affect another team.)
2. If `frontend` team members should be able to manage their own Pods but NOT touch `backend`'s
or `data`'s, what does that suggest about how Namespaces relate to access control? (You'll
learn the actual mechanism, RBAC, in ROOT 09 — for now just reason about why Namespace
boundaries make that kind of restriction possible at all.)
3. Would you put the shared logging dashboard in one of the three team Namespaces, or somewhere
else? Justify your answer.
4. If two teams both need to run something they each call `web`, what would happen if they were
in the SAME Namespace? What changes if they're in separate Namespaces?
## A reasonable answer to check yourself against
1. Three Namespaces (`frontend`, `backend`, `data`) — one per team, so each team's objects are
isolated from the others and a mistake in one Namespace (deleting the wrong Pod, applying a
bad manifest) can't accidentally affect a different team's workloads.
2. Namespace boundaries are what RBAC rules can be scoped TO — "this person can manage Pods in
the `frontend` Namespace" is a real, common permission model. Without separate Namespaces,
you'd have to distinguish access by resource name pattern instead of by boundary, which is
far more error-prone.
3. A separate Namespace (e.g. `shared-tools` or `platform`) — it isn't owned by any one team,
and mixing it into a team's Namespace would make it unclear who's responsible for it or
accidentally couple its lifecycle to that team's.
4. In the SAME Namespace, object names must be unique — a second `web` Deployment would either
fail to create or overwrite the first, depending on how it's applied. In SEPARATE
Namespaces, both `frontend/web` and `backend/web` can coexist without conflict — the
Namespace is part of what makes a name unique, not just the name itself.
If your answer differed meaningfully from this, re-read the lesson's Namespace section before
continuing to the Lab.
## Instructor Notes
**Facilitation:** Good as a 2-3 person small-group discussion in a live classroom before
revealing the reasonable answer — question 4 in particular often produces a genuine "oh" moment
once students realize name uniqueness is Namespace-scoped, not cluster-wide.
**Approximate timing:** 10 minutes live with discussion; 10 minutes self-paced.
**Optional extension:** ask what would need to be true for the `data` team to be ALLOWED to read
(but not modify) something in `shared-tools` — a good bridge to RBAC's read vs. write verbs,
covered properly in ROOT 09.
@@ -0,0 +1,44 @@
# Launch Lab: Working with Namespaces
Time to apply the mental model for real — you'll create an actual isolated Namespace and deploy
into it, in your own disposable Lab environment.
## What you're doing
You'll create a Namespace (`team-alpha`) and deploy an application into it with a specific
replica count — directly applying the "isolated area within one cluster" idea from the guided
exercise, and the desired-state model from the lesson (once you declare `replicas: 2`, that
becomes the standing desired state for that Deployment, enforced continuously).
## Before you start
Think back to the demonstration: once your Deployment exists with its desired replica count,
Kubernetes will keep it there on its own. You don't need to "check back" and manually fix
anything — if you want to prove this to yourself, try deleting one of your Pods after you've
finished the task and watch what happens.
## Your task in this Lab
Create the Namespace, then deploy the application inside it with the exact replica count
specified. Use `kubectl get namespace` and `kubectl get deployment -n <namespace>` to confirm
your work — the same verification pattern you'll use constantly throughout ROOT_.
This Lab has its own in-context hints if you get stuck. Try working from the lesson and guided
exercise first — the commands you need are exactly the ones described there.
When you're done, return to this module to close out with a reflection on why this isolation
matters beyond just "keeping things organized."
## Instructor Notes
**Classroom flow:** This is most students' first real `kubectl apply`/`create` in ROOT_ beyond
whatever pilot exposure they've had — a good moment to circulate and check that basic
`kubectl` command syntax isn't the blocker (as opposed to the Namespace concept itself, which
the guided exercise should have already addressed).
**Approximate timing:** 10-15 minutes.
**Common mistake to watch for:** creating the Deployment in the DEFAULT namespace instead of
`team-alpha` (forgetting `-n team-alpha` or the namespace field in the manifest) — this is a
very common first mistake and a good real-world illustration of why Namespace isolation only
works if you're actually specifying it every time.
@@ -0,0 +1,44 @@
# Reflection: Why Desired State Matters in Production
You just created an isolated Namespace and a Deployment with a declared replica count. Take a
few minutes to connect this to why it matters beyond passing this Lab's validation.
## Reflection questions
1. If your `team-alpha` Namespace were a real team's production environment, what's one mistake
in a DIFFERENT namespace that Namespace isolation would have protected you from?
2. You declared `replicas: 2` rather than manually creating two Pods. What would you have had
to do manually, over and over, if Kubernetes only supported creating individual Pods with no
concept of desired state?
3. Think of one non-Kubernetes system you've used (a thermostat, a spell-checker, an
auto-save feature) that also works by continuously comparing "what is" to "what should be."
What made you think of it?
## Real Ops takeaway
**Namespaces are your first real production safety boundary, and they're free.** A
misconfigured `kubectl delete` scoped to the wrong Namespace is a common, entirely preventable
class of incident — teams that consistently namespace their environments (dev/staging/prod, or
per-team) limit the blast radius of almost any operator mistake to just that Namespace. This
costs nothing to set up and prevents a real, common category of outage.
**Desired state is what makes self-healing possible at all.** Every "the platform recovered on
its own" story you'll hear about Kubernetes in production traces back to exactly the mechanism
you just watched in the demonstration: a controller comparing actual state to desired state and
correcting the gap. Understanding this loop is what lets you predict Kubernetes' behavior
instead of being surprised by it.
This module is complete. Continue to the next module.
## Instructor Notes
**Discussion questions for a live classroom:** use all three reflection questions as discussion
prompts. Question 3 in particular tends to produce good analogies that make the abstract
"controller loop" concept concrete and memorable.
**Approximate timing:** 8-10 minutes live discussion; 5 minutes self-paced writing.
**Optional extension:** ask students to name one place in their own life (not computing) where
"declare the goal and let a system enforce it" would work better than "manually do the thing
every time" — a genuinely reflective prompt that reinforces the mental model beyond Kubernetes
specifically.
@@ -0,0 +1,49 @@
# Building kubectl Fluency: Why Twenty Small Reps Beat One Big Lecture
## What this module is, and why it looks different from the last one
ROOT 05's mental model module (m00) taught you the CONCEPT of desired state and gave you your
first `kubectl` commands. This module does something deliberately different: it's a breadth
drill across the everyday `kubectl` vocabulary you'll type dozens of times a day as a working
Kubernetes engineer — labels, JSONPath, dry-run manifests, patch, field selectors, sort-by,
diff, init containers, static Pods, and more.
There's no single narrative thread connecting these twenty tasks, and that's intentional. Real
`kubectl` fluency isn't one big insight — it's having enough of these small, specific commands
already in muscle memory that you're not stopping to look up syntax while you're mid-incident.
Each task below explains the specific concept it covers in its own description; your job is to
work through them, not to memorize a lecture first.
## How to approach this module
- **Don't try to do them in one sitting if you don't need to.** These are independent — skip
around, come back to ones you found awkward, and don't feel obligated to treat this as one
continuous session.
- **Read each task's own explanation before reaching for the hint.** Every one of these
twenty tasks includes real context in its description — that context IS the teaching content
for that specific command. The hint is there for when you're stuck, not as the first thing to
read.
- **Notice which commands you reach for confidently versus which ones you have to think about.**
That gap is exactly what the reflection at the end of this module asks you to name.
- **This breadth matters more than it looks like it does.** `kubectl explain`, `--dry-run`,
JSONPath output, and `kubectl patch` in particular are commands experienced engineers use
constantly and newer engineers often don't know exist at all — this module exists specifically
to close that gap early.
## Instructor Notes
**Lesson goal:** Frame this module correctly before students hit it — it's a breadth-practice
bank, not a single-concept lesson, and the individual task descriptions carry the teaching
content. Without this framing, students sometimes expect a lecture and are confused when they
land directly on twenty short tasks.
**Approximate timing:** 5 minutes to introduce; the module itself runs long (20 tasks) and is
meant to be worked through at each student's own pace, live or between sessions.
**Facilitation for a live cohort:** Consider live-demoing 2-3 of the less obvious ones
(`kubectl explain`, JSONPath output, `kubectl patch`) rather than all twenty — the goal is to
unblock students on the commands they're least likely to have seen before, not to walk through
every task.
**Expected takeaway:** Kubernetes fluency is built from a large vocabulary of small, reliable
commands, not just conceptual understanding — this module is where that vocabulary gets built.
@@ -0,0 +1,40 @@
# Reflection: Naming Your Gaps While They're Still Cheap to Close
You just worked through twenty independent `kubectl` tasks. Before moving on, take a few
minutes to consolidate what actually stuck versus what you'd need to look up again.
## Reflection questions
1. Which THREE of these twenty commands do you feel genuinely confident using again right now,
without a hint? Name them specifically.
2. Which ONE command surprised you the most — either because you didn't know it existed, or
because it did something different than you expected?
3. `kubectl explain`, `kubectl patch`, and JSONPath output are commands a lot of engineers never
learn early and end up rediscovering the hard way, mid-incident, years later. Now that you've
used all three, where do you think you'd actually reach for one of them in a real debugging
situation?
## Real Ops takeaway
**Fluency gaps are cheap to close now and expensive to close later.** Right now, not knowing
`kubectl patch` costs you a few minutes rereading this module. In a live incident six months
from now, not knowing it costs you minutes you don't have while a Service is down. Naming your
gaps here, while the stakes are zero, is how you make sure you close them before they matter.
**Breadth practice is not busywork.** It's easy to dismiss twenty small tasks as less valuable
than one big project — but the twenty small tasks are what make the big project fast to execute
later. This is the same discipline you'll rely on for the rest of ROOT_: build the vocabulary
now, apply it under pressure later.
This module is complete. Continue to the next module.
## Instructor Notes
**Discussion questions for a live classroom:** question 3 works well as a round-robin — go
person to person and have each name a different command, so the group collectively surfaces
most of the twenty rather than converging on the same one or two obvious answers.
**Approximate timing:** 8-10 minutes live discussion; 5 minutes self-paced writing.
**Expected takeaway:** Students should leave this module able to name their own remaining gaps
specifically, not just feel generally "done" with kubectl basics.
@@ -0,0 +1,81 @@
# Why Deployments Exist & How a Rolling Update Progresses
## The problem a bare Pod doesn't solve
A Pod, on its own, has no self-healing and no concept of "there should be more than one of
these." If a Pod crashes, nothing recreates it. If you need 3 copies for capacity, you'd have to
create 3 separately-named Pods by hand and keep track of all of them yourself. If you need to
update the container image, you'd have to delete and recreate every one of them manually,
almost certainly causing a visible outage while you do it.
**A Deployment solves all three problems at once**: it declares how many replicas of a Pod
template should exist, keeps that count true continuously (the exact desired-state mechanism
from ROOT 05), and knows how to change the Pod template over time — a version upgrade — without
just deleting everything and starting over.
## ReplicaSets: the layer underneath, briefly
You'll rarely touch a ReplicaSet directly, but understanding it clarifies what a Deployment
actually does. A Deployment doesn't manage Pods directly — it manages a ReplicaSet, which is
what actually keeps N Pods matching a label selector running. When you update a Deployment's
Pod template (like changing the image), the Deployment creates a NEW ReplicaSet with the new
template and gradually shifts replicas from the old ReplicaSet to the new one. The old
ReplicaSet isn't deleted — it's scaled to zero and kept around, which is exactly what makes
rollback possible (more on that below).
## How a rolling update actually progresses
By default, a Deployment updates using the `RollingUpdate` strategy: it doesn't take all old
Pods down and bring all new ones up at once (that would cause an outage). Instead, it
incrementally creates new-version Pods and removes old-version ones, always keeping enough
capacity available:
```
Before: [v1] [v1] [v1]
Step 1: [v1] [v1] [v1] [v2] <- one new Pod created
Step 2: [v1] [v1] [v2] <- one old Pod removed once new one is Ready
Step 3: [v1] [v1] [v2] [v2]
Step 4: [v1] [v2] [v2]
...
After: [v2] [v2] [v2]
```
Crucially, a new Pod only counts as "up" for the purposes of proceeding once it passes its
readiness probe (see ROOT 10's gold-standard module) — this is exactly why a misconfigured
readiness probe can make a rollout hang indefinitely instead of failing fast: Kubernetes is
correctly waiting for a "ready" signal that will never come.
## Rollback: what it actually restores
`kubectl rollout undo` doesn't magically know how to "undo" arbitrary changes — it works because
of the old ReplicaSet still sitting around at zero replicas, from the mechanism described above.
Rolling back means: scale the OLD ReplicaSet back up and the current (broken) one back down,
using the exact same rolling mechanism as a forward update. This is why rollback is fast and
safe — it's not a special operation, it's the same well-tested rolling-update machinery running
in reverse, against a Pod template that's already known to have worked before.
## Instructor Notes
**Approximate timing:** 15 minutes read, 10-12 minutes live with the ReplicaSet diagram.
**Demo plan:** Draw (or show) the ReplicaSet-swap diagram above on a whiteboard before the
demonstration activity — students who see the "old ReplicaSet scaled to zero, not deleted"
detail up front connect rollback to rollout much faster.
**Likely misconceptions to watch for:**
- Students often think rollback "restores a backup" or reverts a Git-style diff. Correct this:
it's scaling an existing (still-present) ReplicaSet back up, nothing is being restored from
storage.
- "Rolling update" is sometimes assumed to mean "one Pod at a time, always." The actual pace is
controlled by `maxSurge`/`maxUnavailable` (not covered in depth here, but worth naming so
students don't over-generalize from the simplified diagram).
**Discussion question:** "Why does a rolling update need readiness probes to work correctly,
specifically?" (Answer: without a readiness signal, Kubernetes has no way to know when a new
Pod is actually safe to receive traffic before removing an old one — it would either guess via a
fixed delay, unreliable, or not wait at all, unsafe.)
**Optional extension:** ask what would happen to an in-progress rollout if the new image simply
doesn't exist (a typo in the tag) — a good bridge to real troubleshooting: the rollout stalls
indefinitely with `ImagePullBackOff` on the new Pods, never completing and never rolling back on
its own.
@@ -0,0 +1,77 @@
# Demonstration: Watching a Rollout, Good and Bad
## A healthy rollout
A Deployment `app-v1` running `nginx:1.24` at 3 replicas gets updated to `nginx:1.25`:
```
$ kubectl set image deployment/app-v1 nginx=nginx:1.25
deployment.apps/app-v1 image updated
$ kubectl rollout status deployment/app-v1
Waiting for deployment "app-v1" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "app-v1" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "app-v1" rollout to finish: 1 old replicas are pending termination...
deployment "app-v1" successfully rolled out
```
`kubectl rollout status` blocks and reports exactly the incremental progress described in the
lesson — this is your primary tool for watching (or scripting a wait on) a rollout, rather than
repeatedly polling `kubectl get pods` yourself.
## Checking revision history
```
$ kubectl rollout history deployment/app-v1
REVISION CHANGE-CAUSE
1 <none>
2 <none>
```
Revision 1 is the original ReplicaSet (now scaled to zero, still present); revision 2 is the
current one. This history is exactly what `kubectl rollout undo` uses.
## A stalled rollout
Now imagine the same update, but to a typo'd tag, `nginx:1.25-typo`, which doesn't exist:
```
$ kubectl rollout status deployment/app-v1
Waiting for deployment "app-v1" rollout to finish: 1 out of 3 new replicas have been updated...
```
...and it just sits there. `kubectl get pods` shows why:
```
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
app-v1-7d8f9c6b5d-abcde 1/1 Running 0 5m
app-v1-7d8f9c6b5d-fghij 1/1 Running 0 5m
app-v1-9c5b7d8f6b-klmno 0/1 ImagePullBackOff 0 30s
```
The new Pod can never become Ready because its image can't even be pulled — so the rollout
correctly refuses to proceed further (removing more old Pods would reduce capacity for no
benefit, since the replacement will never come up). Nothing is "broken" in the platform; the
platform is behaving exactly correctly given bad input.
**The fix here is `kubectl rollout undo`, not waiting longer** — this exact scenario is what
`deployment-rollback`'s Lab, later in ROOT_, gives you real hands-on practice with.
## Instructor Notes
**Demo script:** If time allows, run the healthy rollout live against a throwaway Deployment,
then intentionally break the second one with a bad tag so students see `ImagePullBackOff` occur
in real time rather than only in static text.
**Approximate timing:** 8-10 minutes live; 5-7 minutes self-paced.
**Common misconception to preempt:** students sometimes assume a stalled rollout will
eventually time out and auto-rollback on its own. By default it will not — a Deployment has no
automatic rollback trigger; a human (or automation) has to run `kubectl rollout undo` (or fix
the underlying issue) explicitly. This is worth stating directly since it's a genuine and
consequential production gotcha.
**Optional extension:** ask what `kubectl rollout status --timeout=2m` would change about this
scenario (answer: the command itself would exit with an error after 2 minutes instead of
blocking forever — useful in CI/CD pipelines that need to fail a deploy step rather than hang).
@@ -0,0 +1,48 @@
# Knowledge Check: Deployments & Rollouts
---
**1. A rolling update to a new image version has been "in progress" for 20 minutes with no
error message, and `kubectl rollout status` never returns. What's the first thing you should
check, and why?**
*Answer:* `kubectl get pods` to see the new Pod's status, then `kubectl describe pod` on it if
it isn't `Running`. A stalled rollout with no explicit error almost always means a new Pod can't
become Ready — commonly `ImagePullBackOff` (bad image/tag) or a failing readiness probe. The
rollout isn't broken; it's correctly refusing to proceed until the new Pod proves healthy.
---
**2. True or false: `kubectl rollout undo` restores your application from a backup of its
previous state.**
*Answer:* False. It scales the previous (still-present, just scaled-to-zero) ReplicaSet back up
and the current one back down — the same rolling mechanism as a forward update, just pointed at
an older, already-proven-working Pod template. Nothing is being restored from storage.
---
**3. A Deployment has `replicas: 3`. You need it to handle more traffic. Which is the correct
action: edit the Pod template's resource limits, or change the replica count? Justify your
answer for a stateless web application.**
*Answer:* Change the replica count (scale out) for a stateless application handling more
concurrent traffic — more identical copies sharing the load. Changing resource limits affects
how much each individual Pod can use, not how many Pods exist; it doesn't add capacity for more
concurrent requests the way scaling does.
---
**4. Why does a Deployment keep an old, scaled-to-zero ReplicaSet around instead of deleting it
immediately after a successful rollout?**
*Answer:* That scaled-down ReplicaSet is exactly what makes fast, safe rollback possible later —
deleting it immediately would mean a "rollback" would have to rebuild the old Pod template from
scratch (or fail entirely) instead of simply scaling existing, already-defined Pods back up.
## Instructor Notes
**Facilitation:** Question 1 is the most important one to discuss out loud — it's the exact
diagnostic instinct students will need repeatedly in ROOT 10.
**Approximate timing:** 5 minutes.
@@ -0,0 +1,54 @@
# Guided Exercise: Planning a Safe Rollout Strategy
Paper-and-thinking exercise — work through this before the Lab.
## The scenario
Your team runs `api-server`, a Deployment with 6 replicas serving live traffic. You need to
ship a new image version. The team lead asks: "what could go wrong, and how would we know
before it takes down the whole service?"
## Your task
Work through these questions:
1. If the new image has a bug that crashes the container immediately on startup, at what point
in the rolling update would you expect to notice — after 1 new Pod, after all 6, or somewhere
in between? Why?
2. What's the difference in blast radius between that crash-on-startup bug and a subtler bug
that only fails under load, several minutes after a Pod starts serving traffic?
3. Given your answer to question 2, what's the practical value of watching `kubectl rollout
status` and `kubectl get pods` DURING a rollout, rather than only checking after it reports
"successfully rolled out"?
4. If you noticed a problem partway through, what's the single command that gets you back to
the last known-good state fastest?
## A worked answer to check yourself against
1. After the FIRST new Pod — a crash-on-startup bug means the new Pod never becomes Ready
(`CrashLoopBackOff`), so the rollout stalls immediately, well before more old Pods are
removed. This is actually the SAFEST kind of bug, because the blast radius stays tiny.
2. A crash-on-startup bug is naturally self-limiting (the rollout stalls itself, as above). A
bug that only appears under load is NOT self-limiting the same way — by the time it's
noticeable, several old (working) replicas may already have been replaced, meaning a larger
fraction of real traffic is already hitting the broken version.
3. Watching during the rollout, not just checking the final "successfully rolled out" message,
is what lets you catch the load-dependent case in question 2 early — waiting for the final
status alone would let the rollout complete entirely (100% on the broken version) before you
noticed anything wrong.
4. `kubectl rollout undo deployment/api-server` — assuming the previous ReplicaSet is still
present (the default), this is faster than trying to hand-diagnose and hot-fix the new
version under pressure.
## Instructor Notes
**Facilitation:** Question 2's distinction (self-limiting vs. not) is the core insight worth
making sure lands — it's why "the rollout said success" is necessary but not sufficient
evidence that a deploy actually went well.
**Approximate timing:** 10-15 minutes live with discussion; 10 minutes self-paced.
**Optional extension:** ask what monitoring/observability signal (beyond `kubectl` output) a
real team would want during a rollout to catch the load-dependent case even faster (a good
forward pointer to ROOT 10's observability themes, without needing to build any actual
monitoring infrastructure here).
@@ -0,0 +1,39 @@
# Launch Lab: Perform a Rolling Update
Time to run a real rolling update against a real Deployment in your disposable Lab.
## What you're doing
A Deployment `app-v1` is running an older image version. Your task is to update it to a newer
version and confirm every replica ends up running the new image — exactly the mechanism from
the lesson and demonstration, on a real cluster this time.
## Before you start
Recall the demonstration: `kubectl rollout status` is your tool for watching progress rather
than repeatedly polling `kubectl get pods` yourself, and a stalled rollout with no obvious error
usually means a new Pod can't become Ready. If your update seems to hang, that's exactly the
diagnostic instinct from the knowledge check to apply.
## Your task in this Lab
Update the Deployment's image and verify all replicas are running the new version. Use
`kubectl rollout status` to confirm completion, not just a single `kubectl get pods` snapshot.
This Lab has its own in-context hints if you get stuck.
When you're done, return to this module for a short reflection on what "successfully rolled
out" actually guarantees — and doesn't.
## Instructor Notes
**Classroom flow:** A good moment to have students keep `kubectl get pods --watch` open in a
second terminal (if their Lab environment supports multiple panes) so they see the incremental
old-Pod/new-Pod transition happen live, reinforcing the demonstration.
**Approximate timing:** 10-15 minutes.
**Common mistake to watch for:** running `kubectl edit deployment` and forgetting to save/apply
correctly, or editing the wrong field (e.g. `replicas` instead of the container image) — a good
moment to reinforce reading the task description precisely rather than pattern-matching to a
similar-looking previous Lab.
@@ -0,0 +1,43 @@
# Reflection: Rollback Is a Plan, Not a Panic Button
You just completed a real rolling update. Take a few minutes to connect it to how this plays out
in a real production team.
## Reflection questions
1. "Successfully rolled out" (from `kubectl rollout status`) tells you every replica is Running
and Ready. What does it NOT tell you about whether the new version is actually correct?
2. If you were on-call and a rollout you just ran seemed to be causing problems, would you reach
for `kubectl rollout undo` immediately or try to hot-fix the new version first? What would
change your answer?
3. Think back to the guided exercise's crash-on-startup vs. load-dependent distinction. Which
one would a typical automated health check (like a readiness probe, ROOT 10) catch, and
which one might slip through even with probes configured correctly?
## Real Ops takeaway
**A rollout completing is not the same as a deploy succeeding.** "Every Pod is Running and
Ready" only proves the new code started up and passed whatever checks are configured — it says
nothing about whether the new version is functionally correct under real traffic. Real teams
pair a rollout with actual monitoring (error rates, latency) before calling a deploy genuinely
successful, not just a green `kubectl rollout status`.
**Rollback should be a rehearsed, low-drama action, not a last resort.** Teams that treat
`kubectl rollout undo` as a normal, expected tool — something you reach for calmly the moment
something looks wrong — recover from bad deploys in seconds. Teams that treat rollback as an
admission of failure hesitate, try to hot-fix forward under pressure, and turn a small,
recoverable mistake into a longer outage.
This module is complete. Continue to the next module.
## Instructor Notes
**Discussion questions for a live classroom:** question 2 is worth pushing on — many students'
instinct is to try to fix forward, and it's worth surfacing WHY that instinct is often wrong
under time pressure (unknown time-to-fix vs. known, fast rollback).
**Approximate timing:** 8-10 minutes live discussion; 5 minutes self-paced writing.
**Optional extension:** ask students to write the one-sentence Slack message they'd post to
their team the moment they decide to roll back a bad deploy — reinforces that rollback is a
normal, communicable operational action, not a private failure.
@@ -0,0 +1,85 @@
# Jobs, CronJobs, and DaemonSets: The Other Workload Shapes
Everything in the previous module assumed a long-running application that should always have N
replicas up. Not every workload looks like that. Kubernetes has three other primitives for
genuinely different shapes of work — using a Deployment for any of these would be the wrong
tool, not just a less elegant one.
## Job: run to completion, exactly once (or a fixed number of times)
A Job creates one or more Pods and considers itself done when they exit successfully — it does
NOT restart a Pod that exits with success, and it DOES retry one that fails, up to a configured
limit. This is fundamentally different from a Deployment, which expects its Pods to run forever
and treats any exit as something to replace.
Use a Job for: a database migration, a batch data-processing task, a one-off cleanup script —
anything with a defined "finished" state.
```yaml
apiVersion: batch/v1
kind: Job
metadata:
name: migrate-db
spec:
template:
spec:
containers:
- name: migrate
image: myapp:latest
command: ["./migrate.sh"]
restartPolicy: Never
backoffLimit: 3
```
## CronJob: a Job, on a schedule
A CronJob doesn't run Pods itself — it creates a new Job on a cron schedule (`schedule:
"0 2 * * *"` for "every day at 2am"), and that Job behaves exactly as described above. Use a
CronJob for anything that should run periodically and finish: nightly backups, scheduled
reports, periodic cleanup.
A CronJob that reliably fails to complete (an image that always crashes, for example) will keep
creating new Jobs on schedule, each failing the same way — this is a real, common way to
silently exhaust cluster resources (job/pod history accumulating) if nobody's watching.
## DaemonSet: exactly one Pod per node, automatically
A DaemonSet doesn't declare a replica count at all — it ensures exactly one matching Pod runs
on every node in the cluster (or every node matching a selector), and automatically adds a Pod
when a new node joins. Use a DaemonSet for per-node infrastructure: a log-shipping agent, a
node-level monitoring exporter, a network plugin component — things that fundamentally need to
exist once per machine, not "N copies somewhere."
## The decision in one pass
| You need... | Use |
|---|---|
| Always-on, N identical copies, restarted forever | Deployment |
| Run once (or a few times) until it finishes successfully | Job |
| The above, but on a recurring schedule | CronJob |
| Exactly one copy on every node | DaemonSet |
## Instructor Notes
**Approximate timing:** 15 minutes read; 10 minutes live.
**Demo plan (if extending this PARTIAL module later):** live-create a Job with a command that
deliberately fails once then succeeds (`backoffLimit: 2`), and watch `kubectl get pods` show
the failed attempt alongside the eventual successful one — makes the retry-until-backoffLimit
behavior concrete.
**Likely misconceptions to watch for:**
- Students often assume a Job that finishes successfully gets cleaned up automatically. It
doesn't by default — the completed Pod/Job objects remain until manually deleted or a
`ttlSecondsAfterFinished` is configured (not covered here, worth a one-line mention if asked).
- "DaemonSet" is sometimes confused with "one Pod, cluster-wide" (singular). It's the opposite —
one Pod PER NODE, which can be many Pods total in a multi-node cluster.
**Discussion question:** "Why doesn't a Deployment support 'run once and be done'?" (Answer: a
Deployment's whole contract is maintaining a standing replica count forever — repurposing it for
one-shot work would fight that contract rather than use a primitive designed for it.)
**Optional extension:** ask what would go wrong if a CronJob's schedule ran more often than a
single Job execution takes to complete (answer: overlapping Job runs, unless
`concurrencyPolicy` is set to `Forbid` or `Replace` — a real gotcha worth naming even without
building a full lab around it).
@@ -0,0 +1,45 @@
# Knowledge Check: Choosing the Right Workload Primitive
---
**1. You need to run a database migration script exactly once when a new version deploys. Which
primitive fits, and why not a Deployment?**
*Answer:* A Job. A Deployment expects its Pods to run forever and replaces any that exit; a
migration script is supposed to exit successfully when done, which a Deployment would treat as
a failure to "fix" by restarting it — the wrong behavior for a task that's supposed to finish.
---
**2. A CronJob scheduled for "every 5 minutes" keeps creating Jobs that fail immediately due to
a bad image tag. What actually happens over the next hour if nobody notices?**
*Answer:* A new failing Job (and its Pod) gets created roughly every 5 minutes, accumulating
failed Job/Pod objects and consuming cluster resources — nothing stops this automatically. This
is exactly the kind of slow, easy-to-miss resource exhaustion a real on-call engineer should
watch for.
---
**3. True or false: a DaemonSet's Pod count scales with a `replicas` field, the same as a
Deployment.**
*Answer:* False. A DaemonSet has no `replicas` field — its Pod count is implicitly "one per
matching node," and it grows or shrinks automatically as nodes join or leave the cluster.
---
**4. You need a log-shipping agent that must run on every node, including any new node added
later. Which primitive, and what would go wrong if you used a Deployment with a high replica
count instead?**
*Answer:* A DaemonSet. A Deployment's replicas are placed by the scheduler wherever it decides
is best — nothing guarantees one per node, and a newly-added node wouldn't automatically get a
copy the way a DaemonSet guarantees.
## Instructor Notes
**Facilitation:** Question 2 is worth emphasizing — it's a genuinely realistic "how does a
cluster quietly get into trouble" scenario, not just trivia.
**Approximate timing:** 5 minutes.
@@ -0,0 +1,72 @@
# Demonstration: Watching a Job Actually Complete
## Creating the Job
```
$ kubectl apply -f pi-job.yaml
job.batch/pi-job created
$ kubectl get jobs
NAME COMPLETIONS DURATION AGE
pi-job 0/1 3s 3s
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
pi-job-8x2kq 1/1 Running 0 3s
```
The Job's own status (`0/1` completions) is tracked separately from its Pod's status — this
matters because a Job can retry a FAILED Pod attempt without that looking like the healthy
restart behavior you'd see on a Deployment.
## Once the Pod's process actually exits successfully
```
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
pi-job-8x2kq 0/1 Completed 0 8s
$ kubectl get jobs
NAME COMPLETIONS DURATION AGE
pi-job 1/1 7s 8s
```
`STATUS: Completed` — not `Running`, not `CrashLoopBackOff`. This is the ONE status a
long-running Deployment's Pod is never supposed to reach on its own; for a Job, it's success.
The Job object itself now reports `1/1` and stops — nothing recreates this Pod, because
"finished successfully" is the desired end state here, unlike a Deployment where exiting is
always treated as something to fix.
## What a FAILING Job attempt looks like, briefly
```
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
bad-job-abc12 0/1 Error 0 5s
bad-job-def34 0/1 Error 0 2s
$ kubectl describe job bad-job
...
Events:
Warning BackoffLimitExceeded Job has reached the specified backoff limit
```
Two DIFFERENT Pods (`abc12`, `def34`), not one Pod restarting twice — each failed attempt gets
its own new Pod, up to `backoffLimit`. Once that limit is hit, the Job stops retrying and
reports the failure via Events rather than trying forever.
## Instructor Notes
**Demo script:** Run the successful case live; showing the FAILING case is optional if time is
short, but the `Completed` vs `Running`/`Error` distinction is worth pausing on regardless.
**Approximate timing:** 8 minutes live; 5 minutes self-paced.
**Common misconception to preempt:** students who've only seen Deployments sometimes expect
`kubectl get pods` to show the Job's Pod disappear once it's done. It doesn't, by default — it
stays as `Completed` until manually cleaned up or a TTL is configured, which is worth mentioning
briefly (not a hands-on topic here).
**Optional extension:** ask what `kubectl get pods` would show for a CronJob's Pods across
several scheduled runs (answer: one Completed Pod per past run, accumulating — the same
"nothing auto-cleans this up" behavior, multiplied by the schedule).
@@ -0,0 +1,50 @@
# Guided Exercise: Choosing the Right Workload Primitive
Paper-and-thinking exercise before the Labs.
## The scenario
Your team needs four things running in the cluster:
1. A nightly database backup that must run at 2am and finish before morning.
2. A log-shipping agent that must run on every node, including nodes added later.
3. A one-time script to backfill a new database column, run once during this deploy.
4. The main web application, always up, serving live traffic.
## Your task
For each of the four, name the correct primitive (Deployment, Job, CronJob, or DaemonSet) and
answer: what would go wrong if you used a Deployment for it instead?
## A worked answer to check yourself against
1. **CronJob.** A Deployment has no concept of "run on a schedule" — you'd need external
tooling to trigger it repeatedly, reinventing what CronJob already does. Using a bare Job
would run it once; CronJob is what recreates that Job on schedule.
2. **DaemonSet.** A Deployment's replica count is a number, not "one per node" — nothing
guarantees even coverage, and a newly added node wouldn't automatically get a copy the way a
DaemonSet guarantees.
3. **Job.** A Deployment expects its Pod to run forever; a script that's SUPPOSED to exit when
done would be treated as a crash and restarted forever, which is wrong for a one-time task.
4. **Deployment.** This is the actual always-on, N-replicas case the other three are explicitly
NOT designed for.
## Self-check
If you answered "Job" for #4 or "Deployment" for #1-3, revisit the lesson's decision table
before the Labs — getting this choice right the first time avoids building something that
technically runs but behaves wrong in a way that's easy to miss until it causes a real problem
(a backup CronJob accidentally deployed as a Job that only ever runs once, for example).
## Instructor Notes
**Facilitation:** Good as a quick individual exercise before revealing the answer — the "what
would go wrong" framing forces reasoning about the CONSEQUENCE of a wrong choice, not just
recall of a definition.
**Approximate timing:** 8-10 minutes live with discussion; 8 minutes self-paced.
**Optional extension:** ask what would need to be true for #1 (the nightly backup) to actually
be SAFE to run as a plain Job triggered manually once, instead of a CronJob — a good check that
students understand CronJob's value is specifically the SCHEDULING, not something inherent to
backups.
@@ -0,0 +1,40 @@
# Reflection: The Right Primitive Is a Correctness Decision, Not a Style Choice
You just built a one-shot Job and a scheduled CronJob for real. Take a few minutes to connect
this to production thinking.
## Reflection questions
1. If your team accidentally deployed the database backup script as a Deployment instead of a
CronJob, what's the FIRST symptom you'd likely notice — and would it look like a bug in the
backup script itself, or something else?
2. A CronJob that fails silently every night for a week is a real, common incident pattern. What
would you want monitored to catch this quickly?
3. Why does "it ran once and worked" not prove a CronJob is configured correctly?
## Real Ops takeaway
**Workload-primitive mistakes often look like application bugs at first.** A backup script
"crashing" every few minutes on a Deployment isn't a bug in the script — the script did exactly
what it was told to do (exit when finished) and the Deployment is "correctly" restarting it
according to its own contract. Recognizing "wrong primitive," not "buggy code," as the root
cause saves real debugging time.
**A schedule you never verify is a schedule you can't trust.** A CronJob failing every single
run since deployment is easy to miss if nobody checks — unlike a Deployment's crash-looping
Pods, which are visible in `kubectl get pods` by default, a CronJob's failed history requires
someone to actually look at `kubectl get jobs` or the CronJob's own status. Real teams monitor
CronJob success/failure explicitly rather than assuming silence means success.
This module is complete. Continue to the next module.
## Instructor Notes
**Discussion questions for a live classroom:** question 1 is worth pushing on — many students'
first instinct is to debug the "crashing" script rather than question the workload type it's
running under.
**Approximate timing:** 8 minutes live discussion; 5 minutes self-paced writing.
**Optional extension:** ask students to name one alerting rule they'd add for a critical nightly
CronJob (e.g. "alert if no successful completion recorded in the last 25 hours").
@@ -0,0 +1,84 @@
# Externalizing Configuration: ConfigMaps & Secrets
## The problem: hardcoded configuration doesn't scale
If your application's database URL, log level, or feature flags are baked directly into its
container image, changing any of them means rebuilding and redeploying the image — for a
one-line config change. Worse, the SAME image can't be reused across dev/staging/production if
each environment needs different settings compiled in.
Kubernetes solves this by letting you inject configuration into a container at RUNTIME,
separately from the image itself — the same image, different config per environment.
## ConfigMap: configuration that isn't secret
A **ConfigMap** holds arbitrary key/value configuration data — log levels, feature flags, URLs,
whole config files. It's a plain Kubernetes object, readable by anyone with API access to that
Namespace, so it's for configuration that ISN'T sensitive.
Two ways to consume a ConfigMap in a Pod:
- **As environment variables** — each key becomes an env var the application reads normally.
- **As a mounted volume** — each key becomes a FILE inside the container, useful for
applications that expect to read configuration from disk rather than environment variables.
```yaml
envFrom:
- configMapRef:
name: app-env
```
## Secret: the same idea, for sensitive data
A **Secret** has the identical shape and consumption pattern as a ConfigMap (env vars or a
mounted volume) but is meant for sensitive values — passwords, API keys, tokens. Kubernetes
stores Secret data base64-ENCODED, not encrypted — this is an important distinction:
base64 is trivially reversible, so a Secret's real protection comes from Kubernetes' RBAC
(who can even read Secret objects at all, ROOT 09) and, in a properly configured cluster,
encryption at rest — not from the encoding itself.
```yaml
apiVersion: v1
kind: Secret
metadata:
name: app-secret
type: Opaque
data:
api-key: c3VwZXJzZWNyZXQ= # base64 of "supersecret" — NOT encrypted
```
## Why the same mechanism, two different objects
You might ask why Kubernetes doesn't just have one "Config" object with a "sensitive" flag.
Keeping them separate makes the DISTINCTION visible and enforceable — RBAC rules can grant
"read ConfigMaps" without granting "read Secrets," and a quick `kubectl get configmaps` never
accidentally exposes something that should have been a Secret. The separation is a deliberate
safety boundary, not redundant API surface.
## Instructor Notes
**Approximate timing:** 15 minutes read; 10 minutes live.
**Demo plan:** Before the demonstration activity, it's worth showing `kubectl get secret
app-secret -o yaml` live and pointing out the base64 value, then decoding it on the spot
(`echo <value> | base64 -d`) — the "this isn't actually encrypted" realization lands much better
seen than described.
**Likely misconceptions to watch for:**
- Students very commonly assume Secrets are encrypted by Kubernetes automatically. Correct this
explicitly and early — it's a genuine, consequential security misunderstanding.
- The env-var vs. volume-mount choice is sometimes seen as arbitrary. Note one real
consideration: env vars are visible via `kubectl describe pod` and process inspection tools
inside the container more easily than a mounted file's contents — not a hard rule, but a
reason some teams prefer volume mounts specifically for Secrets.
**Discussion question:** "If someone with `kubectl get secrets` access on your namespace can
trivially read every Secret's real value, what is a Secret actually protecting you from?"
(Answer: it protects against accidental exposure through normal channels like
`kubectl get configmaps`, source control, or logs — not against a determined actor with direct
API read access, which is exactly why RBAC restricting who has that access is the real control.)
**Optional extension:** ask what changes about this picture on a cluster with encryption-at-rest
enabled for Secrets (answer: the ETCD-stored data itself becomes encrypted, protecting against
someone who gains access to the underlying etcd storage directly — still separate from and in
addition to RBAC controlling API-level access).
@@ -0,0 +1,66 @@
# Demonstration: A Secret Is Encoded, Not Encrypted
## Creating a Secret
```
$ kubectl create secret generic app-secret --from-literal=api-key=supersecret
secret/app-secret created
```
## Looking at it
```
$ kubectl get secret app-secret -o yaml
apiVersion: v1
kind: Secret
metadata:
name: app-secret
data:
api-key: c3VwZXJzZWNyZXQ=
type: Opaque
```
## Decoding it — no special permission needed beyond reading the Secret itself
```
$ echo "c3VwZXJzZWNyZXQ=" | base64 -d
supersecret
```
Anyone who can run `kubectl get secret app-secret -o yaml` (i.e. anyone RBAC allows to READ
Secrets in this Namespace) can trivially recover the real value with a standard, built-in
command — no cracking, no special tooling. This is exactly why the lesson's point matters: the
protection here is WHO can read the Secret object at all (RBAC), not the base64 encoding itself.
## Comparing to a ConfigMap
```
$ kubectl get configmap app-config -o yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: app-config
data:
environment: production
log_level: info
```
A ConfigMap's values are stored in PLAIN TEXT — no encoding step at all. This is consistent with
ConfigMaps being explicitly for non-sensitive data; there's no pretense of protection to begin
with.
## Instructor Notes
**Demo script:** Run this live, actually decoding the value in front of the class — the gap
between "Secret" (a name implying protection) and what the command output shows is the entire
point, and it lands far better performed than read.
**Approximate timing:** 6-8 minutes live; 5 minutes self-paced.
**Common misconception to preempt:** this demonstration directly targets the single most common
misconception about Kubernetes Secrets — do not skip it even if time is short, since a wrong
belief here has real security consequences later.
**Optional extension:** ask what a `SealedSecret` or an external secrets manager (mentioned by
name only, not built as a hands-on topic here) is trying to solve that a plain Kubernetes Secret
does not — a good forward pointer without requiring new infrastructure in this course.
@@ -0,0 +1,48 @@
# Knowledge Check: Configuration & Secrets
---
**1. A teammate says "our database password is safe because it's stored as a Kubernetes
Secret." What's missing from that statement?**
*Answer:* A Secret's data is base64-ENCODED, not encrypted — anyone who can read the Secret
object at all can trivially decode it. The real protection is RBAC (who is allowed to read
Secrets in that namespace) and, on a properly configured cluster, encryption at rest — not the
Secret object type itself.
---
**2. Why does Kubernetes have two separate objects (ConfigMap and Secret) instead of one
"Config" object with a sensitivity flag?**
*Answer:* Keeping them separate makes the distinction enforceable via RBAC — a role can grant
"read ConfigMaps" without granting "read Secrets" — and makes it visible at a glance
(`kubectl get configmaps` never risks accidentally listing something sensitive).
---
**3. An application needs its configuration to change without rebuilding its container image
across dev/staging/production. Which Kubernetes mechanism solves this, and why not just bake
different configs into three separate images?**
*Answer:* ConfigMaps (and Secrets, for sensitive values) injected at runtime via env vars or a
mounted volume. Baking config into the image means maintaining three separate images for what
should be identical application code — externalizing configuration lets the SAME image run
correctly in every environment.
---
**4. True or false: mounting a ConfigMap as a volume and consuming it as environment variables
are functionally identical from the application's perspective.**
*Answer:* False. An application reading environment variables needs code that reads env vars; an
application expecting a config FILE on disk needs the volume-mount approach. The right choice
depends on how the application itself is written to read configuration, not just personal
preference.
## Instructor Notes
**Facilitation:** Question 1 is the single most important takeaway from this module — confirm
it's genuinely understood, not just correctly guessed, before moving to the Labs.
**Approximate timing:** 5-7 minutes.
@@ -0,0 +1,50 @@
# Guided Exercise: ConfigMap, Secret, Env Var, or Volume?
Paper-and-thinking exercise before the Labs.
## The scenario
Your application needs four pieces of configuration:
1. A log level (`debug`/`info`/`warn`) that ops sometimes needs to flip without redeploying.
2. A database password.
3. A large JSON config file the application reads from a specific path at startup.
4. A feature flag read once at container startup via an environment variable, per the
application's existing code.
## Your task
For each, decide: ConfigMap or Secret, and env var or volume mount — then justify each choice.
## A worked answer to check yourself against
1. **ConfigMap, env var.** Not sensitive; the application likely already reads it as an env var
(a common convention), and env vars are simple to change and trigger a rolling restart with.
2. **Secret, either — but env var is common here too**, since most applications expect
credentials via environment variables by convention. The KEY decision is Secret (sensitive),
not the delivery mechanism.
3. **ConfigMap, volume mount.** The application expects a FILE at a path, not an environment
variable — volume mount is the only mechanism that presents a ConfigMap as an actual file.
4. **ConfigMap (assuming not sensitive) or Secret (if it is), env var** — the application's
EXISTING code dictates this; you don't get to choose a mechanism the application wasn't
written to read.
## Self-check
If you chose "volume mount" for #4 or "env var" for #3, revisit the lesson — the deciding
factor here isn't which mechanism seems more "correct" in the abstract, it's what the
application actually expects. Getting this backwards means the application simply won't find
its configuration at all.
## Instructor Notes
**Facilitation:** Emphasize that #4's answer depends entirely on EXISTING application
behavior — a good moment to note that platform engineers frequently have to adapt Kubernetes
config delivery to match an application's assumptions, not the reverse.
**Approximate timing:** 8-10 minutes live with discussion; 8 minutes self-paced.
**Optional extension:** ask what would happen if a ConfigMap consumed as a volume mount is
UPDATED after the Pod is already running — a good bridge to the real (if not hands-on-tested
here) behavior that mounted ConfigMap files update automatically after a short delay, while
env-var-injected values do NOT update without a Pod restart.
@@ -0,0 +1,39 @@
# Reflection: Configuration Is Infrastructure, Secrets Are a Trust Boundary
You just externalized configuration and handled a real Secret. Take a few minutes to connect
this to production practice.
## Reflection questions
1. Before this module, would you have assumed a Kubernetes Secret was encrypted? What's the
real-world consequence of a team believing that when it isn't true?
2. If your team's RBAC accidentally granted a low-trust role `get secrets` access across an
entire namespace, what's actually exposed — and would your team necessarily notice?
3. Why does separating configuration from the container image matter beyond convenience — think
about what it enables for testing or incident response.
## Real Ops takeaway
**A misunderstood Secret is a false sense of security.** Teams that believe Secrets are
encrypted by default sometimes skip additional controls (proper RBAC scoping, external secrets
management, encryption at rest) that they'd otherwise recognize as necessary. The gap between
perceived and actual protection is exactly where real incidents happen.
**Externalized configuration is what makes rollback and incident response fast.** If a bad
config value ships (a wrong feature flag, an incorrect log level flooding storage), fixing it
means updating a ConfigMap/Secret and letting Pods pick up the change — not rebuilding and
redeploying an entire image. This is a direct, practical payoff of the separation this module
taught, not just an architectural nicety.
This module is complete. Continue to the next module.
## Instructor Notes
**Discussion questions for a live classroom:** question 2 works well as a small-group exercise —
have students actually list what a namespace's Secrets typically contain in a real application
(DB creds, API keys, TLS certs) to make the exposure concrete.
**Approximate timing:** 8 minutes live discussion; 5 minutes self-paced writing.
**Optional extension:** ask students to draft a one-line policy statement their hypothetical
team would adopt about who is allowed `get`/`list` access to Secrets in production namespaces.
@@ -0,0 +1,87 @@
# The Service/Endpoints/DNS Request Path
## The problem: Pod IPs are not stable
Every Pod gets its own IP address, but that IP is NOT durable — when a Pod is replaced (a
rollout, a crash recovery, a reschedule), the new Pod gets a brand new IP. If another part of
your application had that old IP hardcoded, it would silently start talking to nothing. This is
the exact problem in `expose-service-basics`'s scenario: "other pods cannot reliably connect
because pod IPs change when they restart."
## A Service: a stable address in front of a moving set of Pods
A Service is a stable virtual IP and DNS name that automatically routes to whichever Pods
currently match its label selector — regardless of how many there are or how often they change.
You never talk to a Pod's IP directly in normal operation; you talk to the Service, and the
Service's Endpoints (a live, continuously-updated list of the matching Pods' IPs) determines
where traffic actually goes.
```yaml
apiVersion: v1
kind: Service
metadata:
name: redis-svc
spec:
selector:
app: redis # <- matches Pods by label, same mechanism from ROOT 05
ports:
- port: 6379
```
This is the same label-selector mechanism you learned in ROOT 05 — a Service doesn't know or
care about specific Pod names, only which Pods currently carry a matching label.
## The three Service types you'll use most
- **ClusterIP** (the default) — reachable only from inside the cluster. Use this for
internal-only communication, like one microservice talking to another or to a database.
- **NodePort** — additionally exposes the Service on a static port on every node's own IP,
reachable from outside the cluster. Mostly used for development/testing or when there's no
cloud load balancer available.
- **LoadBalancer** — asks the cloud provider to provision a real external load balancer pointed
at the Service. This is the standard way to expose something to the public internet on a
cloud-hosted cluster.
Choosing wrong isn't just a style preference — a ClusterIP Service is architecturally
unreachable from outside the cluster no matter how correctly everything else is configured.
## DNS: how Pods actually find a Service by name
Every Service automatically gets a DNS name inside the cluster
(`<service-name>.<namespace>.svc.cluster.local`, or usually just `<service-name>` from within
the same Namespace) — this is why application code can use a plain hostname like `redis-svc`
instead of an IP address at all. Kubernetes' internal DNS resolves that name to the Service's
stable virtual IP, which then routes to whichever Pods are currently healthy.
## The full request path, end to end
```
Client -> DNS lookup "redis-svc" -> Service's stable virtual IP
-> Service consults its Endpoints (live list of matching, ready Pods)
-> traffic forwarded to one of those Pod IPs
```
Every one of these steps can fail independently, which is exactly why "the app can't reach
redis" requires checking each layer rather than guessing — the next activity shows you what
that looks like when the Endpoints layer is empty.
## Instructor Notes
**Approximate timing:** 15 minutes read, 10-12 minutes live with the request-path diagram.
**Demo plan:** Before the next activity, it's worth live-running `kubectl get endpoints
<service-name>` against a healthy Service so students see a populated Endpoints list before
they see an empty one in the demonstration.
**Likely misconceptions to watch for:**
- Students often think a Service "is" a load balancer process running somewhere. It's closer to
a routing rule maintained by `kube-proxy` on every node — worth clarifying that there's no
single "Service Pod" to go find and debug.
- ClusterIP vs. NodePort vs. LoadBalancer is sometimes memorized without the "why" — the
knowledge check's Service-type question is designed to test the underlying reasoning, not
just recall.
**Discussion question:** "If a Service's selector matches zero Pods, does the Service itself
fail to exist?" (Answer: no — the Service object exists fine; its Endpoints list is just empty,
which looks like connection failure from the client's side without any error at the Service
level itself. This is exactly the next activity's topic.)
@@ -0,0 +1,82 @@
# Demonstration: A Service With Zero Endpoints
This is one of the most common "it just doesn't work" incidents in real Kubernetes operation —
and it produces no error message at all at the Service level.
## The setup
A Service `api-svc` selects Pods labeled `app: api`. A Deployment was recently updated and its
Pods now carry the label `app: api-v2` instead (a rename during a refactor, deliberately or
accidentally).
## What you'd see from the client side
```
$ curl http://api-svc:8080/health
curl: (7) Failed to connect to api-svc port 8080: Connection refused
```
Just a generic connection failure — nothing here says "your Service's selector doesn't match
any Pods."
## What the Service object itself shows
```
$ kubectl get service api-svc
NAME TYPE CLUSTER-IP PORT(S) AGE
api-svc ClusterIP 10.96.34.201 8080/TCP 2h
```
Completely normal-looking. The Service exists, has a stable IP, is not in any kind of "error"
state — because from the Service's own perspective, nothing is wrong. It's doing exactly what
it's configured to do: route to whatever matches its selector, which currently happens to be
nothing.
## Where the real evidence is: Endpoints
```
$ kubectl get endpoints api-svc
NAME ENDPOINTS AGE
api-svc <none> 2h
```
This is the actual signal. `<none>` means the Service's selector currently matches zero Pods —
this is the single fastest check for "is this a Service-to-Pod mismatch" any time something
reachable-in-principle looks like a connection failure.
## Confirming the mismatch
```
$ kubectl get pods --show-labels
NAME READY STATUS LABELS
api-abc123 1/1 Running app=api-v2
```
The Pod is healthy and running — `app=api-v2`, not `app=api`. The Service's selector
(`app: api`) matches nothing, and nothing about the Pod itself is broken.
## The general diagnostic shape
`curl`/application-level connection failure tells you *that* something's wrong, but nothing
about *where*. `kubectl get endpoints <service>` is the fastest way to distinguish "the Service
has no matching Pods" from "the Pods exist and are ready but something else is wrong" —
reach for it early whenever a Service-fronted connection fails for no obvious reason.
## Instructor Notes
**Demo script:** Live-create this exact mismatch (a Service with one selector, a Deployment
with a slightly different label) and walk through `kubectl get service` -> `kubectl get
endpoints` -> `kubectl get pods --show-labels` in front of the class, narrating what each
command is and isn't telling you.
**Approximate timing:** 8-10 minutes live; 5-7 minutes self-paced.
**Common misconception to preempt:** students often check the Service object first and, seeing
nothing obviously wrong, assume the problem must be elsewhere entirely (DNS, firewall, the
application itself) — reinforce that `kubectl get endpoints` is the specific, fast check for
this exact failure mode, before reaching for anything more complex.
**Optional extension:** ask what `kubectl describe service api-svc` would additionally show
compared to `kubectl get service` (answer: the same selector the Service is using, which lets
you directly compare it against `kubectl get pods --show-labels`'s actual Pod labels side by
side, rather than inferring the mismatch).
@@ -0,0 +1,47 @@
# Knowledge Check: Services & Service Discovery
---
**1. A Deployment has three replicas but `kubectl get endpoints` shows `<none>` for its Service.
Which part of the system should you inspect first, and why?**
*Answer:* Compare the Service's selector (`kubectl describe service`) against the actual Pods'
labels (`kubectl get pods --show-labels`) — a Service with a healthy backend Deployment but zero
Endpoints almost always means the selector doesn't match the Pods' current labels, not that the
Pods themselves are unhealthy.
---
**2. You need a microservice reachable ONLY from other services inside the same cluster, never
from the public internet. Which Service type, and why not LoadBalancer?**
*Answer:* ClusterIP — it's reachable only from inside the cluster by design, which is exactly
the requirement. A LoadBalancer would provision a real external endpoint reachable from the
public internet, which is more exposure than the requirement calls for, and depends on cloud
provider support this scenario doesn't need.
---
**3. True or false: application code that connects to `redis-svc` by that plain hostname is
relying on Kubernetes' internal DNS to resolve it.**
*Answer:* True. Every Service automatically gets a DNS name; that's specifically what lets code
use a stable hostname instead of hardcoding a Pod IP that could change at any time.
---
**4. A Service's Endpoints list is non-empty, but the application still can't connect. Does
this rule out a Service-level problem entirely?**
*Answer:* Not entirely — a non-empty Endpoints list means at least one Pod currently matches the
selector, but the connection could still fail for other reasons (wrong port number on the
Service, a NetworkPolicy blocking the traffic, the application inside the Pod not actually
listening). A populated Endpoints list narrows the problem, it doesn't prove the whole path
works.
## Instructor Notes
**Facilitation:** Question 1 is the single most transferable diagnostic habit from this module —
worth explicitly calling out as "the first thing to check" for this entire class of problem.
**Approximate timing:** 5 minutes.
@@ -0,0 +1,49 @@
# Guided Exercise: Choosing the Right Service Type
Paper-and-thinking exercise before the Lab.
## The scenario
You're deploying three components for a class project on a cloud-hosted cluster:
1. A Postgres database that only your own backend application should ever talk to.
2. A backend API that the frontend (also running in the cluster) calls internally.
3. The frontend web application itself, which needs to be reachable by users on the public
internet.
## Your task
For each of the three components, decide: ClusterIP, NodePort, or LoadBalancer — and justify
each choice in one sentence referencing WHO needs to reach it.
## A worked answer to check yourself against
1. **Postgres — ClusterIP.** Only your own backend (running inside the same cluster) needs to
reach it; there is no reason for it to be reachable from outside the cluster at all, and
exposing a database externally is a real, avoidable security risk.
2. **Backend API — ClusterIP.** The frontend calls it from inside the cluster too — same
reasoning as the database. External users never talk to the backend directly; they only ever
go through the frontend.
3. **Frontend — LoadBalancer** (on a real cloud-hosted cluster). Public internet users need to
reach it, and a cloud provider's LoadBalancer is the standard mechanism for that. NodePort
would technically work but is typically used for development/testing rather than a real
public-facing service on a cloud cluster.
## Self-check
If you chose LoadBalancer for the database or the backend API, reconsider — "internet-reachable
by default" is not the safe default for anything that doesn't need to be, and getting this
choice right the first time avoids a real security exposure later, not just a style issue.
## Instructor Notes
**Facilitation:** Push on the "why not just LoadBalancer everything, it works" instinct some
students will have — this is a good early moment to connect Service-type choice to the security
mindset that ROOT 09 builds on more fully (exposure should be intentional, not the path of
least resistance).
**Approximate timing:** 10 minutes live with discussion; 10 minutes self-paced.
**Optional extension:** ask what would need to be added (not covered in this module) to let
public users reach the frontend at a real domain name over HTTPS rather than a bare
LoadBalancer IP — a natural forward pointer to Ingress, covered in the next module.
@@ -0,0 +1,40 @@
# Launch Lab: Exposing an Application
Time to create a real Service in your disposable Lab and connect it to a real Deployment.
## What you're doing
A `redis` Deployment is running, but other Pods can't reliably reach it because Pod IPs change
whenever a Pod restarts — exactly the problem the lesson opened with. Your task is to create a
stable ClusterIP Service in front of it.
## Before you start
Recall the request path from the lesson: your Service's `selector` has to match the Deployment
Pods' actual labels, or you'll end up with the empty-Endpoints situation from the
demonstration. Check the Deployment's Pod labels before writing your Service's selector, don't
guess at them.
## Your task in this Lab
Create a ClusterIP Service exposing the `redis` Deployment on the correct port. Once it's
created, verify it the same way the demonstration did: `kubectl get service`, then `kubectl get
endpoints` to confirm it actually has a non-empty Endpoints list, not just that the Service
object exists.
This Lab has its own in-context hints if you get stuck.
When you're done, return to this module for a reflection on why service discovery is
infrastructure, not magic.
## Instructor Notes
**Classroom flow:** Encourage students to run `kubectl get endpoints` as their OWN verification
step even though the Lab's automated validation will check this too — building the habit of
checking Endpoints, not just "the Service exists," is the actual transferable skill here.
**Approximate timing:** 10-15 minutes.
**Common mistake to watch for:** a selector that matches on the wrong label key/value, or a
port/targetPort mismatch (the Service's `port` is what clients connect to; `targetPort` is what
the container actually listens on — conflating the two is a very common first-time error).
@@ -0,0 +1,40 @@
# Reflection: Service Discovery Is Infrastructure, Not Magic
You just created a Service that gives a moving set of Pods a stable address. Take a few minutes
to connect this to real operational thinking.
## Reflection questions
1. Before this module, "the app can just connect to redis by name" might have seemed obvious or
automatic. What specific mechanism actually makes that true, and what would break it?
2. If your Service's selector had been wrong (matching zero Pods), what would the SYMPTOM have
looked like from the application's side, versus what the actual CAUSE would have been?
3. Why is checking `kubectl get endpoints` before assuming a networking/DNS/firewall problem a
good default habit, given how common a selector mismatch actually is?
## Real Ops takeaway
**"It's probably a networking issue" is often wrong, and Endpoints is how you find out fast.** A
huge fraction of real "service X can't reach service Y" incidents in Kubernetes are Service
selector mismatches, not actual network problems — checking `kubectl get endpoints` first, before
escalating to firewall rules or DNS debugging, saves real time precisely because this failure
mode produces zero errors at the layers people instinctively check first.
**Stable addressing is what makes rolling updates and self-healing actually usable.** Everything
you learned in ROOT 06 about Pods being replaced during a rollout or after a crash only works
smoothly in practice because Services give callers a stable target that survives those
replacements — without this layer, every Pod replacement would require updating every caller's
configuration by hand.
This module is complete. Continue to the next module.
## Instructor Notes
**Discussion questions for a live classroom:** use question 2 as a discussion prompt — the
symptom/cause gap is exactly why this class of bug is often under-diagnosed in real teams.
**Approximate timing:** 8-10 minutes live discussion; 5 minutes self-paced writing.
**Optional extension:** ask students to write the one-line runbook entry they'd want a future
on-call engineer to have for "Service X seems unreachable" — a good exercise in turning a
lesson into something operationally reusable.
@@ -0,0 +1,90 @@
# Ingress and NetworkPolicy: What They Architecturally Do
## Ingress: routing external HTTP(S) traffic by host/path
You already know a LoadBalancer Service can expose something to the public internet. But a real
application usually needs more than "one raw IP to one Service" — you might want
`api.example.com` and `app.example.com` to route to different backend Services, all through one
entry point, with TLS termination handled in one place instead of per-Service.
**Ingress** is the Kubernetes object for exactly this: a set of HTTP routing rules (by hostname
and/or URL path) that an Ingress controller reads and turns into actual load-balancer/proxy
configuration.
```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: app-ingress
spec:
rules:
- host: app.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: frontend-svc
port: { number: 80 }
```
An important distinction: Ingress is a set of ROUTING RULES, not a load balancer itself — an
Ingress controller (a separate piece of software running in the cluster) is what actually reads
these rules and does the routing. Without a controller installed, an Ingress object exists in
the API but nothing acts on it.
## NetworkPolicy: traffic reachability as something you declare, not assume
By default, every Pod in a Kubernetes cluster can reach every other Pod — there is no built-in
network isolation between them at all. This surprises people coming from a "obviously the
network is locked down unless configured otherwise" assumption. It is exactly backwards from
that assumption: **everything is reachable until you say otherwise.**
A NetworkPolicy changes this for the Pods it selects, by declaring which traffic IS allowed —
everything not explicitly allowed to a Pod covered by at least one NetworkPolicy is then denied.
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-frontend-to-backend
spec:
podSelector:
matchLabels: { app: backend }
ingress:
- from:
- podSelector:
matchLabels: { app: frontend }
```
This example allows traffic to `backend` Pods only from `frontend` Pods — once this policy
exists and selects `backend`, every OTHER Pod trying to reach `backend` is now blocked, even
though nothing was true about that before the policy existed.
## Why this matters architecturally, before it matters as a security control
Right now, think of NetworkPolicy purely as "declaring intended reachability, not assuming it."
ROOT 09 comes back to NetworkPolicy specifically as a SECURITY control (default-deny postures,
least-privilege network access) — this module is deliberately just the architectural mental
model: Kubernetes doesn't isolate network traffic for you by default, and NetworkPolicy is the
mechanism for declaring the reachability you actually intend.
## Instructor Notes
**Approximate timing:** 15 minutes read; 10 minutes live.
**Likely misconceptions to watch for:**
- Students often assume Ingress by itself "does" the routing. Emphasize the controller
requirement — an Ingress object with no controller installed does nothing.
- The "everything reachable by default" fact genuinely surprises most students the first time
they hear it — it's worth stating plainly and letting it land rather than rushing past it.
**Discussion question:** "If NO NetworkPolicy exists anywhere in a Namespace, what's the actual
reachability between Pods in that Namespace?" (Answer: completely open — every Pod can reach
every other Pod, matching the "everything reachable by default" rule.)
**Optional extension:** ask what would happen if you created a NetworkPolicy that selects
`frontend` Pods but declares no `ingress` rules at all (answer: `frontend` Pods would become
completely unreachable by anything — an empty ingress list inside a NetworkPolicy that HAS an
ingress policyType means "deny all incoming," not "no policy applies").
@@ -0,0 +1,46 @@
# Knowledge Check: Ingress & NetworkPolicy Foundations
---
**1. You create an Ingress object with correct routing rules, but no traffic is ever routed
according to them. What's the most likely reason?**
*Answer:* No Ingress controller is installed/running in the cluster. Ingress objects are just
routing rules; a separate controller has to actually read and act on them, or nothing happens.
---
**2. True or false: by default, a Pod in Namespace A cannot reach a Pod in Namespace B unless a
NetworkPolicy explicitly allows it.**
*Answer:* False — the opposite is true by default. With NO NetworkPolicy in place at all,
every Pod can reach every other Pod regardless of Namespace. NetworkPolicy is what RESTRICTS
this default-open behavior, not what's required to enable connectivity in the first place.
---
**3. A NetworkPolicy selects Pods labeled `app: backend` and declares one ingress rule allowing
traffic from Pods labeled `app: frontend`. What happens to traffic from a Pod labeled
`app: admin-tool` trying to reach `backend`?**
*Answer:* It's blocked. Once a NetworkPolicy selects a Pod, only traffic matching one of its
explicit rules is allowed — everything else to that Pod is denied, including traffic that would
have worked fine before the policy existed.
---
**4. Why is Ingress the right tool for exposing an HTTP application by hostname, rather than
creating a separate LoadBalancer Service per hostname?**
*Answer:* A LoadBalancer Service typically provisions its own external IP/load balancer per
Service — one per hostname would mean paying for and managing N separate external endpoints.
Ingress lets one entry point route to many backend Services by hostname/path, which is both
cheaper and centralizes concerns like TLS termination.
## Instructor Notes
**Facilitation:** Question 2 catches a very common and consequential misconception — worth
confirming out loud that everyone actually updated their mental model, not just guessed
correctly.
**Approximate timing:** 5 minutes.
@@ -0,0 +1,72 @@
# Demonstration: Tracing a Request Through Ingress
## The setup
An Ingress `web-ingress` routes `app.example.com/` to Service `web-svc`, which fronts a
Deployment `web-app`. Nothing works yet from outside the cluster — let's see why, step by step.
## Step 1: is the Ingress object even doing anything?
```
$ kubectl get ingress web-ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
web-ingress traefik app.example.com 80 2m
```
`ADDRESS` is empty. On a real cluster, this column fills in once the Ingress controller has
actually processed the object and assigned it a reachable address — an empty address after a
couple of minutes is the first sign to check the controller itself, not your YAML.
## Step 2: is the controller running at all?
```
$ kubectl get pods -n kube-system -l app.kubernetes.io/name=traefik
NAME READY STATUS RESTARTS AGE
traefik-7d8f9c6b5d-abcde 1/1 Running 0 10d
```
Running fine — so the ADDRESS delay in this case is transient, not a missing controller. (If
this returned no Pods at all, THAT would be the actual root cause — no controller means no
Ingress object ever gets acted on, exactly like the demonstration in ROOT 07's Ingress lesson
described.)
## Step 3: once ADDRESS populates, does the route actually work?
```
$ kubectl get ingress web-ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
web-ingress traefik app.example.com 10.0.1.50 80 3m
$ curl -H "Host: app.example.com" http://10.0.1.50/
<html>...web-app's real response...</html>
```
Success — but notice the `-H "Host: app.example.com"` header. Ingress routes by HOSTNAME, not
just by IP — hitting the Ingress controller's address WITHOUT the matching Host header would
route differently (or not at all), because that's literally what the Ingress rule is matching
on.
## The general shape
Symptom (nothing reachable from outside) has at least three DIFFERENT possible layers: the
Ingress object itself, the controller processing it, and the Service/Pod chain underneath it
(already covered in ROOT 07's Services module). Check them in that order — Ingress ADDRESS,
then controller health, then the underlying Service — rather than guessing at DNS or firewall
issues first.
## Instructor Notes
**Demo script:** If a real Ingress controller is available in the demo environment, run this
live including the Host-header nuance — students are frequently surprised that `curl <ip>`
alone doesn't work the way they expect.
**Approximate timing:** 8-10 minutes live; 5-7 minutes self-paced.
**Common misconception to preempt:** students often expect an Ingress's IP/address to work like
a Service's ClusterIP (reachable directly, no extra header needed). Reinforce that Ingress
routing is fundamentally HOSTNAME/PATH-based, which is exactly what makes one Ingress able to
route many different domains through one entry point.
**Optional extension:** ask what would happen if TWO Ingress objects both claimed the same host
+ path — a good forward pointer to why Ingress configuration conflicts are a real operational
concern on a shared cluster, without needing to build a hands-on lab around it here.
@@ -0,0 +1,47 @@
# Guided Exercise: Planning Network Isolation
Paper-and-thinking exercise before the Lab.
## The scenario
A `database` Pod (label `app=database`) in the `netpol` namespace should only ever be reachable
by Pods labeled `role=backend` — not the frontend, not anything else in the cluster.
## Your task
1. Before writing any YAML: right now, with NO NetworkPolicy in place, can the frontend reach
the database? Why?
2. What would a NetworkPolicy that ONLY specifies `podSelector: { matchLabels: { app: database
} }` and no `ingress` rules at all do to traffic reaching the database — including from
`role=backend` Pods?
3. Sketch (in words, not full YAML) what the `ingress` rule needs to say to allow ONLY
`role=backend` traffic in.
4. If a new Pod is later created with `role=backend` in a DIFFERENT namespace, would your policy
as sketched in #3 allow it? What would need to change if cross-namespace access should be
allowed?
## A worked answer to check yourself against
1. Yes — by default, every Pod can reach every other Pod regardless of labels or namespace
(ROOT 07's Ingress & NetworkPolicy lesson). Nothing restricts this until a policy exists.
2. It would block ALL incoming traffic to the database, including from `role=backend` — an
empty `ingress` list under a policy that selects the Pod means "no traffic is allowed in,"
not "no policy applies."
3. An `ingress` rule with `from: [{ podSelector: { matchLabels: { role: backend } } }]` — this
allows traffic specifically from Pods carrying that label, and nothing else.
4. Not automatically — a bare `podSelector` in the `from` clause matches Pods in the SAME
namespace as the policy by default. Allowing a different namespace requires adding a
`namespaceSelector` alongside the `podSelector` in the same `from` entry.
## Instructor Notes
**Facilitation:** Question 2 is the one most likely to trip students up — it directly reinforces
the "empty ingress rules means deny-all, not no-op" point from the lesson, which is worth
confirming actually landed before the Lab.
**Approximate timing:** 10-12 minutes live with discussion; 10 minutes self-paced.
**Optional extension:** ask what a security-minded team would ALSO want to restrict about this
database Pod's OWN outbound (egress) traffic, beyond who can reach it — a light forward pointer
to ROOT 09's security-control framing of NetworkPolicy, without requiring students to build an
egress rule here.
@@ -0,0 +1,40 @@
# Reflection: Reachability Should Be a Decision, Not a Default
You just configured real Ingress routing and real network isolation. Take a few minutes to
connect this to production thinking.
## Reflection questions
1. Before this module, would you have assumed a fresh Kubernetes cluster isolates Pods from
each other by default? Has that changed?
2. If your team runs a multi-tenant cluster (several unrelated applications sharing it), what's
the risk of NEVER applying any NetworkPolicy at all?
3. Ingress lets many different domains share one entry point. What's the operational benefit of
that beyond just "fewer objects to manage"?
## Real Ops takeaway
**"No NetworkPolicy" is not neutral — it's full reachability.** Exactly like an unhardened
Pod's SecurityContext (ROOT 09 covers this in depth), the absence of a NetworkPolicy isn't a
safe default; it's the MOST permissive configuration available. On any cluster where more than
one team or application shares infrastructure, that's a real, often-overlooked exposure.
**Centralizing external entry points through Ingress reduces attack surface, not just object
count.** Every separate LoadBalancer Service is a separate externally-reachable endpoint to
secure, monitor, and patch. Routing through one (or a small number of) Ingress controllers means
security controls — TLS termination, rate limiting, WAF rules — are configured and audited in
one place instead of scattered across every application team's own Service definitions.
This module is complete. Continue to the next module.
## Instructor Notes
**Discussion questions for a live classroom:** question 2 is worth pushing on with a concrete
scenario — ask what a compromised low-value application (a demo app, a staging environment)
could reach on a cluster with zero NetworkPolicies, versus one with sensible defaults.
**Approximate timing:** 8 minutes live discussion; 5 minutes self-paced writing.
**Optional extension:** ask students to sketch (in words) what a reasonable DEFAULT-DENY policy
for an entire new namespace would look like, before any application-specific allow rules are
added — previews the security-control framing ROOT 09 builds on.
@@ -0,0 +1,74 @@
# Ephemeral vs. Persistent Storage
## The default: a container's filesystem is thrown away
By default, anything a container writes to its own filesystem disappears the moment that
container is replaced — a crash, a rollout, a reschedule, anything. This is a deliberate design
choice (it's part of what makes a Pod safely disposable/replaceable), but it means "just write
files to disk" silently loses data unless you use one of Kubernetes' actual storage mechanisms.
## emptyDir: shared, but still temporary
An `emptyDir` volume gives a Pod's containers a shared directory that survives a container
restart WITHIN the same Pod, but is deleted the moment the Pod itself is removed. It's the right
tool for two containers in one Pod that need to share files with each other (a sidecar writing
logs another container reads, for example) — not for anything that needs to outlive the Pod.
```yaml
volumes:
- name: shared-data
emptyDir: {}
```
## PersistentVolume & PersistentVolumeClaim: storage that outlives the Pod
A **PersistentVolume (PV)** represents an actual piece of storage (a cloud disk, an NFS share,
whatever the cluster's infrastructure provides) as a Kubernetes object. A
**PersistentVolumeClaim (PVC)** is a Pod's REQUEST for storage matching some criteria (size,
access mode) — the Pod references the PVC, not a PV directly, and Kubernetes binds the PVC to a
matching PV.
```
Pod ──references──> PVC ("I need 5Gi, ReadWriteOnce") ──binds to──> PV (actual storage)
```
This indirection matters: a Pod's manifest doesn't need to know or care exactly which physical
storage backs its PVC — that's resolved separately, and the same PVC keeps working even if the
Pod behind it is replaced.
## StorageClass & dynamic provisioning
Manually creating a PV for every PVC doesn't scale. A **StorageClass** describes a category of
storage (e.g. "fast SSD" vs. "cheap archival") and, when a PVC requests that class, the cluster
can automatically provision a matching PV on demand — this is "dynamic provisioning," and it's
the normal way storage works on any real cloud-hosted cluster. A StorageClass also controls the
**reclaim policy** — what happens to the underlying storage after its PVC is deleted (`Delete`
the data, or `Retain` it for manual recovery).
## The decision in one pass
| You need... | Use |
|---|---|
| Temporary scratch space shared between containers in ONE Pod | `emptyDir` |
| Storage that survives Pod replacement, manually provisioned | PV + PVC |
| The above, provisioned automatically on demand | PVC + StorageClass (dynamic provisioning) |
## Instructor Notes
**Approximate timing:** 15 minutes read; 10 minutes live with the PV/PVC/StorageClass diagram.
**Likely misconceptions to watch for:**
- Students often assume `emptyDir` is "persistent" because the name doesn't obviously say
otherwise. Be explicit: it survives a CONTAINER restart, not a POD deletion.
- The PV/PVC indirection can seem like unnecessary ceremony at first — frame it as the same
"declare what you want, let the system resolve it" pattern from ROOT 05's desired-state
lesson, applied to storage instead of Pod count.
**Discussion question:** "If a Pod using a PVC is deleted and replaced, what happens to its
data?" (Answer: it's preserved — the PVC and its bound PV are separate objects from the Pod,
and a new Pod referencing the same PVC picks up right where the data was left, which is the
entire point of using persistent storage over `emptyDir`.)
**Optional extension:** ask what a `Retain` reclaim policy is protecting against, compared to
`Delete` (answer: accidental data loss from deleting a PVC by mistake — `Retain` keeps the
underlying storage around for manual recovery instead of immediately destroying it).
@@ -0,0 +1,44 @@
# Knowledge Check: Storage Fundamentals
---
**1. Two containers in the same Pod need to share files with each other, but nothing needs to
survive the Pod being replaced. What's the simplest correct choice?**
*Answer:* `emptyDir` — shared within the Pod, deliberately temporary, exactly matching the
requirement. Reaching for a PV/PVC here would be more machinery than the problem needs.
---
**2. A Pod using a PVC is deleted and a new Pod is created referencing the SAME PVC. What
happens to the data?**
*Answer:* It's preserved. The PVC and its bound PV exist independently of any specific Pod — the
new Pod picks up exactly where the data was left, which is the entire reason to use persistent
storage instead of `emptyDir` for anything that needs to outlive a single Pod's lifetime.
---
**3. True or false: a Pod's manifest must specify exactly which PersistentVolume backs its
storage request.**
*Answer:* False. A Pod references a PVC (a request), not a PV directly — the binding between
PVC and an actual PV is resolved separately, which is exactly the indirection that lets the same
Pod manifest work regardless of which physical storage ends up behind it.
---
**4. What does a StorageClass's reclaim policy actually control, and why would a team choose
`Retain` over `Delete` for a production database's storage?**
*Answer:* It controls what happens to the underlying storage after its PVC is deleted. `Retain`
keeps the data around for manual recovery instead of immediately destroying it — a safety
margin against an accidental `kubectl delete pvc` on something genuinely important, at the cost
of needing manual cleanup later.
## Instructor Notes
**Facilitation:** Question 2 is worth confirming lands correctly — it's the single fact that
justifies the entire PV/PVC indirection existing at all.
**Approximate timing:** 5 minutes.
@@ -0,0 +1,86 @@
# How the Scheduler Places Pods, and How Governance Shapes It
## The scheduler's job
Every time a new Pod needs to run, the scheduler decides which node it lands on — you never
manually assign a Pod to a node in normal operation (ROOT 05's mental model). By default, the
scheduler picks based on available resources (does this node have enough free CPU/memory), but
several mechanisms let you INFLUENCE or CONSTRAIN that decision when the default isn't good
enough.
## Steering placement: nodeSelector, taints/tolerations, affinity
- **nodeSelector** — the simplest constraint: "only schedule this Pod on nodes with THIS
label" (e.g. `disktype: ssd`). A blunt but clear tool.
- **Taints and tolerations** — the inverse relationship. A NODE is tainted (marked "don't
schedule here unless you specifically tolerate this taint") and a Pod TOLERATES a specific
taint to be allowed to land there anyway. Commonly used to reserve nodes for specific
workloads (e.g. GPU nodes tainted so only GPU-requiring Pods land there) or to keep general
workloads off nodes reserved for something else.
- **Affinity/anti-affinity** — more expressive rules: "prefer/require scheduling near Pods with
label X" (affinity) or "prefer/require scheduling AWAY from Pods with label X"
(anti-affinity). A common real use: anti-affinity between replicas of the same application, so
a single node failure can't take down every replica at once.
## Priority: which Pods matter more when resources are tight
A **PriorityClass** lets you mark some Pods as more important than others. When the cluster is
under resource pressure, a higher-priority pending Pod can even cause a lower-priority RUNNING
Pod to be evicted (preempted) to make room — this is a real, consequential mechanism, not just a
scheduling hint, and misconfiguring it can mean an unimportant batch job unexpectedly evicts
something that actually mattered.
## PodDisruptionBudget: protecting against VOLUNTARY disruption
A **PodDisruptionBudget (PDB)** limits how many replicas of an application can be taken down
AT ONCE during voluntary disruptions — a node drain for maintenance, a cluster upgrade — by
declaring a minimum number (or percentage) that must stay available. This does NOT protect
against involuntary disruption (a node crashing unexpectedly) — it specifically constrains
operations that Kubernetes itself controls the timing of.
```yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: web-pdb
spec:
minAvailable: 2
selector:
matchLabels: { app: web }
```
Without a PDB, a node drain during routine maintenance could legally take down every replica of
`web` simultaneously if they all happened to be on nodes being drained together — a PDB is what
prevents that from being "technically allowed."
## ResourceQuota & LimitRange: governance at the Namespace level
A **ResourceQuota** caps the TOTAL resources (CPU, memory, object counts) a Namespace can
consume — useful for multi-team clusters where one team's runaway workload shouldn't be able to
starve everyone else. A **LimitRange** sets DEFAULT and/or min/max resource requests/limits for
individual containers within a Namespace, so a Pod created without explicit resource settings
still gets sensible ones rather than none at all.
## Instructor Notes
**Approximate timing:** 20 minutes read; 15 minutes live given the number of concepts covered.
**Likely misconceptions to watch for:**
- Taints/tolerations are frequently confused with affinity — clarify the directionality: a
taint REPELS by default (node says no); a toleration is what lets a specific Pod through
anyway. Affinity is the Pod expressing a preference, not the node restricting access.
- Students sometimes assume a PDB prevents ALL disruption. Be explicit that it only constrains
VOLUNTARY disruption (things Kubernetes/an operator controls the timing of) — a hardware
failure ignores PDBs entirely, because nobody is "asking permission" for a crash.
**Discussion question:** "Why would a real team use anti-affinity between replicas of the same
application instead of just trusting the scheduler's default spread?" (Answer: the default
scheduler does consider spread as one factor among several, but doesn't GUARANTEE it the way an
explicit anti-affinity rule does — for something where "don't lose all replicas to one node
failure" is a hard requirement, not just a nice-to-have, an explicit rule is safer than an
implicit tendency.)
**Optional extension:** ask what would happen to a Namespace's Pods if a ResourceQuota were
added AFTER Pods without explicit resource requests already existed there (a good bridge to why
LimitRange defaults matter — a Quota alone doesn't retroactively fix Pods created without
resource requests, it just constrains what NEW Pods can consume going forward).
@@ -0,0 +1,48 @@
# Knowledge Check: Scheduling & Resource Governance
---
**1. A node is tainted `dedicated=gpu:NoSchedule`. A Pod with no matching toleration is
pending and never gets scheduled anywhere else, even though other untainted nodes have free
capacity. What's actually going on?**
*Answer:* Nothing is wrong — the taint only affects the tainted node specifically; it doesn't
explain why the Pod isn't landing elsewhere. If it's stuck pending entirely, the cause is
something else (insufficient resources cluster-wide, a nodeSelector/affinity rule that can't be
satisfied anywhere) — the taint itself only ever repels from the ONE node it's applied to.
---
**2. During a planned node drain for maintenance, would you expect a PodDisruptionBudget to
prevent an UNRELATED node from crashing unexpectedly at the same time?**
*Answer:* No. A PDB only constrains VOLUNTARY disruptions — operations Kubernetes/an operator
controls the timing of, like a drain. An unrelated hardware failure is involuntary and ignores
PDBs entirely; nothing is "asking permission" for a crash.
---
**3. True or false: a higher-priority pending Pod can cause a lower-priority, already-RUNNING
Pod to be evicted to make room.**
*Answer:* True — this is what Pod priority/preemption actually does under resource pressure,
not just a scheduling hint. Misconfigured priorities can have real, disruptive consequences for
this reason.
---
**4. A team notices new Pods in their Namespace keep getting created with no CPU/memory
requests at all, even though nobody set that intentionally. What governance mechanism would fix
this going forward, and what would it NOT fix?**
*Answer:* A LimitRange with default requests/limits would apply sensible defaults to future
Pods created without explicit values. It would NOT retroactively change Pods that already exist
— it only affects new Pod creation from the point it's added.
## Instructor Notes
**Facilitation:** Question 2 reinforces a genuinely important, often-missed distinction
(voluntary vs. involuntary disruption) — worth confirming it lands, not just that the answer was
guessed correctly.
**Approximate timing:** 5-7 minutes given the density of this module's lesson.
@@ -0,0 +1,87 @@
# Defense in Depth & the Kubernetes Security Model
## Security is a posture, not a single setting
There is no single "make it secure" switch in Kubernetes. Real security comes from **defense in
depth** — multiple independent layers, each reducing what an attacker (or a bug) can do if an
earlier layer fails. If a container has a vulnerability that lets an attacker execute arbitrary
code, the question that actually matters is: what can that code DO once it's running? A
container running as root, with every Linux capability, on a writable filesystem, with no
restrictions at all, can do far more damage than one deliberately hardened to need almost
nothing.
## Least privilege: grant only what's actually needed
The organizing principle behind everything in this course is **least privilege**: a workload
should have exactly the permissions and capabilities it needs to do its job, and nothing more.
This sounds obvious stated plainly, but the DEFAULT behavior of a plain container is closer to
"maximum privilege unless told otherwise" — root user, all standard Linux capabilities, a
writable filesystem. Hardening a Pod means deliberately narrowing that down.
## SecurityContext: where you actually configure this
`securityContext` (settable at the Pod level and per-container) is where you declare these
restrictions:
```yaml
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1000
containers:
- name: app
securityContext:
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
```
- **`runAsNonRoot` / `runAsUser`** — the container process runs as a specific non-root user
instead of root. If a container image was built to require root and you set this, it will
fail to start — which is a good, loud failure, not a silent security gap. Hardening
sometimes surfaces an image's OWN bad assumptions.
- **`capabilities: drop: ["ALL"]`** — Linux capabilities are fine-grained permissions
(`NET_BIND_SERVICE`, `SYS_ADMIN`, dozens more) a process can hold even without full root.
Dropping all of them, then adding back only the specific ones actually needed, is the
least-privilege pattern applied to capabilities specifically.
- **`readOnlyRootFilesystem: true`** — the container's own filesystem cannot be written to at
all (use an explicit `emptyDir` volume, ROOT 08, for anything that genuinely needs to write
temporary files). This closes off a whole category of attack that relies on writing a
malicious file to disk.
## Why this matters even when "nothing is currently wrong"
None of these settings fix a bug or a vulnerability — they limit the BLAST RADIUS if one exists
that you don't yet know about. A container hardened with non-root, dropped capabilities, and a
read-only filesystem gives a successful exploit dramatically less to work with than an
unhardened one, even against the exact same vulnerability. This is the entire point of defense
in depth: you don't rely on any single layer being perfect.
## Instructor Notes
**Approximate timing:** 15 minutes read; 10-12 minutes live with the demonstration.
**Demo plan:** Have the demonstration activity ready to run right after this lesson — seeing a
concrete before/after Pod definition lands the abstract "least privilege" principle far better
than the lesson text alone.
**Likely misconceptions to watch for:**
- Students sometimes think hardening settings are only relevant "if you expect to be attacked."
Reframe: defense in depth assumes SOMETHING will eventually go wrong somewhere (a
vulnerability, a bug, a misconfiguration) and prepares for that, rather than betting
everything on nothing ever going wrong.
- `runAsNonRoot: true` failing a Pod's startup is sometimes read as "the security setting is
broken." It's the opposite — it's correctly refusing to let a root-requiring image start,
which is valuable, actionable information about the image itself.
**Discussion question:** "If dropping all capabilities breaks a container that turns out to
genuinely need one specific capability, what's the right fix — removing the drop, or something
more targeted?" (Answer: add back only that ONE specific capability rather than removing the
blanket drop — least privilege means "exactly what's needed," not "give up and grant
everything again.")
**Optional extension:** ask students to identify one setting from this lesson that would have
made zero difference against a vulnerability that lets an attacker read environment variables
(a good check that they understand these are layered, not universal, protections — the answer
is that none of SecurityContext's settings here directly protect secrets stored in env vars;
that's a different concern, covered later in this course).
@@ -0,0 +1,87 @@
# Demonstration: A Pod Before and After Hardening
## Before: the default, unhardened Pod
```yaml
apiVersion: v1
kind: Pod
metadata:
name: app-unhardened
spec:
containers:
- name: app
image: myapp:latest
```
Checking what this actually runs as:
```
$ kubectl exec app-unhardened -- id
uid=0(root) gid=0(root) groups=0(root)
$ kubectl exec app-unhardened -- touch /etc/anything-i-want
$ kubectl exec app-unhardened -- ls -la /etc/anything-i-want
-rw-r--r-- 1 root root 0 Jan 1 00:00 /etc/anything-i-want
```
Running as root, with a fully writable filesystem, including system directories. This is not a
misconfiguration — it's simply what happens when nothing restricts it.
## After: the same application, hardened
```yaml
apiVersion: v1
kind: Pod
metadata:
name: app-hardened
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1000
containers:
- name: app
image: myapp:latest
securityContext:
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
```
```
$ kubectl exec app-hardened -- id
uid=1000 gid=0 groups=0
$ kubectl exec app-hardened -- touch /etc/anything-i-want
touch: /etc/anything-i-want: Read-only file system
```
Same application image, same functionality (assuming the app doesn't itself require root or
filesystem writes it wasn't told about) — but a compromise of this container now can't write
to the filesystem, can't use the elevated capabilities that were dropped, and never had root to
begin with.
## What this actually buys you
Nothing here fixed a bug in `myapp:latest` — the image is identical in both cases. What changed
is what an attacker (or a bug behaving unexpectedly) could DO if something inside that container
went wrong. The unhardened version had almost no limits; the hardened version has several
independent ones, any of which alone would have blocked the `touch` command above.
## Instructor Notes
**Demo script:** Run both Pods live if time allows — watching `touch` succeed on one and fail
on the other in real time is far more convincing than reading it.
**Approximate timing:** 8-10 minutes live; 5-7 minutes self-paced.
**Common misconception to preempt:** students sometimes read the hardened Pod's failed `touch`
as "hardening broke something." Reinforce: the FUNCTIONALITY didn't change (the application
itself never needed to write to `/etc`) — only an unnecessary, unused capability was removed.
If hardening breaks something the application genuinely needs, that's real information about
the application, not a reason to skip hardening.
**Optional extension:** ask what would happen if `myapp:latest` were built assuming it could
write a cache file to `/tmp` inside the container — a good bridge to the fact that
`readOnlyRootFilesystem` often needs an explicit `emptyDir` volume mounted at the specific path
an application legitimately needs to write to (ROOT 08), rather than being an all-or-nothing
choice.
@@ -0,0 +1,47 @@
# Knowledge Check: Security Model & SecurityContext
---
**1. A container is hardened with `capabilities: drop: ["ALL"]` and then fails because it
genuinely needs to bind to a low-numbered network port, which requires `NET_BIND_SERVICE`.
What's the least-privilege fix?**
*Answer:* Add back only `NET_BIND_SERVICE` specifically (`capabilities: add:
["NET_BIND_SERVICE"]` alongside the drop-all), not remove the blanket drop entirely.
Least privilege means granting exactly what's needed, not reverting to granting everything.
---
**2. True or false: setting `runAsNonRoot: true` fixes a vulnerability inside the application
itself.**
*Answer:* False. It doesn't fix anything about the application's code — it limits what a
compromise of that code could DO, by ensuring the process never has root privileges to begin
with. This is the defense-in-depth distinction: reducing blast radius, not eliminating bugs.
---
**3. Why might `runAsNonRoot: true` cause a Pod to fail to start entirely, and is that a
problem with the setting?**
*Answer:* If the container image was built assuming it runs as root (e.g. it tries to bind a
privileged port, or write to a root-owned directory), the Pod will fail to start under
`runAsNonRoot`. This is NOT a problem with the setting — it's valuable, actionable information
that the image itself makes an assumption that should probably be fixed.
---
**4. A container has `readOnlyRootFilesystem: true` and needs to write temporary cache files.
What's the correct way to allow that without abandoning the read-only filesystem entirely?**
*Answer:* Mount an explicit `emptyDir` volume (ROOT 08) at the specific path the application
needs to write to. This keeps the rest of the filesystem read-only while allowing the one
legitimately-needed writable location.
## Instructor Notes
**Facilitation:** Question 1 and question 4 both reinforce the same "narrow the exception, don't
remove the protection" pattern — worth calling out that they're the same principle applied
twice.
**Approximate timing:** 5-7 minutes.
@@ -0,0 +1,51 @@
# Guided Exercise: Planning a Pod Hardening Checklist
Paper-and-thinking exercise before the Labs.
## The scenario
You're handed a Pod spec for a small internal API with NO `securityContext` set at all — the
fully-default, unhardened case from the demonstration. Your team lead asks you to harden it
before it ships.
## Your task
Without writing the actual YAML yet, answer these:
1. List the settings you'd add, in the order you'd verify them (hint: there are at least three
distinct concerns from the lesson — user/root, filesystem, capabilities).
2. For each setting, name ONE way you'd notice if you'd broken the application by adding it
(what would the failure actually look like?).
3. If the application genuinely needs to write a cache file to disk, does that mean you should
skip `readOnlyRootFilesystem` entirely? What's the more precise fix?
4. Suppose after hardening, the container still runs fine but a security scan flags that it
still has the `NET_RAW` capability. What's the most likely reason, and what's the fix?
## A worked answer to check yourself against
1. In a reasonable order: (a) `runAsNonRoot`/`runAsUser` — a fundamental identity change,
surfaces image assumptions fastest; (b) `capabilities: drop: ["ALL"]` — narrow, usually safe
for a typical web API; (c) `readOnlyRootFilesystem: true` — most likely to need an
accompanying volume, so verify last once the others are confirmed working.
2. (a) Pod fails to start entirely (`CrashLoopBackOff` or an immediate error) if the image
assumed root. (b) The application errors on a specific operation that needed a capability
you dropped (e.g. failing to bind a privileged port). (c) A write operation fails at runtime
(a log line, an exception) the first time the app tries to write to disk.
3. No — the fix is mounting an explicit `emptyDir` volume at the specific path the cache needs,
keeping the rest of the filesystem read-only rather than abandoning the protection entirely.
4. Most likely: `capabilities: drop: ["ALL"]` was never actually applied (a YAML indentation
mistake, or it was added to the Pod's securityContext instead of the container's — capability
drops are container-level, not Pod-level). The fix is verifying the drop is actually present
on the CONTAINER's securityContext specifically.
## Instructor Notes
**Facilitation:** Question 4 is deliberately tricky — Pod-level vs. container-level
`securityContext` fields is a real, common source of "I set this and it didn't work" confusion
worth surfacing here before students hit it in a Lab.
**Approximate timing:** 15 minutes live with discussion; 12-15 minutes self-paced.
**Optional extension:** ask students to identify which of the settings in this exercise would
be MOST valuable to apply first if they could only add one to a large, existing fleet of
unhardened Pods under time pressure, and why.
@@ -0,0 +1,42 @@
# Reflection: Least Privilege Is a Default, Not an Afterthought
You just hardened real Pods across several Labs — non-root, dropped capabilities, a read-only
filesystem, and a full SecurityContext review. Take a few minutes to connect this to real
practice.
## Reflection questions
1. Before this module, would you have described "runs as root with a fully writable
filesystem" as insecure, or just as "the default, unremarkable configuration"? Has that
changed?
2. None of the settings you configured fixed a bug in any application. What did they actually
change, precisely?
3. If your team had to choose between hardening every new Pod by default (as a policy) versus
hardening Pods only after an incident reveals a gap, which produces less total risk over
time, and why?
## Real Ops takeaway
**Unhardened is not neutral — it's the maximum-privilege default.** A container with no
SecurityContext configured isn't "unopinionated"; it's root, every capability, fully writable —
the MOST privileged configuration possible, not a safe middle ground. Teams that treat hardening
as optional are choosing maximum blast radius by default, usually without realizing they're
choosing anything at all.
**Defense in depth means no single mistake should be catastrophic.** The value of stacking
non-root + dropped capabilities + read-only filesystem isn't that any ONE of them is perfect —
it's that a real compromise would have to defeat several independent restrictions simultaneously
instead of exploiting one gap that grants everything. This is exactly why security-minded teams
apply all of these by default rather than picking "the one that matters most."
This module is complete. Continue to the next module.
## Instructor Notes
**Discussion questions for a live classroom:** question 3 is worth pushing on directly — it's
the actual business case for security-by-default, not just a technical preference.
**Approximate timing:** 8-10 minutes live discussion; 5 minutes self-paced writing.
**Optional extension:** ask students to draft a one-paragraph "default Pod hardening policy"
their hypothetical team would adopt, based on what they've learned in this module.
@@ -0,0 +1,90 @@
# Identity for Workloads: ServiceAccounts & RBAC
## Workloads need an identity too, separate from yours
When YOU run `kubectl`, you're authenticated as yourself. But a Pod running INSIDE the cluster
that needs to call the Kubernetes API (to list other Pods, read a ConfigMap, whatever) needs its
own identity — it can't and shouldn't use your personal credentials. That identity is a
**ServiceAccount**.
Every Pod runs as SOME ServiceAccount — if you don't specify one, it uses the `default`
ServiceAccount in its Namespace, which by default has essentially no permissions. A Pod's
ServiceAccount token is automatically mounted into the container unless you explicitly disable
that (`automountServiceAccountToken: false`) — worth doing for any Pod that never actually needs
to call the Kubernetes API at all, since an unused, mounted credential is still something an
attacker could steal and use.
## RBAC: what that identity is actually allowed to do
Having an identity (a ServiceAccount) doesn't grant any permissions by itself — **RBAC
(Role-Based Access Control)** is the separate system that defines what an identity can DO.
- A **Role** (namespace-scoped) or **ClusterRole** (cluster-wide) defines a set of permissions:
which API resources, which verbs (get/list/watch/create/update/delete).
- A **RoleBinding** (or **ClusterRoleBinding**) connects a Role/ClusterRole to a specific
identity (a ServiceAccount, a user, or a group) — the Role alone grants nothing to anyone
until it's bound.
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: pod-reader
namespace: default
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: read-pods
namespace: default
subjects:
- kind: ServiceAccount
name: my-app
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.io
```
This grants the `my-app` ServiceAccount permission to `get`/`list` Pods in the `default`
Namespace — and nothing else. It cannot delete Pods, cannot touch Secrets, cannot do anything
not explicitly listed.
## Role vs. ClusterRole: scope matters
A Role only grants permissions within the ONE Namespace it's defined in. A ClusterRole can grant
permissions across the entire cluster (or, bound via a RoleBinding instead of a
ClusterRoleBinding, can also be reused namespace-by-namespace) — reach for a ClusterRole only
when the workload genuinely needs cluster-wide visibility, not by default.
## Least privilege, applied to permissions specifically
The same principle from the previous module applies directly here: grant the narrowest set of
verbs on the narrowest set of resources that the workload actually needs. A Role granting
`get`/`list` on Pods is very different from one granting `*` on `*` — the latter is
"root-equivalent" API access, and should be treated with exactly that level of caution.
## Instructor Notes
**Approximate timing:** 15 minutes read; 10 minutes live.
**Likely misconceptions to watch for:**
- Students often conflate "ServiceAccount" with "user account for logging in." A ServiceAccount
is specifically an identity for WORKLOADS, not a human login mechanism.
- The Role/RoleBinding two-step (define permissions, then separately grant them to someone) is
sometimes seen as redundant. Emphasize the reuse benefit: the same Role can be bound to
multiple identities without redefining the permission set each time.
**Discussion question:** "Why might you explicitly disable automounting a ServiceAccount token
for a Pod that never calls the Kubernetes API?" (Answer: an unused but present credential is
still something an attacker who compromises the container could steal and use elsewhere —
removing it entirely removes that risk for a Pod that never needed it.)
**Optional extension:** ask what the practical difference is between granting a ClusterRole via
a RoleBinding (scoped to one Namespace) versus a ClusterRoleBinding (cluster-wide) — a good
check that students understand the Role/ClusterRole and Binding-type distinctions are
independent axes, not the same choice.
@@ -0,0 +1,44 @@
# Knowledge Check: ServiceAccounts & RBAC
---
**1. A Pod is created with no ServiceAccount specified. What identity does it actually run as,
and what can that identity do by default?**
*Answer:* The `default` ServiceAccount in its Namespace, which by default has essentially no
permissions — it can authenticate to the API but is granted nothing until an RBAC binding says
otherwise.
---
**2. You create a Role granting `get`/`list` on Pods, but a workload still can't read Pods.
What's the most likely missing piece?**
*Answer:* A RoleBinding (or ClusterRoleBinding) connecting that Role to the workload's actual
ServiceAccount. A Role by itself grants nothing to anyone — it has to be bound to an identity.
---
**3. True or false: a ClusterRole can only ever be applied cluster-wide.**
*Answer:* False. A ClusterRole can be bound via a RoleBinding to apply within just ONE
Namespace (reusing a cluster-wide-DEFINED permission set in a namespace-SCOPED way), or via a
ClusterRoleBinding to apply cluster-wide. The Role-type and Binding-type are independent
choices.
---
**4. Why would a security review flag a Role granting `verbs: ["*"]` on `resources: ["*"]`
even if the workload "seems to work fine"?**
*Answer:* That grant is root-equivalent API access — it violates least privilege regardless of
whether the workload happens to only use a small subset of it today. "Works fine" doesn't mean
"correctly scoped" — an over-broad grant is a real risk the moment that workload (or its
credentials) is compromised, whether or not it's ever exploited in practice.
## Instructor Notes
**Facilitation:** Question 4 is worth discussing even if everyone answers correctly — the "it
works, so what's the problem" instinct is exactly what least-privilege review exists to counter.
**Approximate timing:** 5 minutes.
@@ -0,0 +1,62 @@
# Demonstration: Watching RBAC Actually Deny Something
## The setup
A ServiceAccount `readonly-bot` has a Role granting only `get`/`list` on `pods` — nothing else.
## What it CAN do
```
$ kubectl auth can-i list pods --as=system:serviceaccount:default:readonly-bot
yes
$ kubectl get pods --as=system:serviceaccount:default:readonly-bot
NAME READY STATUS RESTARTS AGE
web-abc123 1/1 Running 0 2h
```
## What it CANNOT do
```
$ kubectl auth can-i delete pods --as=system:serviceaccount:default:readonly-bot
no
$ kubectl delete pod web-abc123 --as=system:serviceaccount:default:readonly-bot
Error from server (Forbidden): pods "web-abc123" is forbidden:
User "system:serviceaccount:default:readonly-bot" cannot delete resource "pods"
in API group "" in the namespace "default"
```
This is what least privilege actually looks like in practice — not a policy document, a real,
enforced denial with a specific, readable error naming exactly which permission was missing.
`kubectl auth can-i` is your fastest way to CHECK a permission before attempting an action that
might fail — useful both for debugging "why can't my workload do X" and for verifying a Role is
actually as narrow as intended.
## What happens when the ServiceAccount tries something it was never even considered for
```
$ kubectl auth can-i get secrets --as=system:serviceaccount:default:readonly-bot
no
```
Notice this returns `no` the same way `delete pods` did — RBAC has no concept of "not yet
decided," only "explicitly allowed" or "everything else, denied." This is the practical meaning
of default-deny applied to permissions: nothing is grey area.
## Instructor Notes
**Demo script:** Run all three `kubectl auth can-i`/action pairs live — the FORBIDDEN error
message's exact wording (naming the verb, resource, and namespace) is worth reading aloud, since
that's exactly the evidence a student would use to debug a real permissions issue later.
**Approximate timing:** 8 minutes live; 5 minutes self-paced.
**Common misconception to preempt:** students sometimes think `kubectl auth can-i` REQUIRES the
action to actually be attempted to know the answer. It doesn't — it's a dry-run permission
check against the same RBAC rules, safe to run repeatedly without side effects.
**Optional extension:** ask what the error message would look like if `readonly-bot` tried to
act on a DIFFERENT namespace than the one its Role/RoleBinding applies to — a good check that
students understand Role scope is namespace-specific unless a ClusterRole+ClusterRoleBinding is
used instead.
@@ -0,0 +1,45 @@
# Guided Exercise: Writing a Least-Privilege Role
Paper-and-thinking exercise before the Labs.
## The scenario
A monitoring agent needs to run inside your cluster. Its ONLY job is to periodically list Pods
and read their status, across the whole cluster, to report health metrics elsewhere. It never
needs to create, modify, or delete anything, and it never needs to touch Secrets, ConfigMaps, or
any object type other than Pods.
## Your task
1. Should this use a Role or a ClusterRole? Justify your answer based on scope.
2. Which verbs does it actually need? Which common verbs (create, update, delete, patch) should
be deliberately left OUT?
3. Does it need access to Secrets? Why is it tempting to grant broader access "just in case,"
and why should you resist that?
4. What would you check, using `kubectl auth can-i`, to CONFIRM your Role is exactly as narrow
as intended before deploying it?
## A worked answer to check yourself against
1. **ClusterRole** (bound via a ClusterRoleBinding) — it needs to see Pods across the WHOLE
cluster, not just one namespace, so a namespace-scoped Role can't satisfy the requirement.
2. **`get`, `list`, `watch`** on `pods` — read-only verbs only. Deliberately exclude `create`,
`update`, `patch`, `delete` — this workload never legitimately needs to change anything.
3. No — it never touches Secrets, so it should get zero permission on them. The temptation to
"just in case" grant broader access is exactly what least privilege exists to resist: every
unused permission is pure risk with no corresponding benefit if this workload is ever
compromised.
4. Run `kubectl auth can-i delete pods --as=<the ServiceAccount>` and confirm it returns `no` —
and similarly check a Secrets verb returns `no` — proving the Role is bounded, not just
assuming it from reading the YAML.
## Instructor Notes
**Facilitation:** Question 3 is the core judgment call this exercise is testing — push on WHY
"just in case" access is a real anti-pattern, not just point out that it is one.
**Approximate timing:** 10-12 minutes live with discussion; 10 minutes self-paced.
**Optional extension:** ask what would change about the answer if this monitoring agent later
needed to also restart crash-looping Pods automatically — a good check that students can reason
about incrementally expanding a Role's scope only as far as a genuinely new requirement demands.
@@ -0,0 +1,42 @@
# Reflection: Permissions Should Reflect What a Workload Actually Needs
You just configured real ServiceAccounts and RBAC rules. Take a few minutes to connect this to
production security practice.
## Reflection questions
1. Before this module, would you have thought twice about a workload's ServiceAccount having
broad, unused permissions "just in case"? Has that changed?
2. If a workload with cluster-admin-equivalent RBAC access is ever compromised (a vulnerable
dependency, a supply-chain issue), what's the realistic blast radius compared to one scoped
to exactly what it needs?
3. How would you discover, on a real cluster you didn't build, whether any ServiceAccount has
more access than it actually uses?
## Real Ops takeaway
**RBAC over-permissioning is a silent, cumulative risk.** Unlike a misconfigured probe or a
crashing Pod, an over-broad Role doesn't cause a visible symptom on a healthy day — it just sits
there as unused risk until the one day a workload using it is compromised. This makes RBAC
review a discipline that has to be practiced deliberately, not one that gets forced on you by
an obvious failure.
**Least privilege is a default you set once, not a fix you apply after an incident.** Every
example in this module started from "grant nothing, add exactly what's needed" rather than
"grant broadly, restrict later" — the second approach almost never actually happens once
something works, because there's no pressure forcing the cleanup.
This module is complete. Continue to the next module.
## Instructor Notes
**Discussion questions for a live classroom:** question 3 is a good one to leave partially open
— acknowledge that real answers involve tooling beyond plain `kubectl` (RBAC audit tools) that
this course doesn't build hands-on, while still reinforcing that the QUESTION itself is the
right one to ask.
**Approximate timing:** 8 minutes live discussion; 5 minutes self-paced writing.
**Optional extension:** ask students to identify one ServiceAccount permission from earlier in
ROOT_ (any Lab so far) that, in a real production system, they'd want to double-check was
actually necessary.
@@ -0,0 +1,76 @@
# NetworkPolicy as a Security Control & Supply Chain Trust
## NetworkPolicy, revisited as a security posture
ROOT 07 introduced NetworkPolicy architecturally: Kubernetes doesn't isolate Pod-to-Pod traffic
by default, and NetworkPolicy is how you declare intended reachability. Here, the same
mechanism is the point of a deliberate SECURITY practice: **default deny**.
A default-deny policy blocks all traffic to selected Pods, and you then explicitly allow only
what's actually needed:
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
spec:
podSelector: {} # selects EVERY Pod in the namespace
policyTypes: ["Ingress"]
```
An empty `podSelector: {}` with no ingress rules means "select every Pod, allow nothing in" —
the network equivalent of the SecurityContext principle from earlier in this course: start
from nothing, add back only what's proven necessary. This is exactly the opposite of
Kubernetes' un-configured default (everything open), which is precisely why it has to be
declared explicitly rather than relying on any built-in setting.
**Egress restriction** works the same way in reverse — restricting what a Pod can call OUT to,
not just what can call IN. A compromised Pod that can freely make outbound connections to
anything is far more dangerous (data exfiltration, calling out to an attacker's server) than one
restricted to only the specific destinations it legitimately needs.
## Supply chain: trusting what you run before it ever starts
Everything covered so far in this course assumes the container image itself is trustworthy.
Supply-chain security is about that assumption specifically:
- **Image-pull secrets** — when pulling from a private registry, the cluster needs credentials
to authenticate. Storing and using these correctly (rather than, say, a publicly-readable
registry with no access control) is a basic supply-chain control: only images from a trusted,
access-controlled source get pulled at all.
- **Minimizing image footprint** (referenced here, not a hands-on Lab in this module) — a
smaller image means fewer packages, fewer known CVEs, and a smaller attack surface if the
image is ever compromised.
## TLS at the Ingress layer: protecting data in transit
Recall ROOT 07's Ingress lesson — routing external HTTP(S) traffic by hostname/path. Configuring
TLS termination at that same Ingress layer means traffic between users and your cluster is
encrypted, protecting against interception on the network path. This is a security concern
layered on top of the same architectural mechanism, the same relationship NetworkPolicy has to
its ROOT 07 introduction.
## Instructor Notes
**Approximate timing:** 15 minutes read; 10 minutes live.
**Likely misconceptions to watch for:**
- Students sometimes think "default deny" means blocking ALL traffic including what the
application needs to function. Clarify: default deny is the STARTING point; you then add
specific allow rules for exactly what's needed — the goal state is fully functional AND fully
intentional, not non-functional.
- Egress restriction is sometimes seen as less important than ingress. Emphasize the
exfiltration angle explicitly — a compromised Pod's outbound reach is just as much a real
risk as inbound exposure.
**Discussion question:** "Why is 'default deny, then explicitly allow' a better starting
posture than 'default allow, then explicitly block known-bad traffic'?" (Answer: default-deny
means a NEW, unanticipated attempt to reach or leave a Pod is blocked by default; default-allow
means anything not YET identified as bad gets through — you're always one step behind an
attacker under the second model.)
**Optional extension:** ask what a real team would want to verify before trusting a
publicly-available container image from an unfamiliar source, beyond just "does it work" — a
good bridge into image scanning/provenance concepts (not built as a hands-on Lab in this course,
but worth naming as a real practice).
@@ -0,0 +1,45 @@
# Knowledge Check: Network & Supply Chain Security
---
**1. A NetworkPolicy has `podSelector: {}` with `policyTypes: ["Ingress"]` and no ingress rules
defined. What's the practical effect on every Pod in that namespace?**
*Answer:* Every Pod is selected, and since no ingress rule allows anything, all incoming traffic
to every Pod is denied — this is the "default deny" starting posture, applied namespace-wide.
---
**2. Why is restricting EGRESS (outbound) traffic from a Pod a real security control, not just
a mirror-image nicety of ingress restriction?**
*Answer:* A compromised Pod with unrestricted egress can exfiltrate data to an attacker-controlled
destination or call out to download additional malicious payloads. Restricting egress to only
what's legitimately needed limits what a compromise can accomplish, independent of whether
inbound access was ever an issue.
---
**3. True or false: "default deny, then explicitly allow" and "default allow, then explicitly
block known-bad traffic" provide equivalent security given the same set of rules.**
*Answer:* False. Default-deny blocks anything NOT yet explicitly allowed, including new/
unanticipated attempts. Default-allow only blocks what's already been identified as bad — an
attacker doing something not yet on the block list gets through under the second model but not
the first.
---
**4. Why does minimizing a container image's footprint (fewer packages) improve supply-chain
security specifically, rather than just making the image smaller/faster to pull?**
*Answer:* Fewer installed packages means fewer potential known vulnerabilities (CVEs) present in
the image at all, and a smaller attack surface if the image is ever compromised — it's a
security property, not merely a performance optimization.
## Instructor Notes
**Facilitation:** Question 3 is the core conceptual point of this module — worth confirming it's
genuinely understood, not just correctly guessed.
**Approximate timing:** 5-7 minutes.
@@ -0,0 +1,89 @@
# Demonstration: Default-Deny, Before and After
## Before: everything can reach everything
```
$ kubectl run test-pod --image=busybox --rm -it -- wget -qO- --timeout=2 http://backend-svc
<real response from backend>
```
Any Pod in the namespace can reach `backend-svc`, regardless of what it is or who deployed it —
this is the un-configured default the ROOT 07 lesson described, now made concrete from a
security angle.
## Applying a default-deny policy
```
$ kubectl apply -f default-deny.yaml
networkpolicy.networking.k8s.io/default-deny created
```
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
spec:
podSelector: {}
policyTypes: ["Ingress"]
```
## After: the same request, now blocked
```
$ kubectl run test-pod --image=busybox --rm -it -- wget -qO- --timeout=2 http://backend-svc
wget: download timed out
```
No error naming a permission (that's RBAC's job, a different layer) — just a timeout, because
the packets are being dropped at the network layer before they ever reach the Pod. This is a
DIFFERENT failure signature than an RBAC denial, worth being able to tell apart: RBAC denials
are explicit and immediate; NetworkPolicy denials look like the destination is simply
unreachable.
## Explicitly allowing only what's needed
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-frontend-to-backend
spec:
podSelector:
matchLabels: { app: backend }
ingress:
- from:
- podSelector:
matchLabels: { app: frontend }
```
```
$ kubectl run frontend-test --labels="app=frontend" --image=busybox --rm -it -- wget -qO- --timeout=2 http://backend-svc
<real response from backend>
$ kubectl run other-test --labels="app=other" --image=busybox --rm -it -- wget -qO- --timeout=2 http://backend-svc
wget: download timed out
```
Same destination, same request — the ONLY difference is the caller's label. This is
least-privilege network access made concrete: reachability now reflects an explicit decision,
not an accident of both Pods existing in the same cluster.
## Instructor Notes
**Demo script:** Run all three states live if a real cluster is available — the timeout (not an
error) for the denied case is worth calling out explicitly, since it looks identical to a
genuinely broken/unreachable service from the caller's perspective.
**Approximate timing:** 8-10 minutes live; 5-7 minutes self-paced.
**Common misconception to preempt:** students sometimes expect a NetworkPolicy denial to
produce a clear "forbidden" error like RBAC does. It doesn't — reinforce that a NetworkPolicy-
caused timeout is genuinely hard to distinguish from an actual network/DNS problem without
already suspecting NetworkPolicy, which is exactly why checking existing policies is a real
diagnostic step worth knowing.
**Optional extension:** ask how a student would distinguish "blocked by NetworkPolicy" from
"the backend Pod is actually down" purely from the caller's side (answer: check the backend
Pod's own status/logs directly — if it's healthy and Ready, but still unreachable from one
caller and reachable from another with a different label, NetworkPolicy is the likely cause).
@@ -0,0 +1,51 @@
# Guided Exercise: Planning Egress Restriction
Paper-and-thinking exercise before the Labs.
## The scenario
A `payment-processor` Pod handles sensitive transactions. It legitimately needs to reach:
- an internal `payments-db` Service, and
- one external payment gateway API.
It should NOT be able to reach anything else — including the public internet in general, other
internal services, or any other namespace.
## Your task
1. Why is restricting this Pod's OUTBOUND (egress) traffic just as important as restricting who
can reach IT (ingress), given what this Pod handles?
2. If this Pod were compromised via a vulnerable dependency, what would UNRESTRICTED egress let
an attacker do that restricted egress would prevent?
3. Sketch (in words) what an egress policy for this Pod needs to allow, and — just as
important — what it must NOT allow by omission.
4. Why might a security review flag a Pod with `policyTypes: ["Egress"]` but an EMPTY `egress`
list as suspicious, rather than assume it's simply "not configured yet"?
## A worked answer to check yourself against
1. A Pod handling sensitive data that's compromised is far more dangerous if it can freely
exfiltrate that data anywhere (unrestricted egress) than if its only reachable destinations
are pre-approved and narrow.
2. Unrestricted egress would let an attacker exfiltrate transaction data to an external server,
or download additional malicious tooling. Restricted egress limits it to only the
pre-approved destinations, even after compromise.
3. Allow: egress to `payments-db` (internal, by label/namespace selector) and the specific
external payment gateway (by IP range or, if supported, a domain-based egress rule). Must NOT
allow: general internet access, other internal Services, or any other namespace by default.
4. An empty `egress` list under `policyTypes: ["Egress"]` means DENY ALL outbound — exactly like
the ingress case from the lesson. A reviewer seeing this should confirm it's intentional
(a Pod that genuinely needs zero outbound network access) rather than assume it's an
oversight, since the two look identical in the YAML.
## Instructor Notes
**Facilitation:** Question 4 reinforces the same "empty rules list means deny-all, not
unconfigured" point from earlier lessons — a good moment to confirm it generalizes correctly to
egress, not just ingress.
**Approximate timing:** 10-12 minutes live with discussion; 10 minutes self-paced.
**Optional extension:** ask what additional control (beyond NetworkPolicy) a team might want for
truly sensitive egress, like restricting DNS resolution itself for this Pod — a forward pointer
to defense-in-depth without requiring a hands-on Lab here.
@@ -0,0 +1,41 @@
# Reflection: Reachability and Trust Are Decisions You Make on Purpose
You just applied default-deny NetworkPolicy, egress restriction, TLS ingress, and image-pull
authentication for real. Take a few minutes to connect these to a single underlying principle.
## Reflection questions
1. Four different mechanisms in this module (NetworkPolicy ingress, NetworkPolicy egress, TLS,
image-pull secrets) all express the same underlying idea in a different layer. What is that
idea, stated in one sentence?
2. If you had to prioritize implementing ONLY ONE of this module's controls on a brand-new,
unsecured cluster, which would you pick first, and why?
3. How would you explain, to someone who thinks "our cluster is internal, so it doesn't need
this," why internal-only still benefits from all four controls?
## Real Ops takeaway
**Trust should be explicit, narrow, and verifiable — never assumed from proximity.** A Pod on
the same cluster, a request from inside the corporate network, an image from "a registry we
usually use" — none of these are actually trustworthy by default. Every control in this module
replaces an assumption ("it's internal, it's probably fine") with an explicit, checkable
decision (this specific Pod can reach this specific destination; this specific registry is
authenticated; this specific connection is encrypted).
**Supply-chain and network security compound.** A compromised image with unrestricted egress on
an open network is a far worse incident than the same compromised image on a hardened one — the
controls in this module don't individually prevent every incident, but together they
dramatically shrink what any single failure can turn into.
This module is complete. Continue to the next module.
## Instructor Notes
**Discussion questions for a live classroom:** question 1 is the module's thesis — make sure
students can actually articulate it in their own words, not just recognize it when stated.
**Approximate timing:** 8-10 minutes live discussion; 5 minutes self-paced writing.
**Optional extension:** ask students to rank this module's four controls by how much
IMPLEMENTATION EFFORT each typically requires on a real team, separately from security value —
a good exercise in real-world prioritization tradeoffs.
@@ -0,0 +1,64 @@
# Pod Security Admission & a Survey of Runtime Hardening
This module is deliberately a survey, not a deep dive — the topics here are real and important,
but lean toward cluster-operator/platform-team concerns rather than the daily-driver skills most
ROOT_ learners will use constantly. Treat the labs in this module as good-to-know, not required
mastery, before moving on.
## Pod Security Admission: enforcing a baseline across a Namespace
Everything you configured by hand in earlier modules (non-root, dropped capabilities, read-only
filesystem) can also be ENFORCED automatically at the Namespace level via **Pod Security
Admission (PSA)** — a label on the Namespace itself that rejects any Pod violating a chosen
security standard (`privileged`, `baseline`, or `restricted`, from least to most strict):
```yaml
apiVersion: v1
kind: Namespace
metadata:
name: production
labels:
pod-security.kubernetes.io/enforce: restricted
```
This replaces the older PodSecurityPolicy (PSP) mechanism, which was removed from Kubernetes —
if you encounter PSP in older documentation or an older cluster, know that PSA is its current
replacement.
## Immutable Secrets: preventing a class of accidental change
A Secret can be marked `immutable: true`, which prevents its data from being changed after
creation (a new Secret with a new name has to be created instead, and workloads updated to
reference it). This trades a small amount of operational flexibility for a real safety
property: an accidental or malicious edit to a live Secret can't silently happen.
## A survey of further topics (labs available, treat as optional depth)
- **Audit policy** — configuring what the Kubernetes API server logs about requests made against
it (who did what, when) — genuinely important for incident investigation, but a cluster-admin
concern, not something most workload teams configure themselves.
- **RuntimeClass** — lets specific Pods run under a different container runtime configuration
(e.g. an extra-sandboxed runtime like gVisor or Kata Containers) for workloads that need
stronger isolation than the default runtime provides — relevant mainly for genuinely
untrusted or high-risk workloads, not typical applications.
## Instructor Notes
**Approximate timing:** 15 minutes read (shorter than other lessons — deliberately survey-depth).
**Framing for instructors:** Be explicit with students that this module is intentionally
lighter than the earlier ones in this course — the goal is awareness ("this exists, here's
roughly what it's for") rather than hands-on fluency, consistent with not every CKS-style topic
belonging in the core learning path.
**Discussion question:** "Why would a platform team prefer enforcing security standards via
Pod Security Admission at the Namespace level, instead of trusting every team to configure
SecurityContext correctly on every Pod by hand?" (Answer: PSA provides a hard backstop —
correctly-configured SecurityContext relies on every individual Pod author doing it right every
time; PSA rejects violations automatically regardless of whether any individual author
remembered to.)
**Optional extension:** ask which of `privileged`/`baseline`/`restricted` a team should start a
brand-new, security-conscious project with, and why not jump straight to the strictest option
on a cluster already running older, unaudited workloads (a good practical judgment question
about migration risk, not just definitions).
@@ -0,0 +1,86 @@
# Why Health Checks Exist & Probe Types
Kubernetes cannot see inside your application. It doesn't know if your process is deadlocked,
stuck in an infinite loop, or quietly failing every request while the process itself stays
alive. Without a way to ask "are you actually working?", the only signal Kubernetes has is
whether the container's process is still running — and a hung process still counts as running.
**A health probe is how your application tells Kubernetes the truth.** It's a small check —
an HTTP request, a TCP connection attempt, or a command run inside the container — that
Kubernetes runs on a schedule and uses to make real decisions: restart this container, or stop
sending it traffic.
## The three probe types
Kubernetes supports three distinct probes, and they answer three different questions. Mixing
them up is the single most common probe mistake in production.
**Liveness probe — "Is this container still alive, or should it be restarted?"**
If a liveness probe fails repeatedly, the kubelet kills the container and it gets restarted per
its restart policy. Use this for detecting a genuinely stuck process — a deadlock, a crashed
event loop. Don't use liveness to mean "not ready for traffic yet" — that's what readiness is
for, and confusing the two causes the exact incident you're about to fix.
**Readiness probe — "Is this container ready to receive traffic right now?"**
If a readiness probe fails, the Pod is removed from the Service's list of endpoints —
Kubernetes stops routing traffic to it, but does **not** restart the container. This is the
right tool for "still starting up," "warming a cache," or "temporarily overloaded, back off for
a bit." A Pod can be Running and healthy but still correctly Not Ready.
**Startup probe — "Has this slow-starting container finished starting up yet?"**
Some applications take a long time to become ready (loading a large dataset, running
migrations). A startup probe disables liveness/readiness checks until it succeeds once, so a
slow-but-healthy startup doesn't get killed by an impatient liveness probe. Once it succeeds,
liveness and readiness take over as normal.
## How a probe is actually defined
Every probe has a **mechanism** (how the check runs) and a **schedule** (how often, how long to
wait, how many failures before acting):
- `httpGet` — a GET request to a path and **port** on the container. A 2xx or 3xx response
counts as success.
- `tcpSocket` — a TCP connection attempt to a **port**. A successful connection counts as
success — this doesn't check application-level health, just "is something listening."
- `exec` — runs a command inside the container. Exit code `0` counts as success.
Timing fields (`initialDelaySeconds`, `periodSeconds`, `timeoutSeconds`, `failureThreshold`,
`successThreshold`) control how patient Kubernetes is before it acts — but no amount of correct
timing fixes a probe that's pointed at the wrong port or path in the first place.
## Common production mistakes
1. **Wrong port.** The probe's `httpGet.port` (or `tcpSocket.port`) doesn't match the port the
application actually listens on. The container is completely healthy — Kubernetes is just
asking the wrong door to open. This is the exact failure you'll diagnose in this module's lab.
2. **Liveness probe too aggressive on a slow-starting app**, with no startup probe — the app
gets killed for "not being ready yet," restarts, and never gets far enough to pass.
3. **Readiness probe that always passes** (e.g. checking a static file instead of real
dependencies) — traffic gets routed to a Pod that can't actually serve it.
4. **No probes configured at all** — Kubernetes has no way to know the container is stuck, so a
deadlocked process just sits there serving nothing, forever, looking "Running" the whole time.
## Instructor Notes
**Approximate timing:** 12-15 minutes for a live read-through with pauses for questions;
10 minutes for a self-paced read.
**Demo plan:** Before moving to the guided exercise, it's worth live-editing one Deployment's
`readinessProbe.httpGet.port` in front of the class to an obviously wrong value and running
`kubectl get pods` / `kubectl describe pod` together, so students see the Ready column and the
Events section change in real time before they do it themselves.
**Likely misconceptions to watch for:**
- Students often assume a failing liveness probe and a failing readiness probe look the same in
`kubectl get pods` — they don't (liveness failures show restart counts climbing; readiness
failures show `0/1` Ready with no restarts). Worth calling out explicitly.
- Students sometimes think `tcpSocket` checks are "less safe" than `httpGet` — reframe as
"checks a different, shallower thing," not "worse."
**Discussion question:** "Why might a team deliberately choose a readiness probe that fails
under high load, even though it takes healthy capacity out of rotation?" (Answer: shedding load
from an overloaded Pod is often better than serving every request slowly or with errors.)
**Optional extension:** Ask students to sketch what would happen if liveness and readiness
probes were swapped (readiness probe restarts on failure, liveness probe just removes from
Service) — this clarifies the distinction by making the wrong version visibly worse.
@@ -0,0 +1,71 @@
# Demonstration: Probes in Action
Watch what a probe failure actually looks like from the outside, before you're the one
diagnosing it.
Picture a Deployment called `web` running `nginx`, which listens on port 80. Someone adds a
readiness probe but mistypes the port as `8080`:
```yaml
readinessProbe:
httpGet:
path: /
port: 8080
initialDelaySeconds: 3
periodSeconds: 5
```
**What you'd see immediately after this rolls out:**
```
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
web-7d8f9c6b5d-abcde 0/1 Running 0 45s
```
`STATUS` says `Running` — the container started fine, nginx is up. But `READY` says `0/1` — the
readiness probe is failing, so this Pod has never been added to its Service's endpoints. No
traffic reaches it. `RESTARTS` stays at `0` because this is a readiness failure, not a liveness
failure — nothing is being killed, it's just being excluded.
**The trail that explains why**, from `kubectl describe pod`:
```
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 50s default-scheduler Successfully assigned default/web-... to node-1
Normal Pulled 49s kubelet Container image "nginx:1.25" already present on machine
Normal Created 49s kubelet Created container web
Normal Started 49s kubelet Started container web
Warning Unhealthy 15s (x6 over 40s) kubelet Readiness probe failed: Get "http://10.244.0.7:8080/": dial tcp 10.244.0.7:8080: connect: connection refused
```
That last line is the whole story: kubelet tried to connect to port `8080` and nothing was
listening there — because nginx is listening on `80`. The container was never broken. The probe
was asking the wrong question.
**The general diagnostic shape** for any probe issue: `kubectl get pods` tells you *that*
something is wrong (Ready count, restart count). `kubectl describe pod` and its Events section
tell you *what* is wrong, usually in plain English, including the exact port/path that was
tried and the exact error. Reach for Events before logs — logs show you what the application
did; Events show you what Kubernetes tried to do and how it failed.
## Instructor Notes
**Demo script:** Live-run this exact sequence against a throwaway Deployment (or the classroom
projector) rather than only showing static output — `kubectl get pods`, then
`kubectl describe pod <name>`, scrolling straight to Events. Ask the class to predict the
`READY` column and `RESTARTS` count *before* you show it.
**Approximate timing:** 8-10 minutes live; ~5 minutes as a self-paced read.
**Common misconception to preempt:** students often reach for `kubectl logs` first when
something isn't Ready. For a probe misconfiguration, logs are frequently silent or unhelpful —
the application is healthy and has nothing to say. This is a good moment to reinforce
"Events explain what Kubernetes tried; logs explain what the app did," and that reading Events
first is usually faster for this class of failure.
**Optional extension:** show the same failure signature for a `tcpSocket` probe (a bare
"connection refused" with no HTTP verb in the Events message) so students can tell the two
mechanisms apart from Events alone.
@@ -0,0 +1,56 @@
# Knowledge Check: Health Probes
Answer each question yourself before reading the answer that follows it. This isn't graded —
it's a checkpoint to catch a misunderstanding before you're troubleshooting a live Pod in the
next activity.
---
**1. A Pod shows `RESTARTS: 12` and `READY: 1/1`. What kind of probe failure, if any, is most
consistent with this?**
*Answer:* A liveness probe that was failing and has now recovered (or is intermittently
failing) — the climbing restart count is the signature of liveness failures, since a failed
liveness check causes the kubelet to kill and restart the container. `READY: 1/1` means it's
currently passing its readiness check right now.
---
**2. A Pod shows `RESTARTS: 0` and `READY: 0/1`, and has looked this way for several minutes.
What kind of probe failure is most consistent with this?**
*Answer:* A readiness probe that is failing. The container is running fine (no restarts), but
Kubernetes has decided it isn't ready to receive traffic and has excluded it from the Service's
endpoints. This is exactly the symptom you'll investigate in this module's lab.
---
**3. Why does Kubernetes offer a separate startup probe instead of just making
`initialDelaySeconds` on the liveness probe longer?**
*Answer:* `initialDelaySeconds` is a fixed, one-size-fits-all wait before checks even begin —
set it too short and a normally-slow startup gets killed; set it too long and every fast
startup pays that same delay before liveness/readiness protection is even active. A startup
probe instead keeps checking until the application actually reports ready (bounded by its own
failure threshold), so liveness/readiness only take over once startup is confirmed done —
correct for both a consistently slow app and one whose startup time varies.
---
**4. True or false: a readiness probe failure will cause Kubernetes to restart the container.**
*Answer:* False. A readiness probe failure only removes the Pod from Service endpoints — it
does not restart anything. Only a liveness (or startup, before it succeeds once) probe failure
causes a restart.
## Instructor Notes
**Facilitation:** In a live classroom, ask these out loud and take verbal answers before
revealing each one, rather than having students read this silently — the goal is to surface a
wrong mental model while it's still cheap to correct.
**Approximate timing:** 5 minutes.
**Likely misconception this catches:** question 2 specifically catches students who default to
"something is broken, kubectl logs will show an error" — for a pure readiness-probe
misconfiguration, logs are usually clean, which is disorienting the first time you hit it.
@@ -0,0 +1,61 @@
# Guided Exercise: Configure a Readiness Probe
This is a paper-and-thinking exercise, not a Lab task — you won't run these commands against a
real cluster here (that comes next). The goal is to practice reading and writing a probe
definition correctly before you're diagnosing someone else's broken one.
## The scenario
You're deploying a small HTTP API that listens on port `3000` and exposes a health endpoint at
`/healthz`. It takes a couple of seconds to finish connecting to its database on startup — if a
request arrives before that finishes, it correctly returns a `503` at `/healthz` until it's
ready.
## Your task
Work out, on paper (or in a scratch file), what you'd write for this container's readiness
probe. Think through:
1. Which mechanism fits — `httpGet`, `tcpSocket`, or `exec`? (Hint: you have a real health
endpoint that returns a meaningful status code — use it rather than a shallower check.)
2. What `port` and `path` should the probe target?
3. Roughly what `initialDelaySeconds` makes sense, given the database connection takes a couple
of seconds?
4. Should this be a `readinessProbe`, a `livenessProbe`, or both? Justify your answer using the
distinction from the lesson — what actually goes wrong if you get this specific choice wrong
here?
## A worked shape to check yourself against
Your answer should end up structurally similar to this (exact field names matter more than the
precise numbers you chose):
```yaml
readinessProbe:
httpGet:
path: /healthz
port: 3000
initialDelaySeconds: 3
periodSeconds: 5
```
If you reached for `livenessProbe` instead of `readinessProbe` for this specific scenario,
re-read the lesson's distinction before continuing — "returns 503 while warming up" is
precisely the readiness case, not the liveness case. A liveness probe here would restart a
perfectly healthy, still-starting container over and over, which never lets it finish starting.
## Instructor Notes
**Facilitation:** Have students write their probe block on paper or a shared doc before showing
the worked shape — resist showing the answer early, since the value here is in students
committing to an answer and comparing, not passively reading a correct one.
**Approximate timing:** 10-15 minutes live with discussion; 10 minutes self-paced.
**Common mistake to watch for:** choosing `livenessProbe` because "the app isn't healthy yet"
— this is the single most common real-world probe mistake and exactly what question 4 is
designed to surface before the next activity's real diagnostic task.
**Optional extension:** ask what would need to change in this same YAML if the health endpoint
took up to 60 seconds to become available on cold start (answer: a `startupProbe` with a
generous `failureThreshold` × `periodSeconds`, so liveness/readiness don't engage too early).
@@ -0,0 +1,47 @@
# Launch Lab: Stuck Not-Ready: The Probe Port Mismatch
Time to apply everything from this module against a real, running incident — not a diagram, an
actual Deployment in your own disposable Lab environment.
## What you're walking into
A deployment called `api-gateway` has been rolled out. The pods start, the container runs — but
the deployment never reaches Ready, and traffic isn't being routed to it. Somewhere on-call, an
alert is firing.
You now have everything you learned in this module to work with:
- **The lesson** told you what "stuck at 0/2 Ready with 0 restarts" actually means (a readiness
problem, not a crash).
- **The demonstration** showed you exactly where to look for the explanation — `kubectl get
pods`, then `kubectl describe pod` and its Events section — and what a port-mismatch failure
looks like in that output.
- **The guided exercise** had you reason through what a correct probe definition looks like,
which is exactly what you'll need to compare the broken one against.
## Your task in this Lab
Diagnose why the pods are failing their readiness check, correct the probe configuration, and
restore the deployment to fully Ready. Start with pod status and events, the same way you just
practiced — read before you patch.
This Lab has its own in-context hints if you get stuck, revealed one at a time — use the
concepts from this module first; reach for a hint only if you're genuinely stuck, not as the
first move.
When the Lab confirms your fix, come back to this module to close it out with a short
reflection on what actually caused this.
## Instructor Notes
**Classroom flow:** if teaching this live, this is the natural point to have students launch
their own Lab environment and work independently or in pairs, while you circulate. The
Real-Time Classroom dashboard's help-request and progress views (see the instructor Curriculum
view) show who's stuck without needing to walk around asking.
**Approximate timing:** 15-20 minutes for most students; some will resolve it in under 5 once
the module's concepts have landed.
**Common mistake to watch for:** students jumping straight to `kubectl logs` here, which will
be unhelpful (nginx has nothing to say about a probe pointed at the wrong port) — this is a good
moment to reinforce the demonstration's point about reading Events first for this failure class.
@@ -0,0 +1,61 @@
# Root-Cause Reflection & Real Ops Takeaway
You just fixed a real incident pattern, not a puzzle. Before moving on, take a few minutes to
think through what actually happened and why it matters beyond this one Lab.
## What actually went wrong
The container was never broken. `nginx` started fine and was serving requests correctly the
entire time. The only thing wrong was that the readiness probe was configured to check a port
nginx wasn't listening on — so Kubernetes, working exactly as designed, correctly reported the
Pod as Not Ready based on the information it had. The system didn't fail; it was given wrong
information and acted rationally on it.
This is worth sitting with: most "Kubernetes is broken" incidents are actually "Kubernetes is
doing exactly what its configuration tells it to do, and the configuration is wrong."
## Reflection questions
Think through your own answers to these — there's no single correct wording, but you should be
able to explain each one out loud:
1. If you had jumped straight to `kubectl logs` instead of `kubectl describe pod`, how long do
you think it would have taken you to find this? What would the logs have actually shown you?
2. This Lab's port mismatch was introduced deliberately. What real-world change could introduce
the exact same failure by accident? (Consider: someone changes the container's listening
port in a new image version, but forgets the probe definition was hand-written separately.)
3. Why does a rolling update stall — not fail outright, just hang — when new Pods can't pass
their readiness probe? What is Kubernetes protecting you from by refusing to proceed?
## Real Ops takeaway: prevention, not just fixes
In a real production system, you'd want to prevent this class of incident from recurring, not
just fix it once:
- **Keep the probe port derived from the same source as the container's actual port**, where
your tooling allows it (a shared value in a Helm chart or Kustomize overlay), instead of two
independently hand-typed numbers that can drift apart silently.
- **Treat a stalled rollout as a first-class alert**, not just "the deploy is taking a while" —
a Deployment sitting at partial readiness for an unusual length of time is exactly this
failure mode, and catching it via rollout status is faster than waiting for a user-facing
outage report.
- **Practice reading Events before logs** for anything in the "Running but not Ready" state —
you just did this under low stakes; the habit is what pays off during a real incident at 2am.
This module is complete. Return to the course to continue with the next module.
## Instructor Notes
**Discussion questions for a live classroom:** use the three reflection questions above as
discussion prompts rather than a silent worksheet — question 2 in particular tends to produce
good real-world examples from students who've touched CI/CD pipelines before.
**Approximate timing:** 10 minutes for a live discussion; 5-8 minutes self-paced writing.
**Optional extension:** ask students to write the one-sentence postmortem summary they'd post
in a team channel for this incident (real on-call teams write these) — a good answer names the
root cause (probe port mismatch) and the fix (corrected port), not just "it's fixed now."
**Why this activity is a reflection, not another knowledge check:** the goal here is
consolidation and transfer to real-world judgment, not testing recall — resist turning this
into another right/wrong quiz.
@@ -0,0 +1,92 @@
# The RootNode Troubleshooting Framework
## Why a framework, not just "get good at kubectl"
The gold-standard Health Probes module taught you a lot about ONE specific failure mode
(readiness probe misconfiguration) and the specific commands to diagnose it. This module steps
back and asks: what's the repeatable PROCESS that works for almost any Kubernetes failure, not
just the ones you've already seen before?
Guessing under pressure — trying random fixes and seeing what sticks — is slow, often makes
things worse, and doesn't build a transferable skill. A framework does.
## Observe → Gather Evidence → Form Hypothesis → Test → Repair → Validate → Prevent Recurrence
**Observe** — what's the actual, specific symptom? Not "it's broken," but "the Deployment shows
0/3 Ready" or "requests are timing out" or "the Pod restarts every 30 seconds." Precision here
saves time later.
**Gather Evidence** — before touching anything, collect what the system itself is telling you:
`kubectl get pods`, `kubectl describe pod`, `kubectl logs`, `kubectl get events`. Resist the urge
to start changing things before you've looked.
**Form Hypothesis** — based on the evidence, state a SPECIFIC guess about the cause. "The
container is crashing because the command it's told to run doesn't exist" is testable. "Something's
wrong with networking" is not specific enough to act on yet.
**Test** — check whether your hypothesis is actually true, ideally without yet making the fix
(read the exact command being run, compare against what the image expects) — confirm before you
commit to a repair.
**Repair** — make the minimal change that addresses the confirmed cause. Not a bigger change
"while you're in there" — exactly the fix the evidence supports.
**Validate** — confirm the fix actually worked, using the same kind of evidence you gathered
initially (not just "I made a change, it should be fine now").
**Prevent Recurrence** — the step most often skipped under time pressure. Could this class of
mistake happen again? Is there a smaller, structural change (a default, a validation, a check)
that would catch it earlier next time?
## Applying it: a CrashLoopBackOff
- **Observe:** `crash-app`'s Pods show `STATUS: CrashLoopBackOff`, restart count climbing.
- **Gather Evidence:** `kubectl describe deployment crash-app` (check the configured command),
`kubectl logs -l app=crash-app` (what did the container actually print before dying, if
anything).
- **Form Hypothesis:** "The command override is invalid/wrong for this image, so the container
exits immediately every time it starts."
- **Test:** compare the command actually configured against what the image's default/expected
entrypoint should be.
- **Repair:** fix the command to what the container image actually expects.
- **Validate:** `kubectl get pods` shows `Running`, restart count stops climbing.
- **Prevent Recurrence:** a code-review habit of never overriding a container's command without
verifying it against the image's documentation first.
## Applying it: a broken Service selector
- **Observe:** traffic isn't reaching `my-svc`'s backend at all.
- **Gather Evidence:** `kubectl get endpoints my-svc` (empty?), `kubectl get pods
--show-labels` (what labels do the actual Pods have?).
- **Form Hypothesis:** "The Service's selector doesn't match the Pods' actual labels."
- **Test:** compare `kubectl describe service my-svc`'s selector against the Pods' real labels
directly, side by side.
- **Repair:** correct the selector to match.
- **Validate:** `kubectl get endpoints my-svc` now shows the Pod IPs.
- **Prevent Recurrence:** keeping label keys/values consistent (or generated from one source)
between a Deployment's Pod template and any Service that targets it.
## Instructor Notes
**Approximate timing:** 15 minutes read; 10-12 minutes live walking through both worked
examples.
**Demo plan:** Use the demonstration activity right after this lesson to show the CrashLoopBackOff
example live, narrating each framework step explicitly as you go rather than just running
commands.
**Likely misconceptions to watch for:**
- Students under time pressure often skip straight to "Repair" without confirming a hypothesis
first. Reinforce that a wrong guess implemented as a "fix" can make evidence harder to read
afterward, not easier.
- "Prevent Recurrence" is the step most likely to be treated as optional/skippable. Frame it as
what separates a team that keeps hitting the same incident from one that doesn't.
**Discussion question:** "What's the cost of skipping 'Gather Evidence' and jumping straight to
a guessed fix, even if the guess happens to be right?" (Answer: even a correct guess doesn't
teach you whether your reasoning was right — you got lucky, and the same class of mistake could
still recur since you never confirmed the actual cause.)
**Optional extension:** ask students to apply this exact framework, from memory, to the gold
module's `rootnode-probe-port-mismatch` incident they already solved — a good retroactive
exercise connecting a lab they already completed to the generalized framework.
@@ -0,0 +1,91 @@
# Demonstration: Applying the Framework to a CrashLoopBackOff
Watch the full Observe → Evidence → Hypothesis → Test → Repair → Validate → Prevent sequence
against a real symptom, narrated step by step.
## Observe
```
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
crash-app-6d9f8b7c5d-abcde 0/1 CrashLoopBackOff 5 4m
```
Specific symptom: `CrashLoopBackOff`, restart count climbing (5 and counting) — the container
is starting, exiting, and being restarted repeatedly, not just failing to start once.
## Gather Evidence
```
$ kubectl logs crash-app-6d9f8b7c5d-abcde
nginx: [emerg] invalid command "daemon-off": expected valid command
```
```
$ kubectl describe deployment crash-app
...
Command:
nginx
-g
daemon-off;
...
```
The logs show nginx itself rejecting the command it was given. The Deployment's configured
command shows exactly why: `daemon-off;` (a typo — missing space, should be `daemon off;`).
## Form Hypothesis
"The container's command override has a syntax error that nginx can't parse, so nginx exits
immediately every time it starts — which is why it's crash-looping rather than just failing to
start once."
## Test
Compare the broken command (`daemon-off;`) against nginx's actual expected syntax
(`daemon off;`, two separate words) — confirmed: this is exactly the kind of typo that would
produce exactly this error.
## Repair
```
$ kubectl patch deployment crash-app --type='json' \
-p='[{"op":"replace","path":"/spec/template/spec/containers/0/command","value":["nginx","-g","daemon off;"]}]'
```
## Validate
```
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
crash-app-7f8c9d6b5d-fghij 1/1 Running 0 15s
```
New Pod, `Running`, `RESTARTS: 0` — the fix is confirmed, not just assumed.
## Prevent Recurrence
A command override like this is a hand-typed string with no validation until the container
actually tries to run it. A team that hit this once might decide: don't override a well-known
image's default command unless there's a specific reason to, and if you must, test the exact
command string locally (`docker run` or similar) before deploying it — catching a typo in
seconds locally beats debugging a CrashLoopBackOff in the cluster.
## Instructor Notes
**Demo script:** Run this live if time allows, deliberately narrating which framework step
you're on at each command — students retain the framework far better tied to real commands than
as an abstract list.
**Approximate timing:** 8-10 minutes live; 5-7 minutes self-paced.
**Common misconception to preempt:** students sometimes skip straight from "Observe" to
"Repair" once they spot the typo, without stating a hypothesis or testing it. Reinforce that
even an "obvious" fix benefits from the explicit hypothesis step — it's what makes the eventual
Validate step meaningful (you're confirming a SPECIFIC claim, not just "trying something").
**Optional extension:** ask what evidence you'd gather differently if the logs had been
completely empty instead of showing a clear nginx error (a good bridge to cases where the
container dies before it can log anything at all — `kubectl describe pod`'s Events section
becomes the primary evidence source in that case, same as the gold module's probe-mismatch
lesson).
@@ -0,0 +1,50 @@
# Knowledge Check: The Troubleshooting Framework
---
**1. You spot what looks like an obvious typo causing a failure and fix it immediately without
writing down a hypothesis first. What's the actual cost of skipping that step, even if the fix
works?**
*Answer:* You don't get a confirmed, testable claim about the cause — if the fix happens to
"work" for an unrelated reason (a coincidental restart, a different underlying issue also
resolving), you'd have no way to distinguish that from your fix actually being correct. The
hypothesis step is what makes Validate meaningful.
---
**2. During the "Gather Evidence" step for a CrashLoopBackOff, which two commands give you the
most direct evidence, and what does each one tell you that the other doesn't?**
*Answer:* `kubectl logs` (what the container itself printed, if anything, before exiting) and
`kubectl describe pod`/`describe deployment` (the container's actual configuration, plus recent
Events). Logs show application-level output; describe/Events show what Kubernetes itself
observed and configured — a crash with no logs at all often means the failure happened before
the application could log anything, which describe/Events can still reveal.
---
**3. True or false: "Prevent Recurrence" only applies to major incidents, not small
one-off mistakes like a typo.**
*Answer:* False. A small mistake that recurs repeatedly across a team becomes a real, ongoing
cost even if no single instance is severe — the framework applies the same reasoning regardless
of incident size: could a small, structural change catch this earlier next time?
---
**4. A broken Service selector (endpoint-fix-task's scenario) and a CrashLoopBackOff
(crashloop-fix's scenario) are very different failures. What's the SAME about how you'd
approach both?**
*Answer:* The same seven-step framework — Observe the specific symptom, Gather Evidence before
guessing, form a specific testable Hypothesis, Test it, make the minimal Repair the evidence
supports, Validate with the same kind of evidence you started with, and consider how to Prevent
Recurrence. The framework doesn't change; only the specific commands and evidence do.
## Instructor Notes
**Facilitation:** Question 4 is the whole point of this module — make sure it's discussed even
if answered correctly, since it's the transferability claim the entire framework rests on.
**Approximate timing:** 5-7 minutes.
@@ -0,0 +1,52 @@
# Guided Exercise: Applying the Framework to a Broken Service, on Paper First
Before the `endpoint-fix-task` Lab, walk through the framework yourself against a described
symptom — no cluster needed yet.
## The scenario
A teammate says: "`my-svc` isn't routing any traffic to our Pods, and I don't know why — the
Pods look healthy in `kubectl get pods`."
## Your task
Write out, step by step, what you would actually DO for each framework stage, before looking at
the worked answer:
1. **Observe** — what's the precise, specific symptom here (not "it's broken")?
2. **Gather Evidence** — name the exact `kubectl` commands you'd run, in order, and what each
one is checking for.
3. **Form Hypothesis** — based on what ROOT 07's Services lesson taught you, what's the most
likely cause of "Pods healthy, but Service routes nowhere"?
4. **Test** — how would you CONFIRM that hypothesis before changing anything?
5. **Repair** — what's the minimal fix, assuming your hypothesis is correct?
6. **Validate** — what would you check afterward to PROVE the fix worked, not just assume it?
## A worked answer to check yourself against
1. Traffic sent to `my-svc` never reaches any Pod, despite the Pods themselves reporting
`Running`/`Ready` — this rules out "the Pods are unhealthy" as the cause.
2. `kubectl get endpoints my-svc` (does it show any Pod IPs at all?), then `kubectl describe
service my-svc` (what selector is it actually using?), then `kubectl get pods --show-labels`
(what labels do the real Pods actually carry?).
3. A Service-to-Pod label selector mismatch — exactly the failure mode ROOT 07's demonstration
walked through, where a Service's selector no longer matches the Pods' actual labels.
4. Directly compare the Service's selector (from `describe service`) against the Pods' labels
(from `get pods --show-labels`) side by side — confirmed if they genuinely don't match.
5. Correct the Service's selector (or the Pods' labels, whichever is actually wrong) so they
match again.
6. `kubectl get endpoints my-svc` now shows the Pod IPs populated — not just "the YAML looks
right now," but the actual Endpoints object reflecting real, matched Pods.
## Instructor Notes
**Facilitation:** This exercise deliberately reuses ROOT 07's own content — a good moment to
point out explicitly that the framework's value is applying PRIOR knowledge systematically, not
learning something entirely new each time.
**Approximate timing:** 10-12 minutes live with discussion; 10 minutes self-paced.
**Optional extension:** ask what would change about steps 2-4 if the symptom were instead
"Pods healthy, but ONE specific caller can't reach the Service while others can" — a good bridge
to considering NetworkPolicy (ROOT 09) as an alternative hypothesis once the simpler
selector-mismatch explanation is ruled out.
@@ -0,0 +1,45 @@
# Reflection: A Framework You've Actually Used Is One You'll Trust Under Pressure
You just solved two real, different failures — a CrashLoopBackOff and a broken Service
selector — using the same seven-step framework both times. Take a few minutes to consolidate
this.
## Reflection questions
1. Which framework step did you personally find easiest to skip or rush through across both
Labs? Why do you think that step specifically tempts shortcuts?
2. Compare how you approached `crashloop-fix` versus `endpoint-fix-task`. What was IDENTICAL
about your process, and what was different only because the evidence looked different?
3. Now that you've practiced this framework on two failures you'd never seen exact instructions
for, how confident do you feel applying it to a THIRD, completely unfamiliar Kubernetes
failure? What would you still want to double-check?
## Real Ops takeaway
**The framework's value shows up most on failures you've never seen before.** Anyone can
memorize "if X symptom, do Y fix" for a handful of common problems — that knowledge doesn't
transfer. Observe → Evidence → Hypothesis → Test → Repair → Validate → Prevent Recurrence
transfers to failures nobody has specifically taught you, because it's a PROCESS, not a lookup
table.
**"Prevent Recurrence" is what separates incident response from firefighting.** Both Labs in
this module had a real, nameable prevention practice (verify command strings before deploying;
keep label keys consistent between a Deployment and its Service). Teams that consistently ask
this question after every fix accumulate fewer repeat incidents over time — not because they got
lucky, but because they closed the actual gap each time.
This module is complete. Continue to the next module.
## Instructor Notes
**Discussion questions for a live classroom:** question 1 tends to produce honest, useful
answers — "Prevent Recurrence" and "Form Hypothesis" (skipping straight to a guessed fix) are
the two most commonly admitted shortcuts, both worth normalizing as a real temptation to resist
rather than treating as a personal failing.
**Approximate timing:** 8-10 minutes live discussion; 5-7 minutes self-paced writing.
**Optional extension:** ask students to apply the framework, from memory and without re-reading
this module, to a THIRD failure type they haven't practiced yet (e.g., a Pod stuck `Pending` —
not built as a hands-on Lab in this module, just a thought exercise) to test genuine
transferability.
@@ -0,0 +1,88 @@
# DevOps as an Operating Model, Not a Job Title
## You've already been doing this — just not naming it yet
Every course before this one taught you a Kubernetes skill: declaring desired state, hardening
a Pod, diagnosing a failure with a repeatable framework, running an incident with real
discipline. This module doesn't teach a new hands-on skill — it names the OPERATING MODEL those
skills are all part of, so you can recognize it and apply it beyond Kubernetes specifically.
## DevOps: a set of practices, not a team you join
"DevOps" is often misused as a job title ("I'm a DevOps engineer") or a team name ("throw it
over to DevOps"). Properly understood, DevOps is a set of PRACTICES for how development and
operations work together: shorter feedback loops, automation over manual process, and shared
ownership of what happens after code ships — not a separate team that operations gets outsourced
to.
Concretely, you already practiced core DevOps ideas without the label:
- **Declarative thinking** (ROOT 05) — describing desired state and letting a system enforce it,
instead of manually executing steps, is the same principle behind Infrastructure as Code.
- **Fast, safe rollback** (ROOT 06) — treating recovery as a rehearsed, ordinary action rather
than a crisis is a direct DevOps practice (fast feedback, low-drama recovery).
- **A repeatable troubleshooting framework** (ROOT 10) and **structured incident response**
(ROOT 12) — both are operational discipline, the "Ops" half of DevOps, practiced for real.
## Platform Engineering: what DevOps looks like at scale
**Platform engineering** is the next layer: instead of every application team reinventing how
to deploy safely, monitor, and secure their own workloads, a platform team builds SELF-SERVICE
tooling and golden paths that let other engineers move fast without needing deep infrastructure
expertise themselves. The Kubernetes cluster, the CI/CD pipeline, the observability stack — all
of it becomes a PRODUCT a platform team builds for INTERNAL customers (other engineers), not
just infrastructure that happens to exist.
A useful lens: platform engineering exists to **reduce cognitive load**. A developer who can
deploy a well-configured, secure, observable service through a simple, guided path doesn't need
to personally understand every Kubernetes concept from ROOT 05-10 to ship safely — the platform
encodes that expertise once, for everyone.
## Where Kubernetes fits in the larger picture
Kubernetes is INFRASTRUCTURE, not the whole platform. A real platform engineering practice
typically layers several things around it:
- **Declarative delivery** — describing what should be deployed and letting automation apply it
(the GitOps model — future ROOT 11 content, requires a Git+deployment-automation environment
this course doesn't yet provide).
- **Continuous integration/delivery** — automatically testing and shipping code changes (CI/CD —
future content, requires a CI runner environment).
- **Infrastructure as Code** — declaring cloud/infrastructure resources the same way you declare
Kubernetes objects (future content, requires a provisioning environment like Terraform).
- **Observability** — knowing what's actually happening in production beyond "is the Pod
Running" (future content, requires a metrics/logging/tracing stack).
None of these require abandoning what you learned in ROOT 05-10 — they build directly on the
same mental models (desired vs. actual state, evidence-based diagnosis, least privilege) applied
to a wider set of systems.
## Instructor Notes
**Lesson goal:** Give students a name and a framework for practices they've already exercised,
and an honest map of what ROOT 11 will eventually cover once appropriate lab environments exist —
without pretending those labs exist today.
**Approximate timing:** 20 minutes read; 15 minutes live with discussion.
**Preparation:** None beyond having read `docs/ROOT11_DEVOPS_PLATFORM_SCOPE.md` to speak
accurately about what's planned versus what exists.
**Likely misconceptions to watch for:**
- "DevOps" as a job title/team is a very common prior belief — correct this directly and early,
it's the central point of the lesson.
- Platform engineering is sometimes conflated with "just Kubernetes." Emphasize the self-service/
golden-path angle — the PRODUCT framing, not just the technology.
**Questions to ask students:** "Which ROOT_ course so far most directly practiced a DevOps
principle, even though it was never labeled that way?" (Most will cite ROOT 06's rollback
discipline or ROOT 12's incident framework — both are legitimate, good answers.)
**When to let a student struggle vs. provide a hint:** Not applicable — this is an orientation
lesson with no hands-on task.
**Optional extension:** ask students to name one manual process at a company they've worked for
(or heard about) that platform engineering-style self-service tooling could have eliminated.
**Expected takeaway:** DevOps and platform engineering are the organizing ideas behind almost
everything ROOT_ has already taught — not a new, separate subject to learn from scratch.
@@ -0,0 +1,46 @@
# Knowledge Check: DevOps & Platform Thinking
---
**1. A colleague says "DevOps is the team that handles our servers." What's wrong with that
statement?**
*Answer:* DevOps describes a set of PRACTICES (shared ownership, fast feedback loops,
automation) for how development and operations work together — not a separate team that
operations gets handed off to. Treating it as a team name misses the point that these practices
are meant to be shared across engineering, not isolated in one group.
---
**2. Which ROOT_ course's central skill is most directly an example of "declarative thinking"
applied outside Kubernetes objects specifically?**
*Answer:* ROOT 05's desired-state model is the direct root of declarative thinking generally —
Infrastructure as Code applies the exact same principle (describe what should exist, let
automation enforce it) to cloud infrastructure rather than Kubernetes objects specifically.
---
**3. True or false: platform engineering's main goal is building the most technically
sophisticated internal tooling possible.**
*Answer:* False. Its main goal is reducing COGNITIVE LOAD for other engineers via self-service,
golden-path tooling — sophistication that doesn't make another engineer's job easier isn't the
point, even if it's technically impressive.
---
**4. Why can't this course build a hands-on GitOps or CI/CD Lab the same way ROOT 05-10 built
Kubernetes Labs?**
*Answer:* Those require different infrastructure the current Lab platform doesn't provide (a
Git repository + deployment-automation tool like ArgoCD for GitOps; a CI runner for CI/CD) — see
`docs/ROOT11_DEVOPS_PLATFORM_SCOPE.md`. Building a fake version of either would teach an
inaccurate mental model, which is worse than honestly deferring the hands-on practice.
## Instructor Notes
**Facilitation:** Question 4 is worth stating plainly in a live classroom — students should
understand this is a genuine platform limitation being handled honestly, not an oversight.
**Approximate timing:** 5 minutes.
@@ -0,0 +1,45 @@
# Reflection: Where You Already Practiced This
You've now named the operating model behind most of what ROOT_ taught. Take a few minutes to
trace it back through your own experience in this program.
## Reflection questions
1. Pick one specific moment from ROOT 05-10 or ROOT 12 where you practiced a DevOps/platform
engineering principle without it being labeled that way. Name the principle explicitly now.
2. If you were designing a platform team's "golden path" for deploying a new service safely,
which THREE things from ROOT_ (health probes, RBAC, resource limits, NetworkPolicy, etc.)
would you bake in as defaults rather than leave to each team to remember?
3. What's one thing about GitOps, CI/CD, Infrastructure as Code, or observability you're
genuinely curious about, now that you have the Kubernetes foundation to eventually build
hands-on skill in it?
## Real Ops takeaway
**The skills you already have transfer further than "just Kubernetes."** Desired-state
thinking, least privilege, evidence-based troubleshooting, and structured incident response
aren't Kubernetes-specific facts — they're how experienced platform engineers reason about ANY
complex system. Recognizing that transfer is what turns "I learned Kubernetes" into "I think
like a platform engineer."
**Honest scope beats fake completeness.** This course didn't pretend to teach you GitOps or
CI/CD hands-on today — it told you exactly what's missing and why, and pointed at what it would
take to build it for real. That's the same standard the rest of ROOT_ held itself to
throughout — a module that doesn't exist yet is more useful, honestly labeled, than one that
exists in name only.
This module is complete.
## Instructor Notes
**Discussion questions for a live classroom:** question 2 works well as a small-group exercise —
have groups compare their "golden path defaults" lists and discuss disagreements.
**Approximate timing:** 10 minutes live discussion; 8 minutes self-paced writing.
**Optional extension:** ask students to research (outside class) one real company's public
engineering blog post about their platform engineering practice, and bring one concrete example
to share next session.
**Expected takeaway:** ROOT_ built real, transferable engineering judgment — this module makes
that transfer explicit rather than leaving students to discover it on their own.
@@ -0,0 +1,88 @@
# The RootNode Incident-Response Framework
## This module feels different on purpose
Everything before ROOT 12 asked you to complete an exercise. This module asks something
different: **you are responsible for this environment.** The incidents in this module are
modeled on real operational failures — not textbook exercises designed to teach one clean
concept, but the kind of messy, "why is this actually broken" situations a real on-call engineer
gets paged for.
## Detect → Triage → Diagnose → Mitigate → Recover → Verify → Learn
This is a lightweight incident-response framework — not enterprise incident management
tooling, just a way of thinking that scales down to "one engineer, one broken thing" as well as
it scales up to a real incident-response team.
**Detect** — something alerted you (a monitoring page, a user report, or in this Lab's case,
the scenario telling you something is wrong). You know an incident exists before you know what
it is.
**Triage** — how bad is this, right now, before you dive into root cause? Is anything actively
getting worse? Is this affecting one thing or cascading into others? Triage decides how urgently
you act, before you fully understand why.
**Diagnose** — this is where ROOT 10's troubleshooting framework (Observe, Gather Evidence, Form
Hypothesis, Test) does its work — the specific mechanics of finding the actual cause.
**Mitigate** — the fastest safe action that stops things from getting worse, which is not always
the same as the final fix. Sometimes mitigation and repair are the same step; sometimes
mitigation buys time (e.g. temporarily reducing load) while a proper fix is prepared.
**Recover** — restore the system to its actual working state — not just "stopped getting worse"
but genuinely operational again.
**Verify** — confirm recovery with real evidence, the same discipline as ROOT 10's Validate
step. "I think it's fixed" is not the same as checking.
**Learn** — what would prevent this exact class of incident from happening again? This is where
`rootnode-enforce-limits` picks up directly after `rootnode-runaway-monitor` — not a separate
topic, but the deliberate "Learn" step of the SAME incident, made into its own scenario.
## What this module will NOT tell you
The specific fixes for each incident are not in this lesson, and instructor notes on the
individual incident activities go deeper than what's shown here — this module gives you the
FRAMEWORK for approaching an unfamiliar incident, not a cheat sheet for these specific four.
That's deliberate: a framework you had to apply yourself is one you'll actually remember under
real pressure.
## Instructor Notes
**Lesson goal:** Set the tone shift for this entire module — from "complete an exercise" to
"you own this." This lesson is short by design; the real teaching happens through the incidents
themselves.
**Approximate timing:** 15 minutes read; 10 minutes live framing discussion.
**Preparation:** Read each incident's own instructor notes (in the Lab activities below) before
teaching this module live — several contain deeper solution guidance not repeated here,
specifically so students still have to work for it.
**Demonstration plan:** Not applicable to this framing lesson — the four incidents themselves
serve as the demonstrations.
**Likely misconceptions to watch for:**
- Students sometimes treat "Mitigate" and "Recover" as the same step. They can be, but aren't
always — worth naming a real-world example (rolling back a bad deploy mitigates immediately;
fully recovering might still require redeploying the CORRECT version afterward).
- "Learn" is the step most likely to get skipped once a Lab is marked complete. Explicitly
connect it to why `rootnode-enforce-limits` exists as its own scenario.
**Questions to ask students:** "Which step of this framework do you think real teams are worst
at, under real pressure, and why?" (There's no single right answer — the goal is getting
students to reason about incident response as a discipline, not just a checklist.)
**When to let a student struggle vs. provide a hint:** These incidents are intentionally less
guided than earlier modules. Let a student sit with "I don't know where to start" for a few
minutes before nudging — the discomfort of a genuinely unfamiliar failure is part of what this
module is teaching. Redirect toward the Diagnose step's ROOT 10 framework if truly stuck, rather
than pointing at the specific fix.
**Optional extension:** ask students to write a two-sentence incident summary for
`rootnode-runaway-monitor` as if posting it to a real team channel — Detect/Diagnose/Recover in
plain language, no jargon.
**Expected takeaway:** A framework applied under pressure beats improvisation, even for
engineers who are technically skilled enough to eventually stumble into the right answer anyway
— the framework is what makes the RIGHT answer arrive faster and more reliably.

Some files were not shown because too many files have changed in this diff Show More