feat(root): author workloads curriculum

root-06-m02-deployments-scaling-updates — a new, full gold-standard-
pattern module: why Deployments/ReplicaSets exist, how a rolling update
actually progresses, and what rollback restores. Lesson, demonstration
(a healthy rollout vs. one stalled on ImagePullBackOff), knowledge
check, guided exercise (planning a safe rollout strategy), real Labs
(rolling-update-task, deployment-rollback), and a reflection.

root-06-m03-jobs-cronjobs-daemonsets — a new PARTIAL module: Jobs,
CronJobs, and DaemonSets as distinct workload shapes from a
Deployment's always-on model. Lesson + knowledge check + real Labs
(job-task, cronjob-task).

Co-Authored-By: Claude Sonnet 5 <[email protected]>
This commit is contained in:
2026-08-26 09:06:00 -05:00
co-authored by Claude Sonnet 5
parent 25409d3bd7
commit 86ea035e9b
8 changed files with 472 additions and 0 deletions
@@ -0,0 +1,81 @@
# Why Deployments Exist & How a Rolling Update Progresses
## The problem a bare Pod doesn't solve
A Pod, on its own, has no self-healing and no concept of "there should be more than one of
these." If a Pod crashes, nothing recreates it. If you need 3 copies for capacity, you'd have to
create 3 separately-named Pods by hand and keep track of all of them yourself. If you need to
update the container image, you'd have to delete and recreate every one of them manually,
almost certainly causing a visible outage while you do it.
**A Deployment solves all three problems at once**: it declares how many replicas of a Pod
template should exist, keeps that count true continuously (the exact desired-state mechanism
from ROOT 05), and knows how to change the Pod template over time — a version upgrade — without
just deleting everything and starting over.
## ReplicaSets: the layer underneath, briefly
You'll rarely touch a ReplicaSet directly, but understanding it clarifies what a Deployment
actually does. A Deployment doesn't manage Pods directly — it manages a ReplicaSet, which is
what actually keeps N Pods matching a label selector running. When you update a Deployment's
Pod template (like changing the image), the Deployment creates a NEW ReplicaSet with the new
template and gradually shifts replicas from the old ReplicaSet to the new one. The old
ReplicaSet isn't deleted — it's scaled to zero and kept around, which is exactly what makes
rollback possible (more on that below).
## How a rolling update actually progresses
By default, a Deployment updates using the `RollingUpdate` strategy: it doesn't take all old
Pods down and bring all new ones up at once (that would cause an outage). Instead, it
incrementally creates new-version Pods and removes old-version ones, always keeping enough
capacity available:
```
Before: [v1] [v1] [v1]
Step 1: [v1] [v1] [v1] [v2] <- one new Pod created
Step 2: [v1] [v1] [v2] <- one old Pod removed once new one is Ready
Step 3: [v1] [v1] [v2] [v2]
Step 4: [v1] [v2] [v2]
...
After: [v2] [v2] [v2]
```
Crucially, a new Pod only counts as "up" for the purposes of proceeding once it passes its
readiness probe (see ROOT 10's gold-standard module) — this is exactly why a misconfigured
readiness probe can make a rollout hang indefinitely instead of failing fast: Kubernetes is
correctly waiting for a "ready" signal that will never come.
## Rollback: what it actually restores
`kubectl rollout undo` doesn't magically know how to "undo" arbitrary changes — it works because
of the old ReplicaSet still sitting around at zero replicas, from the mechanism described above.
Rolling back means: scale the OLD ReplicaSet back up and the current (broken) one back down,
using the exact same rolling mechanism as a forward update. This is why rollback is fast and
safe — it's not a special operation, it's the same well-tested rolling-update machinery running
in reverse, against a Pod template that's already known to have worked before.
## Instructor Notes
**Approximate timing:** 15 minutes read, 10-12 minutes live with the ReplicaSet diagram.
**Demo plan:** Draw (or show) the ReplicaSet-swap diagram above on a whiteboard before the
demonstration activity — students who see the "old ReplicaSet scaled to zero, not deleted"
detail up front connect rollback to rollout much faster.
**Likely misconceptions to watch for:**
- Students often think rollback "restores a backup" or reverts a Git-style diff. Correct this:
it's scaling an existing (still-present) ReplicaSet back up, nothing is being restored from
storage.
- "Rolling update" is sometimes assumed to mean "one Pod at a time, always." The actual pace is
controlled by `maxSurge`/`maxUnavailable` (not covered in depth here, but worth naming so
students don't over-generalize from the simplified diagram).
**Discussion question:** "Why does a rolling update need readiness probes to work correctly,
specifically?" (Answer: without a readiness signal, Kubernetes has no way to know when a new
Pod is actually safe to receive traffic before removing an old one — it would either guess via a
fixed delay, unreliable, or not wait at all, unsafe.)
**Optional extension:** ask what would happen to an in-progress rollout if the new image simply
doesn't exist (a typo in the tag) — a good bridge to real troubleshooting: the rollout stalls
indefinitely with `ImagePullBackOff` on the new Pods, never completing and never rolling back on
its own.
@@ -0,0 +1,77 @@
# Demonstration: Watching a Rollout, Good and Bad
## A healthy rollout
A Deployment `app-v1` running `nginx:1.24` at 3 replicas gets updated to `nginx:1.25`:
```
$ kubectl set image deployment/app-v1 nginx=nginx:1.25
deployment.apps/app-v1 image updated
$ kubectl rollout status deployment/app-v1
Waiting for deployment "app-v1" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "app-v1" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "app-v1" rollout to finish: 1 old replicas are pending termination...
deployment "app-v1" successfully rolled out
```
`kubectl rollout status` blocks and reports exactly the incremental progress described in the
lesson — this is your primary tool for watching (or scripting a wait on) a rollout, rather than
repeatedly polling `kubectl get pods` yourself.
## Checking revision history
```
$ kubectl rollout history deployment/app-v1
REVISION CHANGE-CAUSE
1 <none>
2 <none>
```
Revision 1 is the original ReplicaSet (now scaled to zero, still present); revision 2 is the
current one. This history is exactly what `kubectl rollout undo` uses.
## A stalled rollout
Now imagine the same update, but to a typo'd tag, `nginx:1.25-typo`, which doesn't exist:
```
$ kubectl rollout status deployment/app-v1
Waiting for deployment "app-v1" rollout to finish: 1 out of 3 new replicas have been updated...
```
...and it just sits there. `kubectl get pods` shows why:
```
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
app-v1-7d8f9c6b5d-abcde 1/1 Running 0 5m
app-v1-7d8f9c6b5d-fghij 1/1 Running 0 5m
app-v1-9c5b7d8f6b-klmno 0/1 ImagePullBackOff 0 30s
```
The new Pod can never become Ready because its image can't even be pulled — so the rollout
correctly refuses to proceed further (removing more old Pods would reduce capacity for no
benefit, since the replacement will never come up). Nothing is "broken" in the platform; the
platform is behaving exactly correctly given bad input.
**The fix here is `kubectl rollout undo`, not waiting longer** — this exact scenario is what
`deployment-rollback`'s Lab, later in ROOT_, gives you real hands-on practice with.
## Instructor Notes
**Demo script:** If time allows, run the healthy rollout live against a throwaway Deployment,
then intentionally break the second one with a bad tag so students see `ImagePullBackOff` occur
in real time rather than only in static text.
**Approximate timing:** 8-10 minutes live; 5-7 minutes self-paced.
**Common misconception to preempt:** students sometimes assume a stalled rollout will
eventually time out and auto-rollback on its own. By default it will not — a Deployment has no
automatic rollback trigger; a human (or automation) has to run `kubectl rollout undo` (or fix
the underlying issue) explicitly. This is worth stating directly since it's a genuine and
consequential production gotcha.
**Optional extension:** ask what `kubectl rollout status --timeout=2m` would change about this
scenario (answer: the command itself would exit with an error after 2 minutes instead of
blocking forever — useful in CI/CD pipelines that need to fail a deploy step rather than hang).
@@ -0,0 +1,48 @@
# Knowledge Check: Deployments & Rollouts
---
**1. A rolling update to a new image version has been "in progress" for 20 minutes with no
error message, and `kubectl rollout status` never returns. What's the first thing you should
check, and why?**
*Answer:* `kubectl get pods` to see the new Pod's status, then `kubectl describe pod` on it if
it isn't `Running`. A stalled rollout with no explicit error almost always means a new Pod can't
become Ready — commonly `ImagePullBackOff` (bad image/tag) or a failing readiness probe. The
rollout isn't broken; it's correctly refusing to proceed until the new Pod proves healthy.
---
**2. True or false: `kubectl rollout undo` restores your application from a backup of its
previous state.**
*Answer:* False. It scales the previous (still-present, just scaled-to-zero) ReplicaSet back up
and the current one back down — the same rolling mechanism as a forward update, just pointed at
an older, already-proven-working Pod template. Nothing is being restored from storage.
---
**3. A Deployment has `replicas: 3`. You need it to handle more traffic. Which is the correct
action: edit the Pod template's resource limits, or change the replica count? Justify your
answer for a stateless web application.**
*Answer:* Change the replica count (scale out) for a stateless application handling more
concurrent traffic — more identical copies sharing the load. Changing resource limits affects
how much each individual Pod can use, not how many Pods exist; it doesn't add capacity for more
concurrent requests the way scaling does.
---
**4. Why does a Deployment keep an old, scaled-to-zero ReplicaSet around instead of deleting it
immediately after a successful rollout?**
*Answer:* That scaled-down ReplicaSet is exactly what makes fast, safe rollback possible later —
deleting it immediately would mean a "rollback" would have to rebuild the old Pod template from
scratch (or fail entirely) instead of simply scaling existing, already-defined Pods back up.
## Instructor Notes
**Facilitation:** Question 1 is the most important one to discuss out loud — it's the exact
diagnostic instinct students will need repeatedly in ROOT 10.
**Approximate timing:** 5 minutes.
@@ -0,0 +1,54 @@
# Guided Exercise: Planning a Safe Rollout Strategy
Paper-and-thinking exercise — work through this before the Lab.
## The scenario
Your team runs `api-server`, a Deployment with 6 replicas serving live traffic. You need to
ship a new image version. The team lead asks: "what could go wrong, and how would we know
before it takes down the whole service?"
## Your task
Work through these questions:
1. If the new image has a bug that crashes the container immediately on startup, at what point
in the rolling update would you expect to notice — after 1 new Pod, after all 6, or somewhere
in between? Why?
2. What's the difference in blast radius between that crash-on-startup bug and a subtler bug
that only fails under load, several minutes after a Pod starts serving traffic?
3. Given your answer to question 2, what's the practical value of watching `kubectl rollout
status` and `kubectl get pods` DURING a rollout, rather than only checking after it reports
"successfully rolled out"?
4. If you noticed a problem partway through, what's the single command that gets you back to
the last known-good state fastest?
## A worked answer to check yourself against
1. After the FIRST new Pod — a crash-on-startup bug means the new Pod never becomes Ready
(`CrashLoopBackOff`), so the rollout stalls immediately, well before more old Pods are
removed. This is actually the SAFEST kind of bug, because the blast radius stays tiny.
2. A crash-on-startup bug is naturally self-limiting (the rollout stalls itself, as above). A
bug that only appears under load is NOT self-limiting the same way — by the time it's
noticeable, several old (working) replicas may already have been replaced, meaning a larger
fraction of real traffic is already hitting the broken version.
3. Watching during the rollout, not just checking the final "successfully rolled out" message,
is what lets you catch the load-dependent case in question 2 early — waiting for the final
status alone would let the rollout complete entirely (100% on the broken version) before you
noticed anything wrong.
4. `kubectl rollout undo deployment/api-server` — assuming the previous ReplicaSet is still
present (the default), this is faster than trying to hand-diagnose and hot-fix the new
version under pressure.
## Instructor Notes
**Facilitation:** Question 2's distinction (self-limiting vs. not) is the core insight worth
making sure lands — it's why "the rollout said success" is necessary but not sufficient
evidence that a deploy actually went well.
**Approximate timing:** 10-15 minutes live with discussion; 10 minutes self-paced.
**Optional extension:** ask what monitoring/observability signal (beyond `kubectl` output) a
real team would want during a rollout to catch the load-dependent case even faster (a good
forward pointer to ROOT 10's observability themes, without needing to build any actual
monitoring infrastructure here).
@@ -0,0 +1,39 @@
# Launch Lab: Perform a Rolling Update
Time to run a real rolling update against a real Deployment in your disposable Lab.
## What you're doing
A Deployment `app-v1` is running an older image version. Your task is to update it to a newer
version and confirm every replica ends up running the new image — exactly the mechanism from
the lesson and demonstration, on a real cluster this time.
## Before you start
Recall the demonstration: `kubectl rollout status` is your tool for watching progress rather
than repeatedly polling `kubectl get pods` yourself, and a stalled rollout with no obvious error
usually means a new Pod can't become Ready. If your update seems to hang, that's exactly the
diagnostic instinct from the knowledge check to apply.
## Your task in this Lab
Update the Deployment's image and verify all replicas are running the new version. Use
`kubectl rollout status` to confirm completion, not just a single `kubectl get pods` snapshot.
This Lab has its own in-context hints if you get stuck.
When you're done, return to this module for a short reflection on what "successfully rolled
out" actually guarantees — and doesn't.
## Instructor Notes
**Classroom flow:** A good moment to have students keep `kubectl get pods --watch` open in a
second terminal (if their Lab environment supports multiple panes) so they see the incremental
old-Pod/new-Pod transition happen live, reinforcing the demonstration.
**Approximate timing:** 10-15 minutes.
**Common mistake to watch for:** running `kubectl edit deployment` and forgetting to save/apply
correctly, or editing the wrong field (e.g. `replicas` instead of the container image) — a good
moment to reinforce reading the task description precisely rather than pattern-matching to a
similar-looking previous Lab.
@@ -0,0 +1,43 @@
# Reflection: Rollback Is a Plan, Not a Panic Button
You just completed a real rolling update. Take a few minutes to connect it to how this plays out
in a real production team.
## Reflection questions
1. "Successfully rolled out" (from `kubectl rollout status`) tells you every replica is Running
and Ready. What does it NOT tell you about whether the new version is actually correct?
2. If you were on-call and a rollout you just ran seemed to be causing problems, would you reach
for `kubectl rollout undo` immediately or try to hot-fix the new version first? What would
change your answer?
3. Think back to the guided exercise's crash-on-startup vs. load-dependent distinction. Which
one would a typical automated health check (like a readiness probe, ROOT 10) catch, and
which one might slip through even with probes configured correctly?
## Real Ops takeaway
**A rollout completing is not the same as a deploy succeeding.** "Every Pod is Running and
Ready" only proves the new code started up and passed whatever checks are configured — it says
nothing about whether the new version is functionally correct under real traffic. Real teams
pair a rollout with actual monitoring (error rates, latency) before calling a deploy genuinely
successful, not just a green `kubectl rollout status`.
**Rollback should be a rehearsed, low-drama action, not a last resort.** Teams that treat
`kubectl rollout undo` as a normal, expected tool — something you reach for calmly the moment
something looks wrong — recover from bad deploys in seconds. Teams that treat rollback as an
admission of failure hesitate, try to hot-fix forward under pressure, and turn a small,
recoverable mistake into a longer outage.
This module is complete. Continue to the next module.
## Instructor Notes
**Discussion questions for a live classroom:** question 2 is worth pushing on — many students'
instinct is to try to fix forward, and it's worth surfacing WHY that instinct is often wrong
under time pressure (unknown time-to-fix vs. known, fast rollback).
**Approximate timing:** 8-10 minutes live discussion; 5 minutes self-paced writing.
**Optional extension:** ask students to write the one-sentence Slack message they'd post to
their team the moment they decide to roll back a bad deploy — reinforces that rollback is a
normal, communicable operational action, not a private failure.
@@ -0,0 +1,85 @@
# Jobs, CronJobs, and DaemonSets: The Other Workload Shapes
Everything in the previous module assumed a long-running application that should always have N
replicas up. Not every workload looks like that. Kubernetes has three other primitives for
genuinely different shapes of work — using a Deployment for any of these would be the wrong
tool, not just a less elegant one.
## Job: run to completion, exactly once (or a fixed number of times)
A Job creates one or more Pods and considers itself done when they exit successfully — it does
NOT restart a Pod that exits with success, and it DOES retry one that fails, up to a configured
limit. This is fundamentally different from a Deployment, which expects its Pods to run forever
and treats any exit as something to replace.
Use a Job for: a database migration, a batch data-processing task, a one-off cleanup script —
anything with a defined "finished" state.
```yaml
apiVersion: batch/v1
kind: Job
metadata:
name: migrate-db
spec:
template:
spec:
containers:
- name: migrate
image: myapp:latest
command: ["./migrate.sh"]
restartPolicy: Never
backoffLimit: 3
```
## CronJob: a Job, on a schedule
A CronJob doesn't run Pods itself — it creates a new Job on a cron schedule (`schedule:
"0 2 * * *"` for "every day at 2am"), and that Job behaves exactly as described above. Use a
CronJob for anything that should run periodically and finish: nightly backups, scheduled
reports, periodic cleanup.
A CronJob that reliably fails to complete (an image that always crashes, for example) will keep
creating new Jobs on schedule, each failing the same way — this is a real, common way to
silently exhaust cluster resources (job/pod history accumulating) if nobody's watching.
## DaemonSet: exactly one Pod per node, automatically
A DaemonSet doesn't declare a replica count at all — it ensures exactly one matching Pod runs
on every node in the cluster (or every node matching a selector), and automatically adds a Pod
when a new node joins. Use a DaemonSet for per-node infrastructure: a log-shipping agent, a
node-level monitoring exporter, a network plugin component — things that fundamentally need to
exist once per machine, not "N copies somewhere."
## The decision in one pass
| You need... | Use |
|---|---|
| Always-on, N identical copies, restarted forever | Deployment |
| Run once (or a few times) until it finishes successfully | Job |
| The above, but on a recurring schedule | CronJob |
| Exactly one copy on every node | DaemonSet |
## Instructor Notes
**Approximate timing:** 15 minutes read; 10 minutes live.
**Demo plan (if extending this PARTIAL module later):** live-create a Job with a command that
deliberately fails once then succeeds (`backoffLimit: 2`), and watch `kubectl get pods` show
the failed attempt alongside the eventual successful one — makes the retry-until-backoffLimit
behavior concrete.
**Likely misconceptions to watch for:**
- Students often assume a Job that finishes successfully gets cleaned up automatically. It
doesn't by default — the completed Pod/Job objects remain until manually deleted or a
`ttlSecondsAfterFinished` is configured (not covered here, worth a one-line mention if asked).
- "DaemonSet" is sometimes confused with "one Pod, cluster-wide" (singular). It's the opposite —
one Pod PER NODE, which can be many Pods total in a multi-node cluster.
**Discussion question:** "Why doesn't a Deployment support 'run once and be done'?" (Answer: a
Deployment's whole contract is maintaining a standing replica count forever — repurposing it for
one-shot work would fight that contract rather than use a primitive designed for it.)
**Optional extension:** ask what would go wrong if a CronJob's schedule ran more often than a
single Job execution takes to complete (answer: overlapping Job runs, unless
`concurrencyPolicy` is set to `Forbid` or `Replace` — a real gotcha worth naming even without
building a full lab around it).
@@ -0,0 +1,45 @@
# Knowledge Check: Choosing the Right Workload Primitive
---
**1. You need to run a database migration script exactly once when a new version deploys. Which
primitive fits, and why not a Deployment?**
*Answer:* A Job. A Deployment expects its Pods to run forever and replaces any that exit; a
migration script is supposed to exit successfully when done, which a Deployment would treat as
a failure to "fix" by restarting it — the wrong behavior for a task that's supposed to finish.
---
**2. A CronJob scheduled for "every 5 minutes" keeps creating Jobs that fail immediately due to
a bad image tag. What actually happens over the next hour if nobody notices?**
*Answer:* A new failing Job (and its Pod) gets created roughly every 5 minutes, accumulating
failed Job/Pod objects and consuming cluster resources — nothing stops this automatically. This
is exactly the kind of slow, easy-to-miss resource exhaustion a real on-call engineer should
watch for.
---
**3. True or false: a DaemonSet's Pod count scales with a `replicas` field, the same as a
Deployment.**
*Answer:* False. A DaemonSet has no `replicas` field — its Pod count is implicitly "one per
matching node," and it grows or shrinks automatically as nodes join or leave the cluster.
---
**4. You need a log-shipping agent that must run on every node, including any new node added
later. Which primitive, and what would go wrong if you used a Deployment with a high replica
count instead?**
*Answer:* A DaemonSet. A Deployment's replicas are placed by the scheduler wherever it decides
is best — nothing guarantees one per node, and a newly-added node wouldn't automatically get a
copy the way a DaemonSet guarantees.
## Instructor Notes
**Facilitation:** Question 2 is worth emphasizing — it's a genuinely realistic "how does a
cluster quietly get into trouble" scenario, not just trivia.
**Approximate timing:** 5 minutes.