docs(phase3b): document the learning and assessment engine
New docs/PHASE3B_LEARNING_AND_ASSESSMENT.md covers the attempt/learning-record model, progressive hints (including the Gateway response-rewrite mechanism), curriculum completion rules, the assessment/exam engine and its timer/draw integrity guarantees, the RootNode Capstone design, achievement records, the deliberately minimal versioning model, instructor analytics, the Phase 3B RBAC matrix, security/integrity assumptions, dev/test commands, known limitations, and the Phase 4 boundary. Updates ENGINEERING_SOP.md (roadmap items 2/7/8 now done, new section 12, corrects a stale capstone-placeholder description in section 4), LAB_ARCHITECTURE.md's status line, README.md (corrects the 'Lab image unmodified' claim now that its frontend bundle carries Phase 3B UX, plus new platform properties), platform/README.md, and platform/api/README.md.
This commit is contained in:
+46
-12
@@ -71,14 +71,18 @@ Two separate structures, don't confuse them:
|
||||
- **RootNode path** (`scenarios/rootnode-path.json`) — the guided onboarding sequence,
|
||||
separate from bundles on purpose (it's *staged*, bundles are *flat*, and the bundle loader
|
||||
would choke on a staged file if it were dropped in `scenarios/bundles/`). Loaded by
|
||||
`loadCurriculumPath()`/`GET /api/curriculum/path`. Stages are Foundation → Workloads →
|
||||
Networking → Configuration → Troubleshooting → Real Ops → Capstone, currently built by
|
||||
reusing existing `1-k8s-basics` and `rootnode-real-ops` scenarios grouped by their existing
|
||||
`category` field — no new scenario content was written for this. **Capstone stage is a
|
||||
locked placeholder** (`"locked": true`, empty `scenario_ids`) — the capstone engine itself
|
||||
is a separate, not-yet-built roadmap item (see §7, item 6). Do not put real scenarios in it
|
||||
until that engine exists, or you'll reproduce the "duplicate beginner scenarios as capstone"
|
||||
anti-pattern this path was built to avoid.
|
||||
`loadCurriculumPath()`/`GET /api/curriculum/path` (raw content) and, as of Phase 3B,
|
||||
`computeCurriculumState()`/`GET /api/curriculum/state` (rule-driven lock/complete/next-scenario
|
||||
state — see `docs/PHASE3B_LEARNING_AND_ASSESSMENT.md` §4). Stages are Foundation → Workloads →
|
||||
Networking → Configuration → Troubleshooting → Real Ops → Capstone, built by reusing existing
|
||||
`1-k8s-basics` and `rootnode-real-ops` scenarios grouped by their existing `category` field —
|
||||
no new scenario content was written for this. Each stage carries a `completion_rule`
|
||||
(`all_required`/`threshold`/`prerequisites`). **Capstone is no longer a hardcoded placeholder**
|
||||
— it's `"type": "capstone"` with an `"unlock_rule"` and a `"capstone_assessment_id"` pointing
|
||||
at the `rootnode-capstone` assessment definition (Postgres, `assessment_definitions` table),
|
||||
which is where its actual scenario draw lives — see the Phase 3B doc §7. Don't put real
|
||||
scenarios directly into the JSON stage's `scenario_ids` for capstone; that field stays empty
|
||||
by design, the draw is defined in the assessment definition instead.
|
||||
|
||||
To add/reorder a stage: edit `rootnode-path.json` directly, referencing existing
|
||||
`scenarios/data/*.json` ids. No backend code change needed — it's read fresh via
|
||||
@@ -138,7 +142,9 @@ Order agreed 2026-08-17/18. Status as of this doc:
|
||||
1. RootNode identity/branding pass — **partial.** Shell banner fixed
|
||||
(`⎈ RootNode Academy Lab — node: k8s-lab`). Upstream KubeKosh attribution left untouched in
|
||||
footer/README/LICENSE. Broader "make Real Ops first-class" UI work not started.
|
||||
2. RootNode curriculum mode — **backend done** (§4), frontend not started.
|
||||
2. RootNode curriculum mode — **done**. Rule-driven stage completion/unlock as of Phase 3B
|
||||
(§12), student-facing tabs (My Learning/Practice/Assessments/Capstone/Achievements) at
|
||||
`frontend/src/platform/`.
|
||||
3. Student accounts / persistent progress — **done**, Postgres-authoritative as of Phase 2B
|
||||
(§9), sessions persistent as of Phase 3A (§10). Roster name-picking is still classroom/pilot
|
||||
identification, not strong student auth — documented, not silently glossed over
|
||||
@@ -156,15 +162,18 @@ Order agreed 2026-08-17/18. Status as of this doc:
|
||||
documented adapter, not a working integration (docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md
|
||||
§16).
|
||||
6. Scenarios → teaching modules (incident context/investigation/prevention) — not started.
|
||||
7. Capstone/exam engine — not started (placeholder stage only, see §4).
|
||||
8. Progressive hint levels — not started.
|
||||
7. Capstone/exam engine — **done** as of Phase 3B (§12). RootNode Capstone (Real Ops-weighted,
|
||||
unlockable by curriculum-completion rule, not a permanent placeholder) plus a general exam
|
||||
engine (definitions, cohort assignment, random scenario draw, server-authoritative timer).
|
||||
8. Progressive hint levels — **done** as of Phase 3B (§12). Server-enforced sequential reveal,
|
||||
tracked per attempt, policy-aware under assessment mode (free/disabled/flagged).
|
||||
9. Production observability — not started.
|
||||
10. CI scenario testing — not started.
|
||||
11. Container/build optimization — explicitly deprioritized by design (platform works, not urgent).
|
||||
|
||||
Next slice: production OIDC/SSO, the production two-VM Proxmox deployment, and teaching-module
|
||||
scenario content are the biggest remaining items. See
|
||||
`docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md` §18 for the full BUILT NOW vs. FUTURE split.
|
||||
`docs/PHASE3B_LEARNING_AND_ASSESSMENT.md` §21 for the full Phase 4 boundary.
|
||||
|
||||
## 8. Platform Provisioner (Phase 1)
|
||||
|
||||
@@ -246,3 +255,28 @@ Going forward: after any `npm install` that changes a service's dependencies, th
|
||||
`package-lock.json` is part of the same commit as the `package.json` change — never left
|
||||
untracked, never regenerated-and-ignored. Validate a lockfile actually matches its
|
||||
`package.json` with `npm ci` (fails loudly on drift) before committing either.
|
||||
|
||||
## 12. Learning & Assessment Engine (Phase 3B)
|
||||
|
||||
Full design: `docs/PHASE3B_LEARNING_AND_ASSESSMENT.md`.
|
||||
|
||||
- **What it is:** the teaching/assessment layer on top of everything Phases 1–3A built —
|
||||
detailed per-attempt telemetry (`scenario_attempts`, evolved from Phase 2B's one-row-per-click
|
||||
shape into a real attempt lifecycle), server-enforced progressive hints, rule-driven
|
||||
curriculum stage/capstone unlock logic, a full exam + capstone assessment engine
|
||||
(`assessment_definitions`/`assessment_sessions`/`assessment_session_scenarios`), and
|
||||
persistent `achievements` records.
|
||||
- **Where the boundary moved:** the Gateway now rewrites one Lab response
|
||||
(`GET /lab/:token/api/scenarios/:scenarioId`, stripping `hints`) so progressive hints can be
|
||||
genuinely server-gated rather than just UI-collapsed — the Lab image's own code/bytes are
|
||||
still untouched; this is a response rewrite at the Gateway boundary, same pattern as every
|
||||
other Lab-mediation decision made in earlier phases. The Lab's own frontend bundle *did*
|
||||
change this phase (`ScenarioPanel.jsx`, new `ProgressiveHints`/`AssessmentBanner` components)
|
||||
— Phase 3B explicitly authorized enhancing Lab UX, unlike prior phases.
|
||||
- **Test procedure:** `platform/test/e2e-phase3b.js` — covers attempt telemetry, hint
|
||||
sequencing/persistence-through-Lab-reset, curriculum/capstone lock-unlock, the exam engine
|
||||
(random draw, timer, expiry, restart persistence), two-student isolation, and instructor
|
||||
analytics/RBAC.
|
||||
- **Known limitations:** see the Phase 3B doc §20 — no automatic abandoned-attempt detection, no
|
||||
per-cohort needs-attention threshold configuration, raw-JSON pool_spec authoring in the admin
|
||||
UI (correct, not yet a guided form).
|
||||
|
||||
@@ -2,14 +2,16 @@
|
||||
|
||||
Status: **Phase 1 (platform foundation) + Phase 2A (Gateway + student flow) + Phase 2B
|
||||
(Platform API split, Postgres academic data, server sessions, admin portal) + Phase 3A
|
||||
(persistent sessions, per-instructor accounts, RBAC, audit log) IMPLEMENTED, 2026-08-18** — see
|
||||
`docs/PHASE1_PLATFORM_FOUNDATION.md`, `docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md`,
|
||||
`docs/PHASE2B_ADMIN_AND_PLATFORM_DATA.md`, and `docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md`
|
||||
for concrete schema/API/test results, and `platform/README.md` for how to run the stack. This
|
||||
document remains the architecture record for the full ephemeral-lab direction; production
|
||||
OIDC/SSO (an adapter interface exists, not a working integration — see the Phase 3A doc §16)
|
||||
and production Proxmox deployment are still future work (Phase 3B+ — see the Phase 3A doc §18).
|
||||
Supersedes the namespace-isolation
|
||||
(persistent sessions, per-instructor accounts, RBAC, audit log) + Phase 3B (learning/assessment
|
||||
engine: attempt telemetry, progressive hints, curriculum rules, exam + capstone engine,
|
||||
achievements) IMPLEMENTED, 2026-08-18** — see `docs/PHASE1_PLATFORM_FOUNDATION.md`,
|
||||
`docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md`, `docs/PHASE2B_ADMIN_AND_PLATFORM_DATA.md`,
|
||||
`docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md`, and
|
||||
`docs/PHASE3B_LEARNING_AND_ASSESSMENT.md` for concrete schema/API/test results, and
|
||||
`platform/README.md` for how to run the stack. This document remains the architecture record
|
||||
for the full ephemeral-lab direction; production OIDC/SSO (an adapter interface exists, not a
|
||||
working integration) and production Proxmox deployment are still future work (Phase 4 — see the
|
||||
Phase 3B doc §21). Supersedes the namespace-isolation
|
||||
(`shared_namespace`/`ephemeral_cluster`) approach that was in flight before this redirect — see
|
||||
`scenarios/SCHEMA.md`'s `isolation_type` note for the pointer back here.
|
||||
|
||||
|
||||
@@ -0,0 +1,370 @@
|
||||
# Phase 3B — Learning & Assessment Engine
|
||||
|
||||
Turns the platform infrastructure built in Phases 1–3A (identity, sessions, RBAC, disposable
|
||||
Labs, Gateway routing, admin portal) into a real teaching and assessment system: detailed
|
||||
attempt telemetry, progressive hints, rule-driven curriculum progression, a proper exam engine,
|
||||
a RootNode Capstone, and persistent achievement records. Builds on
|
||||
`docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md` — read that first if you haven't.
|
||||
|
||||
## 1. What changed, in one paragraph
|
||||
|
||||
Phase 2B's `scenario_attempts` table was one row per validation click, written atomically at
|
||||
completion, with `hint_count` deliberately left unmeasured. Phase 3B evolves it into a real
|
||||
attempt lifecycle — one row per continuous working session on a scenario, with live counters
|
||||
(validation attempts/passes/fails, hint reveals, duration) updated in place as the student
|
||||
works. On top of that: a rule-driven curriculum-progression engine that actually locks/unlocks
|
||||
stages instead of a permanent capstone placeholder; a progressive hint system that's genuinely
|
||||
server-enforced (not just collapsed-by-default in the UI); a full exam/capstone assessment
|
||||
engine with server-authoritative timers and random scenario draws; and persistent achievement
|
||||
records for stage/curriculum/assessment/capstone completion.
|
||||
|
||||
## 2. Learning record model
|
||||
|
||||
`scenario_attempts` (migration `012`) gained: `cohort_id`, `curriculum_stage` (both snapshotted
|
||||
at attempt-start, so a later cohort/curriculum change never rewrites history), `status`
|
||||
(`in_progress`/`completed`/`abandoned`), `validation_attempt_count`, `validation_pass_count`,
|
||||
`validation_fail_count`, `highest_hint_level`, `duration_seconds`, `assessment_session_id`.
|
||||
`result` (Phase 2B) is now nullable — meaningful only once `status = 'completed'`.
|
||||
`hint_count` (Phase 2B, always `NULL`) finally gets a `DEFAULT 0` and real increments.
|
||||
|
||||
This is an **evolution, not a new table** — Phase 2B's historical rows are backfilled (one
|
||||
validation performed, matching their original one-row-per-click semantics), not dropped or
|
||||
reshaped. `platform/api/src/attempts.js` owns the lifecycle:
|
||||
|
||||
- `getOrCreateOpenAttempt` — finds the open (`in_progress`) attempt for
|
||||
`(student, scenario, environment)`, or starts one. Environment-scoped so a destroyed/recreated
|
||||
Lab never silently resumes an attempt tied to compute that no longer exists.
|
||||
- `recordValidation` — increments the attempt/pass/fail counters; closes the attempt
|
||||
(`status='completed'`, `duration_seconds` computed) only on a pass.
|
||||
- `recordHintReveal` — sequential only: level *N* can't be revealed before *N-1* has been.
|
||||
Returns a 409 once every hint is exhausted.
|
||||
|
||||
No telemetry is invented that the Lab doesn't actually emit — `duration_seconds` is measured
|
||||
server-side from `started_at` to completion (not client-reported), and hint reveals only exist
|
||||
because Phase 3B builds a real reveal endpoint (§3), not because the old static hint array was
|
||||
reinterpreted as "measured."
|
||||
|
||||
## 3. Validation telemetry & progressive hints
|
||||
|
||||
`POST /api/scenarios/:id/start` — fire-and-forget from the Lab UI when a scenario becomes
|
||||
active (both task and MCQ types); creates/reopens the attempt with a real `started_at`.
|
||||
|
||||
`POST /api/progress/record` (Gateway-relayed from the Lab's existing validate/answer calls,
|
||||
unchanged mechanism from Phase 2A/2B) now routes through `attempts.recordValidation` instead of
|
||||
a flat insert, and — on a pass — checks curriculum/capstone/assessment consequences (§4, §6).
|
||||
|
||||
`POST /api/scenarios/:id/hint` reveals hints **one at a time, server-enforced**. This required a
|
||||
real architectural change: Phase 2B's Lab backend (unmodified, per the "Lab image stays
|
||||
unmodified" convention through Phase 3A) returns the scenario's full `hints` array in
|
||||
`GET /api/scenarios/:id` — always did, since the Lab has no student/attempt concept at all. A
|
||||
client could already see hint 3 in the raw response even with the UI collapsed. Making the gate
|
||||
real meant the Gateway now **strips `hints` from that response** (replacing it with a bare
|
||||
`hint_count`) for exactly the multi-student, Platform-mediated deployment
|
||||
(`platform/gateway/src/server.js`, `GET /lab/:token/api/scenarios/:scenarioId`) — the Lab
|
||||
image's own bytes are still untouched; this is a response rewrite at the boundary, the same
|
||||
pattern already used for the `hints`-stripping shape as everything else the Gateway mediates.
|
||||
Standalone/single-container dev mode never goes through the Gateway, so it keeps the original
|
||||
fully-preloaded array unchanged — the frontend detects which mode it's in
|
||||
(`scenario.hints === undefined && typeof scenario.hint_count === 'number'`) and switches to
|
||||
`ProgressiveHints` (`frontend/src/components/ProgressiveHints.jsx`) only in the gated case.
|
||||
|
||||
Instructor-visible attempt summary matches the shape in the Phase 3B spec directly:
|
||||
|
||||
```
|
||||
Student: Alice Scenario: Probe Port Mismatch Attempts: 2
|
||||
Validation: Failed 2, Passed 1 Hints used: 1/3 Time: 14m 22s Final: Completed
|
||||
```
|
||||
|
||||
— all straight off `scenario_attempts` columns, no derived/estimated numbers, and explicitly
|
||||
**not** turned into a score. See §12 for where instructors see this.
|
||||
|
||||
## 4. Curriculum completion rules
|
||||
|
||||
`scenarios/rootnode-path.json` gained a `version` field and, per stage, a `completion_rule`:
|
||||
|
||||
```json
|
||||
{ "type": "all_required" } // every scenario_id in the stage must be completed (default)
|
||||
{ "type": "threshold", "percent": 80 } // at least N% completed
|
||||
{ "type": "prerequisites", "scenario_ids": [...] } // only these specific ids must be completed
|
||||
```
|
||||
|
||||
The capstone stage is no longer a hardcoded `"locked": true` placeholder — it carries
|
||||
`"type": "capstone"`, an `"unlock_rule"` (`{"type":"stages_complete","stage_ids":[...]}`), and a
|
||||
`"capstone_assessment_id"` pointing at the assessment definition that drives it (§6/§7).
|
||||
|
||||
`platform/api/src/curriculum-rules.js`'s `computeCurriculumState(path, progress, capstoneStatus)`
|
||||
is the rule engine — pure, synchronous, no DB access — answering: per-stage locked/complete/
|
||||
percent/next-scenario, the student's current stage, overall percent complete, and the capstone's
|
||||
unlock status (server route supplies `capstoneStatus` after checking for a
|
||||
`capstone_completed` achievement, since that's Postgres state this module doesn't touch).
|
||||
Exposed at `GET /api/curriculum/state` (new; `GET /api/curriculum/path` from Phase 2A/2B is
|
||||
unchanged and still used by the pre-existing dashboard summary).
|
||||
|
||||
Stage/curriculum completion also triggers achievement recording (§8) automatically, from inside
|
||||
`POST /api/progress/record` — no separate "check for completion" step the student or frontend
|
||||
has to remember to call.
|
||||
|
||||
## 5 & 6. Assessment model and exam engine
|
||||
|
||||
`PRACTICE` is just... practice: the existing curriculum flow, hints freely available (subject to
|
||||
the scenario actually having any), unrestricted retries, no timer, no session concept.
|
||||
|
||||
`ASSESSMENT` (exam or capstone — same underlying engine, §7 covers what makes a capstone
|
||||
different) is a **definition** (the reusable template: pool, duration, hint policy, completion
|
||||
rule) producing **sessions** (one per student attempt, immutable snapshot of what was drawn and
|
||||
under what rules). New tables (migration `013`):
|
||||
|
||||
```
|
||||
assessment_definitions slug PK, type ('exam'|'capstone'), title, duration_minutes,
|
||||
hint_policy ('free'|'disabled'|'flagged'), pool_spec JSONB,
|
||||
completion_rule JSONB, unlock_rule JSONB (capstone only),
|
||||
version INTEGER, status ('active'|'inactive')
|
||||
assessment_cohort_assignments which cohorts a definition is available to
|
||||
assessment_sessions one per student attempt — snapshots type/title/duration_minutes/
|
||||
hint_policy from the definition at draw time, plus
|
||||
definition_version, started_at, server-computed expires_at,
|
||||
submitted_at, random_seed, result JSONB
|
||||
assessment_session_scenarios the drawn set, ordered, with per-scenario status + a real FK
|
||||
back to the scenario_attempts row once worked
|
||||
```
|
||||
|
||||
`exam_sessions` (Phase 1/2B, historical-import-only from the old Lab-local SQLite exam
|
||||
bookkeeping) is **left untouched** — it was already scoped as dead historical data, never the
|
||||
live mechanism, so there was nothing to migrate into the tables above without inventing data
|
||||
that was never real. This is a deliberate non-duplication: the assessment tables above *are*
|
||||
the real thing Phase 2B's note said didn't exist yet.
|
||||
|
||||
`pool_spec` shape:
|
||||
|
||||
```json
|
||||
{ "required": ["scenario-id", ...], "random": [{ "count": 1, "from": ["a", "b", "c"] }, ...] }
|
||||
```
|
||||
|
||||
`platform/api/src/assessments.js`'s `drawScenarios` includes every `required` id, plus `count`
|
||||
random picks from each `random` group, shuffled together into presentation order. The draw uses
|
||||
a tiny deterministic PRNG (mulberry32, seeded from `crypto.randomBytes` — no new dependency)
|
||||
purely so the seed can be recorded for audit; a student never sees the seed or the undrawn pool
|
||||
before starting.
|
||||
|
||||
**One assessment at a time per student** — `startSession` refuses if the student already has a
|
||||
`not_started`/`in_progress` session for *any* definition. This keeps "which scenario is this
|
||||
validate call happening inside" unambiguous for hint-policy enforcement (§19) without needing
|
||||
the client to declare a mode.
|
||||
|
||||
**Timer**: `expires_at` is computed once, server-side, at `started_at + duration_minutes`, and
|
||||
never recomputed from anything the client sends. Expiry is checked **lazily** — whenever a
|
||||
session is read or written, if `now() > expires_at` and it's still `in_progress`, it flips to
|
||||
`expired` right then. No background cron (explicitly out of scope this phase — see the
|
||||
non-goals list). Verified live: a 1-minute test definition correctly transitioned to `expired`
|
||||
on the first request after the minute passed, and unblocked starting a new assessment.
|
||||
|
||||
## 7. RootNode Capstone
|
||||
|
||||
A capstone is an `assessment_definitions` row with `type='capstone'` — same engine as an exam,
|
||||
different framing and unlock gate. The shipped default (seeded in migration `013`, idempotent
|
||||
`INSERT ... ON CONFLICT DO NOTHING`) is `rootnode-capstone`:
|
||||
|
||||
- **Required** (always drawn): all four RootNode Real Ops scenarios
|
||||
(`rootnode-runaway-monitor`, `broken-deployment`, `rootnode-probe-port-mismatch`,
|
||||
`rootnode-enforce-limits`) — "feature prominently," per the spec, satisfied by making them
|
||||
non-optional rather than just likely.
|
||||
- **Random**: one Workloads task, one Configuration task, one Troubleshooting task — task-type
|
||||
scenarios preferred over MCQ, since the point is operational ability, not recall.
|
||||
- 90 minutes, `hint_policy: 'flagged'` (available, but every use is recorded — appropriate for
|
||||
a capstone that should mostly reflect independent work without being punitive about needing a
|
||||
nudge).
|
||||
- `unlock_rule`: every non-capstone curriculum stage complete (Foundation through Real Ops).
|
||||
|
||||
`GET /api/capstone` is a student-friendly view combining `computeCurriculumState`'s capstone
|
||||
block with the definition and the student's own session history —
|
||||
`locked` → `unlocked` → `completed`, with a prerequisite checklist while locked. Verified live,
|
||||
end to end: locked with all six blocking stage ids, unlocked to zero blocking ids the moment the
|
||||
last stage completed, a real draw (7 scenarios, all four Real Ops present), completion of all
|
||||
seven finalizing the session and recording a `capstone_completed` achievement.
|
||||
|
||||
## 8. Achievement / completion records
|
||||
|
||||
`achievements` (migration `014`): `student_id`, `achievement_type`
|
||||
(`stage_completed`/`curriculum_completed`/`assessment_completed`/`capstone_completed`),
|
||||
`curriculum_path_id`/`curriculum_version` (for stage/curriculum types),
|
||||
`assessment_session_id` (for assessment/capstone types), `achieved_at`, `metadata`.
|
||||
|
||||
`recordAchievement` is idempotent via a `WHERE NOT EXISTS` guard (not a unique-index/
|
||||
`ON CONFLICT` dance — simple and correct at this write volume) — calling it again for something
|
||||
already recorded is a safe no-op. Triggered automatically: `checkStageAndCurriculumAchievements`
|
||||
runs inside `POST /api/progress/record` after every pass; `recordAssessmentAchievement` runs
|
||||
after `assessments.recordScenarioOutcome`/`submitSession` finalizes a session, and only on an
|
||||
actual pass (defensive — today's only `completion_rule` is `all_scenarios_passed`, which already
|
||||
guarantees a `completed` session passed, but a future threshold-based rule shouldn't silently
|
||||
grant achievements for a partial result).
|
||||
|
||||
**Not a certificate.** No public/shareable artifact is generated — this is explicitly a Phase
|
||||
4/5 concern (see the non-goals list); a certificate can be rendered from these records later
|
||||
without needing new data.
|
||||
|
||||
## 9. Assessment/exam Lab behavior
|
||||
|
||||
An assessment doesn't provision a separate Lab — the student works through their drawn
|
||||
scenarios inside the **same** disposable environment their curriculum work already uses. This
|
||||
is what makes "Lab reset doesn't erase assessment state" nearly free: assessment state
|
||||
(`assessment_sessions`, `assessment_session_scenarios`, `scenario_attempts`) lives entirely in
|
||||
Platform Postgres; the Lab environment is purely compute. Resetting/recreating it changes the
|
||||
container, never the rows. Verified live: reset an environment mid-assessment,
|
||||
`expires_at`/`status` on the session were unchanged afterward.
|
||||
|
||||
**Assessment-mode detection is entirely server-derived** — `findActiveAssessmentContextForScenario`
|
||||
looks up whether the scenario a `validate`/`hint` call is happening against is currently in the
|
||||
student's own active session's drawn set. Nothing the client sends declares "I'm in assessment
|
||||
mode"; the server's own `assessment_session_scenarios` table is the only source of truth (see
|
||||
§19).
|
||||
|
||||
If Lab infrastructure fails mid-assessment, the assessment session itself is unaffected — an
|
||||
instructor can reset (or, if truly needed, destroy/reprovision) the student's Lab via the
|
||||
existing admin environment controls (Phase 2B) without touching assessment state at all; there
|
||||
is no code path that resets an assessment as a side effect of a Lab operation.
|
||||
|
||||
## 10 & 11. Student UI and Lab UX
|
||||
|
||||
`frontend/src/platform/StudentDashboard.jsx` is now tabbed: **My Learning** (the original
|
||||
Phase 2A/2B "Up Next" + Launch/Reset Lab flow, extracted unchanged into `MyLearningTab.jsx` and
|
||||
extended with an overall percent-complete bar from the new rule-driven curriculum state),
|
||||
**Practice** (scenario/stage status + recent attempt history), **Assessments** (available/
|
||||
active-with-live-countdown/history), **Capstone** (locked checklist → unlocked → start/resume →
|
||||
completed, with attempt history), **Achievements** (a flat, dated list — not a leaderboard).
|
||||
|
||||
Inside the Lab UI itself (`frontend/src/components/ScenarioPanel.jsx`, `App.jsx`):
|
||||
`ProgressiveHints` replaces the static hint array render when the Gateway has stripped hints
|
||||
(§3); an `AssessmentBanner` (new, top-level, polls `/api/assessments/active`) shows a live
|
||||
countdown and per-scenario completion count whenever an assessment is running — distinct from,
|
||||
and rendered alongside, the pre-existing Lab-local exam-mode UI (`ExamTimer`/`ExamSession` etc.,
|
||||
upstream KubeKosh feature, untouched). Both render nothing in standalone/single-container dev
|
||||
mode (no Platform API to poll there), so neither regresses that deployment path.
|
||||
|
||||
## 12. Instructor learning analytics
|
||||
|
||||
`/admin` → **Learning**: cohort summary (student count, 7-day active-learner count, average
|
||||
completion, per-stage completion counts), a **Needs Attention** list, and a per-student
|
||||
drill-down (curriculum state, full attempt history with validation/hint counts, assessment
|
||||
history). `/admin` → **Assessments**: create/edit definitions (admin/owner), assign to a cohort,
|
||||
view/cancel sessions — visible to every instructor role, cohort-scoped server-side for the plain
|
||||
`instructor` role the same way Phase 3A's other cohort-scoped routes are.
|
||||
|
||||
`GET /api/admin/scenario-analytics` (optional `?cohort_id=`): completions, distinct students,
|
||||
average validation attempts, total failures, total hints used, average completion time — one
|
||||
`GROUP BY scenario_id` query over `scenario_attempts`, nothing pre-aggregated or cached.
|
||||
|
||||
## 13. "Needs Attention"
|
||||
|
||||
`GET /api/admin/cohorts/:id/needs-attention` computes four signals per student, on read (no
|
||||
stored table, no background job): **Repeated Attempts** (an open attempt with ≥3 failed
|
||||
validations), **May Need Help** (an open attempt where a hint has been used but the scenario
|
||||
isn't done), **Needs Attention — inactive** (no activity in 7 days), and **Needs Attention —
|
||||
incomplete assessment** (their most recent assessment session expired without completing). All
|
||||
four thresholds are named constants in `admin-learning-routes.js`
|
||||
(`NEEDS_ATTENTION_FAIL_THRESHOLD`, `NEEDS_ATTENTION_STALE_DAYS`), not hardcoded inline. Wording
|
||||
is deliberately instructional (`Needs Attention`, `May Need Help`, `Repeated Attempts`) — an
|
||||
explicit test asserts none of the returned labels match punitive language.
|
||||
|
||||
## 14 & 15. Assessment/capstone management (admin)
|
||||
|
||||
A structured form (`frontend/src/admin/AssessmentsView.jsx`) — id, title, type, duration, hint
|
||||
policy, and a raw JSON `pool_spec` textarea. Not a drag-and-drop authoring tool, per the explicit
|
||||
scope line; the JSON textarea is honest about what's actually being edited rather than hiding it
|
||||
behind partial UI that can't express every pool shape. Editing a definition's
|
||||
duration/hint_policy/pool_spec/completion_rule bumps its `version` automatically
|
||||
(`admin-learning-routes.js` diffs the incoming fields against the current row) — cosmetic edits
|
||||
(title, description) don't. Create/edit is admin/owner only server-side; assign-to-cohort and
|
||||
session view/cancel are available to a scoped `instructor` too, matching the RBAC matrix (§18).
|
||||
|
||||
## 16. Versioning
|
||||
|
||||
Deliberately minimal, per the explicit "don't overbuild full CMS versioning" instruction:
|
||||
|
||||
- **Curriculum path**: a plain `"version": 1` integer in `rootnode-path.json` itself, recorded
|
||||
on `stage_completed`/`curriculum_completed` achievements as a snapshot.
|
||||
- **Assessment definitions**: a plain incrementing `version` integer, bumped only on a semantic
|
||||
change (duration/hint_policy/pool_spec/completion_rule — anything that would change what a
|
||||
session is graded against). `assessment_sessions.definition_version` snapshots which version
|
||||
was live at draw time, alongside a full redundant snapshot of the fields themselves
|
||||
(title/duration/hint_policy) — so a later definition edit, or even deletion, can never change
|
||||
what an already-started session is measured against.
|
||||
|
||||
No full version-history table exists for either — if that's ever needed, `version` is already a
|
||||
column to build on, not a retrofit.
|
||||
|
||||
## 17. RBAC matrix (Phase 3B additions)
|
||||
|
||||
| Action | owner | admin | instructor | student |
|
||||
|---|---|---|---|---|
|
||||
| Create/edit assessment definitions | ✅ | ✅ | ❌ | ❌ |
|
||||
| Assign an assessment to a cohort | ✅ | ✅ | ✅ (own cohorts) | ❌ |
|
||||
| View/cancel assessment sessions | ✅ (any) | ✅ (any) | ✅ (own cohorts) | ❌ |
|
||||
| View cohort learning summary / needs-attention / scenario analytics | ✅ (global) | ✅ (global) | ✅ (own cohorts) | ❌ |
|
||||
| View a student's learning detail | ✅ | ✅ | ✅ (if enrolled in an owned cohort) | own only |
|
||||
| Start/work an assessment, view own attempts/hints/achievements | ❌ | ❌ | ❌ | ✅ (own only) |
|
||||
|
||||
Enforced entirely server-side, reusing Phase 3A's `rbac.js` (`requireAdminOrOwner`,
|
||||
`requireAnyInstructor`, `requireCohortAccess`, `requireStudentCohortAccess`) — no new
|
||||
authorization primitives were needed, only new routes wired through the existing ones.
|
||||
|
||||
## 18. Security / integrity
|
||||
|
||||
- **Timer is server-authoritative** (§6) — `expires_at` fixed at draw time, never accepts a
|
||||
client value, checked lazily on every touch.
|
||||
- **Scenario draw is server-stored** — `assessment_session_scenarios` is the only source of
|
||||
truth for what a student is supposed to be working on; nothing the client sends can add,
|
||||
remove, or reorder it.
|
||||
- **A student cannot submit/read another student's session** — every assessment route filters
|
||||
by `student_id = req.studentId` server-side; a foreign session id returns 404, not 403 (same
|
||||
"don't confirm existence" posture as Phase 3A's RBAC denials) — verified by an explicit test.
|
||||
- **Gateway ownership** (Phase 2A/2B/3A) is unchanged and still enforced on every proxied Lab
|
||||
request, HTTP and WebSocket, including the new `/start` and `/hint` relays.
|
||||
- **Lab recreation never resets attempt/exam state** (§9) — verified live.
|
||||
- **Hidden solution/pool data**: undrawn pool alternatives are never returned to the student
|
||||
(only the actual draw); hints beyond the currently-unlocked level are never sent (§3, the
|
||||
actual mechanism, not just a UI convention).
|
||||
- **No invasive anti-cheat** — no browser surveillance, no lockdown mode. Explicitly out of
|
||||
scope, per the non-goals list.
|
||||
|
||||
## 19. Dev/test commands
|
||||
|
||||
```bash
|
||||
cd platform && docker compose -f docker-compose.dev.yml up -d
|
||||
(cd api && PLATFORM_DB_URL=postgres://postgres:postgres@localhost:5433/rootnode_platform \
|
||||
node bootstrap-owner.js [email protected] "Your Name" "a-real-password-12-chars-plus")
|
||||
|
||||
OWNER_EMAIL=[email protected] OWNER_PASSWORD=... node test/e2e-phase2a.js
|
||||
OWNER_EMAIL=[email protected] OWNER_PASSWORD=... node test/e2e-phase2b.js
|
||||
OWNER_EMAIL=[email protected] OWNER_PASSWORD=... node test/e2e-phase3a.js
|
||||
OWNER_EMAIL=[email protected] OWNER_PASSWORD=... node test/e2e-phase3b.js
|
||||
```
|
||||
|
||||
`e2e-phase3b.js` additionally shells out to `docker compose restart api` for its restart-
|
||||
persistence case (set `SKIP_RESTART_TEST=1` to skip that one case if running somewhere that
|
||||
isn't safe to restart).
|
||||
|
||||
## 20. Known limitations
|
||||
|
||||
- **No automatic "abandoned" detection** — an attempt a student never returns to stays
|
||||
`in_progress` forever; there's no reaper tying Lab destruction back to attempt state. A
|
||||
documented, deliberate scope line (would need wiring to Provisioner `environment_events`,
|
||||
more machinery than the instructional-visibility goal needs).
|
||||
- **`cohort_instructors.role_in_cohort`** (Phase 3A) still isn't consulted for finer-grained
|
||||
assessment permissions — any cohort assignment grants the full scoped-instructor action set.
|
||||
- **Needs-attention thresholds are fixed constants**, not per-cohort configurable yet.
|
||||
- **No pagination** on the audit log (Phase 3A limitation, unchanged) or on scenario analytics.
|
||||
- **Assessment pool_spec authoring is raw JSON** in the admin UI — correct and complete, but not
|
||||
a guided form; a validation-assist UI is a natural but unbuilt next step.
|
||||
- **One assessment at a time per student** — intentional (§6), but means a student can't have a
|
||||
capstone and an exam "in flight" simultaneously even if that were ever desired.
|
||||
|
||||
## 21. Phase 4 boundary
|
||||
|
||||
Everything in this document runs on a single dev machine against
|
||||
`platform/docker-compose.dev.yml`. **Not built, and explicitly out of scope this phase:**
|
||||
production Hetzner/Proxmox deployment, production OIDC provider integration, a parent portal,
|
||||
payments, public signup, automated email, equipment inventory, a multi-host scheduler,
|
||||
Grafana/Prometheus production dashboards, public certificates, a marketplace/self-paced commerce
|
||||
layer, the six held KubeKosh scenarios, invasive proctoring, and AI-driven grading. All of that
|
||||
stays exactly where the Phase 3A doc's non-goals list and this document's §21 leave it — Phase 4
|
||||
territory.
|
||||
Reference in New Issue
Block a user