ced4568
bf1e1b5c76
Master overview (PHASE4_PRODUCTION_INFRASTRUCTURE.md) plus five detail docs (PRODUCTION_DEPLOYMENT, PRODUCTION_NETWORK_SECURITY, BACKUP_DR, OBSERVABILITY, CAPACITY_PLANNING), a master PRODUCTION_VALIDATION_CHECKLIST, and 12 runbooks (first deployment, normal deploy, rollback, Postgres restore, VM rebuild for both hosts, Docker TLS rotation, TLS renewal, owner recovery, stuck Lab cleanup, failed migration, backup failure, disk failure, Lab Host resource pressure). PRODUCTION_NETWORK_SECURITY.md prominently documents the one genuine architectural gap this phase surfaced: the Gateway's existing env.internal_ip mechanism for reaching a Lab container assumes Gateway and Lab share a Docker daemon, which breaks once Labs run on a genuinely separate rn-lab-host-01 VM. The network-policy half is built (a firewall allowance for Gateway->Lab-Host on a published port range); the application half (Provisioner/Gateway address handling) is an explicit, reviewed, NOT-YET-BUILT follow-up flagged there rather than silently decided. Updated LAB_ARCHITECTURE.md, ENGINEERING_SOP.md, README.md, and platform/README.md with Phase 4 pointers, consistent with how every prior phase updated these. Status: PHASE 4 BUILD COMPLETE / PRODUCTION DEPLOYMENT PENDING HARDWARE.
246 lines
11 KiB
Markdown
246 lines
11 KiB
Markdown
<div align="center">
|
||
|
||
<img src="frontend/public/logo.svg" alt="KubeKosh" height="96" />
|
||
|
||
# KubeKosh — RootNode Academy Edition
|
||
|
||
**A browser-based Kubernetes lab for hands-on learners.**
|
||
Real K3s. Real `kubectl`. Instant validation. No cloud account, no local cluster.
|
||
|
||
[](LICENSE)
|
||
[](#)
|
||
[](https://k3s.io)
|
||
[](https://docs.docker.com/get-docker/)
|
||
[](https://github.com/ced4568)
|
||
|
||
</div>
|
||
|
||
---
|
||
|
||
> **Attribution.** This is a branded, modified fork of the excellent
|
||
> [`zeborg/kubekosh`](https://github.com/zeborg/kubekosh), licensed under Apache 2.0.
|
||
> RootNode Academy adds custom scenarios, branding, and multi-student deployment tooling on
|
||
> top of the upstream project. Original copyright and license notices are preserved in
|
||
> [`LICENSE`](LICENSE). Modified files are marked per Apache 2.0 §4. Full credit to
|
||
> the original author for the core platform.
|
||
|
||
---
|
||
|
||
## What It Is
|
||
|
||
KubeKosh runs a real single-node [K3s](https://k3s.io) cluster inside one Docker
|
||
container and pairs it with a browser terminal and automated scenario validation.
|
||
A learner opens a tab, gets a live cluster, runs real `kubectl`, and clicks **Validate**
|
||
to check their work against actual cluster state.
|
||
|
||
In the RootNode Academy curriculum this is the **capstone rung** — the Kubernetes & DevOps
|
||
track that sits on top of the Linux / Git / Docker foundations. It is aimed at the
|
||
advanced learner (roughly 14+), not the absolute beginner.
|
||
|
||
---
|
||
|
||
## Quick Start
|
||
|
||
**Prerequisite:** [Docker](https://docs.docker.com/get-docker/)
|
||
|
||
```bash
|
||
docker run -itd --name kubekosh --privileged -p 7554:80 ced4568/kubekosh:latest
|
||
```
|
||
|
||
Open **http://localhost:7554** and wait ~30s for the *Cluster Ready* indicator to turn green.
|
||
|
||
> `--privileged` is required — K3s needs kernel namespaces and cgroups.
|
||
> **Do not expose this container directly to the public internet.** It is a teaching
|
||
> sandbox. Multi-student exposure is handled by the deployment layer below, not by
|
||
> publishing the raw container.
|
||
|
||
### Persist Progress
|
||
|
||
```bash
|
||
docker run -itd --name kubekosh --privileged -p 7554:80 \
|
||
-v <host_dir>:/data ced4568/kubekosh:latest
|
||
```
|
||
|
||
Progress is stored in SQLite at `/data/progress.db`. Mount a host directory to `/data`
|
||
to keep progress across restarts — one volume per student.
|
||
|
||
### Build From Source
|
||
|
||
```bash
|
||
docker build -t kubekosh .
|
||
# multi-platform
|
||
docker buildx build --platform linux/amd64,linux/arm64 -t kubekosh .
|
||
```
|
||
|
||
---
|
||
|
||
## What's Inside
|
||
|
||
| Bundle | Focus | Exam Mode |
|
||
| ------ | ----- | --------- |
|
||
| 🌱 Kubernetes Basics | Core concepts | 60 min |
|
||
| 🧑✈️ Kubernetes Administrator | CKA | 120 min |
|
||
| 🛠️ Kubernetes Developer | CKAD | 120 min |
|
||
| 🛡️ Kubernetes Security | CKS | 120 min |
|
||
|
||
**Scenario types**
|
||
|
||
- **Task** — hands-on challenge in the live terminal; **Validate** auto-checks cluster state.
|
||
- **MCQ** — multiple choice with a detailed explanation on submission.
|
||
|
||
Custom RootNode Academy scenarios live alongside upstream ones in `scenarios/data/` and are
|
||
grouped into bundles in `scenarios/bundles/`. See [`scenarios/SCHEMA.md`](scenarios/SCHEMA.md).
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
```mermaid
|
||
flowchart TD
|
||
subgraph Container["Single Docker Image (--privileged)"]
|
||
NGINX["nginx :80 — reverse proxy"]
|
||
FE["Frontend<br/>React + Vite + xterm.js"]
|
||
BE["Backend<br/>Node / Express + node-pty (WebSocket PTY)"]
|
||
K3S["K3s<br/>single-node in-container cluster"]
|
||
DB[("SQLite<br/>/data/progress.db")]
|
||
|
||
NGINX --> FE
|
||
NGINX --> BE
|
||
BE -->|kubectl / PTY| K3S
|
||
BE --> DB
|
||
end
|
||
|
||
Student["Learner browser"] -->|:7554| NGINX
|
||
```
|
||
|
||
| Component | Technology |
|
||
| --------- | ---------- |
|
||
| Frontend | React + Vite, `xterm.js` |
|
||
| Backend | Node.js / Express, `node-pty` WebSocket PTY |
|
||
| Cluster | K3s (single-node, in-container) |
|
||
| Proxy | nginx on port `80`, mapped to host `7554` |
|
||
| Storage | SQLite (`better-sqlite3`) at `/data/progress.db` |
|
||
|
||
Everything runs inside one image managed by `scripts/entrypoint.sh`.
|
||
|
||
---
|
||
|
||
## Multi-Student Deployment (RootNode Academy addition)
|
||
|
||
The upstream project assumes one learner, one container — this image is still exactly that.
|
||
Its backend/entrypoint/Dockerfile are unmodified except for a node-pty build fix (see
|
||
`docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md` §1); its frontend bundle gained progressive-hint
|
||
controls and an assessment-mode indicator in Phase 3B (`docs/PHASE3B_LEARNING_AND_ASSESSMENT.md`
|
||
§10–11) — additive UI, not a rewrite of the Lab experience. RootNode Academy multiplies the
|
||
image into a real platform via separate provisioning and gateway layers rather than changing
|
||
its core:
|
||
|
||
- `platform/provisioner/` — Postgres-backed environment registry, per-student Docker
|
||
networking, firewall-enforced egress, idle reaper (`docs/PHASE1_PLATFORM_FOUNDATION.md`).
|
||
- `platform/api/` — lightweight Platform API: students, cohorts, curriculum, progress, detailed
|
||
scenario attempt telemetry, progressive hints, assessment/capstone engine, achievements,
|
||
persistent student/instructor sessions, per-instructor accounts (RBAC: owner/admin/
|
||
instructor), audit log, instructor/admin control plane. No k3s, no `--privileged`
|
||
(`docs/PHASE2B_ADMIN_AND_PLATFORM_DATA.md`, `docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md`,
|
||
`docs/PHASE3B_LEARNING_AND_ASSESSMENT.md`).
|
||
- `platform/gateway/` — the only student-facing path to a Lab: resolves identity from a
|
||
server-controlled, persistent session cookie (not a client-suppliable header/param), proxies
|
||
HTTP/WebSocket into that one student's container, never a raw container IP/port, and
|
||
server-gates progressive hint reveal by stripping the Lab's own hint payload before it reaches
|
||
the browser (`docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md`, session model in
|
||
`docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md` §3, hint gating in
|
||
`docs/PHASE3B_LEARNING_AND_ASSESSMENT.md` §3).
|
||
- The student flow (roster login → curriculum dashboard → Launch Lab → practice/assessments/
|
||
capstone → achievements) lives in `frontend/src/platform/`; the instructor/admin portal
|
||
(per-instructor login, cohorts, students, instructor accounts, audit log, classroom
|
||
"Prepare Labs," environment controls, learning analytics, assessment/capstone management)
|
||
lives in `frontend/src/admin/` at the `/admin` route — both additions alongside the existing
|
||
Lab UI, not a rewrite of it.
|
||
|
||
**Never publish the raw privileged container** — front it with isolation (that's what the
|
||
Gateway is). The one-container `docker run --privileged -p 7554:80 ...` command above stays
|
||
supported for local development and scenario authoring; it is not how this gets deployed to
|
||
real students.
|
||
|
||
Key properties of the platform (not this image):
|
||
|
||
- One container **per student**, its own Docker network, its own firewall chain — never a
|
||
shared cluster between learners. Verified with two simultaneous students: separate clusters,
|
||
separate progress, no cross-access.
|
||
- Progress/identity live outside the disposable container, in the Platform API's own
|
||
PostgreSQL (authoritative as of Phase 2B) — destroying or recreating a student's Lab never
|
||
loses their academic record. Verified with a full class-sized (3-student) cohort workflow:
|
||
provision, complete different scenarios, reset one student's Lab, destroy all Labs after
|
||
class — progress survives every step.
|
||
- Instructors sign in with their own account (no shared password) and manage cohorts, students,
|
||
curriculum assignment, and classroom Lab prep (provision an entire cohort before class, verify
|
||
readiness, destroy Labs after) from `/admin` — no manual database, Docker, or Kubernetes
|
||
access needed. Owners manage instructor accounts and roles; every instructor/admin action is
|
||
recorded in an audit log.
|
||
- Sessions (student and instructor) are Postgres-backed and survive a Platform API restart —
|
||
verified by actually restarting it mid-session and confirming access still works.
|
||
- Every validation attempt, hint reveal, curriculum stage completion, exam, and Capstone
|
||
attempt is recorded in Postgres, not the disposable Lab — a Lab reset or recreate never loses
|
||
attempt telemetry, hint progress, or an in-progress assessment's timer. Verified live:
|
||
reset a Lab mid-assessment, timer and drawn scenario set were unchanged afterward.
|
||
- Exams and the RootNode Capstone use a server-authoritative timer and a server-drawn,
|
||
server-stored scenario set — nothing the browser sends can change either.
|
||
- CPU/memory/PID/storage limits are enforced per container, config-driven.
|
||
- Egress is restricted to an allowlisted proxy at the firewall layer, not left open.
|
||
- Idle/abandoned environments are automatically cleaned up, and a student can launch a fresh
|
||
one and keep going without losing progress.
|
||
|
||
Production topology and what's built vs. not yet: `docs/LAB_ARCHITECTURE.md`,
|
||
`docs/PHASE1_PLATFORM_FOUNDATION.md`, `docs/PHASE2A_GATEWAY_AND_STUDENT_FLOW.md`,
|
||
`docs/PHASE3A_IDENTITY_AND_INSTRUCTOR_ACCESS.md`, `docs/PHASE3B_LEARNING_AND_ASSESSMENT.md`.
|
||
|
||
The production infrastructure package (Proxmox/Ansible/Terraform/firewall/monitoring/backup/CI)
|
||
that runs all of this on real hardware lives in `infra/` — see
|
||
`docs/PHASE4_PRODUCTION_INFRASTRUCTURE.md`. Built and locally validated now; deployment is
|
||
pending purchase of the target Hetzner dedicated server.
|
||
|
||
---
|
||
|
||
## Repository Layout
|
||
|
||
```
|
||
scenarios/
|
||
├── data/ # one JSON file per scenario -> <scenario-id>.json
|
||
├── bundles/ # one JSON file per bundle -> <bundle-id>.json
|
||
└── SCHEMA.md # full schema reference
|
||
|
||
backend/ server.js # Express API + WebSocket PTY
|
||
frontend/ src/ # React + Vite SPA
|
||
scripts/ entrypoint.sh # container startup (k3s -> API -> nginx)
|
||
nginx.conf # reverse-proxy config
|
||
```
|
||
|
||
---
|
||
|
||
## Authoring Scenarios
|
||
|
||
Each scenario is one JSON file in `scenarios/data/`; each bundle is one JSON file in
|
||
`scenarios/bundles/`. See [`scenarios/SCHEMA.md`](scenarios/SCHEMA.md) for the full schema.
|
||
|
||
**Task checklist**
|
||
- `validation.commands` — idempotent `kubectl` only
|
||
- `setup_commands` / `teardown_commands` — `kubectl` or native Ubuntu only
|
||
|
||
**MCQ checklist**
|
||
- `correct_option` must match an `options[].id`
|
||
- always include an `explanation`
|
||
|
||
Hot-reload while authoring (no rebuild):
|
||
|
||
```bash
|
||
docker run --rm -itd --privileged -p 7554:80 --name kubekosh \
|
||
-v $PWD/scenarios:/app/scenarios ced4568/kubekosh:latest
|
||
curl -X POST http://localhost:7554/api/cache/reload
|
||
```
|
||
|
||
---
|
||
|
||
## License
|
||
|
||
Apache 2.0 — see [`LICENSE`](LICENSE). Original work © the `zeborg/kubekosh` authors.
|
||
Modifications © RootNode Academy / Chase Dumphord (ced4568). |