Initial commit: Ced's K3s HomeLab

2026-08-13 06:24:02 +00:00 · 2025-12-06 22:22:42 -06:00
commit 188651954e
20 changed files with 1078 additions and 0 deletions
@@ -0,0 +1,340 @@
+# Ced's K3s Cluster Setup – End-to-End
+
+This document describes how your current K3s cluster is built so you can re‑create it later or explain it in GitHub/OneNote.
+
+---
+
+## 1. Base Networking
+
+**HomeLab VLAN:** `10.10.30.0/24`  
+**Gateway:** UniFi Dream Router  
+**DHCP:** Configured to *exclude* `.251–.254` so MetalLB can safely use that range.
+
+- Proxmox VE: `10.10.30.250`
+- MetalLB pool: `10.10.30.251–10.10.30.254`
+
+Subnets you have overall (for reference):
+
+- Default: `10.10.1.0/24` (not used)
+- Main: `10.10.10.0/24`
+- IoT: `10.10.20.0/24`
+- HomeLab: `10.10.30.0/24`
+- Guest: `10.10.99.0/24`
+
+---
+
+## 2. K3s Control Plane
+
+### Nodes
+
+- `k3s-django-1` – `10.10.30.72`  (control-plane, etcd, master)
+- `k3s-django-2` – `10.10.30.245` (control-plane, etcd, master)
+- `k3s-django-3` – `10.10.30.128` (control-plane, etcd, master)
+
+All three are Debian 12 (Bookworm, ARM64 on Raspberry Pi).
+
+### 2.1 Install K3s on the first control-plane node
+
+On `k3s-django-1`:
+
+```bash
+curl -sfL https://get.k3s.io | sh -s - server
+```
+
+Then capture the version, token, and kubeconfig:
+
+```bash
+k3s --version
+
+sudo cat /var/lib/rancher/k3s/server/node-token
+
+mkdir -p ~/.kube
+sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
+sudo chown $(id -u):$(id -g) ~/.kube/config
+```
+
+Update the kubeconfig to point to the node's real IP instead of `127.0.0.1`:
+
+```bash
+sed -i 's/https:\/\/127.0.0.1:6443/https:\/\/10.10.30.72:6443/' ~/.kube/config
+export KUBECONFIG=$HOME/.kube/config
+```
+
+Check:
+
+```bash
+kubectl get nodes -o wide
+```
+
+You should see `k3s-django-1` as `Ready control-plane,etcd,master`.
+
+### 2.2 Join additional control-plane nodes
+
+Get the join token from `k3s-django-1`:
+
+```bash
+TOKEN=$(sudo cat /var/lib/rancher/k3s/server/node-token)
+echo "$TOKEN"
+```
+
+On `k3s-django-2` and `k3s-django-3`:
+
+```bash
+export K3S_URL="https://10.10.30.72:6443"
+export K3S_TOKEN="<PASTE TOKEN FROM k3s-django-1>"
+
+curl -sfL https://get.k3s.io |   K3S_URL=$K3S_URL   K3S_TOKEN=$K3S_TOKEN   sh -s - server
+```
+
+Verify from `k3s-django-1`:
+
+```bash
+kubectl get nodes -o wide
+```
+
+You should now see all three `k3s-django-*` nodes as control-plane,etcd,master.
+
+---
+
+## 3. Worker Nodes
+
+You have 9 worker nodes grouped as:
+
+- Ingress: `k3s-node-1`, `k3s-node-2`, `k3s-node-3`
+- Data:    `k3s-node-4`, `k3s-node-5`, `k3s-node-6`
+- Monitoring: `k3s-node-7`, `k3s-node-8`, `k3s-node-9`
+
+### 3.1 Join workers as agents
+
+On each worker node (`k3s-node-N`), use:
+
+```bash
+export K3S_URL="https://10.10.30.72:6443"
+export K3S_TOKEN="<same token from k3s-django-1>"
+
+curl -sfL https://get.k3s.io |   K3S_URL=$K3S_URL   K3S_TOKEN=$K3S_TOKEN   sh -s - agent
+```
+
+Confirm from `k3s-django-1`:
+
+```bash
+kubectl get nodes -o wide
+```
+
+All 9 agents should show as `Ready` with no special roles yet.
+
+### 3.2 Label nodes by function
+
+From `k3s-django-1` (or any node with kubeconfig):
+
+```bash
+# Ingress group
+kubectl label node k3s-node-1 ingress-node=true
+kubectl label node k3s-node-2 ingress-node=true
+kubectl label node k3s-node-3 ingress-node=true
+
+# Data group
+kubectl label node k3s-node-4 data-node=true
+kubectl label node k3s-node-5 data-node=true
+kubectl label node k3s-node-6 data-node=true
+
+# Monitoring group
+kubectl label node k3s-node-7 monitoring-node=true
+kubectl label node k3s-node-8 monitoring-node=true
+kubectl label node k3s-node-9 monitoring-node=true
+
+kubectl get nodes -L ingress-node -L data-node -L monitoring-node
+```
+
+Additionally, you used `kubectl label node ...` to set the `ROLES` field (`ingress`, `data`, `monitoring`) for documentation.
+
+---
+
+## 4. MetalLB
+
+### 4.1 Install MetalLB
+
+```bash
+kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.8/config/manifests/metallb-native.yaml
+```
+
+Wait for pods:
+
+```bash
+kubectl get pods -n metallb-system -o wide
+```
+
+### 4.2 Configure IPAddressPool and L2Advertisement
+
+MetalLB is configured with an address pool in `manifests/metallb/ipaddresspool.yaml`:
+
+```yaml
+apiVersion: metallb.io/v1beta1
+kind: IPAddressPool
+metadata:
+  name: ceds-pool
+  namespace: metallb-system
+spec:
+  addresses:
+    - 10.10.30.251-10.10.30.254
+```
+
+and `manifests/metallb/l2advertisement.yaml`:
+
+```yaml
+apiVersion: metallb.io/v1beta1
+kind: L2Advertisement
+metadata:
+  name: ceds-l2
+  namespace: metallb-system
+spec:
+  ipAddressPools:
+    - ceds-pool
+```
+
+Apply:
+
+```bash
+kubectl apply -f manifests/metallb/ipaddresspool.yaml
+kubectl apply -f manifests/metallb/l2advertisement.yaml
+```
+
+Check:
+
+```bash
+kubectl get ipaddresspools.metallb.io -n metallb-system
+kubectl get l2advertisements.metallb.io -n metallb-system
+```
+
+---
+
+## 5. Ingress-NGINX
+
+### 5.1 Install Helm & ingress-nginx
+
+On `k3s-django-1`:
+
+```bash
+curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash
+
+helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
+helm repo update
+```
+
+Install ingress-nginx targeting the ingress node group:
+
+```bash
+helm install ingress-nginx ingress-nginx/ingress-nginx   --namespace ingress-nginx   --create-namespace   --set controller.service.type=LoadBalancer   --set controller.nodeSelector.ingress-node="true"
+```
+
+Check:
+
+```bash
+kubectl get pods -n ingress-nginx
+kubectl get svc -n ingress-nginx ingress-nginx-controller
+```
+
+You should see `EXTERNAL-IP` assigned from MetalLB, e.g. `10.10.30.251`.
+
+---
+
+## 6. Demo Ingress App
+
+`manifests/demo-app/demo-nginx.yaml` contains:
+
+- A `Deployment` with 3 replicas of `nginx`
+- A `ClusterIP` Service
+- An Ingress pointing `demo.local` to the service via ingress-nginx
+
+Apply:
+
+```bash
+kubectl apply -f manifests/demo-app/demo-nginx.yaml
+```
+
+Update your desktop hosts file:
+
+```text
+10.10.30.251 demo.local
+```
+
+Open `http://demo.local` in a browser and you should see the nginx welcome page, served via:
+
+Desktop → MetalLB VIP (10.10.30.251) → ingress-nginx → demo-nginx Service → Pods.
+
+---
+
+## 7. Ced's NOC (kube-prometheus-stack)
+
+Namespace:
+
+```bash
+kubectl create namespace monitoring
+```
+
+Helm repo:
+
+```bash
+helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
+helm repo update
+```
+
+Install with your custom values file:
+
+```bash
+helm install ceds-noc prometheus-community/kube-prometheus-stack   --namespace monitoring   -f kube-prom-values.yaml
+```
+
+`kube-prom-values.yaml` (included in this repo) pins Prometheus, Alertmanager, and Grafana to the `monitoring-node=true` nodes, configures PVCs for metrics and Grafana data, and exposes Grafana via a LoadBalancer (MetalLB).
+
+Check pods & services:
+
+```bash
+kubectl get pods -n monitoring
+kubectl get svc -n monitoring
+```
+
+You should see:
+
+- `ceds-noc-grafana` as a `LoadBalancer` with an `EXTERNAL-IP` like `10.10.30.252`
+- Prometheus, Alertmanager, kube-state-metrics, node exporters, etc.
+
+Open Grafana at:
+
+```text
+http://10.10.30.252
+```
+
+Username: `admin`  
+Password: either the one in `kube-prom-values.yaml` (if set) or via:
+
+```bash
+kubectl --namespace monitoring get secret ceds-noc-grafana -o jsonpath="{.data.admin-password}" | base64 -d ; echo
+```
+
+---
+
+## 8. Grafana Dashboards (Ced's NOC)
+
+Use Grafana → Dashboards → New → Import, and upload:
+
+- `dashboards/ced_cluster_overview.json`
+- `dashboards/ced_nodes_detail.json`
+- `dashboards/ced_ingress_metallb.json`
+
+Choose the kube-prometheus-stack Prometheus datasource.
+
+You now have a live NOC for:
+
+- Cluster health (nodes, pods, API server)
+- Per‑node CPU, RAM, disk, pod counts
+- Ingress and MetalLB metrics
+
+---
+
+## 9. Future Enhancements
+
+- Add Loki + Promtail for logs
+- Add Prometheus alerting rules (node down, high CPU, low disk, etc)
+- Add Alertmanager integrations (Discord/Slack/email)
+- Use GitOps (FluxCD or ArgoCD) to manage this repo as the source of truth.