# Ced's K3s Cluster Setup – End-to-End This document describes how your current K3s cluster is built so you can re‑create it later or explain it in GitHub/OneNote. --- ## 1. Base Networking **HomeLab VLAN:** `10.10.30.0/24` **Gateway:** UniFi Dream Router **DHCP:** Configured to *exclude* `.251–.254` so MetalLB can safely use that range. - Proxmox VE: `10.10.30.250` - MetalLB pool: `10.10.30.251–10.10.30.254` Subnets you have overall (for reference): - Default: `10.10.1.0/24` (not used) - Main: `10.10.10.0/24` - IoT: `10.10.20.0/24` - HomeLab: `10.10.30.0/24` - Guest: `10.10.99.0/24` --- ## 2. K3s Control Plane ### Nodes - `k3s-django-1` – `10.10.30.72` (control-plane, etcd, master) - `k3s-django-2` – `10.10.30.245` (control-plane, etcd, master) - `k3s-django-3` – `10.10.30.128` (control-plane, etcd, master) All three are Debian 12 (Bookworm, ARM64 on Raspberry Pi). ### 2.1 Install K3s on the first control-plane node On `k3s-django-1`: ```bash curl -sfL https://get.k3s.io | sh -s - server ``` Then capture the version, token, and kubeconfig: ```bash k3s --version sudo cat /var/lib/rancher/k3s/server/node-token mkdir -p ~/.kube sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config sudo chown $(id -u):$(id -g) ~/.kube/config ``` Update the kubeconfig to point to the node's real IP instead of `127.0.0.1`: ```bash sed -i 's/https:\/\/127.0.0.1:6443/https:\/\/10.10.30.72:6443/' ~/.kube/config export KUBECONFIG=$HOME/.kube/config ``` Check: ```bash kubectl get nodes -o wide ``` You should see `k3s-django-1` as `Ready control-plane,etcd,master`. ### 2.2 Join additional control-plane nodes Get the join token from `k3s-django-1`: ```bash TOKEN=$(sudo cat /var/lib/rancher/k3s/server/node-token) echo "$TOKEN" ``` On `k3s-django-2` and `k3s-django-3`: ```bash export K3S_URL="https://10.10.30.72:6443" export K3S_TOKEN="" curl -sfL https://get.k3s.io | K3S_URL=$K3S_URL K3S_TOKEN=$K3S_TOKEN sh -s - server ``` Verify from `k3s-django-1`: ```bash kubectl get nodes -o wide ``` You should now see all three `k3s-django-*` nodes as control-plane,etcd,master. --- ## 3. Worker Nodes You have 9 worker nodes grouped as: - Ingress: `k3s-node-1`, `k3s-node-2`, `k3s-node-3` - Data: `k3s-node-4`, `k3s-node-5`, `k3s-node-6` - Monitoring: `k3s-node-7`, `k3s-node-8`, `k3s-node-9` ### 3.1 Join workers as agents On each worker node (`k3s-node-N`), use: ```bash export K3S_URL="https://10.10.30.72:6443" export K3S_TOKEN="" curl -sfL https://get.k3s.io | K3S_URL=$K3S_URL K3S_TOKEN=$K3S_TOKEN sh -s - agent ``` Confirm from `k3s-django-1`: ```bash kubectl get nodes -o wide ``` All 9 agents should show as `Ready` with no special roles yet. ### 3.2 Label nodes by function From `k3s-django-1` (or any node with kubeconfig): ```bash # Ingress group kubectl label node k3s-node-1 ingress-node=true kubectl label node k3s-node-2 ingress-node=true kubectl label node k3s-node-3 ingress-node=true # Data group kubectl label node k3s-node-4 data-node=true kubectl label node k3s-node-5 data-node=true kubectl label node k3s-node-6 data-node=true # Monitoring group kubectl label node k3s-node-7 monitoring-node=true kubectl label node k3s-node-8 monitoring-node=true kubectl label node k3s-node-9 monitoring-node=true kubectl get nodes -L ingress-node -L data-node -L monitoring-node ``` Additionally, you used `kubectl label node ...` to set the `ROLES` field (`ingress`, `data`, `monitoring`) for documentation. --- ## 4. MetalLB ### 4.1 Install MetalLB ```bash kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.8/config/manifests/metallb-native.yaml ``` Wait for pods: ```bash kubectl get pods -n metallb-system -o wide ``` ### 4.2 Configure IPAddressPool and L2Advertisement MetalLB is configured with an address pool in `manifests/metallb/ipaddresspool.yaml`: ```yaml apiVersion: metallb.io/v1beta1 kind: IPAddressPool metadata: name: ceds-pool namespace: metallb-system spec: addresses: - 10.10.30.251-10.10.30.254 ``` and `manifests/metallb/l2advertisement.yaml`: ```yaml apiVersion: metallb.io/v1beta1 kind: L2Advertisement metadata: name: ceds-l2 namespace: metallb-system spec: ipAddressPools: - ceds-pool ``` Apply: ```bash kubectl apply -f manifests/metallb/ipaddresspool.yaml kubectl apply -f manifests/metallb/l2advertisement.yaml ``` Check: ```bash kubectl get ipaddresspools.metallb.io -n metallb-system kubectl get l2advertisements.metallb.io -n metallb-system ``` --- ## 5. Ingress-NGINX ### 5.1 Install Helm & ingress-nginx On `k3s-django-1`: ```bash curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx helm repo update ``` Install ingress-nginx targeting the ingress node group: ```bash helm install ingress-nginx ingress-nginx/ingress-nginx --namespace ingress-nginx --create-namespace --set controller.service.type=LoadBalancer --set controller.nodeSelector.ingress-node="true" ``` Check: ```bash kubectl get pods -n ingress-nginx kubectl get svc -n ingress-nginx ingress-nginx-controller ``` You should see `EXTERNAL-IP` assigned from MetalLB, e.g. `10.10.30.251`. --- ## 6. Demo Ingress App `manifests/demo-app/demo-nginx.yaml` contains: - A `Deployment` with 3 replicas of `nginx` - A `ClusterIP` Service - An Ingress pointing `demo.local` to the service via ingress-nginx Apply: ```bash kubectl apply -f manifests/demo-app/demo-nginx.yaml ``` Update your desktop hosts file: ```text 10.10.30.251 demo.local ``` Open `http://demo.local` in a browser and you should see the nginx welcome page, served via: Desktop → MetalLB VIP (10.10.30.251) → ingress-nginx → demo-nginx Service → Pods. --- ## 7. Ced's NOC (kube-prometheus-stack) Namespace: ```bash kubectl create namespace monitoring ``` Helm repo: ```bash helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update ``` Install with your custom values file: ```bash helm install ceds-noc prometheus-community/kube-prometheus-stack --namespace monitoring -f kube-prom-values.yaml ``` `kube-prom-values.yaml` (included in this repo) pins Prometheus, Alertmanager, and Grafana to the `monitoring-node=true` nodes, configures PVCs for metrics and Grafana data, and exposes Grafana via a LoadBalancer (MetalLB). Check pods & services: ```bash kubectl get pods -n monitoring kubectl get svc -n monitoring ``` You should see: - `ceds-noc-grafana` as a `LoadBalancer` with an `EXTERNAL-IP` like `10.10.30.252` - Prometheus, Alertmanager, kube-state-metrics, node exporters, etc. Open Grafana at: ```text http://10.10.30.252 ``` Username: `admin` Password: either the one in `kube-prom-values.yaml` (if set) or via: ```bash kubectl --namespace monitoring get secret ceds-noc-grafana -o jsonpath="{.data.admin-password}" | base64 -d ; echo ``` --- ## 8. Grafana Dashboards (Ced's NOC) Use Grafana → Dashboards → New → Import, and upload: - `dashboards/ced_cluster_overview.json` - `dashboards/ced_nodes_detail.json` - `dashboards/ced_ingress_metallb.json` Choose the kube-prometheus-stack Prometheus datasource. You now have a live NOC for: - Cluster health (nodes, pods, API server) - Per‑node CPU, RAM, disk, pod counts - Ingress and MetalLB metrics --- ## 9. Future Enhancements - Add Loki + Promtail for logs - Add Prometheus alerting rules (node down, high CPU, low disk, etc) - Add Alertmanager integrations (Discord/Slack/email) - Use GitOps (FluxCD or ArgoCD) to manage this repo as the source of truth.