ced-k3s-homelab/k3s-homelab-full/cluster-setup.md

Ced's K3s Cluster Setup – End-to-End

This document describes how your current K3s cluster is built so you can re‑create it later or explain it in GitHub/OneNote.

---

1. Base Networking

HomeLab VLAN: 10.10.30.0/24
Gateway: UniFi Dream Router
DHCP: Configured to exclude .251–.254 so MetalLB can safely use that range.

• Proxmox VE: 10.10.30.250
• MetalLB pool: 10.10.30.251–10.10.30.254

Subnets you have overall (for reference):

• Default: 10.10.1.0/24 (not used)
• Main: 10.10.10.0/24
• IoT: 10.10.20.0/24
• HomeLab: 10.10.30.0/24
• Guest: 10.10.99.0/24

---

2. K3s Control Plane

Nodes

• k3s-django-1 – 10.10.30.72 (control-plane, etcd, master)
• k3s-django-2 – 10.10.30.245 (control-plane, etcd, master)
• k3s-django-3 – 10.10.30.128 (control-plane, etcd, master)

All three are Debian 12 (Bookworm, ARM64 on Raspberry Pi).

2.1 Install K3s on the first control-plane node

On k3s-django-1:

curl -sfL https://get.k3s.io | sh -s - server

Then capture the version, token, and kubeconfig:

k3s --version

sudo cat /var/lib/rancher/k3s/server/node-token

mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/config

Update the kubeconfig to point to the node's real IP instead of 127.0.0.1:

sed -i 's/https:\/\/127.0.0.1:6443/https:\/\/10.10.30.72:6443/' ~/.kube/config
export KUBECONFIG=$HOME/.kube/config

Check:

kubectl get nodes -o wide

You should see k3s-django-1 as Ready control-plane,etcd,master.

2.2 Join additional control-plane nodes

Get the join token from k3s-django-1:

TOKEN=$(sudo cat /var/lib/rancher/k3s/server/node-token)
echo "$TOKEN"

On k3s-django-2 and k3s-django-3:

export K3S_URL="https://10.10.30.72:6443"
export K3S_TOKEN="<PASTE TOKEN FROM k3s-django-1>"

curl -sfL https://get.k3s.io |   K3S_URL=$K3S_URL   K3S_TOKEN=$K3S_TOKEN   sh -s - server

Verify from k3s-django-1:

kubectl get nodes -o wide

You should now see all three k3s-django-* nodes as control-plane,etcd,master.

---

3. Worker Nodes

You have 9 worker nodes grouped as:

• Ingress: k3s-node-1, k3s-node-2, k3s-node-3
• Data: k3s-node-4, k3s-node-5, k3s-node-6
• Monitoring: k3s-node-7, k3s-node-8, k3s-node-9

3.1 Join workers as agents

On each worker node (k3s-node-N), use:

export K3S_URL="https://10.10.30.72:6443"
export K3S_TOKEN="<same token from k3s-django-1>"

curl -sfL https://get.k3s.io |   K3S_URL=$K3S_URL   K3S_TOKEN=$K3S_TOKEN   sh -s - agent

Confirm from k3s-django-1:

kubectl get nodes -o wide

All 9 agents should show as Ready with no special roles yet.

3.2 Label nodes by function

From k3s-django-1 (or any node with kubeconfig):

# Ingress group
kubectl label node k3s-node-1 ingress-node=true
kubectl label node k3s-node-2 ingress-node=true
kubectl label node k3s-node-3 ingress-node=true

# Data group
kubectl label node k3s-node-4 data-node=true
kubectl label node k3s-node-5 data-node=true
kubectl label node k3s-node-6 data-node=true

# Monitoring group
kubectl label node k3s-node-7 monitoring-node=true
kubectl label node k3s-node-8 monitoring-node=true
kubectl label node k3s-node-9 monitoring-node=true

kubectl get nodes -L ingress-node -L data-node -L monitoring-node

Additionally, you used kubectl label node ... to set the ROLES field (ingress, data, monitoring) for documentation.

---

4. MetalLB

4.1 Install MetalLB

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.8/config/manifests/metallb-native.yaml

Wait for pods:

kubectl get pods -n metallb-system -o wide

4.2 Configure IPAddressPool and L2Advertisement

MetalLB is configured with an address pool in manifests/metallb/ipaddresspool.yaml:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: ceds-pool
  namespace: metallb-system
spec:
  addresses:
    - 10.10.30.251-10.10.30.254

and manifests/metallb/l2advertisement.yaml:

apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: ceds-l2
  namespace: metallb-system
spec:
  ipAddressPools:
    - ceds-pool

Apply:

kubectl apply -f manifests/metallb/ipaddresspool.yaml
kubectl apply -f manifests/metallb/l2advertisement.yaml

Check:

kubectl get ipaddresspools.metallb.io -n metallb-system
kubectl get l2advertisements.metallb.io -n metallb-system

---

5. Ingress-NGINX

5.1 Install Helm & ingress-nginx

On k3s-django-1:

curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

Install ingress-nginx targeting the ingress node group:

helm install ingress-nginx ingress-nginx/ingress-nginx   --namespace ingress-nginx   --create-namespace   --set controller.service.type=LoadBalancer   --set controller.nodeSelector.ingress-node="true"

Check:

kubectl get pods -n ingress-nginx
kubectl get svc -n ingress-nginx ingress-nginx-controller

You should see EXTERNAL-IP assigned from MetalLB, e.g. 10.10.30.251.

---

6. Demo Ingress App

manifests/demo-app/demo-nginx.yaml contains:

• A Deployment with 3 replicas of nginx
• A ClusterIP Service
• An Ingress pointing demo.local to the service via ingress-nginx

Apply:

kubectl apply -f manifests/demo-app/demo-nginx.yaml

Update your desktop hosts file:

10.10.30.251 demo.local

Open http://demo.local in a browser and you should see the nginx welcome page, served via:

Desktop → MetalLB VIP (10.10.30.251) → ingress-nginx → demo-nginx Service → Pods.

---

7. Ced's NOC (kube-prometheus-stack)

Namespace:

kubectl create namespace monitoring

Helm repo:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Install with your custom values file:

helm install ceds-noc prometheus-community/kube-prometheus-stack   --namespace monitoring   -f kube-prom-values.yaml

kube-prom-values.yaml (included in this repo) pins Prometheus, Alertmanager, and Grafana to the monitoring-node=true nodes, configures PVCs for metrics and Grafana data, and exposes Grafana via a LoadBalancer (MetalLB).

Check pods & services:

kubectl get pods -n monitoring
kubectl get svc -n monitoring

You should see:

• ceds-noc-grafana as a LoadBalancer with an EXTERNAL-IP like 10.10.30.252
• Prometheus, Alertmanager, kube-state-metrics, node exporters, etc.

Open Grafana at:

http://10.10.30.252

Username: admin
Password: either the one in kube-prom-values.yaml (if set) or via:

kubectl --namespace monitoring get secret ceds-noc-grafana -o jsonpath="{.data.admin-password}" | base64 -d ; echo

---

8. Grafana Dashboards (Ced's NOC)

Use Grafana → Dashboards → New → Import, and upload:

• dashboards/ced_cluster_overview.json
• dashboards/ced_nodes_detail.json
• dashboards/ced_ingress_metallb.json

Choose the kube-prometheus-stack Prometheus datasource.

You now have a live NOC for:

• Cluster health (nodes, pods, API server)
• Per‑node CPU, RAM, disk, pod counts
• Ingress and MetalLB metrics

---

9. Future Enhancements

• Add Loki + Promtail for logs
• Add Prometheus alerting rules (node down, high CPU, low disk, etc)
• Add Alertmanager integrations (Discord/Slack/email)
• Use GitOps (FluxCD or ArgoCD) to manage this repo as the source of truth.