mirror of
https://github.com/ced4568/ceds-observability-stack.git
synced 2026-08-12 22:34:03 +00:00
Update roadmap progress, replace IPs with placeholders in configs
This commit is contained in:
+27
-30
@@ -1,50 +1,47 @@
|
|||||||
|
|
||||||
|
|
||||||
```markdown
|
|
||||||
# Project Roadmap
|
# Project Roadmap
|
||||||
|
|
||||||
## Phase 1 — Repo Foundation
|
## Phase 1 Repo Foundation
|
||||||
|
|
||||||
- [ ] Create repo
|
- [x] Create repo
|
||||||
- [ ] Add README
|
- [x] Add README
|
||||||
- [ ] Add architecture document
|
- [x] Add architecture document
|
||||||
- [ ] Add setup guide
|
- [x] Add setup guide
|
||||||
- [ ] Add roadmap
|
- [x] Add roadmap
|
||||||
- [ ] Add troubleshooting guide
|
- [ ] Add troubleshooting guide
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Phase 2 — Metrics Foundation
|
## Phase 2 Metrics Foundation
|
||||||
|
|
||||||
- [ ] Install node exporter
|
- [x] Install node exporter
|
||||||
- [ ] Add Prometheus config
|
- [x] Add Prometheus config
|
||||||
- [ ] Add K3s node targets
|
- [x] Add K3s node targets
|
||||||
- [ ] Add Proxmox target
|
- [x] Add Proxmox target
|
||||||
- [ ] Add service uptime checks
|
- [x] Add service uptime checks
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Phase 3 — Grafana Dashboards
|
## Phase 3 Grafana Dashboards
|
||||||
|
|
||||||
- [ ] Add Prometheus datasource
|
- [x] Add Prometheus datasource
|
||||||
- [ ] Build infrastructure dashboard
|
- [x] Build infrastructure dashboard
|
||||||
- [ ] Build K3s dashboard
|
- [x] Build K3s dashboard
|
||||||
- [ ] Build Proxmox dashboard
|
- [x] Build Proxmox dashboard
|
||||||
- [ ] Build service uptime dashboard
|
- [x] Build service uptime dashboard
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Phase 4 — Alerting
|
## Phase 4 Alerting
|
||||||
|
|
||||||
- [ ] Add alert rules
|
- [ ] Add alert rules
|
||||||
- [ ] Add Alertmanager
|
- [x] Add Alertmanager
|
||||||
- [ ] Test node down alert
|
- [ ] Test node down alert
|
||||||
- [ ] Test service down alert
|
- [ ] Test service down alert
|
||||||
- [ ] Document alert response process
|
- [ ] Document alert response process
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Phase 5 — Advanced Observability
|
## Phase 5 Advanced Observability
|
||||||
|
|
||||||
- [ ] Add Loki for logs
|
- [ ] Add Loki for logs
|
||||||
- [ ] Add Cloudflare access logs
|
- [ ] Add Cloudflare access logs
|
||||||
@@ -54,11 +51,11 @@
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Phase 6 — Portfolio Polish
|
## Phase 6 Portfolio Polish
|
||||||
|
|
||||||
- [ ] Add screenshots
|
- [x] Add screenshots
|
||||||
- [ ] Add architecture diagram image
|
- [x] Add architecture diagram image
|
||||||
- [ ] Add demo dashboard image
|
- [x] Add demo dashboard image
|
||||||
- [ ] Link to Ced’s HomeLab repo
|
- [x] Link to Ced's HomeLab repo
|
||||||
- [ ] Link to Ced’s NOC repo
|
- [x] Link to Ced's NOC repo
|
||||||
- [ ] Add project summary to portfolio website
|
- [ ] Add project summary to portfolio website
|
||||||
@@ -20,7 +20,7 @@ Unpoller
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Step 1 — Create UniFi Monitoring User
|
## Step 1 Create UniFi Monitoring User
|
||||||
|
|
||||||
In the UniFi Network application, create a local monitoring user.
|
In the UniFi Network application, create a local monitoring user.
|
||||||
|
|
||||||
@@ -33,58 +33,74 @@ Do not commit the password to GitHub.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Step 2 — Run UniFi Exporter / Unpoller
|
## Step 2 Run UniFi Exporter / Unpoller
|
||||||
|
|
||||||
Replace the password before running:
|
Replace the password before running:
|
||||||
|
|
||||||
|
```bash
|
||||||
docker run -d \
|
docker run -d \
|
||||||
--name unifi-exporter \
|
--name unifi-exporter \
|
||||||
--restart unless-stopped \
|
--restart unless-stopped \
|
||||||
-p 9130:9130 \
|
-p 9130:9130 \
|
||||||
-e UP_UNIFI_DEFAULT_URL="https://10.10.1.1" \
|
-e UP_UNIFI_DEFAULT_URL="https://<unifi-controller-ip>" \
|
||||||
-e UP_UNIFI_DEFAULT_USER="prometheus" \
|
-e UP_UNIFI_DEFAULT_USER="prometheus" \
|
||||||
-e UP_UNIFI_DEFAULT_PASS="REPLACE_WITH_YOUR_PASSWORD" \
|
-e UP_UNIFI_DEFAULT_PASS="REPLACE_WITH_YOUR_PASSWORD" \
|
||||||
-e UP_UNIFI_DEFAULT_VERIFY_SSL="false" \
|
-e UP_UNIFI_DEFAULT_VERIFY_SSL="false" \
|
||||||
-e UP_PROMETHEUS_HTTP_LISTEN="0.0.0.0:9130" \
|
-e UP_PROMETHEUS_HTTP_LISTEN="0.0.0.0:9130" \
|
||||||
ghcr.io/unpoller/unpoller:latest
|
ghcr.io/unpoller/unpoller:latest
|
||||||
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Step 3 — Verify Container Is Running
|
## Step 3 Verify Container Is Running
|
||||||
|
|
||||||
|
```bash
|
||||||
docker ps --format "table {{.Names}}\t{{.Ports}}\t{{.Status}}"
|
docker ps --format "table {{.Names}}\t{{.Ports}}\t{{.Status}}"
|
||||||
|
```
|
||||||
|
|
||||||
Expected result:
|
Expected result:
|
||||||
|
|
||||||
|
```
|
||||||
unifi-exporter 0.0.0.0:9130->9130/tcp Up
|
unifi-exporter 0.0.0.0:9130->9130/tcp Up
|
||||||
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Step 4 — Test Metrics Endpoint
|
## Step 4 Test Metrics Endpoint
|
||||||
|
|
||||||
curl http://10.10.30.140:9130/metrics
|
```bash
|
||||||
|
curl http://<exporter-host-ip>:9130/metrics
|
||||||
|
```
|
||||||
|
|
||||||
Expected result:
|
Expected result:
|
||||||
|
|
||||||
|
```
|
||||||
UniFi metrics output
|
UniFi metrics output
|
||||||
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Step 5 — Restart Prometheus
|
## Step 5 Restart Prometheus
|
||||||
|
|
||||||
|
```bash
|
||||||
systemctl restart prometheus
|
systemctl restart prometheus
|
||||||
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Step 6 — Confirm in Prometheus
|
## Step 6 Confirm in Prometheus
|
||||||
|
|
||||||
Open:
|
Open:
|
||||||
|
|
||||||
http://10.10.30.140:9090/targets
|
```
|
||||||
|
http://<prometheus-host-ip>:9090/targets
|
||||||
|
```
|
||||||
|
|
||||||
Expected:
|
Expected:
|
||||||
|
|
||||||
|
```
|
||||||
unifi-exporter UP
|
unifi-exporter UP
|
||||||
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -92,16 +108,18 @@ unifi-exporter UP
|
|||||||
|
|
||||||
Add this to prometheus.yml if not present:
|
Add this to prometheus.yml if not present:
|
||||||
|
|
||||||
|
```yaml
|
||||||
- job_name: "unifi-exporter"
|
- job_name: "unifi-exporter"
|
||||||
static_configs:
|
static_configs:
|
||||||
- targets:
|
- targets:
|
||||||
- "10.10.30.140:9130"
|
- "<exporter-host-ip>:9130"
|
||||||
labels:
|
labels:
|
||||||
instance: "unifi-exporter"
|
instance: "unifi-exporter"
|
||||||
service: "unifi"
|
service: "unifi"
|
||||||
role: "network"
|
role: "network"
|
||||||
vendor: "unifi"
|
vendor: "unifi"
|
||||||
site: "ceds-homelab"
|
site: "ceds-homelab"
|
||||||
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -114,8 +132,10 @@ Exporter not running or not bound to port 9130.
|
|||||||
|
|
||||||
Fix:
|
Fix:
|
||||||
|
|
||||||
|
```bash
|
||||||
docker ps --format "table {{.Names}}\t{{.Ports}}\t{{.Status}}"
|
docker ps --format "table {{.Names}}\t{{.Ports}}\t{{.Status}}"
|
||||||
docker restart unifi-exporter
|
docker restart unifi-exporter
|
||||||
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -133,15 +153,21 @@ Update credentials and restart container.
|
|||||||
|
|
||||||
Test exporter:
|
Test exporter:
|
||||||
|
|
||||||
curl http://10.10.30.140:9130/metrics
|
```bash
|
||||||
|
curl http://<exporter-host-ip>:9130/metrics
|
||||||
|
```
|
||||||
|
|
||||||
Check Prometheus config:
|
Check Prometheus config:
|
||||||
|
|
||||||
|
```bash
|
||||||
promtool check config /etc/prometheus/prometheus.yml
|
promtool check config /etc/prometheus/prometheus.yml
|
||||||
|
```
|
||||||
|
|
||||||
Restart Prometheus:
|
Restart Prometheus:
|
||||||
|
|
||||||
|
```bash
|
||||||
systemctl restart prometheus
|
systemctl restart prometheus
|
||||||
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -157,7 +183,9 @@ Never commit:
|
|||||||
|
|
||||||
Use placeholders:
|
Use placeholders:
|
||||||
|
|
||||||
|
```
|
||||||
REPLACE_WITH_YOUR_PASSWORD
|
REPLACE_WITH_YOUR_PASSWORD
|
||||||
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -165,6 +193,8 @@ REPLACE_WITH_YOUR_PASSWORD
|
|||||||
|
|
||||||
When complete, Prometheus should show:
|
When complete, Prometheus should show:
|
||||||
|
|
||||||
|
```
|
||||||
unifi-exporter UP
|
unifi-exporter UP
|
||||||
|
```
|
||||||
|
|
||||||
Grafana dashboards can now use UniFi metrics.
|
Grafana dashboards can now use UniFi metrics.
|
||||||
+18
-18
@@ -14,27 +14,27 @@ scrape_configs:
|
|||||||
- job_name: "proxmox"
|
- job_name: "proxmox"
|
||||||
static_configs:
|
static_configs:
|
||||||
- targets:
|
- targets:
|
||||||
- "10.10.30.250:9221"
|
- "<proxmox-host-ip>:9221"
|
||||||
|
|
||||||
- job_name: "k3s-control-plane"
|
- job_name: "k3s-control-plane"
|
||||||
static_configs:
|
static_configs:
|
||||||
- targets:
|
- targets:
|
||||||
- "10.10.30.72:9100"
|
- "<control-plane-ip-1>:9100"
|
||||||
- "10.10.30.245:9100"
|
- "<control-plane-ip-2>:9100"
|
||||||
- "10.10.30.128:9100"
|
- "<control-plane-ip-3>:9100"
|
||||||
|
|
||||||
- job_name: "k3s-workers"
|
- job_name: "k3s-workers"
|
||||||
static_configs:
|
static_configs:
|
||||||
- targets:
|
- targets:
|
||||||
- "10.10.30.101:9100"
|
- "<worker-node-ip-1>:9100"
|
||||||
- "10.10.30.102:9100"
|
- "<worker-node-ip-2>:9100"
|
||||||
- "10.10.30.103:9100"
|
- "<worker-node-ip-3>:9100"
|
||||||
- "10.10.30.104:9100"
|
- "<worker-node-ip-4>:9100"
|
||||||
- "10.10.30.105:9100"
|
- "<worker-node-ip-5>:9100"
|
||||||
- "10.10.30.106:9100"
|
- "<worker-node-ip-6>:9100"
|
||||||
- "10.10.30.107:9100"
|
- "<worker-node-ip-7>:9100"
|
||||||
- "10.10.30.108:9100"
|
- "<worker-node-ip-8>:9100"
|
||||||
- "10.10.30.109:9100"
|
- "<worker-node-ip-9>:9100"
|
||||||
|
|
||||||
- job_name: "homelab-services"
|
- job_name: "homelab-services"
|
||||||
metrics_path: /probe
|
metrics_path: /probe
|
||||||
@@ -42,11 +42,11 @@ scrape_configs:
|
|||||||
module: [tcp_connect]
|
module: [tcp_connect]
|
||||||
static_configs:
|
static_configs:
|
||||||
- targets:
|
- targets:
|
||||||
- "10.10.30.250:8006"
|
- "<proxmox-host-ip>:8006"
|
||||||
- "10.10.30.143:80"
|
- "<truenas-ip>:80"
|
||||||
- "10.10.30.210:81"
|
- "<npm-ip>:81"
|
||||||
- "10.10.30.61:4000"
|
- "<dashy-ip>:4000"
|
||||||
- "10.10.30.143:30013"
|
- "<truenas-ip>:30013"
|
||||||
relabel_configs:
|
relabel_configs:
|
||||||
- source_labels: [__address__]
|
- source_labels: [__address__]
|
||||||
target_label: __param_target
|
target_label: __param_target
|
||||||
|
|||||||
Reference in New Issue
Block a user