mirror of
https://github.com/ced4568/ceds-observability-stack.git
synced 2026-08-12 22:34:03 +00:00
Add Phase 3 live observability deployment docs
This commit is contained in:
@@ -46,30 +46,34 @@ It simulates real-world **SRE / Platform Engineering environments**, delivering:
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Linux server or VM
|
||||
- Python 3 installed
|
||||
- Prometheus installed
|
||||
- Grafana installed
|
||||
- Network access to homelab systems
|
||||
* Linux server or VM
|
||||
* Python 3 installed
|
||||
* Prometheus installed
|
||||
* Grafana installed
|
||||
* Network access to homelab systems
|
||||
|
||||
---
|
||||
|
||||
### Run Service Health Check
|
||||
|
||||
bash python3 scripts/service-health-check.py
|
||||
```bash
|
||||
python3 scripts/service-health-check.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Run Prometheus
|
||||
|
||||
bash prometheus --config.file=prometheus/prometheus.yml
|
||||
```bash
|
||||
prometheus --config.file=prometheus/prometheus.yml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Access Services
|
||||
|
||||
- Prometheus: http://localhost:9090
|
||||
- Grafana: http://localhost:3000
|
||||
* Prometheus: http://localhost:9090
|
||||
* Grafana: http://localhost:3000
|
||||
|
||||
---
|
||||
|
||||
@@ -110,15 +114,19 @@ flowchart TD
|
||||
## 📸 Dashboards
|
||||
|
||||
### Infrastructure Overview
|
||||
|
||||
Infrastructure Dashboard
|
||||
|
||||
### K3s Cluster Dashboard
|
||||
|
||||
K3s Dashboard
|
||||
|
||||
### Proxmox HA Dashboard
|
||||
|
||||
Proxmox Dashboard
|
||||
|
||||
### Service Uptime Dashboard
|
||||
|
||||
Services Dashboard
|
||||
|
||||
---
|
||||
@@ -270,9 +278,9 @@ http://<your-server-ip>:3000
|
||||
|
||||
This observability stack is part of a larger ecosystem:
|
||||
|
||||
- Ced’s HomeLab → Infrastructure layer
|
||||
- Ced’s Observability Stack → Metrics + monitoring layer
|
||||
- Ced’s NOC → Visualization and operations layer
|
||||
* Ced’s HomeLab → Infrastructure layer
|
||||
* Ced’s Observability Stack → Metrics + monitoring layer
|
||||
* Ced’s NOC → Visualization and operations layer
|
||||
|
||||
Data flows from monitored systems into Prometheus, is visualized in Grafana, and feeds Ced’s NOC dashboard for real-time system visibility.
|
||||
|
||||
@@ -284,36 +292,40 @@ To verify the system is working correctly:
|
||||
|
||||
### Prometheus Targets
|
||||
|
||||
- Navigate to: http://localhost:9090/targets
|
||||
- Confirm all targets show UP
|
||||
* Navigate to: http://localhost:9090/targets
|
||||
* Confirm all targets show UP
|
||||
|
||||
---
|
||||
|
||||
### Node Exporter
|
||||
|
||||
bash curl http://<node-ip>:9100/metrics
|
||||
```bash
|
||||
curl http://<node-ip>:9100/metrics
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Service Health Check
|
||||
|
||||
bash python3 scripts/service-health-check.py
|
||||
```bash
|
||||
python3 scripts/service-health-check.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Grafana
|
||||
|
||||
- Confirm dashboards display real-time metrics
|
||||
- Verify data source connection to Prometheus
|
||||
- Check for active alerts
|
||||
* Confirm dashboards display real-time metrics
|
||||
* Verify data source connection to Prometheus
|
||||
* Check for active alerts
|
||||
|
||||
---
|
||||
|
||||
### Alert Testing
|
||||
|
||||
- Stop a service or node temporarily
|
||||
- Confirm alert triggers in Prometheus
|
||||
- Confirm alert appears in Grafana
|
||||
* Stop a service or node temporarily
|
||||
* Confirm alert triggers in Prometheus
|
||||
* Confirm alert appears in Grafana
|
||||
|
||||
---
|
||||
|
||||
@@ -333,26 +345,26 @@ It is designed to demonstrate how distributed systems are monitored, analyzed, a
|
||||
|
||||
Key capabilities include:
|
||||
|
||||
- Monitoring a hybrid infrastructure environment (Proxmox HA + K3s cluster)
|
||||
- Collecting and visualizing system and service metrics
|
||||
- Tracking service availability and uptime
|
||||
- Detecting infrastructure and application-level failures
|
||||
- Supporting alert-driven operations
|
||||
- Integrating with a centralized NOC dashboard
|
||||
* Monitoring a hybrid infrastructure environment (Proxmox HA + K3s cluster)
|
||||
* Collecting and visualizing system and service metrics
|
||||
* Tracking service availability and uptime
|
||||
* Detecting infrastructure and application-level failures
|
||||
* Supporting alert-driven operations
|
||||
* Integrating with a centralized NOC dashboard
|
||||
|
||||
This project represents a shift from running infrastructure to actively operating and maintaining it using observability principles aligned with SRE and platform engineering practice
|
||||
This project represents a shift from running infrastructure to actively operating and maintaining it using observability principles aligned with SRE and platform engineering practices.
|
||||
|
||||
---
|
||||
|
||||
## 🧠 Future Improvements
|
||||
## 🧠 Future Improvements
|
||||
|
||||
- Loki log aggregation
|
||||
- Tempo tracing
|
||||
- Cloudflare Access log ingestion
|
||||
- Automated remediation (self-healing infrastructure)
|
||||
- Grafana public demo dashboard
|
||||
- GitOps-based deployment (Argo CD / Flux)
|
||||
- Multi-cluster Kubernetes monitoring
|
||||
* Loki log aggregation
|
||||
* Tempo tracing
|
||||
* Cloudflare Access log ingestion
|
||||
* Automated remediation (self-healing infrastructure)
|
||||
* Grafana public demo dashboard
|
||||
* GitOps-based deployment (Argo CD / Flux)
|
||||
* Multi-cluster Kubernetes monitoring
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -0,0 +1,131 @@
|
||||
# Blackbox Exporter Setup
|
||||
|
||||
## Purpose
|
||||
|
||||
Blackbox Exporter is used to monitor endpoint availability from Prometheus.
|
||||
|
||||
---
|
||||
|
||||
## Config Location
|
||||
|
||||
/opt/blackbox/blackbox.yml
|
||||
|
||||
---
|
||||
|
||||
## Setup
|
||||
|
||||
```bash
|
||||
sudo mkdir -p /opt/blackbox
|
||||
sudo nano /opt/blackbox/blackbox.yml
|
||||
|
||||
## Config File
|
||||
|
||||
Create or edit:
|
||||
|
||||
/opt/blackbox/blackbox.yml
|
||||
|
||||
Paste:
|
||||
|
||||
modules:
|
||||
http_2xx:
|
||||
prober: http
|
||||
timeout: 10s
|
||||
http:
|
||||
method: GET
|
||||
preferred_ip_protocol: ip4
|
||||
|
||||
http_2xx_insecure:
|
||||
prober: http
|
||||
timeout: 10s
|
||||
http:
|
||||
method: GET
|
||||
preferred_ip_protocol: ip4
|
||||
tls_config:
|
||||
insecure_skip_verify: true
|
||||
|
||||
tcp_connect:
|
||||
prober: tcp
|
||||
timeout: 5s
|
||||
|
||||
icmp:
|
||||
prober: icmp
|
||||
timeout: 5s
|
||||
|
||||
---
|
||||
|
||||
## Remove Old Containers
|
||||
|
||||
Run:
|
||||
|
||||
docker rm -f blackbox
|
||||
docker rm -f blackbox-exporter
|
||||
|
||||
---
|
||||
|
||||
## Start Exporter
|
||||
|
||||
Run:
|
||||
|
||||
docker run -d \
|
||||
--name blackbox-exporter \
|
||||
--restart unless-stopped \
|
||||
-p 9115:9115 \
|
||||
-v /opt/blackbox/blackbox.yml:/config/blackbox.yml:ro \
|
||||
prom/blackbox-exporter:latest \
|
||||
--config.file=/config/blackbox.yml
|
||||
|
||||
---
|
||||
|
||||
## Test
|
||||
|
||||
Run:
|
||||
|
||||
curl "http://10.10.30.140:9115/probe?target=http://10.10.30.68:3000&module=http_2xx_insecure"
|
||||
|
||||
Expected output should include:
|
||||
|
||||
probe_success 1
|
||||
|
||||
---
|
||||
|
||||
## Restart Prometheus
|
||||
|
||||
Run:
|
||||
|
||||
systemctl restart prometheus
|
||||
|
||||
---
|
||||
|
||||
## Verify in Prometheus
|
||||
|
||||
Open:
|
||||
|
||||
http://10.10.30.140:9090/targets
|
||||
|
||||
You should now see:
|
||||
|
||||
blackbox-http-internal UP
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Unknown module "http_2xx_insecure"
|
||||
|
||||
Cause:
|
||||
Blackbox Exporter is running with an old or incorrect config.
|
||||
|
||||
Fix:
|
||||
Restart the container after verifying the config file is mounted correctly.
|
||||
|
||||
---
|
||||
|
||||
### Port already allocated
|
||||
|
||||
Cause:
|
||||
Another container is using port 9115.
|
||||
|
||||
Fix:
|
||||
|
||||
docker ps --format "table {{.Names}}\t{{.Ports}}"
|
||||
docker rm -f <container-name>
|
||||
@@ -0,0 +1,60 @@
|
||||
# Phase 3 — Live Deployment Notes
|
||||
|
||||
## Overview
|
||||
|
||||
Phase 3 focused on moving Ced’s Observability Stack from documentation into a working live monitoring environment.
|
||||
|
||||
The goal was to verify that Prometheus could collect real metrics from Ced’s HomeLab infrastructure and that Grafana could visualize those metrics through live dashboards.
|
||||
|
||||
---
|
||||
|
||||
## Environment
|
||||
|
||||
| System | Purpose |
|
||||
|---|---|
|
||||
| Prometheus CT | Metrics collection |
|
||||
| Grafana CT | Dashboard visualization |
|
||||
| Blackbox Exporter | Service availability checks |
|
||||
| Node Exporter | Host-level metrics |
|
||||
| Unpoller / UniFi Exporter | UniFi network metrics |
|
||||
| TrueNAS Graphite Exporter | TrueNAS metrics |
|
||||
| Proxmox HA Cluster | Virtualization platform |
|
||||
| 12-node K3s Cluster | Kubernetes environment |
|
||||
|
||||
---
|
||||
|
||||
## Completed Work
|
||||
|
||||
- Verified Prometheus target health
|
||||
- Repaired Blackbox Exporter configuration
|
||||
- Added custom Blackbox modules
|
||||
- Fixed HTTP probing for internal services
|
||||
- Verified Proxmox node exporter targets
|
||||
- Verified K3s node exporter targets
|
||||
- Installed UniFi exporter using Unpoller
|
||||
- Verified UniFi metrics on port 9130
|
||||
- Confirmed all Prometheus targets are UP
|
||||
- Verified Grafana dashboards are receiving live data
|
||||
|
||||
---
|
||||
|
||||
## Services Confirmed Working
|
||||
|
||||
| Component | Status |
|
||||
|---|---|
|
||||
| Prometheus | UP |
|
||||
| Blackbox Exporter | UP |
|
||||
| Proxmox Node Exporters | UP |
|
||||
| K3s Node Exporters | UP |
|
||||
| Windows Exporter | UP |
|
||||
| TrueNAS Graphite Exporter | UP |
|
||||
| UniFi Exporter / Unpoller | UP |
|
||||
| Grafana Dashboards | Partially complete and receiving live data |
|
||||
|
||||
---
|
||||
|
||||
## Key Outcome
|
||||
|
||||
At the end of this phase, Ced’s Observability Stack successfully collected metrics from multiple platforms across the homelab environment.
|
||||
|
||||
This moved the project from a documentation-only repo into a real operating observability system.
|
||||
Reference in New Issue
Block a user