13 KiB
🤖 CedOps — AI Infrastructure Operations NOC
A live AI operations platform running on Hetzner VPS — 8 autonomous agents orchestrated by Paperclip, monitored in real-time by Prometheus (hosted on Proxmox homelab) scraping over Tailscale, visualized in a Grafana NOC dashboard.
📸 NOC Dashboard
V2 — Current
V1
Live: All 5 public services reporting UP · NOC Health Score 100% · VPS Disk 27% · Prometheus scraping every 30s
🧭 Overview
CedOps is a production-style AI operations platform built to simulate a real-world Platform Engineering and AI Infrastructure environment. The VPS stack runs on Hetzner and is monitored by Prometheus hosted on a Proxmox homelab node, communicating over a private Tailscale network — with everything visualized in a live Grafana NOC dashboard.
The platform combines:
- Paperclip — AI Agent Orchestration Platform managing 8 autonomous agents
- SugeBot — Hermes AI agent deployed as Telegram Operations Assistant (CTO/Ops role)
- n8n — Automation platform for workflow execution and inter-service routing
- OpenRouter — AI model gateway serving as the LLM backbone for all agents
- Prometheus — Hosted on Proxmox homelab, scraping VPS services over Tailscale
- Grafana — Full observability NOC dashboard visualizing the entire stack in real-time
- Nginx Proxy Manager — Reverse proxy + SSL management across all services
- Docker + Portainer — Container infrastructure and management layer
🏗️ Architecture Overview
Internet
│
┌─────────────┴─────────────┐
│ Cloudflare Edge │
│ DNS · Tunnels · Proxy │
└─────────────┬─────────────┘
│
┌─────────────▼─────────────┐
│ Hetzner VPS │
│ synthos-node-1 │
│ │
│ ┌──────────────────────┐ │
│ │ Nginx Proxy Manager │ │
│ │ Reverse Proxy / SSL │ │
│ └──────────┬───────────┘ │
│ │ │
│ ┌──────────▼───────────┐ │
│ │ Portainer │ │
│ │ Docker Management │ │
│ └──────────┬───────────┘ │
│ │ │
│ ┌──────────▼──────────────────────────────────┐ │
│ │ CedOps Platform │ │
│ │ │ │
│ │ ┌─────────────┐ ┌──────────────────────┐ │ │
│ │ │ Paperclip │ │ SugeBot │ │ │
│ │ │ AI Orch. │ │ Telegram Ops │ │ │
│ │ │ 8 Agents │ │ (Hermes Agent) │ │ │
│ │ └──────┬──────┘ └──────────┬───────────┘ │ │
│ │ │ │ │ │
│ │ ┌──────▼──────┐ ┌──────────▼───────────┐ │ │
│ │ │ OpenRouter │ │ n8n │ │ │
│ │ │ AI Gateway │ │ Automation │ │ │
│ │ └─────────────┘ └──────────────────────┘ │ │
│ └───────────────────────────────────────────────┘ │
│ ▲ Node Exporter + Blackbox │
└──────────────┼──────────────────────────────────────┘
│ Tailscale (encrypted private tunnel)
┌──────────────┴──────────────────────┐
│ Proxmox Homelab │
│ │
│ ┌─────────────┐ ┌───────────────┐ │
│ │ Prometheus │ │ Grafana │ │
│ │ Metrics │→ │ NOC Dashboard │ │
│ │ Collection │ │ │ │
│ └─────────────┘ └───────────────┘ │
└─────────────────────────────────────┘
🤖 AI Agent Roster
All agents run inside Paperclip — the AI Agent Orchestration Platform at paperclip.synthossystems.com.
| Agent | Role | Status |
|---|---|---|
| Suge | CTO / Operations (Hermes Agent) — also deployed as SugeBot on Telegram | ✅ Live |
| CEO | Executive decision-making and strategic oversight | ✅ Live |
| ClaudeCoder | Software engineering and code generation | ✅ Live |
| Engineer | Infrastructure and systems engineering | ✅ Live |
| Marketing | Marketing strategy and content | ✅ Live |
| Offer Builder Agent | Sales offer generation and pipeline | ✅ Live |
| Research | Market research and intelligence gathering | ✅ Live |
| Sales Agent | Lead qualification and outreach | ✅ Live |
Suge serves dual duty — the primary Hermes-style ops agent inside Paperclip, and the external-facing SugeBot on Telegram for real-time operational commands.
🖥️ Infrastructure Layer
All services run on a Hetzner VPS (synthos-node-1) behind Cloudflare Tunnels with zero exposed ports.
| Service | Role | URL |
|---|---|---|
| VPS / Portainer | Docker container management | Internal |
| Nginx Proxy Manager | Reverse proxy + SSL termination | Internal |
| Homepage | CedOps command center dashboard | ops.synthossystems.com |
| Synthos Website | Business-facing website | synthossystems.com |
| Ced's Home Lab | Homelab / Portfolio platform | cedshomelab.com |
Traffic Flow:
Internet → Cloudflare Edge → Tunnel → Nginx Proxy Manager → Docker Services
📊 Observability & NOC
Prometheus runs on the Proxmox homelab and scrapes the Hetzner VPS over a Tailscale private tunnel — keeping metrics collection completely off the public internet. Grafana visualizes everything in the live NOC dashboard.
Executive Health Panel
- Synthos VPS Status — UP/DOWN state with instant alerting
- Public Services Up — count of healthy endpoints (target: 5/5)
- NOC Health Score — composite platform health percentage
- VPS Uptime — continuous uptime tracking in days
VPS Resource Health
- CPU Usage % —
synthos-node-1CPU time series - Memory Utilization % — real-time memory pressure
- Disk Used % — current at 27%
- Network Traffic — RX/TX bandwidth graphs (30s refresh)
Public Service Availability
Tracked via Blackbox Exporter HTTP probes:
| Service | Endpoint | Status |
|---|---|---|
| cedshomelab | cedshomelab.com | ✅ UP |
| n8n | n8n.synthossystems.com | ✅ UP |
| cedops | ops.synthossystems.com | ✅ UP |
| paperclip | paperclip.synthossystems.com | ✅ UP |
| synthos-website | synthossystems.com | ✅ UP |
Reliability Metrics
- HTTP Status Codes per endpoint (all returning
200) - Public Endpoint Response Time (ms) — time series graph
- SSL Expiry Days — tracked per certificate with warning thresholds
🔒 Security Architecture
CedOps is designed with zero public infrastructure exposure.
| Control | Implementation |
|---|---|
| No open ports | Cloudflare Tunnel handles all ingress |
| Private metrics pipeline | Prometheus → Tailscale → VPS; never public |
| SSL everywhere | NPM handles termination for all services |
| Container isolation | Docker network segmentation via Portainer |
| Secret management | No secrets in this repo — handled via environment variables |
📂 Repository Structure
cedops-ai-infrastructure-noc/
│
├── docs/
│ ├── architecture/
│ ├── screenshots/
│ │ ├── dashboard_v1.png
│ │ └── dashboard_v2.png
│ └── sops/
│
├── grafana/
│ ├── cedops-ai-infrastructure-noc-v1.json
│ └── cedops-ai-infrastructure-noc-v2.json
│
├── Prometheus/
│ ├── prometheus-example.yml
│ └── blackbox-targets-example.yml
│
├── diagrams/
│ └── cedops-architecture.mmd
│
└── README.md
Note: Prometheus and Blackbox configs are sanitized for public sharing. Production configs are managed privately via environment variables and are never committed to this repo.
🛠️ Stack
| Layer | Technology |
|---|---|
| Compute | Hetzner VPS (synthos-node-1) |
| Containers | Docker + Portainer |
| Reverse Proxy | Nginx Proxy Manager |
| DNS & Tunnels | Cloudflare |
| Private Network | Tailscale (Proxmox ↔ VPS metrics tunnel) |
| AI Orchestration | Paperclip |
| AI Gateway | OpenRouter |
| Automation | n8n |
| Ops Agent | SugeBot (Hermes / Telegram) |
| Metrics | Prometheus (Proxmox) + Node Exporter + Blackbox Exporter |
| Visualization | Grafana |
| Virtualization | Proxmox VE (homelab, hosts Prometheus + Grafana) |
✅ What's Built
V1 Dashboard
- VPS UP/DOWN status panel
- Public service availability (5 endpoints via Blackbox Exporter)
- VPS resource health (CPU, Memory, Disk, Network)
- HTTP status codes per endpoint
- Public endpoint response times
- SSL certificate expiry tracking
- Cross-environment scraping via Tailscale (Proxmox → Hetzner VPS)
V2 Dashboard
- Agent activity metrics (Paperclip runs, success rate, token spend)
- SugeBot command log panel
- Per-agent health monitoring
- n8n workflow execution tracking
- Telegram alert integration via SugeBot
🔮 Roadmap
- AI Incident Detection — anomaly-triggered agent escalation
- Automated Remediation — n8n workflows triggered by Prometheus alerts
- Multi-environment monitoring (homelab + VPS unified NOC)
- Discord alert channel
👤 Author
Chase Dumphord Digital Systems Engineer | Platform Engineering | AI Infrastructure | Observability
🔗 Related Repos
| Repo | Description |
|---|---|
| ceds-homelab | 6-node Proxmox cluster + 12-node K3s + full homelab infrastructure |
| ced-k3s-homelab | 12-node Raspberry Pi K3s cluster detail |
| ceds-noc | Custom-built public NOC status page |