mirror of
https://github.com/ced4568/cedops-ai-infrastructure-noc.git
synced 2026-08-12 21:34:02 +00:00
"docs: complete README rewrite with agent roster, NOC dashboard, and architecture"
This commit is contained in:
Binary file not shown.
|
Before Width: | Height: | Size: 274 KiB |
@@ -1,166 +1,187 @@
|
|||||||
# 🚀 CedOps — AI Infrastructure Operations NOC
|
# 🤖 CedOps — AI Infrastructure Operations NOC
|
||||||
|
|
||||||
|
> A live AI operations platform running on Hetzner VPS — 8 autonomous agents orchestrated by Paperclip, monitored in real-time by a Prometheus + Grafana NOC dashboard. Built by a Digital Systems Engineer at GE Aerospace.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
[](https://noc.chasedumphord.com)
|
||||||
|
[](https://paperclip.synthossystems.com)
|
||||||
|
[](https://synthossystems.com)
|
||||||
|
[](https://chasedumphord.com)
|
||||||
|
[](#-ai-agent-roster)
|
||||||
|
[](#️-infrastructure-layer)
|
||||||
|
[](#-observability--noc)
|
||||||
|
|
||||||
## Overview
|
---
|
||||||
|
|
||||||
CedOps is an **AI-Assisted Infrastructure Operations Platform** designed to provide centralized monitoring, observability, uptime management, and operational visibility across a hybrid environment of:
|
## 📸 NOC Dashboard
|
||||||
|
|
||||||
- 🏠 Homelab Infrastructure
|
[](docs/screenshots/dashboard_v1.png)
|
||||||
- ☁️ VPS Cloud Infrastructure
|
|
||||||
- 🤖 AI Operations Systems
|
> **Live:** All 5 public services reporting UP · NOC Health Score 100% · VPS Disk 27% · Prometheus scraping every 30s
|
||||||
- 🌐 Public-Facing Services
|
|
||||||
- 📡 Network & Platform Health
|
---
|
||||||
|
|
||||||
|
## 🧭 Overview
|
||||||
|
|
||||||
|
CedOps is a **production-style AI operations platform** built to simulate a real-world Platform Engineering and AI Infrastructure environment. The entire stack runs on a Hetzner VPS and is monitored continuously by a live Grafana NOC dashboard.
|
||||||
|
|
||||||
The platform combines:
|
The platform combines:
|
||||||
|
|
||||||
- **Prometheus** for metrics collection
|
- **Paperclip** — AI Agent Orchestration Platform managing 8 autonomous agents
|
||||||
- **Grafana** for visualization
|
- **SugeBot** — Hermes AI agent deployed as Telegram Operations Assistant (CTO/Ops role)
|
||||||
- **Blackbox Exporter** for service monitoring
|
- **n8n** — Automation platform for workflow execution and inter-service routing
|
||||||
- **Node Exporter** for infrastructure metrics
|
- **OpenRouter** — AI model gateway serving as the LLM backbone for all agents
|
||||||
- **Tailscale** for secure private observability
|
- **Prometheus + Grafana** — Full observability NOC monitoring the entire stack in real-time
|
||||||
- **Proxmox VE** for virtualization
|
- **Nginx Proxy Manager** — Reverse proxy + SSL management across all services
|
||||||
- **Docker** for service deployment
|
- **Docker + Portainer** — Container infrastructure and management layer
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# 🎯 Project Goals
|
## 🏗️ Architecture Overview
|
||||||
|
|
||||||
CedOps was built to simulate a **real-world Platform Engineering / Infrastructure Operations environment**.
|
```
|
||||||
|
Internet
|
||||||
Goals:
|
│
|
||||||
|
┌─────────────┴─────────────┐
|
||||||
✅ Single-pane-of-glass infrastructure visibility
|
│ Cloudflare Edge │
|
||||||
✅ Secure observability without exposing services publicly
|
│ DNS · Tunnels · Proxy │
|
||||||
✅ Hybrid monitoring (on-prem + cloud)
|
└─────────────┬─────────────┘
|
||||||
✅ Public service uptime tracking
|
│
|
||||||
✅ AI platform monitoring
|
┌─────────────▼─────────────┐
|
||||||
✅ Portfolio-grade infrastructure engineering project
|
│ Hetzner VPS │
|
||||||
|
│ synthos-node-1 │
|
||||||
---
|
│ │
|
||||||
|
│ ┌──────────────────────┐ │
|
||||||
# 🏗️ Architecture Overview
|
│ │ Nginx Proxy Manager │ │
|
||||||
|
│ │ Reverse Proxy / SSL │ │
|
||||||
```text
|
│ └──────────┬───────────┘ │
|
||||||
Internet
|
│ │ │
|
||||||
│
|
│ ┌──────────▼───────────┐ │
|
||||||
┌───────────┴───────────┐
|
│ │ Portainer │ │
|
||||||
│ Public Services │
|
│ │ Docker Management │ │
|
||||||
│ synthossystems.com │
|
│ └──────────┬───────────┘ │
|
||||||
│ noc.chasedumphord.com │
|
│ │ │
|
||||||
│ cedshomelab.com │
|
│ ┌──────────▼───────────────────────────────┐ │
|
||||||
└───────────┬───────────┘
|
│ │ CedOps Platform │ │
|
||||||
│
|
│ │ │ │
|
||||||
Blackbox Exporter
|
│ │ ┌─────────────┐ ┌──────────────────┐ │ │
|
||||||
│
|
│ │ │ Paperclip │ │ SugeBot │ │ │
|
||||||
▼
|
│ │ │ AI Orch. │ │ Telegram Ops │ │ │
|
||||||
Prometheus Monitoring
|
│ │ │ 8 Agents │ │ (Hermes Agent) │ │ │
|
||||||
(10.10.30.140)
|
│ │ └──────┬──────┘ └────────┬─────────┘ │ │
|
||||||
│
|
│ │ │ │ │ │
|
||||||
┌───────────────┼────────────────┐
|
│ │ ┌──────▼──────┐ ┌────────▼─────────┐ │ │
|
||||||
│ │ │
|
│ │ │ OpenRouter │ │ n8n │ │ │
|
||||||
▼ ▼ ▼
|
│ │ │ AI Gateway │ │ Automation │ │ │
|
||||||
Proxmox Nodes K3s Cluster Hetzner VPS
|
│ │ └─────────────┘ └──────────────────┘ │ │
|
||||||
Node Exporter Monitoring synthos-node-1
|
│ └───────────────────────────────────────────┘ │
|
||||||
│
|
│ │
|
||||||
Node Exporter
|
│ ┌───────────────────────────────────────┐ │
|
||||||
│
|
│ │ Observability Stack │ │
|
||||||
Tailscale
|
│ │ Prometheus → Grafana NOC Dashboard │ │
|
||||||
|
│ │ Blackbox Exporter → Service Uptime │ │
|
||||||
|
│ │ Node Exporter → VPS Resource Health │ │
|
||||||
|
│ └───────────────────────────────────────┘ │
|
||||||
|
└───────────────────────────────────────────────┘
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# 🖥️ Dashboard Features
|
## 🤖 AI Agent Roster
|
||||||
|
|
||||||
## Executive Health
|
All agents run inside **Paperclip** — the AI Agent Orchestration Platform at `paperclip.synthossystems.com`.
|
||||||
|
|
||||||
- VPS Status
|
| Agent | Role | Status |
|
||||||
- Public Services Status
|
|---|---|---|
|
||||||
- NOC Health Score
|
| **Suge** | CTO / Operations (Hermes Agent) — also deployed as SugeBot on Telegram | ✅ Live |
|
||||||
- Uptime Metrics
|
| **CEO** | Executive decision-making and strategic oversight | ✅ Live |
|
||||||
|
| **ClaudeCoder** | Software engineering and code generation | ✅ Live |
|
||||||
|
| **Engineer** | Infrastructure and systems engineering | ✅ Live |
|
||||||
|
| **Marketing** | Marketing strategy and content | ✅ Live |
|
||||||
|
| **Offer Builder Agent** | Sales offer generation and pipeline | ✅ Live |
|
||||||
|
| **Research** | Market research and intelligence gathering | ✅ Live |
|
||||||
|
| **Sales Agent** | Lead qualification and outreach | ✅ Live |
|
||||||
|
|
||||||
## Infrastructure Monitoring
|
> **Suge** serves dual duty — the primary Hermes-style ops agent inside Paperclip, and the external-facing **SugeBot** on Telegram for real-time operational commands.
|
||||||
|
|
||||||
- CPU Usage
|
|
||||||
- Memory Utilization
|
|
||||||
- Disk Usage
|
|
||||||
- Network Traffic
|
|
||||||
|
|
||||||
## Public Service Monitoring
|
|
||||||
|
|
||||||
Tracked via Blackbox Exporter:
|
|
||||||
|
|
||||||
- CedOps
|
|
||||||
- n8n
|
|
||||||
- Paperclip
|
|
||||||
- Synthos Website
|
|
||||||
- Ced's Home Lab
|
|
||||||
- Grafana
|
|
||||||
- Prometheus
|
|
||||||
|
|
||||||
## Reliability Monitoring
|
|
||||||
|
|
||||||
- HTTP Status Codes
|
|
||||||
- Response Times
|
|
||||||
- Endpoint Availability
|
|
||||||
- Service Reachability
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# 📸 Dashboard Screenshots
|
## 🖥️ Infrastructure Layer
|
||||||
|
|
||||||
## CedOps Infrastructure NOC (V1)
|
All services run on a **Hetzner VPS** (`synthos-node-1`) behind Cloudflare Tunnels with zero exposed ports.
|
||||||
|
|
||||||
|
| Service | Role | URL |
|
||||||
|
|---|---|---|
|
||||||
|
| **VPS / Portainer** | Docker container management | Internal |
|
||||||
|
| **Nginx Proxy Manager** | Reverse proxy + SSL termination | Internal |
|
||||||
|
| **Homepage** | CedOps command center dashboard | `ops.synthossystems.com` |
|
||||||
|
| **Synthos Website** | Business-facing website | `synthossystems.com` |
|
||||||
|
| **Ced's Home Lab** | Homelab / Portfolio platform | `cedshomelab.com` |
|
||||||
|
|
||||||
|
**Traffic Flow:**
|
||||||
|
```
|
||||||
|
Internet → Cloudflare Edge → Tunnel → Nginx Proxy Manager → Docker Services
|
||||||
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# 🔒 Security Architecture
|
## 📊 Observability & NOC
|
||||||
|
|
||||||
CedOps prioritizes **secure observability**.
|
The CedOps NOC dashboard monitors the entire VPS stack in real-time.
|
||||||
|
|
||||||
### No Public Prometheus Exposure
|
### Executive Health Panel
|
||||||
|
- **Synthos VPS Status** — UP/DOWN state with instant alerting
|
||||||
|
- **Public Services Up** — count of healthy endpoints (target: 5/5)
|
||||||
|
- **NOC Health Score** — composite platform health percentage
|
||||||
|
- **VPS Uptime** — continuous uptime tracking in days
|
||||||
|
|
||||||
Infrastructure metrics are collected using:
|
### VPS Resource Health
|
||||||
|
- CPU Usage % — `synthos-node-1` CPU time series
|
||||||
|
- Memory Utilization % — real-time memory pressure
|
||||||
|
- Disk Used % — current at 27%
|
||||||
|
- Network Traffic — RX/TX bandwidth graphs (30s refresh)
|
||||||
|
|
||||||
**Tailscale private networking**
|
### Public Service Availability
|
||||||
|
|
||||||
Benefits:
|
Tracked via **Blackbox Exporter** HTTP probes:
|
||||||
|
|
||||||
- Encrypted communication
|
| Service | Endpoint | Status |
|
||||||
- Zero Trust access
|
|---|---|---|
|
||||||
- No public metrics exposure
|
| cedshomelab | cedshomelab.com | ✅ UP |
|
||||||
- Private infrastructure scraping
|
| n8n | n8n.synthossystems.com | ✅ UP |
|
||||||
|
| cedops | ops.synthossystems.com | ✅ UP |
|
||||||
|
| paperclip | paperclip.synthossystems.com | ✅ UP |
|
||||||
|
| synthos-website | synthossystems.com | ✅ UP |
|
||||||
|
|
||||||
### Monitoring Stack
|
### Reliability Metrics
|
||||||
|
- HTTP Status Codes per endpoint (all returning `200`)
|
||||||
| Tool | Purpose |
|
- Public Endpoint Response Time (ms) — time series graph
|
||||||
|------|---------|
|
- SSL Expiry Days — tracked per certificate with warning thresholds
|
||||||
| Prometheus | Metrics collection |
|
|
||||||
| Grafana | Visualization |
|
|
||||||
| Blackbox Exporter | Uptime checks |
|
|
||||||
| Node Exporter | Linux system metrics |
|
|
||||||
| Tailscale | Secure networking |
|
|
||||||
| Docker | Service deployment |
|
|
||||||
| Proxmox | Virtualization |
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# 📂 Repository Structure
|
## 🔒 Security Architecture
|
||||||
|
|
||||||
```text
|
CedOps is designed with **zero public infrastructure exposure**.
|
||||||
|
|
||||||
|
| Control | Implementation |
|
||||||
|
|---|---|
|
||||||
|
| No open ports | Cloudflare Tunnel handles all ingress |
|
||||||
|
| No public Prometheus | Metrics stay private; only Grafana is exposed |
|
||||||
|
| SSL everywhere | NPM handles termination for all services |
|
||||||
|
| Container isolation | Docker network segmentation via Portainer |
|
||||||
|
| Secret management | No secrets in this repo — handled via environment variables |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📂 Repository Structure
|
||||||
|
|
||||||
|
```
|
||||||
cedops-ai-infrastructure-noc/
|
cedops-ai-infrastructure-noc/
|
||||||
│
|
│
|
||||||
├── docs/
|
├── docs/
|
||||||
│ ├── architecture/
|
│ ├── architecture/
|
||||||
│ ├── screenshots/
|
│ ├── screenshots/
|
||||||
|
│ │ └── dashboard_v1.png
|
||||||
│ └── sops/
|
│ └── sops/
|
||||||
│
|
│
|
||||||
├── grafana/
|
├── grafana/
|
||||||
@@ -176,78 +197,315 @@ cedops-ai-infrastructure-noc/
|
|||||||
└── README.md
|
└── README.md
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
> **Note:** Prometheus and Blackbox configs are sanitized for public sharing. Production configs are managed privately via environment variables and are never committed to this repo.
|
||||||
|
|
||||||
# 🛠️ Skills Demonstrated
|
|
||||||
|
|
||||||
### Platform Engineering
|
|
||||||
- Infrastructure Monitoring
|
|
||||||
- Observability Design
|
|
||||||
- Service Reliability
|
|
||||||
- Secure Networking
|
|
||||||
|
|
||||||
### DevOps
|
|
||||||
- Linux Administration
|
|
||||||
- Docker
|
|
||||||
- YAML Configuration
|
|
||||||
- Troubleshooting
|
|
||||||
|
|
||||||
### Cloud & Infrastructure
|
|
||||||
- Hetzner VPS
|
|
||||||
- Proxmox Virtualization
|
|
||||||
- Hybrid Infrastructure
|
|
||||||
- Private Networking
|
|
||||||
|
|
||||||
### Monitoring & Observability
|
|
||||||
- Prometheus
|
|
||||||
- Grafana
|
|
||||||
- Blackbox Exporter
|
|
||||||
- Alerting
|
|
||||||
- Metrics Design
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# 💼 Resume Relevance
|
## 🛠️ Stack
|
||||||
|
|
||||||
This project demonstrates real-world experience with:
|
| Layer | Technology |
|
||||||
|
|---|---|
|
||||||
**Platform Engineering**
|
| **Compute** | Hetzner VPS (synthos-node-1) |
|
||||||
**Infrastructure Monitoring**
|
| **Containers** | Docker + Portainer |
|
||||||
**Observability Engineering**
|
| **Reverse Proxy** | Nginx Proxy Manager |
|
||||||
**DevOps Workflows**
|
| **DNS & Tunnels** | Cloudflare |
|
||||||
**Cloud + On-Prem Hybrid Systems**
|
| **AI Orchestration** | Paperclip |
|
||||||
|
| **AI Gateway** | OpenRouter |
|
||||||
|
| **Automation** | n8n |
|
||||||
|
| **Ops Agent** | SugeBot (Hermes / Telegram) |
|
||||||
|
| **Metrics** | Prometheus + Node Exporter + Blackbox Exporter |
|
||||||
|
| **Visualization** | Grafana |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# 🔮 Roadmap
|
## 🔮 Roadmap
|
||||||
|
|
||||||
## V2 Dashboard
|
### V2 Dashboard
|
||||||
|
- [ ] Agent activity metrics (Paperclip runs, success rate, token spend)
|
||||||
|
- [ ] SugeBot command log panel
|
||||||
|
- [ ] Per-agent health monitoring
|
||||||
|
- [ ] n8n workflow execution tracking
|
||||||
|
- [ ] Telegram alert integration via SugeBot
|
||||||
|
|
||||||
- SSL Expiration Monitoring
|
### Platform Enhancements
|
||||||
- Alert Feed
|
- [ ] AI Incident Detection — anomaly-triggered agent escalation
|
||||||
- SLA Tracking
|
- [ ] Automated Remediation — n8n workflows triggered by Prometheus alerts
|
||||||
- Reliability Scoring
|
- [ ] Multi-environment monitoring (homelab + VPS unified NOC)
|
||||||
- Capacity Metrics
|
- [ ] Discord alert channel
|
||||||
|
|
||||||
## Future Enhancements
|
|
||||||
|
|
||||||
- Telegram Alerts
|
|
||||||
- Discord Notifications
|
|
||||||
- AI Incident Detection
|
|
||||||
- Automated Remediation
|
|
||||||
- Multi-Environment Monitoring
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# 👨💻 Author
|
## 👤 Author
|
||||||
|
|
||||||
### Chase Dumphord
|
**Chase Dumphord**
|
||||||
|
Digital Systems Engineer | Platform Engineering | AI Infrastructure | Observability
|
||||||
|
|
||||||
Digital Systems Engineer | Infrastructure & Platform Engineering | Observability | AI Systems
|
[](https://www.linkedin.com/in/toochase-dumphord/)
|
||||||
|
[](https://github.com/ced4568)
|
||||||
|
[](https://noc.chasedumphord.com)
|
||||||
|
[](https://chasedumphord.com)
|
||||||
|
|
||||||
GitHub: https://github.com/ced4568
|
---
|
||||||
|
|
||||||
Portfolio:
|
## 🔗 Related Repos
|
||||||
- https://chasedumphord.com
|
|
||||||
- https://noc.chasedumphord.com
|
| Repo | Description |
|
||||||
- https://cedshomelab.com
|
|---|---|
|
||||||
|
| [ceds-homelab](https://github.com/ced4568/ceds-homelab) | 6-node Proxmox cluster + 12-node K3s + full homelab infrastructure |
|
||||||
|
| [ced-k3s-homelab](https://github.com/ced4568/ced-k3s-homelab) | 12-node Raspberry Pi K3s cluster detail |
|
||||||
|
# 🤖 CedOps — AI Infrastructure Operations NOC
|
||||||
|
|
||||||
|
> A live AI operations platform running on Hetzner VPS — 8 autonomous agents orchestrated by Paperclip, monitored in real-time by a Prometheus + Grafana NOC dashboard. Built by a Digital Systems Engineer at GE Aerospace.
|
||||||
|
|
||||||
|
[](https://noc.chasedumphord.com)
|
||||||
|
[](https://paperclip.synthossystems.com)
|
||||||
|
[](https://synthossystems.com)
|
||||||
|
[](https://chasedumphord.com)
|
||||||
|
[](#-ai-agent-roster)
|
||||||
|
[](#️-infrastructure-layer)
|
||||||
|
[](#-observability--noc)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📸 NOC Dashboard
|
||||||
|
|
||||||
|
[](docs/screenshots/dashboard_v1.png)
|
||||||
|
|
||||||
|
> **Live:** All 5 public services reporting UP · NOC Health Score 100% · VPS Disk 27% · Prometheus scraping every 30s
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🧭 Overview
|
||||||
|
|
||||||
|
CedOps is a **production-style AI operations platform** built to simulate a real-world Platform Engineering and AI Infrastructure environment. The entire stack runs on a Hetzner VPS and is monitored continuously by a live Grafana NOC dashboard.
|
||||||
|
|
||||||
|
The platform combines:
|
||||||
|
|
||||||
|
- **Paperclip** — AI Agent Orchestration Platform managing 8 autonomous agents
|
||||||
|
- **SugeBot** — Hermes AI agent deployed as Telegram Operations Assistant (CTO/Ops role)
|
||||||
|
- **n8n** — Automation platform for workflow execution and inter-service routing
|
||||||
|
- **OpenRouter** — AI model gateway serving as the LLM backbone for all agents
|
||||||
|
- **Prometheus + Grafana** — Full observability NOC monitoring the entire stack in real-time
|
||||||
|
- **Nginx Proxy Manager** — Reverse proxy + SSL management across all services
|
||||||
|
- **Docker + Portainer** — Container infrastructure and management layer
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🏗️ Architecture Overview
|
||||||
|
|
||||||
|
```
|
||||||
|
Internet
|
||||||
|
│
|
||||||
|
┌─────────────┴─────────────┐
|
||||||
|
│ Cloudflare Edge │
|
||||||
|
│ DNS · Tunnels · Proxy │
|
||||||
|
└─────────────┬─────────────┘
|
||||||
|
│
|
||||||
|
┌─────────────▼─────────────┐
|
||||||
|
│ Hetzner VPS │
|
||||||
|
│ synthos-node-1 │
|
||||||
|
│ │
|
||||||
|
│ ┌──────────────────────┐ │
|
||||||
|
│ │ Nginx Proxy Manager │ │
|
||||||
|
│ │ Reverse Proxy / SSL │ │
|
||||||
|
│ └──────────┬───────────┘ │
|
||||||
|
│ │ │
|
||||||
|
│ ┌──────────▼───────────┐ │
|
||||||
|
│ │ Portainer │ │
|
||||||
|
│ │ Docker Management │ │
|
||||||
|
│ └──────────┬───────────┘ │
|
||||||
|
│ │ │
|
||||||
|
│ ┌──────────▼───────────────────────────────┐ │
|
||||||
|
│ │ CedOps Platform │ │
|
||||||
|
│ │ │ │
|
||||||
|
│ │ ┌─────────────┐ ┌──────────────────┐ │ │
|
||||||
|
│ │ │ Paperclip │ │ SugeBot │ │ │
|
||||||
|
│ │ │ AI Orch. │ │ Telegram Ops │ │ │
|
||||||
|
│ │ │ 8 Agents │ │ (Hermes Agent) │ │ │
|
||||||
|
│ │ └──────┬──────┘ └────────┬─────────┘ │ │
|
||||||
|
│ │ │ │ │ │
|
||||||
|
│ │ ┌──────▼──────┐ ┌────────▼─────────┐ │ │
|
||||||
|
│ │ │ OpenRouter │ │ n8n │ │ │
|
||||||
|
│ │ │ AI Gateway │ │ Automation │ │ │
|
||||||
|
│ │ └─────────────┘ └──────────────────┘ │ │
|
||||||
|
│ └───────────────────────────────────────────┘ │
|
||||||
|
│ │
|
||||||
|
│ ┌───────────────────────────────────────┐ │
|
||||||
|
│ │ Observability Stack │ │
|
||||||
|
│ │ Prometheus → Grafana NOC Dashboard │ │
|
||||||
|
│ │ Blackbox Exporter → Service Uptime │ │
|
||||||
|
│ │ Node Exporter → VPS Resource Health │ │
|
||||||
|
│ └───────────────────────────────────────┘ │
|
||||||
|
└───────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🤖 AI Agent Roster
|
||||||
|
|
||||||
|
All agents run inside **Paperclip** — the AI Agent Orchestration Platform at `paperclip.synthossystems.com`.
|
||||||
|
|
||||||
|
| Agent | Role | Status |
|
||||||
|
|---|---|---|
|
||||||
|
| **Suge** | CTO / Operations (Hermes Agent) — also deployed as SugeBot on Telegram | ✅ Live |
|
||||||
|
| **CEO** | Executive decision-making and strategic oversight | ✅ Live |
|
||||||
|
| **ClaudeCoder** | Software engineering and code generation | ✅ Live |
|
||||||
|
| **Engineer** | Infrastructure and systems engineering | ✅ Live |
|
||||||
|
| **Marketing** | Marketing strategy and content | ✅ Live |
|
||||||
|
| **Offer Builder Agent** | Sales offer generation and pipeline | ✅ Live |
|
||||||
|
| **Research** | Market research and intelligence gathering | ✅ Live |
|
||||||
|
| **Sales Agent** | Lead qualification and outreach | ✅ Live |
|
||||||
|
|
||||||
|
> **Suge** serves dual duty — the primary Hermes-style ops agent inside Paperclip, and the external-facing **SugeBot** on Telegram for real-time operational commands.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🖥️ Infrastructure Layer
|
||||||
|
|
||||||
|
All services run on a **Hetzner VPS** (`synthos-node-1`) behind Cloudflare Tunnels with zero exposed ports.
|
||||||
|
|
||||||
|
| Service | Role | URL |
|
||||||
|
|---|---|---|
|
||||||
|
| **VPS / Portainer** | Docker container management | Internal |
|
||||||
|
| **Nginx Proxy Manager** | Reverse proxy + SSL termination | Internal |
|
||||||
|
| **Homepage** | CedOps command center dashboard | `ops.synthossystems.com` |
|
||||||
|
| **Synthos Website** | Business-facing website | `synthossystems.com` |
|
||||||
|
| **Ced's Home Lab** | Homelab / Portfolio platform | `cedshomelab.com` |
|
||||||
|
|
||||||
|
**Traffic Flow:**
|
||||||
|
```
|
||||||
|
Internet → Cloudflare Edge → Tunnel → Nginx Proxy Manager → Docker Services
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📊 Observability & NOC
|
||||||
|
|
||||||
|
The CedOps NOC dashboard monitors the entire VPS stack in real-time.
|
||||||
|
|
||||||
|
### Executive Health Panel
|
||||||
|
- **Synthos VPS Status** — UP/DOWN state with instant alerting
|
||||||
|
- **Public Services Up** — count of healthy endpoints (target: 5/5)
|
||||||
|
- **NOC Health Score** — composite platform health percentage
|
||||||
|
- **VPS Uptime** — continuous uptime tracking in days
|
||||||
|
|
||||||
|
### VPS Resource Health
|
||||||
|
- CPU Usage % — `synthos-node-1` CPU time series
|
||||||
|
- Memory Utilization % — real-time memory pressure
|
||||||
|
- Disk Used % — current at 27%
|
||||||
|
- Network Traffic — RX/TX bandwidth graphs (30s refresh)
|
||||||
|
|
||||||
|
### Public Service Availability
|
||||||
|
|
||||||
|
Tracked via **Blackbox Exporter** HTTP probes:
|
||||||
|
|
||||||
|
| Service | Endpoint | Status |
|
||||||
|
|---|---|---|
|
||||||
|
| cedshomelab | cedshomelab.com | ✅ UP |
|
||||||
|
| n8n | n8n.synthossystems.com | ✅ UP |
|
||||||
|
| cedops | ops.synthossystems.com | ✅ UP |
|
||||||
|
| paperclip | paperclip.synthossystems.com | ✅ UP |
|
||||||
|
| synthos-website | synthossystems.com | ✅ UP |
|
||||||
|
|
||||||
|
### Reliability Metrics
|
||||||
|
- HTTP Status Codes per endpoint (all returning `200`)
|
||||||
|
- Public Endpoint Response Time (ms) — time series graph
|
||||||
|
- SSL Expiry Days — tracked per certificate with warning thresholds
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🔒 Security Architecture
|
||||||
|
|
||||||
|
CedOps is designed with **zero public infrastructure exposure**.
|
||||||
|
|
||||||
|
| Control | Implementation |
|
||||||
|
|---|---|
|
||||||
|
| No open ports | Cloudflare Tunnel handles all ingress |
|
||||||
|
| No public Prometheus | Metrics stay private; only Grafana is exposed |
|
||||||
|
| SSL everywhere | NPM handles termination for all services |
|
||||||
|
| Container isolation | Docker network segmentation via Portainer |
|
||||||
|
| Secret management | No secrets in this repo — handled via environment variables |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📂 Repository Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
cedops-ai-infrastructure-noc/
|
||||||
|
│
|
||||||
|
├── docs/
|
||||||
|
│ ├── architecture/
|
||||||
|
│ ├── screenshots/
|
||||||
|
│ │ └── dashboard_v1.png
|
||||||
|
│ └── sops/
|
||||||
|
│
|
||||||
|
├── grafana/
|
||||||
|
│ └── cedops-ai-infrastructure-noc-v1.json
|
||||||
|
│
|
||||||
|
├── prometheus/
|
||||||
|
│ ├── prometheus-example.yml
|
||||||
|
│ └── blackbox-targets-example.yml
|
||||||
|
│
|
||||||
|
├── diagrams/
|
||||||
|
│ └── cedops-architecture.mmd
|
||||||
|
│
|
||||||
|
└── README.md
|
||||||
|
```
|
||||||
|
|
||||||
|
> **Note:** Prometheus and Blackbox configs are sanitized for public sharing. Production configs are managed privately via environment variables and are never committed to this repo.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🛠️ Stack
|
||||||
|
|
||||||
|
| Layer | Technology |
|
||||||
|
|---|---|
|
||||||
|
| **Compute** | Hetzner VPS (synthos-node-1) |
|
||||||
|
| **Containers** | Docker + Portainer |
|
||||||
|
| **Reverse Proxy** | Nginx Proxy Manager |
|
||||||
|
| **DNS & Tunnels** | Cloudflare |
|
||||||
|
| **AI Orchestration** | Paperclip |
|
||||||
|
| **AI Gateway** | OpenRouter |
|
||||||
|
| **Automation** | n8n |
|
||||||
|
| **Ops Agent** | SugeBot (Hermes / Telegram) |
|
||||||
|
| **Metrics** | Prometheus + Node Exporter + Blackbox Exporter |
|
||||||
|
| **Visualization** | Grafana |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🔮 Roadmap
|
||||||
|
|
||||||
|
### V2 Dashboard
|
||||||
|
- [ ] Agent activity metrics (Paperclip runs, success rate, token spend)
|
||||||
|
- [ ] SugeBot command log panel
|
||||||
|
- [ ] Per-agent health monitoring
|
||||||
|
- [ ] n8n workflow execution tracking
|
||||||
|
- [ ] Telegram alert integration via SugeBot
|
||||||
|
|
||||||
|
### Platform Enhancements
|
||||||
|
- [ ] AI Incident Detection — anomaly-triggered agent escalation
|
||||||
|
- [ ] Automated Remediation — n8n workflows triggered by Prometheus alerts
|
||||||
|
- [ ] Multi-environment monitoring (homelab + VPS unified NOC)
|
||||||
|
- [ ] Discord alert channel
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 👤 Author
|
||||||
|
|
||||||
|
**Chase Dumphord**
|
||||||
|
Digital Systems Engineer | Platform Engineering | AI Infrastructure | Observability
|
||||||
|
|
||||||
|
[](https://www.linkedin.com/in/toochase-dumphord/)
|
||||||
|
[](https://github.com/ced4568)
|
||||||
|
[](https://noc.chasedumphord.com)
|
||||||
|
[](https://chasedumphord.com)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🔗 Related Repos
|
||||||
|
|
||||||
|
| Repo | Description |
|
||||||
|
|---|---|
|
||||||
|
| [ceds-homelab](https://github.com/ced4568/ceds-homelab) | 6-node Proxmox cluster + 12-node K3s + full homelab infrastructure |
|
||||||
|
| [ced-noc](https://github.com/ced4568/ceds-noc) | Custom-built public NOC dashboard |
|
||||||
|
| [ced-k3s-homelab](https://github.com/ced4568/ced-k3s-homelab) | 12-nodeRaspberry Pi K3s cluster detail |
|
||||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user