diff --git a/alerting/alerting-strategy.md b/alerting/alerting-strategy.md index e69de29..7a206a8 100644 --- a/alerting/alerting-strategy.md +++ b/alerting/alerting-strategy.md @@ -0,0 +1,69 @@ +# Alerting Strategy + +## Purpose + +The goal of alerting is not to create noise. + +The goal is to identify problems that require action. + +--- + +## Alert Severity Levels + +| Severity | Meaning | Example | +|---|---|---| +| Critical | Immediate action required | Node down | +| Warning | Investigate soon | CPU above 85% | +| Info | Awareness only | Service restarted | + +--- + +## Initial Alert Rules + +### Critical + +- Proxmox node down +- K3s control plane node down +- Core service unavailable +- Storage nearly full +- Cluster quorum issue + +### Warning + +- High CPU usage +- High memory usage +- High disk usage +- Repeated pod restarts +- Slow service response + +--- + +## Alert Response Process + +1. Confirm alert in Grafana +2. Identify affected system +3. Check recent changes +4. Review logs +5. Verify service status +6. Restart or repair if needed +7. Document what happened + +--- + +## Example Incident Notes + +```text +Incident: +Service unavailable alert triggered for Nginx Proxy Manager. + +Impact: +External service routing may be affected. + +Investigation: +Checked host status, container status, and reverse proxy logs. + +Resolution: +Restarted affected container and confirmed service recovery. + +Follow-up: +Add container-level monitoring and restart alert. \ No newline at end of file