Skip to content

Architecture — LastVet Slack enrichment

Placement

Runs on the observability VM (192.168.1.93, MagicDNS observability.tailc3bbdc.ts.net), colocated with Alertmanager, Prometheus, Loki, and Grafana.

Data flow

Prometheus / Loki ruler
        ↓
  Alertmanager (:9093)
        ↓ HTTP POST (localhost)
  lastvet-slack-enrichment (:8088)
        ↓ read-only queries
  Prometheus / Loki / Grafana (127.0.0.1)
        ↓ incoming webhook
  Slack #alerts-lastvet

Critical alerts still send email directly from Alertmanager (unchanged).

Listeners

Bind Port Routes Consumers
127.0.0.1 8088 POST /webhook/alertmanager Alertmanager (same host)
Tailscale IP 8089 GET /health UptimeRobot, operators

Prometheus and Loki queries use loopback (127.0.0.1). Slack button links use Tailnet MagicDNS (observability.tailc3bbdc.ts.net) so they open in the operator browser, not localhost on the operator machine.

Webhook requests from non-localhost addresses receive 403.

Enrichment pipeline

  1. Parse Alertmanager grouped payload (status, labels, annotations, timestamps).
  2. Load per-alert rules from config/enrichment-rules.yaml.
  3. Build Grafana Explore link (alert start −30m → start +10m).
  4. Resolve runbook URL from annotation or lookup table.
  5. Query Loki for last 5 log lines (optional).
  6. Query Prometheus for instant metric snapshot (optional).
  7. Render Slack Block Kit with action buttons: Open Grafana, Runbook, Silence 1h.

Partial enrichment failures append a context note; basic alert still posts if the whole pipeline fails.

Security

  • No auto-remediation; read-only observability queries.
  • Slack webhook URL stored in /etc/lastvet-slack-enrichment/enrichment.env (BWS restore, mode 640).
  • Webhook not exposed off localhost.