Skip to content

LAST1.ID W6B-1 Report — Pre-Cutover Alerting Verification

Date: 2026-08-25
Scope: Verify Postgres exporter scrape, GoTrue/auth 5xx alerting, and critical alert delivery path before DNS cutover.
Status: W6B-1 complete — STOP for Ryan alert-delivery confirmation + DNS creation (W6B-2).

Executive summary

Check Result
Postgres exporter scrape (192.168.1.95:9187) UP — pg_up=1
New critical alert PostgresLast1idDatabaseDown (pg_up==0) Deployed + live-fired during 90s Postgres stop test
GoTrue 5xx error-rate alert Last1idAuth5xxRateHigh Deployed — Caddy metrics scrape on 192.168.1.96:2019 UP
Alert delivery path Same as N8nHostDown — critical → email alerts@last1.enterprises + enriched Slack webhook
Live fire test PostgresLast1idDatabaseDown reached firing in Prometheus and active in Alertmanager

Ryan action: Confirm you received the critical alert (email and/or Slack) from the live Postgres stop test. Then create Cloudflare DNS for auth.last1.id (W6B-2) and report when ready for W6B-3 coordinated deployment.

1. Postgres exporter scraping

Prometheus on observability (192.168.1.93) scrapes:

Target Job Health Metric
192.168.1.95:9187 postgres up pg_up=1
192.168.1.95:9100 node up —
192.168.1.96:9100 node up —

Existing rules (unchanged, verified loaded):

  • PostgresLast1idExporterDown — scrape target down, for: 2m, critical
  • PostgresLast1idDown — sovereign-db VM node_exporter down, for: 3m, critical
  • GoTrueDown / KongDown — auth-last1id VM node_exporter down, for: 3m, critical
  • Last1idHighDbConnections — warning

2. New alerts added (W6B-1)

Critical: PostgresLast1idDatabaseDown

Detects actual database outage while exporter remains reachable (W6A gap: /health stays 200, signup returns 500).

expr: pg_up{instance="192.168.1.95:9187"} == 0
for: 1m
severity: critical
service: last1id

Live test (2026-08-25T18:33Z): Stopped postgresql@17-main on sovereign-db for ~90s.

Phase Result
During outage Prometheus: PostgresLast1idDatabaseDown firing
Alertmanager Alert active (critical route)
After restore pg_up recovered to 1

Warning: Last1idAuth5xxRateHigh

Detects elevated 5xx on the auth ingress path (Caddy → Kong → GoTrue), same pattern as CaddyErrorRateHigh on lastvet-edge.

expr: 5xx rate > 5% on caddy_http_request_duration_seconds_count{job="caddy-last1id"}
for: 5m
severity: warning
service: last1id

Infrastructure fixes applied:

  1. Caddy admin API bound to 192.168.1.96:2019 (was localhost-only)
  2. UFW on auth-last1id: allow 192.168.1.93 → :2019/tcp (mirrors node_exporter :9100 pattern)
  3. Prometheus scrape job caddy-last1id added → up=1

3. Alert delivery path

Same stack as N8nHostDown (verified firing pattern in N8N_MIGRATION_RUNBOOK.md):

Component Config
Prometheus Rules in /etc/prometheus/rules/critical.yml + warnings.yml
Alertmanager severity=critical → receiver critical-email-enriched-slack
Email alerts@last1.enterprises via Gmail SMTP
Slack Enrichment webhook http://127.0.0.1:8088/webhook/alertmanager → #alerts-lastvet

Critical alerts include service: last1id label for filtering.

4. Files changed

File Change
infra/observability/prometheus/prometheus-phase4.yml Added caddy-last1id scrape job
infra/observability/prometheus/rules/critical.yml Added PostgresLast1idDatabaseDown
infra/observability/prometheus/rules/warnings.yml Added Last1idAuth5xxRateHigh
scripts/last1-id/week6b1-verify-alerting.sh Deploy + verify script (includes live fire test)
scripts/last1-id/setup-auth-stack.sh Caddy admin on LAN for metrics
scripts/last1-id/deploy-last1id-observability.sh UFW rule for Caddy metrics

Live VM changes (auth-last1id): Caddyfile admin bind + UFW :2019 from observability.

5. Gate status

Sub-task Status
W6B-1 Alerting verification Complete — pending Ryan delivery confirmation
W6B-2 Ryan DNS creation Not started — Ryan action
W6B-3 Coordinated deployment Blocked on W6B-2
W6B-4 Cutover verification Not started
W6C Soak Not started

6. Ryan checklist

  1. Confirm alert delivery: Did PostgresLast1idDatabaseDown reach email and/or Slack during the live test (~18:33 UTC)?
  2. Approve W6B-1 (or note gaps).
  3. W6B-2: Create Cloudflare DNS: auth.last1.id → Cloudflare Tunnel (lastvet-edge / lastvet-r940, route prepared Week 1).
  4. Report "DNS created" — Cursor executes W6B-3 coordinated deployment (~2–3h window).

W6B-3 will NOT start until Ryan reports DNS created.

References