R940 Observability Stack¶

Status: Phase 3 complete (August 2026). Metrics + log aggregation live from all R940 VMs.

Scope: LastVet, Meridian, Sovereign, Angel (Phases 2-5 add exporters, logs, alerts, external uptime).

Architecture¶

Component	Role	Port	Data path
Prometheus	Metrics TSDB (90d retention)	9090	`/var/lib/observability-data/prometheus`
Grafana	Dashboards + datasource UI	3000	`/var/lib/grafana`
Loki	Log aggregation (filesystem + TSDB index)	3100	`/var/lib/observability-data/loki`
Alertmanager	Alert routing (Phase 4)	9093	`/var/lib/alertmanager`

Phase 1: Prometheus scrapes itself only. Grafana provisioned with Prometheus + Loki datasources (localhost). Alertmanager blackhole receiver only.

Phase 2: Prometheus scrapes node/gpu/postgres/minio/caddy/proxmox targets over LAN (192.168.1.x). Operator UI remains Tailscale-only.

Phase 3: Promtail on 11 VMs ships systemd journal + app logs to Loki over LAN (192.168.1.93:3100). Tiered retention via retention_tier label (system 90d, audit 6yr).

Log topology (Phase 3)¶

flowchart LR
  subgraph vms["11 R940 VMs (100-110)"]
    PT[Promtail :9080]
  end
  subgraph observability["observability VM .93"]
    Loki[Loki :3100]
    Graf[Grafana :3000]
  end
  PT -->|"push LAN"| Loki
  Graf --> Loki

Policy: Promtail pushes outbound to Loki; no new inbound ports on source VMs. Loki ingest: LAN + Tailscale (unchanged from Phase 1). Observability VM does not run Promtail (no self-scrape v1).

Retention tiers (Phase 3)¶

Label	Period	Streams
`retention_tier=system`	90 days	systemd journal on all 11 VMs (audit units excluded)
`retention_tier=audit`	6 years	LastVet API, Postgres, Caddy journal units (Bucket A); MinIO deferred

Set at Promtail ingestion. Loki retention_stream in /etc/loki/loki-config.yml applies per-label periods. Compactor delete_request_store: filesystem required when retention is enabled.

Runbook: add audit-tier logs for a new service

Confirm log source (journal unit name and/or file path).
Extend install-promtail.sh role config or add a PROMTAIL_ROLE scrape block with retention_tier: audit.
Redeploy Promtail on that VM only (deploy-promtail-phase3.sh or manual install-promtail.sh).
Verify in Grafana Explore: {host="<vm>", retention_tier="audit"}.
If HIPAA-relevant, confirm stream is covered by offsite backup follow-up (gap register #6 / #10).

Legacy AWS logs: CloudTrail + CloudWatch exports in /tank-bulk/aws-retirement-preservation/ remain a static archive. Not ingested to Loki (compressed historical data; low query value vs storage cost).

Backup follow-up: Loki data on observability VM is a single point of failure. NAS vzdump covers the VM; audit-tier offsite archive (B2 encrypted) is a separate Muster item (observability-audit-offsite-backup). Monitor loki_ingester_chunks_bytes growth; if projected annual growth exceeds ~200 GB, revisit disk sizing. Phase 4: add alerting on Loki storage growth rate.

Scrape topology (Phase 2)¶

flowchart LR
  subgraph observability["observability VM .93"]
    Prom[Prometheus :9090]
    PVEExp[pve-exporter :9221]
    Graf[Grafana :3000]
  end
  subgraph lan["192.168.1.0/24 LAN"]
    Nodes["12 VMs + pve-r940\nnode_exporter :9100"]
    GPU["inference-01\nnvidia_gpu_exporter :9835"]
    PG["lastvet-db\npostgres_exporter :9187"]
    MinIO["lastvet-imaging\nMinIO :9000/metrics"]
    Caddy["lastvet-edge\nCaddy admin :2019/metrics"]
    PVE["pve-r940 API :8006"]
  end
  Prom -->|"scrape LAN"| Nodes
  Prom --> GPU
  Prom --> PG
  Prom --> MinIO
  Prom --> Caddy
  Prom --> PVEExp
  PVEExp --> PVE
  Graf --> Prom

Policy: Prometheus on .93 scrapes exporter ports on the flat LAN. Each exporter ufw allows inbound only from 192.168.1.93/32. Grafana/Prometheus UI for humans: Tailscale MagicDNS only (unchanged).

VM¶

Property	Value
Proxmox VMID	111
Hostname	`observability`
LAN IP	`192.168.1.93/24`
MagicDNS	`observability.tailc3bbdc.ts.net`
vCPU / RAM	8 / 16 GB
OS disk	50 GB (`tank-fast`, scsi0)
Data disk	450 GB (`tank-fast`, scsi1 → `/var/lib/observability-data`)
Startup order	14, up=30 (after LastVet stack)
OS	Ubuntu 24.04 LTS (noble cloud image)

Access (Tailscale only for UI)¶

Service	URL
Grafana	http://observability.tailc3bbdc.ts.net:3000
Prometheus	http://observability.tailc3bbdc.ts.net:9090
Alertmanager	http://observability.tailc3bbdc.ts.net:9093

Loki push (Phase 3): http://192.168.1.93:3100 from LAN or MagicDNS from Tailscale. UFW allows Tailscale + LAN on 3100 only.

Credentials¶

Secret	BWS key
Grafana admin	`LASTVET_R940_OBSERVABILITY_GRAFANA_ADMIN_PASSWORD`
Postgres exporter DB user	`LASTVET_R940_POSTGRES_EXPORTER_PASSWORD`
MinIO Prometheus JWT (staging scrape)	`LASTVET_R940_STAGING_MINIO_PROMETHEUS_TOKEN`
MinIO Prometheus JWT (prod/bucket scrape)	`LASTVET_R940_PROD_MINIO_PROMETHEUS_TOKEN`
Proxmox API token (pve-exporter)	`LASTVET_R940_PROXMOX_EXPORTER_TOKEN`

Rotation follow-ups (90-day calendar): MinIO Prometheus JWTs, Proxmox exporter token. Regenerate in place, update BWS, reload Prometheus (systemctl reload prometheus after bearer file update).

Never store observability credentials in git or shell history.

Verify on first login: Confirm the BWS-stored admin password actually works. If default admin/admin still works, reset via Grafana's first-login flow and update BWS.

Post-install gotcha (future VM builds): Automated password changes can run before Grafana finishes initial admin user setup. The service may still accept admin/admin until first-login completes. After any install script sets the password, log in once and verify BWS credentials work; if not, use the first-login flow and update BWS before marking Phase 1 done.

Firewall (ufw)¶

Default deny inbound
Grafana/Prometheus/Alertmanager: Tailscale CIDR 100.64.0.0/10 only
Loki: Tailscale + 192.168.1.0/24
SSH: LAN 192.168.1.0/24 only
fail2ban: ignores LAN + Tailscale CIDRs

Service versions (Phase 1 install)¶

Prometheus 2.55.1
Grafana 11.4.0
Loki 3.3.2
Alertmanager 0.27.0

Config paths¶

/etc/prometheus/prometheus.yml
/etc/grafana/grafana.ini + /etc/grafana/provisioning/datasources/observability.yaml
/etc/loki/loki-config.yml (Phase 3: tiered retention_stream)
/etc/promtail/promtail.yml on VMs 100-110 (not observability)
/etc/alertmanager/alertmanager.yml
/etc/prometheus/pve.yml + /etc/prometheus/minio_*_bearer (Phase 2 secrets on disk, mode 600)
/etc/grafana/provisioning/dashboards/ (Phase 2 community dashboards)

Phase 1 complete¶

[x] VM provisioned on R940
[x] Four stack services installed and active
[x] Data disk mounted
[x] Basic configs + ufw
[x] Grafana datasources provisioned
[x] BWS credential for Grafana admin

Exporters (Phase 2)¶

Host	VMID	LAN IP	Exporter	Port	Notes
agent-01	100	.30	node_exporter 1.12.1	9100
inference-01	101	.40	node_exporter + nvidia_gpu_exporter 1.14.0	9100 / 9835	GPU metrics need NVIDIA driver loaded (see surprises)
meridian-eval	102	.41	node_exporter	9100
meridian-rag	103	.42	node_exporter	9100
sovereign-app	104	.70	node_exporter	9100
sovereign-mem	105	.71	node_exporter	9100
angel-local	106	.80	node_exporter	9100
lastvet-api	107	.90	node_exporter	9100	App-level HTTP metrics deferred
lastvet-db	108	.91	postgres_exporter 0.20.1	9187	`--auto-discover-databases` (`lastvet_staging`, `lastvet_prod`)
lastvet-imaging	109	.92	MinIO built-in	9000	JWT bearer auth; staging=cluster metrics, prod=bucket metrics
lastvet-edge	110	.89	Caddy built-in	2019	`admin` + global `metrics` in Caddyfile
observability	111	.93	node_exporter + pve-exporter 3.5.5	9100 / 9221	pve-exporter queries Proxmox API on `.111`
pve-r940 (hypervisor)	—	.111	node_exporter (ZFS collectors)	9100	Complements pve-exporter VM-level metrics

Install scripts: ../../scripts/observability/.

Grafana dashboards (Phase 2)¶

Provisioned from ../../infra/observability/grafana/dashboards/ into folder R940 Platform:

ID	Dashboard	Job / use
1860	Node Exporter Full	`node` (all hosts)
9628	PostgreSQL Database	`postgres`
13502	MinIO Dashboard (official JSON)	`minio-staging`, `minio-prod`
25216	Caddy Standalone Reverse Proxy	`caddy`
10347	Proxmox via Prometheus	`proxmox`
14574	NVIDIA GPU Metrics	`gpu`

Prometheus UI targets: http://observability.tailc3bbdc.ts.net:9090/targets (all jobs up as of Phase 2 deploy).

Phase 2 complete¶

[x] node_exporter on 12 VMs + pve-r940 hypervisor
[x] nvidia_gpu_exporter on inference-01
[x] postgres_exporter with auto-discover-databases on lastvet-db
[x] MinIO JWT scrape (staging cluster + prod bucket jobs)
[x] Caddy metrics on lastvet-edge (admin LAN bind + ufw)
[x] pve-exporter on observability VM (Proxmox API token in BWS)
[x] Prometheus static scrape config (90d retention unchanged)
[x] Community Grafana dashboards file-provisioned

Promtail (Phase 3)¶

Host	VMID	LAN IP	Role	Audit journal units (`retention_tier=audit`)
agent-01	100	.30	default	—
inference-01	101	.40	default	—
meridian-eval	102	.41	default	—
meridian-rag	103	.42	default	—
sovereign-app	104	.70	default	—
sovereign-mem	105	.71	default	—
angel-local	106	.80	default	—
lastvet-api	107	.90	lastvet-api	`lastvet-api-staging.service`, `lastvet-api-prod.service`
lastvet-db	108	.91	lastvet-db	`postgresql@17-main.service`
lastvet-imaging	109	.92	default	MinIO deferred until disk logging enabled
lastvet-edge	110	.89	lastvet-edge	`caddy.service`

Promtail 3.3.2 (matches Loki). Journal is the sole source (no file tailing). Audit units are scraped in dedicated jobs; the system-tier journal job drops those units to avoid duplicate streams. Install scripts: ../../scripts/observability/ (install-promtail.sh, deploy-promtail-phase3.sh, update-loki-phase3.sh).

Bucket A (implemented): infrastructure audit via journal unit tagging above.

Bucket B (deferred): application-level audit event parsing from LastVet API structured logs (regex Promtail stages vs API code changes). Muster: observability-audit-bucket-b.

MinIO audit: deferred until MinIO writes access logs to disk. Revisit when object-access volume warrants it.

pgaudit / log_statement: not enabled; current Postgres logging level unchanged unless a security audit requires it.

Grafana dashboards (Phase 3 — Loki)¶

ID	Dashboard	Use
13639	Loki Dashboard quick search	Log query / Explore patterns
13407	Loki2.0 Global Metrics	Loki ingester/compactor health
15141	Loki + Promtail overview	Promtail shipper status

Phase 3 complete¶

[x] Loki tiered retention (system 90d, audit 6yr) with compactor enabled
[x] Promtail on 11 VMs (100-110), not observability
[x] Audit streams (Bucket A): LastVet API, Postgres, Caddy journal units
[x] All 11 hosts visible in Loki {host=...} label
[x] Loki community dashboards provisioned
[x] Legacy AWS logs left as static archive (not ingested)
[ ] Bucket B application audit parsing (Muster follow-up)
[ ] MinIO audit logging (deferred)
[ ] Offsite audit log archive (Muster follow-up)

Phase 4 complete (August 2026)¶

Alert routing on observability VM (192.168.1.93). Secrets from BWS → /etc/alertmanager/alertmanager.env (mode 600), referenced by systemd EnvironmentFile. At service start, envsubst renders /etc/alertmanager/alertmanager.yml.template → alertmanager.yml (Alertmanager 0.27 has no --config.expand-env).

Routing¶

Setting	Value
`group_by`	`alertname`, `cluster`, `severity`
Missing `severity` label	`enriched-slack-default` receiver
`critical`	Email (`alerts@last1.enterprises`) + enriched Slack via webhook
`warning`	Enriched Slack `#alerts-lastvet` only
`info`	Dropped (`null` receiver)
`send_resolved`	`true` on critical email + enriched Slack

Slack enrichment service (August 2026): Alertmanager posts to http://127.0.0.1:8088/webhook/alertmanager on the observability VM. Service code: services/lastvet-slack-enrichment/. Health: Tailscale :8089/health. Deploy: ./scripts/observability/deploy-slack-enrichment.sh.

Inhibit: LastVetAPIHostDown suppresses LastVetAPIProdSilent / LastVetAPIStagingSilent on same cluster.

Rule sources¶

Engine	Path (repo)	Live path
Prometheus	`infra/observability/prometheus/rules/{critical,warnings}.yml`	`/etc/prometheus/rules/`
Loki ruler	`infra/observability/loki/rules/lastvet.yml`	`/etc/loki/rules/fake/lastvet.yml`

Prometheus also scrapes Loki (job: loki, :3100) for self-metrics. Loki ruler working directory /var/lib/observability-data/loki/ruler must be owned by loki:loki.

Critical alerts (Prometheus)¶

Alert	Signal
`LastVetAPIHostDown`	`node` up on `192.168.1.90:9100`
`PostgresDown`	`postgres` job up
`MinIODown`	`minio-*` job up
`CaddyDown`	`caddy` job up
`R940HostCPUCritical`	pve-r940 CPU >95% (15m)
`DiskFillCritical`	Any monitored FS >90% (10m)
`ZFSPoolDegraded`	`node_zfs_zpool_state` degraded/faulted on pve-r940

Critical alerts (Loki)¶

Alert	Signal
`LastVetAPIProdSilent`	No `lastvet-api-prod.service` lines 5m
`LastVetAPIStagingSilent`	No `lastvet-api-staging.service` lines 5m
`KEKUnwrapFailure`	Log line `imaging_envelope_decrypt_failed`
`PostgresFatal`	Postgres journal FATAL/PANIC

Warning alerts¶

Alert	Source
`HighCPU` / `HighMemory`	Prometheus node
`PostgresConnectionPoolHigh`	`pg_stat_activity_count`
`MinIODiskWarning`	MinIO staging usable capacity
`CaddyErrorRateHigh`	Caddy 5xx rate
`ObservabilityDataDiskHigh`	`/var/lib/observability-data` >70% (30m)
`LastVetAPIErrors`	API JSON `"level":"error"` rate
`SSHBruteForce`	>20 failed SSH passwords / 10m

Runbooks: docs/observability/runbooks/ (GitHub URLs in rule annotations for phone access).

Deploy / verify¶

./scripts/observability/deploy-alerting-phase4.sh
./scripts/observability/verify-alerting-phase4.sh

Deferred (Muster follow-ups)¶

Item	Reason
`observability-vm-stopped-alert`	Need correct pve-exporter series for VM run state
`observability-cert-expiring-alert`	Requires blackbox exporter (Phase 4.5 or 5)
Loki anomaly / backup log alerts	Calibration baseline first
`loki_ingester_chunks_bytes` growth projection	Revisit after 14d baseline

Interim coverage: node up checks + Caddy on-host TLS renewal.

Known false positive scenarios¶

Populated during the 14-day calibration window (empty at Phase 4 launch):

Alert	Scenario	Mitigation

Phase 5 complete (August 2026)¶

External synthetic uptime via UptimeRobot Pro. Independent of R940; catches total site/WAN outages internal Alertmanager cannot see.

Monitors (live)¶

Name	URL	Interval	Timeout	Check	Public status page
LastVet API	`https://api.last.vet/health`	1 min	30s	Keyword `ok` absent → down (`keyword_type=2`, case-insensitive)	Yes
LastVet API staging (internal)	`https://api-staging.last.vet/health`	5 min	30s	Keyword `ok` absent → down (`keyword_type=2`, case-insensitive)	No (ops only)
LastVet Homepage	`https://last.vet/`	5 min	30s	HTTP 200	Yes
My LastVet	`https://my.last.vet/`	5 min	30s	HTTP 200	Yes
LastVet Provider Portal	`https://provider.last.vet/`	5 min	30s	HTTP 200	Yes
LastVet Slack enrichment health	Heartbeat push from observability VM when `http://100.123.175.50:8089/health` OK	5 min	n/a	Heartbeat (type 5); not public HTTP pull (Tailscale IP unreachable from UptimeRobot probes)	No (ops only)

Enrichment safety net: Alertmanager Slack routes through lastvet-slack-enrichment. If that service dies, internal alerts may not reach #alerts-lastvet. The heartbeat monitor above is independent: a systemd timer on the observability VM pings UptimeRobot only when local /health succeeds. Deploy: ./scripts/observability/deploy-uptimerobot-slack-enrichment-health.sh (requires Phase 5 contacts).

Down threshold: 2 consecutive failures (UptimeRobot threshold minutes: prod 2, staging/web 10).

Alert routing¶

Channel	Destination
Email	`alerts@last1.enterprises`
Slack	`#alerts-lastvet` (shared webhook with Alertmanager)

Both channels notify on down and up. UptimeRobot uses its own message format (distinct from Alertmanager).

Secrets and scripts¶

BWS key	Purpose
`LASTVET_UPTIMEROBOT_API_KEY`	UptimeRobot API
`LASTVET_R940_ALERTMANAGER_SLACK_WEBHOOK_URL`	Slack webhook

./scripts/observability/deploy-uptimerobot-phase5.sh
./scripts/observability/verify-uptimerobot-phase5.sh
./scripts/observability/deploy-uptimerobot-phase5-expand.sh
./scripts/observability/verify-uptimerobot-phase5-expand.sh
./scripts/observability/deploy-uptimerobot-slack-enrichment-health.sh

API: Monitors via v2. Email contacts via v3 POST /v3/alert-contacts (v2 blocks email create/edit). Slack via v2 type 11 (surfaced as v3 integration). Public status page via v3 POST/PATCH /v3/psps.

Contact IDs cached locally in .uptimerobot-phase5/contact-ids.env (gitignored). Expansion state (PSP id, urlKey, web monitor ids) in .uptimerobot-phase5/expand-state.env (gitignored).

Public status page¶

Field	Value
URL (custom)	`https://status.last.vet` (requires Cloudflare CNAME; see below)
URL (standard)	`https://stats.uptimerobot.com/jKTZt8e5at` (live before DNS)
Title	LastVet Status
Monitors shown	Customer-facing only: LastVet API, LastVet Homepage, My LastVet, LastVet Provider Portal (staging API excluded)
Uptime %	Per-monitor + overall (90-day bars via `showBars` / `showOverallUptime`)
Response times	Enabled (`enableDetailsPage`)
Incidents / downtime	Enabled (`showOutageDetails`, `showOutageUpdates`)
Password	None (public transparency)
Visitor subscription	Not on current Pro plan (requires whitelabel tier); ops alerts unchanged

Cloudflare DNS (Ryan): CNAME status → stats.uptimerobot.com, DNS only (grey cloud). UptimeRobot serves TLS for the custom domain.

Add/remove monitors on the status page: edit monitor list in deploy-uptimerobot-phase5-expand.sh (ensure_status_page monitor id array) and re-run the script, or update via UptimeRobot dashboard → Status Pages → LastVet Status. New public URL? Add a monitor first (expand script or dashboard), then include its id on the PSP.

Scope: UptimeRobot-hosted only. Do not ingest status page traffic into R940 Loki/Prometheus.

Logo: API accepts PNG/JPG only (not SVG). Upload https://last.vet/brand/lastvet-mark.svg converted to PNG via dashboard if desired.

Runbook: docs/observability/runbooks/uptimerobot-alert.md

Maintenance windows¶

UptimeRobot supports programmatic maintenance windows via v3 (/v3/maintenance-windows) and v2 (newMWindow, editMWindow, getMWindows). Assign windows to specific monitors by ID; alerts are suppressed during the active window (checks still run).

Capability	v3	v2
Create one-off window	`interval: once` + `date`/`time`/`duration`	`type=1` + unix `start_time`
Recurring daily	`interval: daily`	`type=2` + HH:MM `start_time`
Recurring weekly	`interval: weekly` + `days[]`	`type=3` + `value` (e.g. `2-4-5`)
Recurring monthly	`interval: monthly` + `days[]`	`type=4` + `value` (e.g. `10-17-26`)
Per-monitor scope	`monitorIds[]` or `autoAddMonitors`	Assign via `editMonitor` `mwindows`
Alert behavior	Suppressed during window	Suppressed during window

Not automated yet. Procedure: docs/observability/runbooks/maintenance-window-procedure.md. Future: tie to vzdump/PBS cron (Muster uptimerobot-maintenance-window-automation).

DNS verification (status.last.vet):

dig +short status.last.vet
# After Ryan adds CNAME: should resolve via stats.uptimerobot.com
curl -sI --max-time 10 https://status.last.vet | head -3

Runbook: add a new metric source¶

Deploy exporter on target VM (official binary + systemd). Bind :port on all interfaces.
ufw allow from 192.168.1.93 to any port <port> proto tcp on the target.
Add a commented static_configs entry in /etc/prometheus/prometheus.yml on observability.
sudo promtool check config /etc/prometheus/prometheus.yml && sudo systemctl reload prometheus (or restart if lifecycle flag absent).
Confirm on http://observability.tailc3bbdc.ts.net:9090/targets.
Import or extend a Grafana dashboard; commit JSON under infra/observability/grafana/dashboards/ if platform-wide.

Provisioning notes¶

Ubuntu cloud image: noble-server-cloudimg-amd64.img via qm importdisk (see R940 Platform Architecture).
Tailscale join: use BWS LASTVET_R940_TAILSCALE_AUTH_KEY (reusable, tag:r940-vm, 90-day rotation). Ryan generates the key; automation reads from BWS. Deferred: filed as Muster follow-up tailscale-add-reusable-auth-key-for-automated-r9; Phase 2 proceeds with observability VM already joined via one-time browser auth.
Grafana admin UI is Tailscale-only (no LAN :3000). See Admin UI access.
After stack install, verify Grafana admin password per Credentials before closing Phase 1.