Deployment — LastVet Slack enrichment

Target host: observability VM (ryan@192.168.1.93 from agent-01 LAN).

Prerequisites

• BWS key LASTVET_R940_ALERTMANAGER_SLACK_WEBHOOK_URL populated in lastvet project.
• BWS access on observability VM (/etc/bitwarden/access-token or /etc/profile.d/bws.sh).
• Python 3.11+ on observability VM.

Install / upgrade

cd ~/code/last1/lastvet-gates
git pull
chmod +x scripts/observability/deploy-slack-enrichment.sh
chmod +x scripts/observability/restore-slack-enrichment-env-from-bws.sh
./scripts/observability/deploy-slack-enrichment.sh

This script:

1. Rsyncs services/lastvet-slack-enrichment/ to /opt/lastvet-slack-enrichment
2. Creates system user slack-enrichment
3. Installs Python venv + package
4. Restores /etc/lastvet-slack-enrichment/enrichment.env from BWS
5. Installs and starts lastvet-slack-enrichment.service

Alertmanager cutover

Redeploy Alertmanager config (webhook receiver replaces direct Slack):

./scripts/observability/deploy-alerting-phase4.sh

Alertmanager posts to http://127.0.0.1:8088/webhook/alertmanager with send_resolved: true.

Firewall

Allow Tailscale health checks on port 8089:

sudo ufw allow in on tailscale0 to any port 8089 proto tcp comment 'slack-enrichment-health'

Verify

./scripts/observability/verify-slack-enrichment.sh

Check #alerts-lastvet for SlackEnrichmentTestWarning enriched message and auto-resolve.

UptimeRobot (optional)

Add monitor: http://100.123.175.50:8089/health (refresh Tailscale IP if VM changes).

Rollback

1. Restore previous infra/observability/alertmanager/alertmanager.yml with direct slack_configs.
2. ./scripts/observability/deploy-alerting-phase4.sh
3. sudo systemctl stop lastvet-slack-enrichment