Runbook: R940 LastVet production deploy

Replaces: AWS LASTVET_ALLOW_PROD_DEPLOY=1 gate in deploy-api-image.sh (ECS retired August 2026).

Canonical topology: docs/deployment/R940_ENVIRONMENT_TOPOLOGY.md.

---

When to use

Any change that affects production on R940:

• API code deploy (lastvet-api-prod.service restart after build)
• Database migrations against lastvet_prod
• BWS env restore to /etc/lastvet-api/lastvet-api-prod.env
• Provider portal production Cloudflare deploy (if bundled with API behavior change)

Not required for staging-only work, observability changes, or docs.

---

Pre-flight (mandatory)

1. Staging soaked: Staging deploy completed and verified. Minimum 48 hours soak for non-trivial changes (consent, RLS, imaging, auth). Shorter window only for hotfix with Ryan approval.
2. Staging health:

curl -sS https://api-staging.last.vet/health
ssh lastvet-api 'systemctl is-active lastvet-api-staging'

1. Migration state verified (prod vs staging): Before any prod deploy that ships new migration files, confirm ledger parity and pending set (see Migration pre-flight checklist below). Do not restart prod or run prod migrations until this passes.
2. Fixtures current: If the change touches consent/RLS/Part 2, confirm staging fixture veterans A/B/C present (see topology doc).
3. No prod seeding: Confirm deploy plan does not run seed-staging-fixtures.js or any seed against prod DATABASE_APP_URL.

---

Migration pre-flight checklist (prod vs staging)

Run from agent-01 or any host with SSH to lastvet-db and lastvet-api. Read-only until the apply step in Deploy steps §3.

1. Compare ledger (latest applied)

ssh lastvet-db 'sudo -u postgres psql -d lastvet_prod -tAc "SELECT max(filename) FROM schema_migrations;"'
ssh lastvet-db 'sudo -u postgres psql -d lastvet_staging -tAc "SELECT max(filename) FROM schema_migrations;"'

Expect: Staging max filename ≥ prod max filename before prod promotion. If prod lags staging, migrations are in scope for this deploy.

2. Full diff (optional but recommended for hotfixes)

ssh lastvet-db 'sudo -u postgres bash -c "
  psql -d lastvet_prod -tAc \"COPY (SELECT filename FROM schema_migrations ORDER BY filename) TO STDOUT\" > /tmp/prod.txt
  psql -d lastvet_staging -tAc \"COPY (SELECT filename FROM schema_migrations ORDER BY filename) TO STDOUT\" > /tmp/staging.txt
  echo \"=== In staging not prod ===\"
  comm -23 /tmp/staging.txt /tmp/prod.txt
"'

Any file listed here must be applied on prod before or during this deploy.

3. List pending on prod (migrate runner)

lastvet_master is allowed from LAN only in pg_hba.conf. The restore script rewrites MIGRATION_DATABASE_URL to 192.168.1.91; if running manually, apply the same rewrite.

ssh lastvet-api 'sudo bash -c "
  export LASTVET_ALLOW_R940_PROD_DEPLOY=1
  set -a && source /etc/lastvet-api/lastvet-api-prod.env && set +a
  export MIGRATION_DATABASE_URL=\"\${MIGRATION_DATABASE_URL//lastvet-db.tailc3bbdc.ts.net/192.168.1.91}\"
  cd /opt/last.vet-api
  node scripts/migrate.js --list-pending
"'

Expect: Empty, or exactly the migration set you intend to apply. The runner never auto-applies *_down.sql or files under migrations/rollback/ (manual rollback only).

4. Pin expected set before apply

When applying, use --expect so a surprise pending file aborts the run:

node scripts/migrate.js --expect 059_imaging_upload_types.sql,060_imaging_download_audit_action.sql,061_imaging_envelope.sql

5. Snapshot before prod schema change

On lastvet-db, before any prod migration:

SNAPSHOT="/var/backups/postgres/lastvet_prod_pre_migrate_$(date -u +%Y%m%dT%H%M%SZ).dump"
sudo mkdir -p /var/backups/postgres
sudo -u postgres pg_dump -Fc -d lastvet_prod -f "$SNAPSHOT"
ls -lh "$SNAPSHOT"

Confirm size is non-zero. Do not proceed without a verified snapshot.

---

Prod confirmation gate

Production deploy is blocked until the operator sets:

export LASTVET_ALLOW_R940_PROD_DEPLOY=1

Meaning: intentional, reviewed production change. Unset or any other value = do not restart prod or run prod migrations.

Optional typed confirmation (recommended for migrations):

read -r -p "Type prod to confirm R940 production deploy: " confirm
[[ "$confirm" == "prod" ]] || { echo "Aborted."; exit 1; }
export LASTVET_ALLOW_R940_PROD_DEPLOY=1

Document in your session notes: date, git SHA, migration count, staging soak start time.

---

Deploy steps

Run on lastvet-api VM (or via ssh lastvet-api from agent-01).

1. Build

cd /opt/lastvet/last.vet-api
git fetch && git checkout <tag-or-sha>
npm ci
npm run build

2. Restore prod env from BWS

sudo /opt/lastvet-gates/scripts/restore-lastvet-api-env-from-bws.sh --env prod

Script validates required keys (DATABASE_APP_URL, VETERAN_JWT_SECRET, OAUTH_JWT_SECRET, SECURE_LINK_HMAC_SECRET, etc.). It does not print secret values.

3. Migrations (if any)

Pre-requisite: Migration pre-flight checklist completed; snapshot taken if applying.

set -a && source /etc/lastvet-api/lastvet-api-prod.env && set +a

# Gate check
[[ "${LASTVET_ALLOW_R940_PROD_DEPLOY:-}" == "1" ]] || {
  echo "Prod migration blocked: set LASTVET_ALLOW_R940_PROD_DEPLOY=1" >&2
  exit 1
}

# lastvet_master: pg_hba allows LAN source only (restore script sets 192.168.1.91)
export MIGRATION_DATABASE_URL="${MIGRATION_DATABASE_URL//lastvet-db.tailc3bbdc.ts.net/192.168.1.91}"

node scripts/migrate.js --list-pending
node scripts/migrate.js --expect <comma-separated-filenames>   # required when applying

Never use staging DATABASE_APP_URL for prod migrations. Verify DATABASE_APP_URL host/database before running (grep key name only, do not log URL).

4. Restart prod service

[[ "${LASTVET_ALLOW_R940_PROD_DEPLOY:-}" == "1" ]] || {
  echo "Prod restart blocked: set LASTVET_ALLOW_R940_PROD_DEPLOY=1" >&2
  exit 1
}

sudo systemctl restart lastvet-api-prod
sudo systemctl status lastvet-api-prod --no-pager
journalctl -u lastvet-api-prod -n 50 --no-pager

5. Post-deploy verification

curl -sS https://api.last.vet/health
# Authenticated spot-check if change touches veteran or provider paths
ssh lastvet-api 'journalctl -u lastvet-api-prod -n 20 --no-pager | grep -i error || true'

Watch UptimeRobot / Grafana for LastVetAPIProdSilent and error-rate alerts (see docs/observability/runbooks/).

6. Unset gate

unset LASTVET_ALLOW_R940_PROD_DEPLOY

---

Rollback

1. git checkout <previous-good-sha> in /opt/lastvet/last.vet-api, rebuild, restart prod (still requires LASTVET_ALLOW_R940_PROD_DEPLOY=1).
2. Database: migrations are forward-only unless a dedicated down migration exists. Do not restore prod DB from staging.
3. DNS: prod API is Cloudflare → cloudflared → R940. AWS ALB rollback is historical only (stacks decommissioned). See migration plan rollback section if dual-stack window still open.

---

Never

• Restart lastvet-api-prod without explicit LASTVET_ALLOW_R940_PROD_DEPLOY=1
• Run fixture or seed scripts against prod
• Copy staging env file to prod path
• Blind-sync local .env into BWS prod keys

---

Resolution history

Date	Notes
2026-08-18	Migration pre-flight checklist; LAN host for prod migrate; snapshot gate; --expect required when applying
2026-08-18	Initial R940 runbook; replaces AWS LASTVET_ALLOW_PROD_DEPLOY discipline