Skip to content

Environment topology — LastVet R940 (hard walls)

Status: Approved 2026-08-18. Replaces AWS-era ENVIRONMENT_TOPOLOGY.md for all R940 operations.

Platform: Dell R940 Proxmox (pve-r940), operator-controlled. AWS ECS retired August 2026.

Decision: One physical platform with structural hard walls between staging and prod lanes — separate Postgres databases, MinIO buckets/credentials, BWS key prefixes, and systemd services. Not separate hosts at current scale.

Why before VA production access

Prod today holds synthetic data only (one legacy synthetic veteran plus pre-PHI test accounts). After case #00015000 approval, prod holds real PHI and reconfiguring the permanent sandbox lane becomes a migration, not a configuration change.

Staging is the permanent security-boundary test lane: VA Lighthouse sandbox forever, synthetic fixtures, all consent/RLS/imaging proof before prod promotion.

Logical environments (two deployed lanes + local)

Lane Public API Host VA Lighthouse Postgres DB MinIO bucket BWS prefix Purpose
Staging api-staging.last.vet lastvet-api VM (lastvet-api-staging.service) Sandbox only, forever lastvet_staging on lastvet-db staging imaging bucket LASTVET_R940_STAGING_* Dev, security-boundary proof, fixture verification
Production api.last.vet lastvet-api VM (lastvet-api-prod.service) Production only (after P3) lastvet_prod on lastvet-db prod imaging bucket LASTVET_R940_PROD_* App Store / pilot
Local optional (localhost) engineer machine Sandbox staging DB or docker n/a BWS → .env.local Not a deployed env

Ingress: Cloudflare DNS (grey-cloud) → cloudflared on lastvet-edge → Caddy → lastvet-api upstreams. Never orange-cloud API hostnames.

Meridian M0: local Ollama + synthetic fixtures only (last1-meridian). No Meridian model-serving on prod API until eval harness gate + counsel sign-off.

lastvet-api VM: two systemd services (one host, two ports)

At current scale, both deployed API lanes run on the same VM (lastvet-api, LAN 192.168.1.90). This is intentional hard-wall separation by database, secrets prefix, MinIO bucket, and systemd unit, not by physical host.

systemd unit PORT Upstream (Caddy) Public hostname Postgres DB
lastvet-api-staging.service 3000 http://192.168.1.90:3000 api-staging.last.vet lastvet_staging
lastvet-api-prod.service 3001 http://192.168.1.90:3001 api.last.vet lastvet_prod

Operator notes (2026-08-19):

  • ps may show PPID=1 for both Node processes. That is not evidence of an orphan. Check cgroup and listening port instead: systemctl status lastvet-api-{staging,prod} and ss -tlnp | rg 3000\|3001.
  • PIDs change on restart. Prod traffic is always the process listening on :3001 (lastvet-api-prod.service MainPID). Staging is :3000.
  • Each unit has its own EnvironmentFile: /etc/lastvet-api/lastvet-api-{staging,prod}.env. Never copy prod env to staging or vice versa.

Hard walls

Network and data

  • Separate Postgres databases: lastvet_staging vs lastvet_prod on lastvet-db (192.168.1.91 LAN, MagicDNS lastvet-db.tailc3bbdc.ts.net). Distinct DATABASE_APP_URL per lane.
  • Separate MinIO credentials and buckets: LASTVET_R940_{STAGING,PROD}_MINIO_IMAGING_*, IMAGING_S3_BUCKET, IMAGING_MASTER_KEK (restore script aliases MinIO keys to IMAGING_S3_ACCESS_KEY / IMAGING_S3_SECRET_KEY).
  • Separate crypto secrets per lane: VETERAN_JWT_SECRET, OAUTH_JWT_SECRET, SECURE_LINK_HMAC_SECRET, LAST1_TOKEN_ENCRYPTION_KEY must be distinct within and across lanes (verified on VM 2026-08-18).
  • Never seed prod. Fixture scripts refuse prod connection strings.

Secrets authority

Layer Authority
Runtime (R940) Bitwarden lastvet project → /etc/lastvet-api/lastvet-api-{staging,prod}.env via scripts/restore-lastvet-api-env-from-bws.sh
Local dev / iOS sync BWS → .env.local, Secrets.xcconfig (pull only; never blind-push stale local → BWS)
Historical AWS SM Retired. See AWS doc banner for archaeology only.

Manifest: bitwarden/secrets-manifest.json. Naming: docs/architecture/BITWARDEN_NAMING_CONVENTION.md.

VA OAuth credentials (P1)

Lighthouse issues separate sandbox and production OAuth clients. Using prod credentials against sandbox (or sandbox against api.va.gov) is a policy violation.

Credential Staging / Debug iOS App Store iOS Storage
Sandbox OAuth client IDs ✅ ❌ (sandbox VA env only) Secrets.xcconfig + build settings
Production OAuth client IDs ❌ ✅ (after P3) Bitwarden → prod-only xcconfig slot

Enforcement:

Release builds today keep LASTVET_VA_ENVIRONMENT = sandbox until case #00015000 approves prod OAuth apps (P3).

Operator guards (P2)

  • Env restore: sudo scripts/restore-lastvet-api-env-from-bws.sh --env staging|prod on lastvet-api VM only.
  • Prod deploy confirmation: LASTVET_ALLOW_R940_PROD_DEPLOY=1 required before prod service restart or migration (replaces AWS LASTVET_ALLOW_PROD_DEPLOY). See docs/runbooks/r940-prod-deploy.md.
  • Project boundary: Provider portal builds use hardcoded api.last.vet / api-staging.last.vet only. No cross-product host leakage.
  • Security-boundary changes: verify on staging only after fixture seed (below).

Distinct security-domain keys (P2)

Application code fails closed if dedicated keys are missing:

  • veteran_jwt_secret_missing — veteran Bearer JWT signing (VETERAN_JWT_SECRET only; no fallback to OAUTH_JWT_SECRET).
  • secure_link_hmac_key_missing — secure-link OTP HMAC (SECURE_LINK_HMAC_SECRET only).

Generate with openssl rand -base64 32 per key per env in BWS. Do not copy effective values from other keys (that preserves key reuse).

Staging fixture suite (P0)

Source: last.vet-api/scripts/seed-staging-fixtures.js, UUIDs in staging-fixtures.js.

Run on R940 (from repo checkout with staging env loaded):

ssh lastvet-api
cd /opt/lastvet/last.vet-api   # or operator checkout path
set -a && source /etc/lastvet-api/lastvet-api-staging.env && set +a
export MIGRATION_DATABASE_URL   # from same env file
node scripts/seed-staging-fixtures.js

Verify loaded (read-only):

psql "$DATABASE_APP_URL" -tAc "SELECT email FROM veterans WHERE email LIKE 'staging-veteran-%' ORDER BY 1;"
# Expect: staging-veteran-a@last.vet, staging-veteran-b@last.vet, staging-veteran-c@last.vet
Fixture Purpose
Veteran A Merged dual-grant; signed ROI + authorizations; both Part 2 paths — revoked signed_roi row + active sheet row (HL issue #2 defect state)
Veteran B Single grant, revoked
Veteran C Two providers with active grants (RLS cross-veteran denial)
Providers 1 & 2 staging-dual-grant@last.vet, staging-provider-two@last.vet

Bar: Any consent, RLS, revoke, Part 2, or imaging-boundary change is fully verified on staging — never prod once real PHI exists.

Fixtures were confirmed present on R940 staging DB 2026-08-18.

Postgres connectivity (pg_hba + URL hostnames)

Decision (2026-08-18): MIGRATION_DATABASE_URL uses the LAN host 192.168.1.91, not MagicDNS/Tailscale. Do not widen pg_hba.conf to allow lastvet_master from Tailscale source IPs.

Decision (2026-08-20): DATABASE_APP_URL in BWS (LASTVET_R940_{STAGING,PROD}_DATABASE_APP_URL) is canonical at LAN IP 192.168.1.91, matching on-disk env and the live prod process. Investigation showed BWS had MagicDNS (lastvet-db.tailc3bbdc.ts.net) while runtime used LAN; credentials were identical (host-only drift). Both paths authenticate (pg_hba lines 139–142 allow lastvet_app from lastvet-api LAN 192.168.1.90/32 and Tailscale 100.119.251.125/32; prod pool observed on LAN). LAN chosen because pg_hba already pins app sources to specific IPs, LAN routing is the stable default on this topology, and BWS restore is idempotent with on-disk env. MagicDNS remains a valid fallback if LAN routing breaks; no pg_hba change required for that path today.

Role pg_hba source URL hostname (canonical) Why
lastvet_app (runtime API pool) 192.168.1.90/32 (lastvet-api LAN) or 100.119.251.125/32 (lastvet-api Tailscale) 192.168.1.91 (LAN) App connects from lastvet-api; both source IPs allowed; live prod uses LAN path
lastvet_master (migrations only) 192.168.1.0/24 only 192.168.1.91 required Tailscale connections arrive as Tailscale source IP; lastvet_master has no Tailscale rule → migrate fails with pg_hba.conf rejects connection

Enforcement:

  1. BWS keys LASTVET_R940_{STAGING,PROD}_DATABASE_APP_URL store host=192.168.1.91 (updated 2026-08-20).
  2. BWS keys LASTVET_R940_{STAGING,PROD}_MIGRATION_DATABASE_URL should store host=192.168.1.91.
  3. scripts/restore-lastvet-api-env-from-bws.sh rewrites lastvet-db.tailc3bbdc.ts.net → 192.168.1.91 in MIGRATION_DATABASE_URL on every restore (defense in depth if BWS still has MagicDNS).

See also docs/lastvet/README-postgres-recovery.md.

Phase order

Phase When Work
P1 Pre–VA-reply ✅ VA OAuth build-time + runtime guards
P0 Pre–VA-reply ✅ Staging fixture suite on R940
P2 Pre–VA-reply ✅ BWS distinct secrets, prod deploy gate, this doc
P3 VA approval Flip App Store to VAEnvironment.production + prod OAuth IDs; prod never seeded
P4 Engineer hire or counsel Optional second-account / second-site isolation (platform-level; not AWS-specific)

Deploy sequence (R940)

1. STAGING (always first)
   ssh lastvet-api
   cd /opt/lastvet/last.vet-api && git pull && npm ci && npm run build
   sudo /opt/lastvet-gates/scripts/restore-lastvet-api-env-from-bws.sh --env staging
   set -a && source /etc/lastvet-api/lastvet-api-staging.env && set +a
   node scripts/migrate.js --list-pending
   node scripts/migrate.js --expect <comma-separated-filenames>   # when applying; never auto-apply *_down.sql
   node scripts/seed-staging-fixtures.js      # idempotent; after schema changes
   sudo systemctl restart lastvet-api-staging
   curl -sS https://api-staging.last.vet/health
   # Run targeted verify scripts from last.vet-api (document_share, imaging, etc.)

2. PRODUCTION (after staging green + explicit approval)
   export LASTVET_ALLOW_R940_PROD_DEPLOY=1     # intentional confirmation — see runbook
   # Follow docs/runbooks/r940-prod-deploy.md

3. NEVER
   - Seed scripts on prod
   - Sandbox VA credentials in App Store release (P3)
   - Security-boundary verification on prod after real PHI
   - Prod service restart without LASTVET_ALLOW_R940_PROD_DEPLOY=1

AWS ECS scripts (deploy-api-image.sh, run-ecs-seed-staging-fixtures.sh, run-ecs-db-setup.sh) are historical only. Do not use for R940 operations.

Revision history

Date Change
2026-08-18 Document MIGRATION_DATABASE_URL LAN host decision; restore script enforces 192.168.1.91
2026-08-18 R940 canonical topology; BWS authority; prod gate LASTVET_ALLOW_R940_PROD_DEPLOY; supersedes AWS ENVIRONMENT_TOPOLOGY for operations