Skip to content

Environment topology — LastVet AWS (single account, hard walls)

Superseded for R940 (August 2026). Operational hard walls, deploy sequence, and prod confirmation gate live in lastvet-gates/docs/deployment/R940_ENVIRONMENT_TOPOLOGY.md and lastvet-gates/docs/runbooks/r940-prod-deploy.md. This document is retained for AWS-era archaeology only.

Status: Approved plan 2026-07-23. Superseded on R940 2026-08-18. P1–P2 pre–VA-reply; P3 at VA approval; P4 deferred.

AWS account: 525382240840 (us-west-2), BAA executed.
Decision: One account with structural hard walls — not separate prod/non-prod accounts at current scale.

Why before VA production access

Prod today holds one synthetic veteran (Sheba703). After case #00015000 approval, prod holds real PHI and reconfiguring the permanent sandbox lane becomes a migration, not a configuration change. Same reasoning as moving off Railway while the DB was nearly empty.

Second reason: Staging had no seed data. The provider RLS fix (2026-07-21) was verified on production because prod was the only populated environment. Staging must become the permanent security-boundary test lane.

Logical environments (two deployed lanes + local)

Lane API VA Lighthouse Data Purpose
Staging (permanent non-prod) api-staging.last.vet Sandbox only, forever Synthetic fixtures Dev, API matrix experiments, all security-boundary proof
Production api.last.vet Production only (after P3) Real veterans App Store / pilot
Local optional Sandbox Staging RDS or docker Engineer workspace — not a deployed env

Not a fourth deployed env: veteran web client (parked until VA approval) rides staging for build/test; prod web track waits on P3.

Meridian M0: local Ollama + synthetic fixtures only (last1-meridian private repo). No Meridian container on staging/prod API until PHI hosting gate + counsel sign-off. Bedrock is prod-VPC future path only.

Account split trigger (P4 — deferred)

Split prod into a second AWS account. Must complete before the first real veteran. Triggered by whichever comes first:

  1. First engineer hire (month four in the operating plan), or
  2. Counsel explicitly requires account isolation.

Do not wait until the first real veteran is onboarded — that is when migration is most expensive.

Until P4: VPC isolation + IAM-scoped Secrets Manager + operator guards (below) provide the hard walls.

Hard walls

Network and data

  • Separate VPCs: staging 10.10.0.0/16, prod 10.20.0.0/16
  • Separate RDS, ECS, EFS, Secrets Manager namespaces: lastvet-staging/* vs lastvet-prod/*
  • Never seed prod. Fixture scripts refuse prod connection strings.

VA OAuth credentials (P1)

Lighthouse issues separate sandbox and production OAuth clients. Using prod credentials against sandbox (or sandbox against api.va.gov) is a policy violation.

Credential Staging / Debug iOS App Store iOS Storage
Sandbox OAuth client IDs ✅ ❌ (sandbox VA env only) Secrets.xcconfig + build settings
Production OAuth client IDs ❌ ✅ (after P3) Bitwarden → prod-only xcconfig slot

Enforcement:

  • iOS build script scripts/verify-va-oauth-config.sh — fails Release build if LASTVET_VA_ENVIRONMENT=production while prod IDs are empty or match sandbox.
  • Runtime VAOAuthGuard.validateCurrentConfiguration() — blocks VA auth flows if production host + sandbox IDs.

Prod IDs live in Bitwarden keys VA_OAUTH_PROD_* (empty until VA issues apps). Sandbox IDs are public in repo build settings.

Operator guards (P2)

  • load-aws-env.sh staging|prod sets LASTVET_AWS_ENV_GUARD
  • deploy-api-image.sh verifies guard matches target env
  • Prod deploy requires LASTVET_ALLOW_PROD_DEPLOY=1 (intentional confirmation)
  • Project boundary: lastvet-project-boundary.sh rejects non-LastVet hosts (e.g. angel.report) in deploy/build env vars. Provider portal builds must use hardcoded api.last.vet / api-staging.last.vet only.
  • Security-boundary changes: verify on staging only after ./infra/scripts/run-ecs-seed-staging-fixtures.sh

ECS secrets wiring (P2)

Terraform app_secret_keys includes VETERAN_JWT_SECRET and SECURE_LINK_HMAC_SECRET (previously unwired). The compute module is shared — the next terraform apply in either environment registers a task definition that references both keys.

Apply order (mandatory):

  1. Add VETERAN_JWT_SECRET and SECURE_LINK_HMAC_SECRET to both lastvet-staging/app and lastvet-prod/app in AWS Secrets Manager (not staging-first).
  2. Use distinct values (openssl rand -base64 32 each). Do not copy OAUTH_JWT_SECRET or LAST1_TOKEN_ENCRYPTION_KEY — key reuse across domains is the defect being fixed. Invalidation cost: ~60 min veteran JWTs, ~10 min pending OTPs on synthetic data only.
source infra/scripts/load-aws-env.sh staging
./infra/scripts/merge-app-secret-keys.sh staging   # generates + merges; refuses overwrite

source infra/scripts/load-aws-env.sh prod
./infra/scripts/merge-app-secret-keys.sh prod

# Then update Bitwarden lastvet project to match AWS (AWS is authoritative for ECS).
# sync-bitwarden-secrets.sh api — local dev only, never push Bitwarden → AWS blind.
  1. Only then run terraform apply (staging or prod).

Missing JSON keys → ECS task placement fails (ResourceInitializationError). Empty string keys satisfy Terraform but are a footgun: VETERAN_JWT_SECRET="" disables the ?? fallback in environment.ts (service layer still falls through via ||).

Prod apply today (before populate): Running prod tasks stay on revision :7 because the ECS service has lifecycle { ignore_changes = [task_definition] }. A apply creates revision :8 that cannot start until keys exist. deploy-api-image.sh force-new-deployment reuses the service's pinned revision — safe. Unsafe: switching the prod service to the new task definition before both keys exist in SM.

Staging fixture suite (P0)

Run: ./infra/scripts/run-ecs-seed-staging-fixtures.sh staging
Source: scripts/seed-staging-fixtures.js, UUIDs in scripts/staging-fixtures.js

Fixture Purpose
Veteran A Merged dual-grant grant; signed ROI PDF + authorizations; both Part 2 paths — revoked signed_roi row + active sheet row (HL issue #2 defect state for promotion verify)
Veteran B Single grant, revoked
Veteran C Two providers with active grants (RLS cross-veteran denial)
Providers 1 & 2 staging-dual-grant@last.vet, staging-provider-two@last.vet

Bar: Any consent, RLS, revoke, or Part 2 enforcement change is fully verified on staging — never prod once real PHI exists.

Phase order

Phase When Work
P1 Pre–VA-reply ✅ VA OAuth build-time + runtime guards
P0 Pre–VA-reply ✅ Staging fixture suite
P2 Pre–VA-reply ✅ Terraform secret wiring, deploy guards, runbook
P3 VA approval Flip App Store to VAEnvironment.production + prod OAuth IDs; prod never seeded
P4 Engineer hire or counsel Optional second AWS account split

Deploy sequence (three logical steps)

1. STAGING (always first)
   source infra/scripts/load-aws-env.sh staging
   ./infra/scripts/deploy-api-image.sh staging --push-only
   ./infra/scripts/run-ecs-db-setup.sh staging
   ./infra/scripts/run-ecs-seed-staging-fixtures.sh staging
   ./infra/scripts/run-ecs-verify-*.sh staging    # live API paths only — mirror/DB checks do not count
   ./infra/scripts/deploy-api-image.sh staging --rollout-only
   (provider) npm run cf:deploy:dev

   document_share (Gate 1): run-ecs-verify-document-share-guards.sh must POST to
   /veteran/me/secure-links and get 400 document_share_requires_service_records for ROI.
   Until deploy + that live path passes, allowlist is NOT verified — even if migrations 057/058 applied.

2. PRODUCTION (after staging green + explicit approval)
   export LASTVET_ALLOW_PROD_DEPLOY=1
   source infra/scripts/load-aws-env.sh prod
   ./infra/scripts/deploy-api-image.sh prod --push-only
   ./infra/scripts/run-ecs-db-setup.sh prod --expect <migrations>
   ./infra/scripts/deploy-api-image.sh prod --rollout-only
   (provider) cf:deploy prod
   (iOS) App Store with VA production credentials (P3)

3. NEVER
   - Seed scripts on prod
   - Sandbox VA credentials in App Store release
   - Security-boundary verification on prod after real PHI
  • docs/deployment/DEV_STAGING.md — day-to-day ops
  • docs/security/SECURITY_POSTURE_TOKEN_CUSTODY.md — veteran web parked until VA approval
  • docs/runbooks/VA_PRODUCTION_CUTOVER_RUNBOOK.md — P3 VA + iOS cutover
  • docs/compliance/VA_PRODUCTION_ACCESS_TRACKER.md — case #00015000

Revision history

Date Change
2026-07-23 Plan approved: single account + hard walls; P1→P0→P2 order; split trigger = engineer hire or counsel, not first veteran
2026-07-23 document_share: migrations 057/058 applied on staging (injection workaround used initially); live allowlist verified after image deploy + task def :4 + run-ecs-verify-document-share-guards.sh