LastVet Migration Plan — AWS to R940¶
Status: Planning artifact (August 2026). Do not execute from this document without Ryan review.
Timeline: ~4 weeks (August 2026 target), single-track — complete R940 migration before VA production approval.
Decisions locked (August 2026): Case B (R940 before VA prod) · cloudflared public ingress · MinIO imaging with sovereign envelope encryption.
Assumes: R940_PLATFORM_ARCHITECTURE.md (platform topology; this doc is LastVet-specific execution), ANGEL_ARCHITECTURE_V2.md (Meridian eval dependency).
AWS sources of truth:
| Document | Path |
|---|---|
| Environment inventory | ../lastvet/AWS_ENVIRONMENT.md (vendored to docs/lastvet/ August 2026) |
| Stack / ops bible | ../lastvet/LASTVET_BIBLE.md |
| Ops console | ../lastvet/AWS_OPS_CONSOLE.md |
| Topology / hard walls | ../deployment/R940_ENVIRONMENT_TOPOLOGY.md (R940 canonical; AWS doc superseded) |
| Imaging posture | ../../../last.vet-api/docs/security/SECURITY_POSTURE_IMAGING_STORAGE.md |
| Database / RLS | ../../../last.vet-api/docs/security/SECURITY_POSTURE_DATABASE_ACCESS.md |
| Strategy / gates | ../../gates/GATE_0.5_hardening_and_proof.md, ../../../last1-meridian/MERIDIAN_EVAL_SPEC.md |
CARIN / privacy: LastVet attested to the CARIN Code of Conduct (Terms §13.1 in last.vet-web). Migration must preserve veteran audit visibility, consent enforcement, and deletion timelines in privacy policy. NVLSP privacy counsel outreach is ongoing; prepare infrastructure-change documentation for VA (hosting location on R940), not a fresh functional approval — API behavior unchanged.
CARIN authoritative artifacts (signed): ../compliance/CARIN_Code_of_Conduct_2023_signed.pdf (Ryan Curry attestation, Adobe Sign transaction CBJCHBCAABAA5iGak22GtLC840ctqAxiyTGIUFZmVRZy, April 15, 2026); confirmation email ../compliance/CARIN_Confirmation_Email_Faltz_2026-04-15.pdf (Jaclyn Faltz, Leavitt Partners — adherence confirmed pending live FHIR / VA Lighthouse connection; forward-permission grant for future API connection).
Why Case B¶
Decision (August 2026): Complete R940 migration before VA production approval. No conditional AWS-first branch.
- Zero-veteran state — Migration is engineering work, not a HIPAA breach notification event. No production veteran PHI on AWS prod today; moving synthetic infrastructure is not a patient-data migration.
- What VA reviews — Lighthouse and program review focus on API behavior, consent, and policy adherence. Infrastructure is explicitly out of scope for VA review; hosting-location and HIPAA posture rest with Ryan as Privacy Officer, not the VA program office.
- Sovereignty story — First VA production veteran lands on operator-controlled R940, aligned with Last 1's platform narrative from day one. No retroactive "AWS now, migrate later under PHI" narrative.
- DR posture now — Current single-host R940 + Backblaze B2 offsite backup is HIPAA-adequate for pre-production and early prod at zero veterans (
AWS_ENVIRONMENT.md§14 parity target). - HA deferred — Multi-site HA arrives with funding: second Proxmox node, cluster topology, and replication mechanism TBD at funding gate. Not a blocker to Case B or to first VA production veteran on single-host R940.
Objective¶
Complete the LastVet AWS → R940 migration on a definite timeline (~4 weeks) before VA production approval. There is no VA-approved production workload on AWS during this window: prod on AWS holds at most synthetic/pre-PHI data while engineering lands the sovereign stack on R940.
VA path (Case B): Lighthouse review targets API behavior and policy adherence in sandbox/staging against the sovereign R940 version, not AWS hosting. Production cutover on R940 (DNS, prod credentials, checklist) waits until VA prod access is granted — but the migration itself is not conditional on which stack VA first saw.
Preserve technical rollback: AWS stacks stay alive 30+ days after R940 prod cutover as a rollback net until R940 stability is proven. Railway remains a frozen second-order net only (Gate 0.5 invariant: gone before VA production PHI).
Hard gate: Meridian eval harness operational and integrated into LastVet deployment pipeline before LastVet production accepts model-serving requests (not required for API/consent cutover alone, but required before Meridian/M1 features on prod).
What migrates¶
| AWS today | R940 target | Count |
|---|---|---|
ECS Fargate lastvet-{env}-api |
lastvet-api VM per environment + cloudflared daemon (see Public ingress) |
Staging + prod |
RDS Postgres 18 lastvet-{env}-postgres |
lastvet-db VM, Postgres 18, same lastvet_master / lastvet_app role split |
Staging + prod |
EFS lastvet-{env}-uploads |
ZFS dataset on lastvet-api VM (UPLOAD_DIR parity) |
Staging + prod |
S3 + KMS imaging (imaging-storage module) |
lastvet-imaging VM, MinIO + envelope encryption (see Imaging storage) |
Staging + prod |
Secrets Manager lastvet-{env}/app, lastvet-{env}/rds/* |
Bitwarden project lastvet, bws run on API VMs |
Staging + prod |
| ALB + ACM | cloudflared tunnel → lastvet-api (grey-cloud DNS unchanged) |
Staging + prod |
What stays on Cloudflare (does not migrate)¶
| Asset | Hostname | Rule |
|---|---|---|
| Marketing site | last.vet |
Pages |
| Provider portal | provider.last.vet, dev.provider.last.vet |
Pages |
| DNS + tunnel edge | api.last.vet, api-staging.last.vet |
Grey-cloud CNAME (DNS only, no proxy) → cloudflared tunnel; tunnel payload encrypted end-to-end (see Public ingress) |
What decommissions (after cutover window)¶
| Asset | When |
|---|---|
| Railway prod rollback net | Must be gone before VA production PHI (Gate 0.5 invariant); target: after AWS→R940 stable |
| AWS staging + prod stacks | Minimum 30 days after R940 prod stable; keep until clear-signal |
VLAN architecture¶
Implement platform VLANs before LastVet prod traffic lands (see R940_PLATFORM_ARCHITECTURE.md).
| VLAN | LastVet role |
|---|---|
| 40 | Staging: lastvet-api, lastvet-db, lastvet-imaging |
| 50 | Production: same component set; no L3 route to VLAN 40 |
Firewall matrix (minimum):
| From | To | Ports |
|---|---|---|
| Internet ingress (cloudflared tunnel) | lastvet-api |
443 (local reverse proxy → Node) |
lastvet-api |
lastvet-db |
5432 |
lastvet-api |
lastvet-imaging |
9000 (MinIO API) |
meridian-eval |
lastvet-api staging |
443 (eval hook only) |
| Deny | VLAN 40 ↔ VLAN 50 | all |
Compliance controls parity check¶
Derived from Terraform infra/modules/compliance-logging, imaging-storage, ENVIRONMENT_TOPOLOGY.md, and AWS_ENVIRONMENT.md §14 (vendored).
| AWS control (today) | R940 equivalent | Status |
|---|---|---|
Separate VPCs staging 10.10.0.0/16 / prod 10.20.0.0/16 |
VLAN 40 / VLAN 50 + no cross-route | in-progress |
| RDS encryption at rest | ZFS native encryption on lastvet-db datasets |
in-progress |
Dual DB roles (lastvet_master, lastvet_app NOBYPASSRLS) |
Same roles on R940 Postgres; DATABASE_APP_URL / MIGRATION_DATABASE_URL from BWS |
implemented (pattern) |
| RLS on PHI tables | Same migrations; verify via run-ecs-verify-db-role.sh adapted for R940 |
implemented (verify post-migrate) |
| ECS task isolation | Dedicated API VM per env | implemented |
| Secrets Manager | Bitwarden lastvet + bws run |
in-progress (keys exist; VM wiring TBD) |
EFS uploads (UPLOAD_DIR) |
ZFS on lastvet-api |
in-progress |
| S3 imaging SSE-KMS + CMK per env | MinIO + ZFS pool encryption + application envelope encryption (DEK per image, master key in BWS); parity with SSE-KMS semantics | designed (implement Week 2–3) |
| KMS key policy least privilege | MinIO service account + master key in Bitwarden; DEK wrap/unwrap in API only | designed |
| CloudTrail multi-region | Proxmox audit log + API journalctl + app audit tables |
gap-with-mitigation |
| VPC Flow Logs (~6 yr CloudWatch) | VLAN firewall/syslog + archived to tank-bulk |
not-yet-scoped |
| ALB access logs (S3, KMS, ~7 yr) | Reverse proxy access logs → tank-bulk → B2 |
not-yet-scoped |
CloudWatch ECS logs /ecs/lastvet-* |
Structured logs on API VM → log shipper → bulk pool | in-progress |
AWS BAA (account 525382240840) |
On-prem operator responsibility + B2 BAA for offsite backups | gap-with-mitigation |
| AWS Config / GuardDuty | Not deployed today | not-yet-scoped (optional) |
Append-only consent_audit_log |
Same Postgres rules on R940 | implemented |
| 6-year audit retention (policy) | tank-bulk lifecycle + B2 immutability |
not-yet-scoped |
| Grey-cloud API DNS + cloudflared | Grey-cloud CNAME unchanged; tunnel from R940 to Cloudflare edge | designed (Week 3 rehearsal) |
| VA OAuth credential walls | Unchanged client-side; env-specific secrets in BWS | implemented |
| Meridian eval before model serve | meridian-eval + CI gate |
in-progress |
| Railway teardown before VA prod PHI | Process gate | implemented (policy) |
| CARIN transparency / audit UX | Application-layer unchanged | implemented |
| Physical access (HIPAA §164.310) | R940 location controls documented | not-yet-scoped |
| Incident response runbook | R940-specific IR doc | not-yet-scoped |
Imaging parity: AWS imaging uses SSE-KMS without app-layer ciphertext (SECURITY_POSTURE_IMAGING_STORAGE.md §3). R940 adds envelope encryption for KMS-equivalent semantics on sovereign hardware. Application must preserve consent-gated presigned GET, 300s TTL, and audit-at-issuance — not weaker controls than AWS.
Database migration approach¶
Staging first (mandatory)¶
# 1. Provision lastvet-db (VLAN 40), Postgres 18, pgvector if needed later for Meridian
# 2. Dump from RDS (one-off task or temporarily allowed IP)
pg_dump -Fc -h lastvet-staging-postgres....rds.amazonaws.com -U postgres -d lastvet > staging.dump
# 3. Restore on R940
pg_restore -h lastvet-db.staging.last1.internal -U lastvet_master -d lastvet --no-owner staging.dump
# 4. Reapply role passwords from Bitwarden (not in dump)
./infra/scripts/provision-lastvet-app-role.sh # adapt path for R940
# 5. Verify RLS runtime
./infra/scripts/run-ecs-verify-db-role.sh staging # adapt to SSH/systemd on R940
# Expect: "pass": true, current_user: "lastvet_app"
Production¶
Same pattern after staging green + explicit approval. Prod on AWS today is synthetic/pre-PHI only; the R940 prod restore happens before VA production credentials land, so logical dump remains the default (no live veteran PHI migration from AWS prod).
Checks¶
| Check | Command / artifact |
|---|---|
| Migration version parity | SELECT max(version) FROM schema_migrations |
| RLS enabled | \d+ documents shows RLS |
| App role cannot bypass | verify script |
| No staging→prod DB copy | Operational rule |
Imaging storage (MinIO)¶
Decision: MinIO on lastvet-imaging VM with sovereign key custody. Replaces AWS S3 + KMS (imaging-storage module per AWS_ENVIRONMENT.md).
Stack¶
| Layer | Implementation |
|---|---|
| Object store | MinIO (S3-compatible) on lastvet-imaging VM per environment |
| Pool encryption | ZFS native encryption at rest on imaging dataset (e.g. tank-fast/vm-lastvet-imaging or dedicated imaging pool) |
| Network | lastvet-api → MinIO API :9000 (VLAN firewall) |
Application envelope encryption (SSE-KMS equivalent)¶
AWS today: SSE-KMS with CMK per env, no app-layer ciphertext on imaging objects. R940 replicates KMS separation semantics without AWS KMS:
| Step | Action |
|---|---|
| Write | LastVet API generates a data-encryption key (DEK) per image |
| Encrypt image bytes with DEK | |
Wrap DEK with master key (stored in Bitwarden Secrets Manager, lastvet project) |
|
| Store encrypted image + wrapped DEK in MinIO | |
| Read | Fetch encrypted object + wrapped DEK from MinIO |
| Unwrap DEK using master key from BWS | |
| Decrypt image in API memory; stream to client | |
| Master key rotation | Rotate master key in Bitwarden only; re-wrap DEKs, no bulk re-encryption of stored images |
| Presigned GET | Issued by MinIO with expiration (default 300s TTL, consent-gated) |
| Audit | Audit-at-issuance logged application-side when API issues presigned URL: who, image id, timestamp, expiration, request context |
Compliance notes¶
| Topic | R940 posture |
|---|---|
| BAA for storage tier | MinIO on operator hardware — no third-party BAA for the object store itself; sovereignty is genuine |
| Parity with AWS | Envelope encryption gives KMS-equivalent key separation; ZFS adds defense-in-depth at rest |
| CARIN / consent | Presigned URL issuance remains consent-gated; veteran audit surfaces unchanged (see CARIN artifacts above) |
| Migration cutover | aws s3 sync staging → MinIO during maintenance window; prod sync while AWS prod holds synthetic data only |
Gap (honest): Application code must implement envelope encrypt/decrypt and audit-at-issuance — not yet wired on R940. Week 2 staging smoke test is the proof.
Reference: SECURITY_POSTURE_IMAGING_STORAGE.md, LASTVET_BIBLE.md imaging module notes.
Public ingress (cloudflared)¶
Decision: cloudflared tunnel for all public API traffic. Replaces ALB + ACM on AWS.
DNS (unchanged pattern)¶
| Hostname | Cloudflare mode | Target |
|---|---|---|
api-staging.last.vet |
Grey-cloud CNAME (DNS only, no proxy) | cloudflared tunnel hostname (staging) |
api.last.vet |
Grey-cloud CNAME (DNS only, no proxy) | cloudflared tunnel hostname (prod) |
Same rule as current AWS cutover (DEV_STAGING.md, ENVIRONMENT_TOPOLOGY.md): never orange-cloud API hostnames.
Tunnel architecture¶
| Component | Detail |
|---|---|
| cloudflared daemon | Runs on R940 — Week 1 decision: co-locate on lastvet-api VM vs dedicated ingress VM (see Open questions) |
| Traffic path | Veteran → Cloudflare edge → encrypted tunnel → cloudflared → lastvet-api:443 (or local reverse proxy) |
| PHI at edge | No. Tunnel traffic is end-to-end encrypted; Cloudflare edge sees encrypted bytes only |
| Veteran experience | Same hostnames as AWS — zero client config change |
Rationale¶
| Pro | Detail |
|---|---|
| Grey-cloud parity | Preserves current DNS-only pattern; no new veteran-facing hostname |
| HIPAA-safe pattern | Well-documented tunnel model; PHI never decrypted at Cloudflare |
| Vendor overlap | Cloudflare already serves Pages (last.vet, provider.last.vet); no new edge vendor |
Honest compromise¶
Cloudflare remains an edge dependency even though PHI does not touch their infrastructure in plaintext. If a stronger "no third party at edge" story is required later, WireGuard direct is available — but that implies DDoS handling, TLS termination, and a larger public-facing ops surface. Document the tradeoff; do not pretend Cloudflare is absent from the path.
DNS cutover procedure¶
Cloudflare CNAME retarget (document ALB names in rollback runbook before changing):
| Hostname | From | To |
|---|---|---|
api-staging.last.vet |
AWS ALB DNS | Staging cloudflared tunnel CNAME |
api.last.vet |
AWS ALB DNS | Prod cloudflared tunnel CNAME |
Verify after cutover:
curl -sS https://api-staging.last.vet/health
curl -sS https://api.last.vet/health
# Repeat-authenticated Cache-Control test per ARCHITECTURE lessons
Secrets migration¶
| AWS Secrets Manager | Bitwarden lastvet key |
|---|---|
lastvet-staging/app |
Same keys per lastvet-gates/bitwarden/secrets-manifest.json |
lastvet-staging/rds/master |
MIGRATION_DATABASE_URL (staging) |
lastvet-staging/rds/lastvet_app |
DATABASE_APP_URL (staging) |
lastvet-prod/app |
Prod app keys |
lastvet-prod/rds/* |
Prod DB URLs |
Runtime on R940:
ExecStart=/usr/local/bin/bws run --project-id ${LASTVET_BWS_PROJECT_ID} -- \
/usr/bin/node /opt/lastvet/last.vet-api/dist/server.js
Order: Populate BWS from AWS SM once (read-only export), then never push Bitwarden → AWS blind during overlap (ENVIRONMENT_TOPOLOGY.md ECS secret wiring lesson).
Imaging keys: add MINIO_*, IMAGING_*, and imaging master key to manifest before imaging cutover (envelope encryption).
Audit logging equivalents¶
| AWS | R940 |
|---|---|
| CloudTrail | Proxmox host audit + auditd on LastVet VMs |
| VPC Flow Logs | OPNsense/switch syslog or nftables log → archive |
| ALB access logs | cloudflared + reverse proxy access logs → tank-bulk → B2 |
App consent_audit_log |
Same Postgres (authoritative veteran-facing audit) |
| Retention ~6–7 years | tank-bulk datasets + Backblaze B2 immutable bucket |
Gap closure: Written retention policy matching privacy policy ("audit logs retained 6 years") before prod PHI.
Rollback plan (Case B)¶
This plan assumes no VA-approved production on AWS during the migration window. Rollback is not "revert to VA-approved production on AWS." It is revert to last-known-good pre-production state on AWS (synthetic data, idle stacks) while R940 is fixed.
| Window | Action |
|---|---|
| During cutover | Cloudflare grey-cloud CNAME api*.last.vet → AWS ALB DNS names (document in cutover runbook before pointing at cloudflared) |
| AWS retention | Keep staging + prod AWS stacks running (idle or synthetic load only) for minimum 30 days after R940 prod DNS cutover |
| Rollback trigger | R940 prod fails health, E2E, parity, or soak checks — repoint DNS to ALB; investigate on R940 without deleting AWS |
| Decommission AWS | Only after R940 has demonstrated stability under real veteran load for a defined soak window (30-day minimum post-cutover, extend if Gate 0.5 or VA findings open) + Gate 0.5 clear-signal |
| Second order | Railway prod stays warm (frozen synthetic) if both R940 and AWS fail — per existing Gate 0.5 plan; time-boxed; not policy-compliant for VA prod PHI |
Not in scope: shipping VA production approval on AWS first, then migrating PHI-bearing prod later. That path is closed.
Document ALB DNS names in last.vet-api cutover runbook before DNS changes (terraform output alb_dns_name while stacks still live).
Meridian dependency (hard gate)¶
| Requirement | Detail |
|---|---|
| Harness location | meridian-eval VM + last1-meridian repo |
| Integration | LastVet deploy pipeline runs Meridian Suite A/B/C (or agreed subset) before promoting API image that enables Meridian routes |
| Inference | meridian-mistral-7b on inference-01 via /api/chat |
| Not a substitute | Passing Gate 0.5 E2E does not replace Meridian eval for AI features |
Reference: ../../../last1-meridian/MERIDIAN_EVAL_SPEC.md, ../../gates/meridian-implementation-spec-pointer.md PHI hosting gate.
Timeline (4 weeks, single-track)¶
Case B locked. One path — no AWS-first VA prod branch. Grounded in demonstrated Aug 7–9 velocity (bare-metal → Bitwarden pipeline in ~2 days). One primary deliverable per week.
Week 1 — Platform + staging skeleton¶
| Deliverable | Done when |
|---|---|
| VLAN 40/50 designed on switch; LastVet VM templates on Proxmox | VMs exist, no prod traffic |
lastvet-db staging Postgres 18 + ZFS encryption |
psql from admin net |
lastvet-api staging VM + bws run + health endpoint locally |
curl localhost:3000/health |
Bitwarden lastvet keys for staging DB URLs populated |
bws secret list shows URLs |
Confirm docs/lastvet/AWS_ENVIRONMENT.md §14 parity |
Parity table signed |
Week 2 — Staging data + parity proof¶
| Deliverable | Done when |
|---|---|
pg_dump / pg_restore staging RDS → R940 |
Migration count matches |
run-ecs-verify-db-role.sh equivalent pass on R940 staging |
JSON "pass": true |
lastvet-imaging staging MinIO + ZFS encryption + envelope encrypt smoke test |
Presigned upload/download + audit-at-issuance log row |
| EFS-class uploads on API VM ZFS | Document upload path works |
Meridian eval runs from meridian-eval against staging API (synthetic) |
CI job defined (may be manual first) |
Week 3 — Prod parallel + ingress¶
| Deliverable | Done when |
|---|---|
| VLAN 50 prod VMs provisioned (no cross-VLAN routes) | Firewall tested |
| Prod DB restore or fresh migrate + dual roles | Verify script pass |
| cloudflared staging tunnel + grey-cloud DNS rehearsal | api-staging.last.vet → tunnel CNAME in test window |
| Imaging prod MinIO + envelope encryption + master key in BWS | Staging parity proven; counsel packet references CARIN artifacts |
| Rollback runbook with ALB DNS names | Committed to repo |
Week 4 — Cutover + soak¶
| Deliverable | Done when |
|---|---|
| Staging cutover DNS → R940 | 48h soak; fixtures + document_share verifies green |
| Prod cutover DNS → R940 | Health + Gate 0.5 E2E script on R940; VA prod credentials still pending |
| Meridian eval wired to prod deploy pipeline | Block merge without eval pass |
| AWS stacks idle (not torn down), DNS on R940 | 30-day technical rollback clock starts |
| Railway decommission plan scheduled | Before first VA production PHI on R940 |
Open questions (LastVet — require Ryan decision)¶
Resolved (August 2026)¶
| # | Question | Decision |
|---|---|---|
| 1 | Public ingress | cloudflared tunnel — grey-cloud CNAMEs unchanged |
| 2 | VA Case A vs B | Case B — R940 before VA prod approval (Why Case B) |
| 3 | Imaging backend | MinIO + ZFS pool encryption + application envelope encryption |
Remaining¶
- cloudflared placement: daemon on
lastvet-apiVM (simpler) vs dedicated ingress VM (blast-radius isolation)? Week 1 implementation decision. - API runtime: native Node systemd vs Docker-on-VM on
lastvet-api. - VM topology: separate VMs per component vs combined staging/prod footprint (cost vs isolation).
- Reverse proxy on API VM: Caddy/nginx in front of Node for local TLS termination from cloudflared — config TBD Week 3.
- Meridian CI integration point: block R940 deploy, GitHub Actions, or manual gate until automated?
- infra README vs DEV_STAGING: README says Cloudflare proxy OK for API;
DEV_STAGING.mdrequires grey-cloud — confirm R940 ingress doc alignment (decision: grey-cloud; reconcile README). - HA funding gate: second Proxmox node topology and replication mechanism — TBD when funded (not blocking Case B).
Cross-references¶
- R940_PLATFORM_ARCHITECTURE.md — platform VLANs, pools, shared services
../lastvet/AWS_ENVIRONMENT.md— AWS inventory being replaced../lastvet/LASTVET_BIBLE.md— stack reference- SOVEREIGN_ARCHITECTURE_V2.md
- ANGEL_ARCHITECTURE_V2.md
- Gate invariants:
../../.cursor/rules/00-invariants.mdc