Tier 3 T3-2 Report — VM provision + Supabase stack + Gate B + BWS¶
Date: 2026-08-28
Status: T3-2 COMPLETE. STOP here. Await Ryan authorization for T3-3 (Express app deploy + parallel soak on VM 116).
Parent: TIER_3_REALOUTCOMES_T3-1_REPORT.md
Plan: REALOUTCOMES_MIGRATION_PLAN.md
Executive summary¶
T3-2 provisions VM 116/117, deploys self-hosted Supabase on VM 117, completes Gate B dump/restore rehearsal from hosted project cpocwrygxlqhzvmlrpjv, proves Option A Edge Function cron via systemd timers, and populates REALOUTCOMES_APP_PROD_* in BWS.
Highest-risk sub-task (Edge Function cron): PASS. All five functions return HTTP 200 via Kong after Gate B restore.
Production unchanged: Railway realoutcomes.app + hosted Supabase remain live. No DNS cutover. No teardown.
Phase 1 — VM provisioning ✅ STOP¶
| VM | VMID | Hostname | LAN IP | Spec | Status |
|---|---|---|---|---|---|
| App (T3-3) | 116 | realoutcomes-app |
192.168.1.99 | 2 vCPU · 4 GB · 50 GB boot | Provisioned; no app deploy (T3-3) |
| Supabase | 117 | realoutcomes-db |
192.168.1.98 | 4 vCPU · 16 GB · 50 GB boot + 150 GB ZFS (/var/lib/supabase-data) |
Provisioned + SSH OK |
Scripts: scripts/realoutcomes/provision-vm-app.sh, provision-vm-supabase.sh
Note: Cloud-init SSH key injection failed (same as admin/n8n); fixed via qm guest exec on pve-r940.
Phase 2 — Supabase Docker stack ✅ STOP¶
| Item | Value |
|---|---|
| Compose root | /opt/supabase/docker (official Supabase sparse clone) |
| Kong / API | http://192.168.1.98:8000 |
| Postgres pooler | :5432 / :6543 on LAN |
| Realtime | Disabled (unused) |
| Edge Functions | 8 synced from RealOutcomes repo → volumes/functions/ |
| Stack health | All services healthy after Gate B restore |
Script: scripts/realoutcomes/setup-supabase-stack.sh
Lessons (document for T3-3+ ops):
- Do not re-run
generate-keys.shafter DB init without wipingvolumes/db/data(breaksauthenticator→ PostgREST crash loop). - Quote
STUDIO_DEFAULT_ORGANIZATIONin.envif operator scriptssourcethe file (spaces break bash).
Edge Function smoke (post Gate B):
| Function | HTTP | Notes |
|---|---|---|
process-status-changes |
200 | processed: 200 |
run-validations |
200 | found: 0 |
process-score-runs |
200 | processed: 0 |
send-alert-digests |
200 | SendGrid not configured on stack (expected) |
scheduled-recrawl |
200 | No eligible orgs |
Phase 3 — Edge Function cron (Option A) ✅ STOP¶
Approach: systemd timers on VM 117 → curl http://127.0.0.1:8000/functions/v1/<name> with SERVICE_ROLE_KEY Bearer.
Schedules (parity with hosted dashboard / repo migrations):
| Timer | Schedule | Function |
|---|---|---|
realoutcomes-process-status-changes.timer |
*:0/10 |
process-status-changes |
realoutcomes-run-validations.timer |
daily 04:00 UTC | run-validations |
realoutcomes-process-score-runs.timer |
*:0/5 |
process-score-runs |
realoutcomes-send-alert-digests.timer |
hourly | send-alert-digests |
realoutcomes-scheduled-recrawl.timer |
Sun 03:00 UTC | scheduled-recrawl |
Script: scripts/realoutcomes/install-edge-function-cron.sh
Proof: Manual invocation of all 5 functions PASS after Gate B restore. Timers enabled and listed in systemctl list-timers 'realoutcomes-*'.
Option B escalation: Not required. Systemd + Kong path validated.
Note on cron.job: Hosted dashboard schedules are not stored as pg_cron rows in the restored DB (cron.job count = 0). Option A intentionally replaces dashboard cron with systemd; no pg_cron HTTP jobs needed on R940.
Phase 4 — Gate B rehearsal ✅ STOP¶
| Step | Result |
|---|---|
pg_dump hosted (PG 17 client required) |
133 MB custom-format dump (hosted-20260828T175320Z.dump) |
pg_restore into supabase-db |
Completed with 589 non-fatal errors (expected: --clean on auth/storage/realtime system objects, non-superuser event triggers) |
| Stack restart | All services healthy |
Verification:
| Check | Hosted (T3-0) | R940 rehearsal |
|---|---|---|
irs_organizations rows |
~2M | 1,990,071 |
auth.users |
3 | 3 |
| Public RLS policies | 52 | 52 |
storage.objects |
0 | 0 |
| Extensions | pg_cron, pg_net | present |
Script: scripts/realoutcomes/gate-b-rehearsal.sh (updated: PGDG client 17, workdir permissions)
Not in T3-2 scope: Google OAuth reconfiguration on self-hosted GoTrue (T3-4/T3-5).
Phase 5 — BWS REALOUTCOMES_APP_PROD_* ✅ STOP¶
Project: realoutcomes (788ed6ec-0284-4bbc-b57c-b4840004688e)
| Action | Result |
|---|---|
| Railway → BWS sync | 24 keys from RealOutcomes.app service (read-only Railway GraphQL) |
| R940 stack keys | 9 keys created/updated from VM 117 .env (LAN DATABASE_URL, pooler URL, JWT, Supabase URL/keys) |
Total REALOUTCOMES_APP_PROD_* |
27 |
ADMIN_GATEWAY_SECRET |
Skipped (Tier 6 coordination) |
Legacy realoutcomes_* keys |
Retained |
Scripts:
scripts/realoutcomes/sync-app-bws-from-railway.pyscripts/realoutcomes/push-db-stack-bws.py
D-live vendor rotations (Ryan manual — pending)¶
Approved during BWS setup; Railway interim values remain in BWS until Ryan rotates in vendor dashboards:
| GG ID | Secret | T3-2 status |
|---|---|---|
| 29169495 | Stripe webhook | Interim Railway value in BWS; rotate before T3-5 |
| 29169499 | SendGrid | Interim Railway value in BWS; rotate before T3-5 |
| 29169500 | Firecrawl | Interim Railway value in BWS; rotate before T3-5 |
| 29169498 | Postgres | Replaced by R940 DATABASE_URL / POSTGRES_PASSWORD |
| 29169497 | Supabase JWT | New R940 keys in BWS; hosted JWT retired at cutover |
Constraints honored¶
| Constraint | Status |
|---|---|
| No DNS cutover | OK |
| No hosted Supabase teardown | OK |
| No Railway teardown | OK |
| Railway remains production | OK |
| Admin tier (VM 115) untouched | OK |
| last1.app + RadioCheck bundle untouched | OK |
| No cross-project BWS inference | OK |
ADMIN_GATEWAY_SECRET not rotated |
OK |
STOP — Next gate¶
| Phase | Status | Ryan action |
|---|---|---|
| T3-0 Audit | Done | Approved |
| T3-1 Planning | Done | Approved |
| T3-2 VM + Supabase + Gate B + BWS | Done | Approve T3-3 kickoff |
| T3-3 App deploy + parallel soak | Blocked | — |
| T3-4 Edge/cron/Stripe soak | Blocked | — |
| T3-5 DNS cutover | Blocked | — |
| T3-6 Soak 48–72h | Blocked | — |
| T3-7 Teardown | Blocked | — |
T3-3 will: deploy Express monolith + monthly-irs-sync on VM 116, point at R940 Supabase via BWS-rendered env, parallel soak against Railway prod (no DNS change).
Ryan pre-T3-5: Rotate Stripe webhook, SendGrid, Firecrawl in vendor dashboards → update BWS REALOUTCOMES_APP_PROD_*.
Scripts index¶
| Script | Purpose |
|---|---|
provision-vm-app.sh |
VM 116 on pve-r940 |
provision-vm-supabase.sh |
VM 117 + ZFS data disk |
setup-supabase-stack.sh |
Supabase Docker + functions |
install-edge-function-cron.sh |
systemd timers |
gate-b-rehearsal.sh |
Hosted dump → restore → verify |
sync-app-bws-from-railway.py |
Railway env → BWS |
push-db-stack-bws.py |
VM stack keys → BWS |