Tier 3 T3-3 Report — App deploy + parallel soak (VM 116)¶
Date: 2026-08-28
Status: T3-3 COMPLETE. STOP here. Await Ryan authorization for T3-4 (browser E2E on staging URL).
Parent: TIER_3_REALOUTCOMES_T3-2_REPORT.md
Executive summary¶
RealOutcomes Express monolith is deployed on VM 116 (192.168.1.99:5000), connected to R940 Supabase on VM 117, with monthly-irs-sync systemd timer, 30-minute soak probes, and Prometheus alerts. Railway + hosted Supabase remain production. No DNS changes.
1. App deploy (VM 116)¶
| Item | Value |
|---|---|
| Host | realoutcomes-app · 192.168.1.99 · VMID 116 |
| Repo | /opt/realoutcomes/src (rsync from operator; private repo) |
| Env | /opt/realoutcomes/.env (30 keys, mode 600, BWS REALOUTCOMES_APP_PROD_*) |
| Image / container | realoutcomes-app:prod / realoutcomes-app |
| Bind | 192.168.1.99:5000→5000 (LAN, not 127.0.0.1) |
| Docker | Installed (docker-ce 29.x) |
Scripts: render-realoutcomes-env-app.sh, deploy-realoutcomes-app.sh
Health verification (LAN)¶
| Check | Result |
|---|---|
GET / |
HTTP 200 |
GET /api/auth/user (unauthenticated) |
HTTP 401 |
GET /api/directory?limit=1 |
HTTP 200 · 1,990,402 orgs (R940 Postgres) |
| Stripe init | OK in container logs |
Parallel soak env overrides:
PUBLIC_URL=http://192.168.1.99:5000SUPABASE_URL=http://192.168.1.98:8000DATABASE_URL=postgresql://postgres.{tenant}@192.168.1.98:5432/postgres(Supavisor session mode)
2. monthly-irs-sync systemd timer¶
| Item | Value |
|---|---|
| Schedule | 0 3 1 * * UTC (Railway parity) |
| systemd | realoutcomes-monthly-irs-sync.timer |
| Next fire | 2026-09-01 03:00:00 UTC |
| Command | docker run --rm --env-file /opt/realoutcomes/.env realoutcomes-app:prod node dist/cron/monthly-irs.cjs |
| Log | /var/log/realoutcomes/monthly-irs-sync.log |
Script: install-monthly-irs-sync-timer.sh
Railway cron service unchanged (rollback net until T3-7).
3. Parallel soak setup¶
| Stack | Role | Health |
|---|---|---|
Railway (realoutcomes.app) |
Production | Public 200 / auth 401 |
R940 (192.168.1.99:5000) |
Parallel soak (no live traffic) | LAN 200 / auth 401 / directory 200 |
R940 Supabase (192.168.1.98:8000) |
Platform for soak stack | Reachable from VM 116 (Kong 403 on /rest/v1/ = up) |
30-minute cron probe (VM 116)¶
| Item | Value |
|---|---|
| Script | /opt/realoutcomes/soak-health-probe.sh |
| Cron | */30 * * * * |
| Log | /opt/realoutcomes/soak/probe.log |
| Checks | / → 200, /api/auth/user → 401, /api/directory?limit=1 → 200, Supabase Kong reachable |
| Metrics | realoutcomes_app_probe_*, realoutcomes_app_supabase_reachable via node_exporter textfile |
First probe: 2026-08-28T18:05:43Z root=200 auth=401 dir=200 supabase=403 ok=1
Scripts: soak-health-probe.sh, install-soak-cron-app.sh, install-node-exporter-app.sh
Prometheus (observability VM 192.168.1.93)¶
| Alert | Severity | Signal |
|---|---|---|
| RealOutcomesHostDown | critical | up{instance="192.168.1.99:9100"} == 0 for 3m |
| RealOutcomesHealthProbeFailed | warning | realoutcomes_app_probe_success == 0 for 35m |
| RealOutcomesSupabaseUnreachable | warning | realoutcomes_app_supabase_reachable == 0 for 10m |
Scrape target added: 192.168.1.99:9100 (up=1 verified).
Script: configure-prometheus-realoutcomes-app.sh
Runbooks:
docs/observability/runbooks/realoutcomes-host-down.mddocs/observability/runbooks/realoutcomes-health-probe-failed.mddocs/observability/runbooks/realoutcomes-db-unreachable.md
4. Parallel state verification¶
| Signal | Observation |
|---|---|
| Railway production | https://realoutcomes.app/ → 200; still serves all public traffic |
| R940 app | Connected to R940 Supabase; directory returns live Gate B data |
| VM 117 Edge timers | All 5 realoutcomes-* timers active and firing |
| Cross-traffic | None; R940 has no public DNS route in this phase |
| Hosted Supabase | Untouched; Railway app still uses hosted DB for prod |
5. Anomalies / follow-ups¶
| Item | Severity | Notes |
|---|---|---|
| DATABASE_URL Supavisor format | Fixed T3-3 | Bare postgres@host:5432 fails on Supavisor; requires postgres.{POOLER_TENANT_ID}@192.168.1.98:5432. Updated push-db-stack-bws.py + BWS key. |
| VM 117 placeholder tenant/password | Medium | Stack still has .env.example values (your-tenant-id, default postgres password). Works for soak; rotate before T3-5 cutover. |
| UFW on VM 116 | Fixed T3-3 | Prometheus scrape blocked until :9100 opened from 192.168.1.93. |
| Staging tunnel route | Deferred | Not required for LAN soak. T3-4 browser E2E may need realoutcomes-app.last1.enterprises route on lastvet-edge. |
| VM GitHub deploy key | Low | rsync-from-operator pattern (same as admin/n8n). |
No blocking anomalies.
Constraints honored¶
| Constraint | Status |
|---|---|
| No DNS changes | OK |
| No Railway teardown | OK |
| Railway remains production | OK |
| Hosted Supabase untouched | OK |
| Admin tier (paused) untouched | OK |
| STOP before T3-4 browser E2E | OK |
STOP — Next gate¶
| Phase | Status | Ryan action |
|---|---|---|
| T3-3 App + parallel soak | Done | Approve T3-4 kickoff |
| T3-4 Integration + browser E2E | Blocked | Ryan present |
| T3-5 DNS cutover | Blocked | — |
| T3-7 Teardown | Blocked | — |
T3-4 will: crawl/score pipeline smoke, Stripe/SendGrid tests, Ryan browser checklist on R940 stack (optionally via staging tunnel hostname).
Scripts index (T3-3 additions)¶
| Script | Purpose |
|---|---|
render-realoutcomes-env-app.sh |
BWS → /opt/realoutcomes/.env |
deploy-realoutcomes-app.sh |
Docker build + run on VM 116 |
install-monthly-irs-sync-timer.sh |
IRS cron systemd timer |
soak-health-probe.sh |
30-min health + Supabase probe |
install-soak-cron-app.sh |
Install probe cron |
install-node-exporter-app.sh |
node_exporter + textfile collector |
configure-prometheus-realoutcomes-app.sh |
Scrape target + alert rules |