Tier 3 T3-3 Report — Parallel soak + tunnel route¶

Date: 2026-08-28
Status: T3-3 COMPLETE. PAUSED — await upstream product migrations before T3-4. See TIER_3_ADMIN_PAUSED_STATUS.md.
Parent: TIER_3_ADMIN_T3-2_REPORT.md

1. Cloudflare Tunnel route (no DNS change)¶

Item	Value
Tunnel host	`lastvet-edge` (`192.168.1.89`)
Tunnel name	`lastvet-r940`
New ingress	`admin.last1.enterprises` → `http://192.168.1.97:5000`
Config validated	`cloudflared tunnel ingress validate` → OK
Service	`cloudflared` restarted, active

Script: scripts/last1-admin/configure-tunnel-route-r940.sh

Internal reachability (no public DNS cutover)¶

Test	Result
`lastvet-edge` → `http://192.168.1.97:5000/`	HTTP 200
`admin-r940` → `GET /api/auth/user`	HTTP 401 (expected unauthenticated)
Public `https://admin.last1.enterprises/`	HTTP 200, headers include `x-railway-edge: lax1`

Interpretation: Tunnel ingress is registered on Cloudflare edge. Public DNS/origin routing still sends production traffic to Railway until T3-5. No prod cross-traffic to R940 observed.

T3-3 bind change: Container moved from 127.0.0.1:5000 to 192.168.1.97:5000 so tunnel origin is reachable from lastvet-edge. UFW allows 5000 from 192.168.1.89 only.

2. Parallel soak setup¶

Stack	Role	Health
Railway (`admin.last1.enterprises`)	Production auth/admin	Public 200, Railway headers
R940 (`admin-r940` VM 115)	Parallel soak (no live prod traffic)	LAN 200 / auth 401

30-minute cron probe (VM 115)¶

Item	Value
Script	`/opt/last1-admin/soak-health-probe.sh`
Cron	`/30 * * *`
Log	`/opt/last1-admin/soak/probe.log`
Checks	`GET /` → 200, `GET /api/auth/user` → 401
Metrics	Textfile → `admin_r940_probe_*` via node_exporter

Scripts: scripts/last1-admin/soak-health-probe.sh, install-soak-cron-r940.sh

First probe: 2026-08-28T16:59:53Z root=200 auth=401 ok=1

3. Observability + alerts¶

node_exporter (VM 115)¶

Installed on admin-r940; UFW allows :9100 from 192.168.1.93 (observability Prometheus).

Script: scripts/last1-admin/install-node-exporter-r940.sh

Prometheus scrape target added: 192.168.1.97:9100 (job=node). Verified up=1.

Alert rules (observability VM)¶

Alert	Tier 2 analogue	Severity	Signal
AdminHostDown	N8nHostDown / GoTrueDown	critical	`node_exporter` on `.97` down 3m
AdminHealthProbeFailed	— (soak-specific)	warning	`admin_r940_probe_success == 0` for 35m
AdminAuth5xxRateHigh	Last1idAuth5xxRateHigh	warning	`admin_r940_probe_auth_5xx == 1` for 5m

Runbooks:

docs/observability/runbooks/admin-host-down.md
docs/observability/runbooks/admin-health-probe-failed.md
docs/observability/runbooks/admin-auth-5xx-rate-high.md

PostgresDown equivalent: N/A (admin uses hosted Supabase until Tier 6; no Postgres on VM 115).

4. BWS / env verification (VM 115)¶

Check	Result
Rendered env path	`/opt/last1-admin/.env` (mode 600)
Key count	42 lines (36 BWS `LAST1ADMIN_R940_PROD_*` stripped + derived defaults)
`LAST1_ID_API_BASE`	`https://auth.last1.id`
Container env	Loaded via `--env-file`
Docker bind	`192.168.1.97:5000→5000`

No cross-project BWS inference. Railway production vars unchanged.

5. Cross-traffic check¶

Signal	Observation
Public response headers	Still `x-railway-edge`, not R940-only stack
R940 container logs	Startup + ops-digest scheduler only; no public browser/session traffic pattern
Tunnel	Route exists; DNS not cut over

Conclusion: Railway remains production auth for admin.last1.enterprises. R940 is parallel soak only.

6. Anomalies / follow-ups¶

Item	Severity	Notes
T3-2 localhost bind	Fixed in T3-3	Tunnel origin required LAN bind + UFW from edge
Soak probe curl `-f` bug	Fixed	Auth 401 was concatenating `000` in metrics; script updated
VM GitHub deploy key	Low	Still rsync-from-operator pattern (same as n8n Tier 1)
Prometheus rules live on VM	Info	Source scripts/runbooks in git; `/etc/prometheus/rules/*.yml` edited on observability VM (no git mirror yet)

No blocking anomalies.

Constraints honored¶

Constraint	Status
No DNS changes	OK (public still Railway)
No Railway teardown	OK
Railway prod auth for admin	OK
No Supabase migration	OK
STOP before T3-4 browser E2E	OK

STOP — Next gate¶

Phase	Status	Ryan action
T3-3 Parallel soak + tunnel	Done	Approve T3-4 (browser E2E, Ryan present)
T3-4 Browser E2E	Blocked	Ryan present for unit switcher matrix
T3-5 DNS cutover	Blocked	—
T3-7 Railway teardown	Blocked	—

T3-4 will: browser sign-in, unit switcher, proxy smoke across admin business units on R940 stack (Railway still prod until T3-5).