Onset Rollback Runbook
Severity Levels
- SEV-1: App launch crash, auth outage, widespread inability to use core flow.
- SEV-2: AI/protocol degraded but app usable.
- SEV-3: Non-critical regression, cosmetic or limited-scope issue.
- Confirm blast radius:
- iOS app version affected
- function(s) affected (
daily-insight, generate-protocol, client-event, admin-dashboard)
- whether issue is auth, model/provider, DB migration, or client regression
- Freeze deploys:
- stop additional App Store/TestFlight promotions
- stop Supabase function/migration deploys
- Post status update to internal channel with current severity.
Rollback Paths
A) Supabase Function Regression
- Re-deploy previous known-good function source (from git tag/commit).
- Verify with smoke:
python3 scripts/supabase-smoke.py --project-ref <ref>- Confirm app recovers without client update.
B) Migration Regression
- Do not apply new migrations until root cause identified.
- If migration is additive and safe to ignore, hotfix app/function to bypass new objects.
- If migration caused hard failure, apply explicit forward-fix migration.
- Prefer forward-fix over destructive rollback.
C) iOS Client Regression
- If severe, stop phased release / halt rollout in App Store Connect.
- Ship hotfix build (bump versions with
scripts/ios-version.py).
- Keep backend compatible with both current and previous client versions during transition.
D) Model/Provider Outage
- Verify provider status and API key validity.
- Switch to available fallback model ID.
- Keep user-facing fallback messaging active in app.
Verification Checklist
- [ ]
client-event returns 200
- [ ]
daily-insight returns 200 or graceful fallback behavior confirmed
- [ ] Admin metrics RPC returns expected payload
- [ ] iOS app build and launch sanity check passes
- [ ] No new crash spike after rollback
Postmortem (within 24h)
- Timeline of incident + root cause.
- Why checks did not catch it.
- Preventive actions:
- add/adjust CI checks
- add synthetic monitor
- adjust runbook steps