Skip to content

Onset Monitoring & Alerts

Core Signals

1) Backend Health

  • Edge function success rate
  • daily-insight 2xx rate
  • generate-protocol 2xx rate
  • client-event 2xx rate
  • Error buckets
  • 401 auth failures
  • 429 rate limits
  • 5xx model/provider failures

2) Product Funnel Signals

From product_events and get_admin_dashboard_metrics:

  • paywallShown
  • paywallPurchaseCompleted
  • firstCheckInCompleted
  • premiumPreviewViewed
  • conversion ratio (paywallPurchaseCompleted / paywallShown)

3) AI Quality / Reliability

  • daily-insight error rate and median latency
  • model fallback frequency
  • safety validation failure rate
  • SEV-1
  • daily-insight 5xx > 20% for 10 minutes
  • generate-protocol 5xx > 20% for 10 minutes
  • launch crash spike above baseline after release
  • SEV-2
  • 401 auth failures > 5% for 15 minutes
  • 429 rate limits > expected baseline for 30 minutes
  • SEV-3
  • conversion drop > 20% day-over-day without campaign/context explanation

Synthetic Checks

Run every 5-15 minutes from CI or external monitor:

  1. Anonymous auth signup
  2. client-event write check
  3. daily-insight request check
  4. get_admin_dashboard_metrics response shape check

Use:

python3 scripts/supabase-smoke.py --project-ref <project_ref>

Destinations Configuration

Alert routing is implemented in onset-web:

  • Endpoint: POST /api/admin/alerts
  • Channels supported:
  • Email (SendGrid): ALERT_EMAIL_TO
  • Slack webhook: ALERT_SLACK_WEBHOOK_URL
  • PagerDuty Events v2: PAGERDUTY_ROUTING_KEY

Required auth/env in onset-web:

  • ONSET_ADMIN_ALERTS_TOKEN
  • ONSET_API_URL
  • ONSET_API_ANON_KEY
  • ADMIN_DASHBOARD_SECRET
  • SENDGRID_API_KEY

Optional email cadence env:

  • ALERT_EMAIL_WEEKLY_SUNDAY_ONLY (true by default)
  • When true, alert emails send only on the configured weekly day (UTC).
  • ALERT_EMAIL_WEEKLY_DAY_UTC (0 by default)
  • Day of week for weekly email sends (0=Sunday, 1=Monday, ... 6=Saturday).

Optional AI abuse override env (Supabase Edge functions):

  • ONSET_INTERNAL_OVERRIDE_ENABLED (true by default)
  • Master switch for internal tester override behavior.
  • ONSET_ADMIN_OVERRIDE_USER_IDS (comma-separated UUIDs)
  • Users in this list bypass AI cooldown/daily-cap/temporary lock checks.
  • ONSET_ADMIN_OVERRIDE_EMAILS (comma-separated lowercase emails)
  • Email-based bypass for internal QA/tester accounts.

Progressive lock behavior:

  • Repeated AI rate-limit hits within a short window now trigger a temporary lockout.
  • Current default: lock for 15 minutes after 4+ limit hits in 10 minutes.
  • Applies to generate-protocol, daily-insight, and simulate-protocol-lab.

Operational Cadence

  • Daily: review dashboard metrics and error logs.
  • Weekly: review funnel shifts and alert noise; tune thresholds.
  • Per release: run scripts/release-readiness.sh and archive output.