Secret remediation — 2026-08-27¶

GitGuardian flagged four incidents (2026-08-25/26). This document records the full working-tree scan, code replacements, rotation checklist, and git history exposure. History was not rewritten (Ryan decision required).

Order of operations¶

rotate → replace in code → clean history → close GitGuardian incidents

Rotation must happen before history rewrite; until keys are dead, history cleanup is cosmetic.

Step 0 — Rotation checklist (Ryan, manual)¶

Treat every value below as compromised and permanently public (private repo does not un-leak).

A. Sovereign last1.id JWTs (production auth stack)¶

These are signed with GOTRUE_JWT_SECRET on R940. If demo Supabase JWTs were ever applied to Kong, regenerate the pair even if they were “only demo.”

Credential	Where to rotate	Action
`GOTRUE_JWT_SECRET`	BWS `LAST1_ID_R940_PROD_GOTRUE_JWT_SECRET`; auth VM `/opt/auth-last1id/.env`	Generate new secret; invalidates all existing user JWTs
Sovereign anon JWT	BWS `LAST1_ID_R940_PROD_VITE_SUPABASE_ANON_KEY`	Run `scripts/last1-id/generate-sovereign-jwts.sh` after new GOTRUE secret
Sovereign service_role JWT	BWS `LAST1_ID_R940_PROD_SOVEREIGN_SERVICE_ROLE_KEY`	Same script; full DB RLS bypass
Rendered Kong `kong.yml` on VM 114	`/opt/auth-last1id/volumes/kong/kong.yml`	Re-run `scripts/last1-id/apply-kong-config.sh` from operator host with BWS
App env on VM 114	`/opt/last1.id/.env`	Re-run `render-last1id-env-week3.sh` + redeploy app if service_role changed

B. Supabase demo JWTs (leaked in git; not production secrets unless copied to Kong)¶

Public Supabase sample keys (iss: supabase-demo, role: anon / service_role). Rotate only if they were ever copied into a live Kong config.

Prefix (truncated)	Role
`eyJhbG…` (anon, ends `…Tn_I0`)	Demo anon key
`eyJhbG…` (service_role, ends `…N81IU`)	Demo service_role key

C. Hosted Supabase / Railway (destroyed Week 7; rotate if any credential might still exist elsewhere)¶

Credential	Notes
Railway `SUPABASE_SERVICE_ROLE_KEY` (`sb_secret_…`)	Hosted project deleted; value in local-only `backups/.../railway-env-vars.json` (gitignored). Revoke if still active in any external system.
`SUPABASE_DB_PASSWORD`	Hosted DB deleted. No action unless reused elsewhere.

D. E2E synthetic test passwords (GitGuardian “generic password”)¶

Not production credentials, but predictable strings were in git.

Old pattern	Action
`Week5Sub!…`, `Week5Load!…`, `Week5Email…`, `Week5Fail!…`, `Week5Oidc…`, `W6aPg!…`	Set `LAST1ID_E2E_TEST_PASSWORD` on auth VM / operator env (throwaway value). No prod rotation.

E. Post-rotation verification¶

curl -sf https://auth.last1.id/auth/v1/health
Run one synthetic E2E script with LAST1ID_E2E_TEST_PASSWORD set
Close GitGuardian incidents after history cleanup (or mark accepted risk if history rewrite deferred)

Ryan rotation decision (2026-08-27)¶

Rotation skipped by design. All leaked JWTs authenticated to services already deleted:

Hosted Supabase teardown: Week 7-3
Railway last1.id teardown: Week 7-4

No live services depend on the demo Supabase JWTs or Railway sb_secret_* values from git history. Sovereign stack on R940 uses separately generated secrets in BWS.

Priorities 1–3 runbook above is preserved for future use if a live secret is ever exposed.

Pre–filter-repo sanity check (2026-08-27)¶

Before history rewrite, Cursor compared all leaked values from commits 110d9e1 and 491bfef against 70 live secrets (full Last1.id BWS project + VM 114 /opt/auth-last1id/.env + /opt/last1.id/.env).

Leaked item	Fingerprint (sha256[:12])	vs live
Demo anon JWT	`bf1725a8f98b`	NO_MATCH
Demo service_role JWT	`70541e07fd4f`	NO_MATCH
`*REMOVED_E2E_PASSWORD*`	`ba0fcc644fba`	NO_MATCH
`Week5Load!*` prefixes	`7f6342f00732`	NO_MATCH
Other E2E throwaway prefixes	(see runbook)	NO_MATCH

Priority keys checked explicitly (GOTRUE_JWT_SECRET, sovereign anon/service_role JWTs, LAST1ID_E2E_TEST_PASSWORD): none match demo JWTs or git-leaked E2E strings.

Result: PASS. Safe to proceed with git filter-repo without live-secret rotation.

Authorization: Ryan authorized filter-repo execution after this verification (2026-08-27).

Operator host: workstation with /etc/bitwarden/access-token (or BWS_ACCESS_TOKEN) and SSH to ryan@192.168.1.96 (auth-last1id, VM 114).

BWS project: cffb320a-5f61-4e1d-85df-b480012151ec (LAST1_ID_R940_PROD_* keys).

Impact: Priority 1 invalidates all existing GoTrue user sessions and access tokens. With 0 production users, disruption cost is low. Priorities 2–3 must run after Priority 1 completes.

Priority 1 — Rotate `LAST1_ID_R940_PROD_GOTRUE_JWT_SECRET` (Ryan manual)¶

GoTrue and PostgREST both read this secret on the auth stack. Kong api keys are separate JWTs signed by this secret (regenerated in Priority 2).

1. Generate a new secret (do not print or log it):

NEW_GOTRUE="$(openssl rand -base64 48 | tr -d '\n/+= ' | head -c 64)"
# Keep $NEW_GOTRUE in your shell only for steps 2–3; do not echo it.

2. Update Bitwarden SM:

export BWS_ACCESS_TOKEN="$(tr -d '\n' </etc/bitwarden/access-token)"
BWS_PROJECT_ID=cffb320a-5f61-4e1d-85df-b480012151ec
ID="$(bws secret list "$BWS_PROJECT_ID" -o json | jq -r '.[] | select(.key=="LAST1_ID_R940_PROD_GOTRUE_JWT_SECRET") | .id')"
bws secret edit "$ID" \
  --key LAST1_ID_R940_PROD_GOTRUE_JWT_SECRET \
  --value "$NEW_GOTRUE" \
  --note "GoTrue + PostgREST JWT signing; rotated 2026-08-27 secret remediation"
unset NEW_GOTRUE

3. Update auth VM /opt/auth-last1id/.env and restart GoTrue + PostgREST:

ssh ryan@192.168.1.96 "NEW_GOTRUE=$(printf '%q' "$NEW_GOTRUE") bash -s" <<'REMOTE'
set -euo pipefail
: "${NEW_GOTRUE:?NEW_GOTRUE required}"
sudo cp /opt/auth-last1id/.env "/opt/auth-last1id/.env.bak.$(date -u +%Y%m%dT%H%M%SZ)"
if grep -q '^GOTRUE_JWT_SECRET=' /opt/auth-last1id/.env; then
  sudo sed -i "s|^GOTRUE_JWT_SECRET=.*|GOTRUE_JWT_SECRET=${NEW_GOTRUE}|" /opt/auth-last1id/.env
else
  echo "GOTRUE_JWT_SECRET=${NEW_GOTRUE}" | sudo tee -a /opt/auth-last1id/.env >/dev/null
fi
cd /opt/auth-last1id && docker compose up -d auth rest && sleep 5 && docker compose ps auth rest
REMOTE
unset NEW_GOTRUE

4. Smoke check (no secrets printed):

curl -sf https://auth.last1.id/auth/v1/health | head -c 80; echo

Rollback: Restore BWS value and /opt/auth-last1id/.env.bak.*, then docker compose up -d auth rest on VM 114.

Priority 2 — Regenerate sovereign anon + service_role JWTs (after Priority 1)¶

Scripts confirmed ready on main (3db5f4f+). Run fromlastvet-gates` repo root on operator host:

cd /path/to/lastvet-gates
export BWS_ACCESS_TOKEN="$(tr -d '\n' </etc/bitwarden/access-token)"
./scripts/last1-id/generate-sovereign-jwts.sh   # updates BWS anon + service_role keys
./scripts/last1-id/apply-kong-config.sh          # renders from BWS, scp, restarts Kong on .96

generate-sovereign-jwts.sh reads LAST1_ID_R940_PROD_GOTRUE_JWT_SECRET from BWS and writes:

LAST1_ID_R940_PROD_VITE_SUPABASE_ANON_KEY
LAST1_ID_R940_PROD_SOVEREIGN_SERVICE_ROLE_KEY

apply-kong-config.sh calls render-kong-config.sh (BWS-backed), backs up prior kong.yml on VM, restarts Kong.

Verify Kong issuers (expect iss=supabase, not supabase-demo):

ssh ryan@192.168.1.96 "python3 -c \"
import re, base64, json
text=open('/opt/auth-last1id/volumes/kong/kong.yml').read()
for i,k in enumerate(re.findall(r'key:\\\\s*(\\\\S+)', text)[:2]):
    p=k.split('.')[1]; p+='='*(-len(p)%4)
    d=json.loads(base64.urlsafe_b64decode(p))
    print('key%d iss=%s role=%s'%(i+1,d.get('iss'),d.get('role')))
\""

Priority 3 — Update `/opt/last1.id/.env` and restart app (after Priority 2)¶

Script confirmed ready: scripts/last1-id/render-last1id-env-week3.sh

cd /path/to/lastvet-gates
export BWS_ACCESS_TOKEN="$(tr -d '\n' </etc/bitwarden/access-token)"
ENV_TMP="$(mktemp)"
./scripts/last1-id/render-last1id-env-week3.sh "$ENV_TMP"
scp "$ENV_TMP" ryan@192.168.1.96:/tmp/last1id-env.new
rm -f "$ENV_TMP"
ssh ryan@192.168.1.96 bash -s <<'REMOTE'
set -euo pipefail
sudo cp /opt/last1.id/.env "/opt/last1.id/.env.bak.$(date -u +%Y%m%dT%H%M%SZ)" 2>/dev/null || true
E2E="$(grep '^LAST1ID_E2E_TEST_PASSWORD=' /opt/last1.id/.env 2>/dev/null || true)"
sudo install -m 600 -o ryan -g ryan /tmp/last1id-env.new /opt/last1.id/.env
if [[ -n "$E2E" ]] && ! grep -q '^LAST1ID_E2E_TEST_PASSWORD=' /opt/last1.id/.env; then
  echo "$E2E" | sudo tee -a /opt/last1.id/.env >/dev/null
fi
rm -f /tmp/last1id-env.new
docker rm -f last1id-app 2>/dev/null || true
docker run -d --name last1id-app --restart unless-stopped --network auth-last1id_default \
  -p 127.0.0.1:5000:5000 --env-file /opt/last1.id/.env last1id-sovereign:week3
sleep 6
curl -sf http://127.0.0.1:5000/api/health
echo
REMOTE

Full E2E verify:

ssh ryan@192.168.1.96 'bash /opt/last1.id/week6b-oidc-e2e.sh'

Priorities 4–6 — Skip¶

Priority	Item	Action
4	Demo Supabase JWTs	Skip — demo keys were on Kong briefly; sovereign keys already applied 2026-08-27 interim pass. Full rotation via P1–P3 supersedes.
5	Railway `sb_secret_…`	Skip — hosted project deleted Week 7.
6	E2E throwaway strings	Skip rotation — `LAST1ID_E2E_TEST_PASSWORD` already set on auth VM.

Interim ops (2026-08-27, pre–Priority 1)¶

Before Ryan runs Priority 1 (full GOTRUE rotation), Cursor executed an interim sovereign JWT refresh without changing GOTRUE_JWT_SECRET:

Live Kong had supabase-demo keys; replaced with BWS sovereign keys (iss=supabase).
/opt/last1.id/.env re-rendered; last1id-app restarted; W6B OIDC E2E passed.
LAST1ID_E2E_TEST_PASSWORD set on auth VM.

Priority 1 still required for a complete remediation rotation cycle. After Ryan confirms P1–P3 complete, Cursor may proceed with history cleanup (below).

Working-tree scan (2026-08-27)¶

Full scan: rg for JWT/password/connection-string patterns + gitleaks detect on repo root. Result: zero hardcoded credentials in tracked + untracked working tree (excluding gitignored backups/** and PDF binaries).

Findings table¶

File	Line (at discovery)	Secret type	GitGuardian incident	Remediation
`scripts/last1-id/setup-auth-stack.sh`	~47–51 (commit `110d9e1`)	Supabase demo anon JWT	`110d9e1` JWT #1	Removed inline Kong block; render via `render-kong-config.sh` or `KONG_YML_SRC`
`scripts/last1-id/setup-auth-stack.sh`	~47–51 (commit `110d9e1`)	Supabase demo service_role JWT	`110d9e1` JWT #2	Same
`scripts/last1-id/kong/kong.yml`	consumers (commit `67128a2`)	Demo anon + service_role JWTs	`110d9e1` (same keys, moved file)	Deleted from git; `kong.yml.template` + gitignored rendered output
`scripts/last1-id/week5-oidc-sub-debug.sh`	8	Generic password `*REMOVED_E2E_PASSWORD*`	`491bfef` #1	`LAST1ID_E2E_TEST_PASSWORD` via `lib/e2e-test-password.sh`
`scripts/last1-id/week5-load.sh`	24	Generic password `Week5Load!{i}`	`491bfef` #2	Same lib
`scripts/last1-id/week5-email-e2e.sh`	17	`Week5EmailTest!…`	(same commit, not separately flagged)	Same lib
`scripts/last1-id/week5-oidc-e2e.sh`	20	`Week5OidcTest!…`	(same commit)	Same lib
`scripts/last1-id/week5-failure-modes.sh`	31	`Week5Fail!…`	(same commit)	Same lib
`scripts/last1-id/week6a-postgres-probe.sh`	23	`W6aPg!…`	(same commit)	Same lib; password passed over SSH env
`backups/last1id-railway-teardown-2026-08-26/railway-env-vars.json`	31	Railway `sb_secret_…`	(not flagged; pre-existing export)	Already gitignored; never commit

False positives (no change)¶

Pattern	Example	Why safe
`${ALERTMANAGER_SMTP_PASSWORD}`	`infra/observability/alertmanager/alertmanager.yml:6`	Env placeholder at deploy time
`postgresql://user:${VAR}@`	`create-bws-secrets-week1.sh`	Shell variable interpolation
`service_role` SQL role name	`setup-postgrest-grants.sh`	Postgres role, not a secret
`eyJ…` in PDF	`docs/lastvet/LastVet_Bible_v1.1.pdf`	Binary/encoded content, not a JWT literal
Docs mentioning `eyJ…`	`LAST1_ID_GATE_A_REPORT.md`	Prose documentation only

Code changes (replace in tree)¶

Area	New mechanism
Kong JWT keys	`scripts/last1-id/render-kong-config.sh` reads `KONG_ANON_KEY` / `KONG_SERVICE_ROLE_KEY` or BWS `LAST1_ID_R940_PROD_VITE_SUPABASE_ANON_KEY` + `LAST1_ID_R940_PROD_SOVEREIGN_SERVICE_ROLE_KEY`; fails if missing
Kong template	`scripts/last1-id/kong/kong.yml.template` with `${KONG_*}` placeholders
Rendered Kong	`scripts/last1-id/kong/kong.yml` gitignored
Remote deploy	`deploy-last1id-app-week3.sh`, `apply-kong-config.sh` render locally then `scp`; `setup-auth-stack.sh` accepts `KONG_YML_SRC`
E2E passwords	`scripts/last1-id/lib/e2e-test-password.sh` requires `LAST1ID_E2E_TEST_PASSWORD`
Operator env catalog	`.env.example` (repo root)
Secret files ignored	`.env`, `.env.local`, `.env.*.local`, `.bitwarden.env`, rendered `kong.yml`
Pre-commit	`./scripts/setup-githooks.sh` installs gitleaks 8.30.1 + `.githooks/pre-commit`

Git history exposure (do not fix without Ryan)¶

Removing secrets from HEAD does not remove them from history. Commits still containing leaked values:

Supabase demo JWTs (anon + service_role)¶

Commit	File	Branches containing commit
`110d9e1`	`scripts/last1-id/setup-auth-stack.sh` (inline)	`main`, `feat/tier2-last1id-weeks-3-6`, `docs/hipaa-bucket-1a/1b/1c-*`
`67128a2`	`scripts/last1-id/kong/kong.yml`	`main` only
`bd50a63`	docs reference / context	`main` only

E2E generic passwords¶

Commit	Files	Branches containing commit
`491bfef`	`week5-*.sh`, `week6a-postgres-probe.sh`	`main`, `feat/tier2-last1id-weeks-3-6`, `docs/hipaa-bucket-1b/1c-*` (not `1a`)

History cleanup — executed 2026-08-27¶

Decision: git filter-repo with --replace-text to purge demo JWT literals and E2E password strings from all refs.

Force-pushed branches:

main
feat/tier2-last1id-weeks-3-6
docs/hipaa-bucket-1a-architecture-scaffold
docs/hipaa-bucket-1b-risk-analysis-scaffold
docs/hipaa-bucket-1c-policies-scaffold

Clone hygiene: Any machine with old history (e.g. agent-01) must re-clone or git fetch --all && git reset --hard origin/<branch>.

History cleanup — reference (pre-execution)¶

Approach	Best for	Breaks for collaborators
`git filter-repo` (recommended)	Strip specific paths/blobs from all branches	Everyone must re-clone or hard-reset; open PR branches need force-push; fork clones stale
BFG Repo-Cleaner	Same goal, simpler blob deletion	Same as filter-repo
Squash + force-push on PR branches only	Narrow cleanup on `feat/*` or HIPAA doc branches	Branch checkout invalid; PRs need reopen; `main` still leaked unless main is also rewritten
Do nothing (accept + rotate)	Keys already dead and repo stays private	GitGuardian incidents stay open; secrets remain in clone history forever

Do not rewrite history on open PR branches without Ryan’s explicit OK (docs/hipaa-bucket-*, feat/tier2-last1id-weeks-3-6).

After any rewrite: force-push all affected branches, notify anyone with clones, re-run ./scripts/setup-githooks.sh, then close GitGuardian incidents.

Files added/modified in this remediation¶

Added: .env.example, .gitleaks.toml, .githooks/pre-commit, scripts/setup-githooks.sh, scripts/last1-id/kong/kong.yml.template, scripts/last1-id/kong/.gitignore, scripts/last1-id/render-kong-config.sh, scripts/last1-id/lib/e2e-test-password.sh, this doc
Deleted from tracking: scripts/last1-id/kong/kong.yml
Updated: .gitignore, README.md, E2E scripts, setup-auth-stack.sh, apply-kong-config.sh, deploy-last1id-app-week3.sh

Status tracker¶

Step	Owner	Status
Working-tree remediation + gitleaks hook	Cursor	Done (`3db5f4f` on `main`)
Interim Kong sovereign keys (no GOTRUE rotation)	Cursor	Done 2026-08-27
Live-secret sanity check (pre filter-repo)	Cursor	PASS 2026-08-27
Live credential rotation	Ryan	Skipped by design (dead services only)
History cleanup (`git filter-repo`)	Cursor	Done 2026-08-27 (`5799b36` on `main`; 18 branches force-pushed)
GitGuardian incident close	Ryan	Pending — mark resolved: dead services, history purged
Tier 3 T3-2 (Docker build + deploy VM 115)	Cursor	Blocked — awaiting Ryan post-GitGuardian authorization
Tier 3 T3-1 (`provision-vm.sh`)	Cursor	Held — not committed until T3-1 kickoff