Skip to content

MERIDIAN — M0 KICKOFF (eval harness slice)

Confidential — Last 1 Enterprises trade secret

Paste into Cursor in the new, empty, PRIVATE repo: last1-meridian

We're starting Meridian, the Last 1 health intelligence layer. This repo is new and private. First slice: the EVALUATION HARNESS — it precedes and gates everything else, including the model choice. PLAN FIRST — propose, STOP before code.

STEP 1 — Read the two specs I'm providing: - MERIDIAN_SPEC.md (architecture, hard rules, M-gates) - MERIDIAN_EVAL_SPEC.md (the decided safety bar — your requirements document) Every doc you create in this repo carries the header: "Confidential — Last 1 Enterprises trade secret."

STEP 2 — Propose the repo scaffold + harness design:

  1. Stack: propose it (lean Python — the ML ecosystem this grows into for M3 fine-tuning and specialized models — but argue if you disagree).
  2. MeridianModelProvider interface: the swappable model boundary from MERIDIAN_SPEC. For M0, one dev adapter to an API model is PERMITTED because all data is synthetic — plus a trivial MockProvider so the harness runs with zero external calls in CI.
  3. Fixtures: snapshot the VA sandbox synthetic personas' FHIR bundles into versioned fixture files (fetch via the sandbox health API using existing sandbox credentials — coordinate with me for values; NEVER commit credentials, use env). Fixtures are the golden set's ground truth.
  4. Case format: suites A (grounding), B (red team), C (utility) as declarative case files (YAML/JSON): id, suite, category, tier, pressure-variant, prompt(s) (multi-turn supported — the drift variants require it), expected behavior, grading method. Authoring ~30 seed cases per suite is part of this slice — flag any category where you can't write credible cases and I'll supply them.
  5. Runner + graders: deterministic graders (citation-presence structural check, crisis template match, scope check), LLM-judge grader with versioned judge prompts
  6. audit-sampling hooks (10% per MERIDIAN_EVAL_SPEC), human-review queue output for every red-team failure.
  7. Threshold governance: eval-config.yaml (all thresholds, versioned) + THRESHOLD_LOG.md machinery — a threshold change without a log entry should fail CI. Reports: versioned, archived per run (config + model/prompt versions + per-suite/per-tier results incl. Tier-2 over-refusal separately + failure transcripts + threshold log). Never overwritten.
  8. Regression rule: case files are append-only in spirit — build the convention (cases marked superseded, never deleted).

GUARDRAILS: - SYNTHETIC DATA ONLY in this repo until the PHI hosting gate clears (MERIDIAN_SPEC hard rule 5/7). No real veteran data, ever, in fixtures or logs. - The harness is model-agnostic. Nothing above MeridianModelProvider may know which model runs. - No RAG in this slice — that's slice 2. The harness must be able to test a provider WITH or WITHOUT retrieval attached. - No veteran-facing anything. No UI. This is instrumentation. - Zero-tolerance semantics are literal: the report must render ONE crisis/diagnosis/ treatment/scope/injection failure as M1-BLOCKING, unmissably. - Smallest harness that runs all three suites end-to-end against MockProvider in CI and against the dev API adapter locally.

STEP 3 — STOP and present: scaffold tree, case-file schema, grader designs, report format mockup, and the ~90 seed cases list (titles only). Wait for approval.