Sovereign v2 — Session 3 notes (MemPalace + pgvector)¶

Date: 2026-08-20
Status: Mem/data plane complete. E executed. Session 3b (J1–J3) executed. Recall E2E working. Ready for Ryan review + merge decision.
Branch: feat/sovereign-v2-session3 (do not merge to main until Ryan review).
Canonical scope: SOVEREIGN_V2_SESSION1_DISCOVERY.md

Architecture note (important)¶

Session 1 discovery wires memory as:

sovereign-app  --MEMPALACE_PGVECTOR_DSN-->  sovereign-mem:5432

MemPalace upstream is CLI / library / MCP (stdio). It has no first-party HTTP API.

Session 3 therefore ships:

Postgres 17 + pgvector on sovereign-mem (canonical data plane)
MemPalace install at /opt/sovereign-mempalace with a thin FastAPI wrapper (health_api.py) on :7200 for health, metrics, and smoke write/search
Odysseus still uses native ChromaDB on :8100 for VectorRAG / MemoryVectorStore / ToolIndex. That noise is not MemPalace. Clearing it requires Chroma on sovereign-app (or disabling those paths). Not done in Session 3.

Sub-task A — VM preflight¶

Check	Result
Host	`sovereign-mem` VM 105, Ubuntu 24.04.4 LTS
LAN / Tailscale	`192.168.1.71` / `100.125.49.26`
RAM / disk	31 GiB / ~81G free on `/`
Residue	`/opt` empty; no prior MemPalace/Postgres
Reachability	sovereign-app + inference-01 Ollama OK
Obs agents	node_exporter `:9100`, promtail `:9080` already present

Sub-task B — Postgres + pgvector¶

Item	Value
Version	Postgres 17.11 (matches lastvet-db PGDG pattern)
Extension	`vector` 0.8.6 (+ `pg_stat_statements`)
Database	`sovereign_mem`
Roles	`sovereign_mem_app` (app), `sovereign_mem_admin` (migrate), `prometheus_exporter`
SSL	`ssl = on` (snakeoil); clients use `sslmode=require`
Listen	`*` (restricted by `pg_hba`)
`pg_hba`	`hostssl` app from `192.168.1.70` + `100.109.119.54`; local mem LAN/TS/loopback; admin also from agent-01 `192.168.1.30`; reject all else to this DB

Verified: SSL connect from sovereign-app via LAN and Tailscale MagicDNS.

Sub-task C — MemPalace application¶

Item	Value
Source	`thelast1org/mempalace` (`develop`, org mirror of ~3.4.1)
Path	`/opt/sovereign-mempalace`
User	`sovereign-mem` (system)
Data	`/var/lib/sovereign-mempalace/palace` (local KG markers; vectors in Postgres)
Env	`/etc/sovereign/mempalace.env` (mode 640, group `sovereign-mem`)
Unit	`sovereign-mempalace.service` (`Restart=always`)
Health	`http://sovereign-mem.tailc3bbdc.ts.net:7200/health`
Metrics	`http://192.168.1.71:7200/metrics`
Write/search	`POST /v1/memory`, `POST /v1/search`

Health evidence:

{"status": "healthy", "backend": "pgvector", "postgres": {"ok": true, "detail": "vector=0.8.6"}, "drawers": 1, "error": null}

Sub-task D — BWS (project `sovereign`)¶

Key	Secret id prefix	Notes
`SOVEREIGN_R940_PROD_MEMPALACE_DATABASE_URL`	`8280c098`	LAN host `192.168.1.71`, `sslmode=require`
`SOVEREIGN_R940_PROD_MEMPALACE_PGVECTOR_DSN`	`3eff3707`	Same value (MemPalace env name)
`SOVEREIGN_R940_PROD_MEMPALACE_ADMIN_PASSWORD`	`c2b5d57b`	`sovereign_mem_admin`
`SOVEREIGN_R940_PROD_MEMPALACE_API_URL`	`f6260e77`	Tailscale `:7200`
`SOVEREIGN_R940_PROD_MEMPALACE_API_URL_LAN`	`64110f49`	LAN `:7200`

No non-MEMPALACE_* BWS writes in this session.

Sub-task E — Odysseus reconnection (DONE)¶

Ryan decisions (Q1 APPROVED, Q2 DEFER Chroma): applied 2026-08-20.

Env added to `/etc/sovereign/odysseus.env` (backup `.bak.s3e-`)¶

Var	Value
`MEMPALACE_BACKEND`	`pgvector`
`MEMPALACE_BACKEND_EXPLICIT`	`pgvector`
`MEMPALACE_PGVECTOR_DSN`	LAN host `192.168.1.71`, DB `sovereign_mem`, `sslmode=require` (from BWS)
`MEMPALACE_PALACE_PATH` / `SOVEREIGN_PALACE_PATH`	`/var/lib/sovereign/odysseus/palace`
`SOVEREIGN_MEMPALACE_AUTO_RECALL`	`1`
`ODYSSEUS_SKIP_BUILTIN_MCP`	`memory,rag`

Verification after `systemctl restart odysseus`¶

Check	Result
Unit	`active`
`GET /api/health`	`healthy`
Chat PONG 7B / 24B	Pass (short prompt; see 3b note on recall-triggering prompts)
Builtin Memory/RAG MCP	Skipped (INFO), no boot connection errors for those two
Native Chroma `:8100` WARNING	Still present (ToolIndex / VectorRAG) — expected under Q2 defer
Email / image_gen MCP	Still fail — pre-existing, out of scope

No chat regression gate tripped. Proceeded to Session 3b.

Why a naive “point at MemPalace URL” still does not clear all Session 2 noise¶

Odysseus log	Root cause	Cleared by E?
Builtin MCP Memory/RAG connection closed	MCP wrappers over Odysseus Chroma	Yes (skipped)
`ChromaDB … localhost:8100`	Native VectorRAG / MemoryVectorStore / ToolIndex	No (deferred)
MemPalace recall silent	Missing script + `uv` runner	No (fixed in 3b)

Sub-task F — Observability¶

Piece	Status
node_exporter	Already scraped (`192.168.1.71:9100` up)
postgres_exporter	`:9187` on mem; Prometheus job target up
MemPalace `/metrics`	job `sovereign_mempalace` up
Alerts	`MemPalaceDown`, `SovereignMemPostgresExporterDown` in `critical.yml`
Grafana	Dashboard uid `sovereign-mempalace` provisioned
Loki	Promtail journal job active on mem (unit logs via `__journal__systemd_unit`)

Sub-task G — Smoke tests¶

Test	Result
`GET /health`	200 healthy
`POST /v1/memory`	id `api-54f685feda204789`
`POST /v1/search` query `ALPHA-S3-VERIFY`	1 hit, similarity ~0.665
`systemctl restart sovereign-mempalace`	recovers; drawers=1
pgvector table	`mempalace_*_mempalace_drawers` owned by `sovereign_mem_app`

Operator commands¶

ssh sovereign-mem 'systemctl status sovereign-mempalace postgresql postgres_exporter --no-pager'
curl -sS http://sovereign-mem.tailc3bbdc.ts.net:7200/health
curl -sS http://192.168.1.71:7200/metrics | head

Secrets: BWS project sovereign → keys above. Local agent copies (gitignored): lastvet-gates/.local/sovereign-mem-*.

Investigation I1–I3 — memory systems (read-only, 2026-08-20)¶

Scope: Odysseus source on thelast1org/odysseus (clone used for Session 2 + live /opt/sovereign/odysseus). No VM or code changes.

Verdict (one paragraph)¶

v1 ran three different systems, not one redundant memory. MemPalace is the Sovereign long-term “brain” (wings/rooms/drawers, auto-recall into chat). Odysseus native memory is a separate short fact store (memory.json + optional Chroma vectors). Odysseus RAG is a document corpus index (personal dirs/files), also on Chroma. Built-in Memory/RAG MCP servers are thin wrappers over those native stores, not MemPalace. Session 2 “Chroma unreachable” noise is Odysseus HTTP Chroma (:8100), not MemPalace. For v2.0 MVP: MemPalace-primary via pgvector DSN + fix recall bridge; keep memory.json without Chroma; skip installing Chroma unless you need document RAG / tool-index RAG soon.

I1 — What builtin RAG MCP and Memory MCP do¶

RAG MCP (`mcp_servers/rag_server.py`)¶

Aspect	Detail
Tool	`manage_rag` with actions `list` / `add_directory` / `remove_directory`
Stores	Filesystem documents indexed into Chroma collection `odysseus_rag` (via `VectorRAG` / personal docs manager)
Does not store	Chat history, user facts, MemPalace drawers
Data model	Chunked file text + metadata (`source` path, `filename`, `owner`)
Depends on	HTTP Chroma at `CHROMADB_HOST:CHROMADB_PORT` (default `localhost:8100`)

Also used in chat path (chat_processor.build_context_preface): when RAG is healthy, injects “Relevant documents…” into the prompt. Without Chroma, that path no-ops (warning only).

Memory MCP (`mcp_servers/memory_server.py`)¶

Aspect	Detail
Tool	`manage_memory` with `list` / `add` / `edit` / `delete` / `search`
Categories	`fact`, `event`, `contact`, `preference`
System of record	`memory.json` under Odysseus data dir (`MemoryManager`)
Optional vector layer	`MemoryVectorStore` → Chroma collection `odysseus_memories` (same HTTP Chroma)
Does not store	Full chat transcripts (sessions live in SQLite/`app.db`); not MemPalace

Native HTTP API /api/memory/* hits the same MemoryManager. Chat injection uses pinned facts + BM25 hybrid retrieve over memory.json; vector lane is optional enhancement. Write/list/search work without Chroma.

There is also a native agent tool manage_memory in tool_schemas.py (always-available). Builtin Memory MCP is a parallel stdio surface over the same store. Skipping the MCP via ODYSSEUS_SKIP_BUILTIN_MCP=memory,rag does not remove native memory.

How MemPalace differs¶

	MemPalace	Native memory	Native RAG
Role	Long-term structured palace (wings/rooms/drawers, hub news, property agents)	Short user facts / prefs	Document corpus retrieval
Storage (v1)	MemPalace Chroma on disk under palace path + KG SQLite	`memory.json` (+ optional Odysseus HTTP Chroma)	Odysseus HTTP Chroma `odysseus_rag`
Storage (v2 Session 3)	pgvector on `sovereign-mem`	unchanged	still wants `:8100` if enabled
Chat wiring	`src/mempalace_recall.py` injects trusted context before user turn	`chat_processor` preface	same preface
UI	Sovereign “Brain” / wing cards	Odysseus memory panel `/api/memory`	personal docs / RAG

Answer: RAG MCP + Memory MCP + MemPalace are three different things. Naming overlap (“memory”) caused the Session 2 noise to be misread as “MemPalace down.”

Fourth Chroma consumer (not MCP)¶

ToolIndex embeds tool descriptions into Chroma collection odysseus_tool_index for agent tool RAG. If Chroma is down, agent loop falls back to keyword / ALWAYS_AVAILABLE (includes manage_memory). Degraded, not broken.

I2 — v1 archaeology¶

What Chroma held in v1 (two separate Chroma worlds):

MemPalace’s own Chroma (default MemPalace backend): verbatim drawers under %USERPROFILE%\.mempalace\palace\ (or similar). This is what unicorn held as the “brain.” Session 1 decision: discard; fresh palace on pgvector.
Odysseus HTTP Chroma (docker compose service on :8100): collections odysseus_rag, odysseus_memories, odysseus_tool_index. Sidecar for Odysseus features, not the palace.

Short-term vs long-term?

Layer	v1 role
Chat session messages	Short-term conversation (DB), not Chroma
`memory.json`	Durable short facts (remember X)
MemPalace	Long-term knowledge palace; auto-recall + property wings
Odysseus RAG Chroma	Optional document library
Tool-index Chroma	Optional agent convenience

So: Chroma-on-:8100 was not “session memory.” MemPalace-on-disk Chroma was the long-term store. Odysseus HTTP Chroma was partially overlapping in spirit with MemPalace for “things to recall,” but different schemas and product surfaces. Discovery already flagged “dual native Odysseus memory + MemPalace without clear boundary” as rewrite/collapse work.

v1 bridge status (still true on live app):

src/mempalace_recall.py exists and is hooked from routes/chat_helpers.py.
It shells out via uv tool run --from mempalace==3.4.1 to scripts/mempalace-recall-query.py.
That script is missing from the repo (confirmed on clone and live /opt/sovereign/odysseus/scripts/). Auto-recall currently no-ops (debug log only). Hub stacks use the same missing-script pattern.
Env: SOVEREIGN_MEMPALACE_AUTO_RECALL (default on), MEMPALACE_PALACE_PATH / SOVEREIGN_PALACE_PATH.

I3 — v2.0 data path recommendation¶

Options¶

Option	What it means	MVP simplicity	Long-term
A. Dual-write Chroma + MemPalace	Keep Odysseus Chroma + MemPalace	Worst: two vector stacks, two failure modes	Avoid
B. MemPalace-only (DSN)	App talks Postgres via `MEMPALACE_PGVECTOR_DSN`; fix recall bridge in-process or commit missing script	Best for Sovereign brain	Matches discovery; maintainable
C. MemPalace via `:7200` wrapper	Odysseus HTTP to Session 3 health API	Easy smoke; wrong long-term (wrapper is not upstream MemPalace)	Use for ops only, not primary write path
D. Install Chroma on sovereign-app	Clears boot noise; enables doc RAG + vector memory + tool index	Medium ops cost	Keep only if you need those features soon

Recommended for v2.0 MVP¶

Primary long-term memory: MemPalace → pgvector DSN (Option B). Wire env on Odysseus: MEMPALACE_BACKEND=pgvector, MEMPALACE_PGVECTOR_DSN from BWS, local palace path for KG markers. Fix recall (commit adapter or in-process search_memories) so E2E write/recall works. Prefer DSN over :7200 for the product path.
Keep memory.json for explicit “remember this” short facts. No Chroma required (BM25 path already works).
Do not install Chroma for MVP unless Ryan needs personal-doc RAG soon. Silence boot noise with:
ODYSSEUS_SKIP_BUILTIN_MCP=memory,rag (and optionally image_gen,email if unused), and/or
accept degraded VectorRAG / MemoryVectorStore / ToolIndex (already designed for degradation).
Do not dual-write facts into both native memory and MemPalace by default. Policy: MemPalace = durable palace; memory.json = lightweight prefs until a later consolidation pass (discovery “collapse”).
Park document RAG (Odysseus Chroma) until after Mac chat + MemPalace recall are solid. Open question #2 in SOVEREIGN_ARCHITECTURE_V2.md stays open for v2.1.

What Sub-task E should do (when approved)¶

Minimal stability-safe E:

Set MemPalace env (DSN + backend + palace path). Does not require Chroma.
Skip builtin memory,rag MCP to clear those connection errors (native tools remain).
Restart Odysseus; verify chat PONG still works.
Separately (same session or follow-up): land recall-script fix so MemPalace auto-recall actually fires. Without that, E alone wires config but recall stays silent.

Installing Chroma is optional, not a prerequisite for E or MemPalace.

Ryan decisions (locked this session)¶

Q	Decision
Q1 / E	APPROVED — DSN + restart
Q2 / Chroma	DEFER — `ODYSSEUS_SKIP_BUILTIN_MCP=memory,rag`
Q3 / Recall	Session 3b immediately after E

Remaining: approve merge of feat/sovereign-v2-session3 to main after review.

Session 3b — Recall bridge (J1–J4)¶

J1 — Contract investigation¶

Original scripts/mempalace-recall-query.py was never committed (missing from Odysseus clone and live tree). Contract reconstructed from callers in src/mempalace_recall.py and src/hub_stacks.py:

Command	Args	Expected JSON
`search`	`--palace` `--query` `--limit` `[--wing]`	`{"results":[{text,wing,room,source_file,similarity,...}]}`
`wakeup`	`--palace`	`{"text":"..."}`
`list-drawers`	`--palace` `--wing` `[--room]` `--limit`	`{"drawers":[...]}`
`add-drawer`	`--palace` `--wing` `--room` `--content` `[--source]` `[--added-by]`	success dict

Historical runner: uv tool run --from mempalace==3.4.1 python <script> …
Live blockers before 3b: script missing; uv not installed; mempalace not in Odysseus venv.

Also found path bug: _RECALL_SCRIPT used Path(__file__).parents[2] → /opt/sovereign/scripts/… instead of repo-root parents[1] → /opt/sovereign/odysseus/scripts/….

J2 — Implementation (live on `sovereign-app`)¶

Installed mempalace[pgvector]==3.4.1 into /opt/sovereign/odysseus/venv.
Wrote /opt/sovereign/odysseus/scripts/mempalace-recall-query.py (review copy: artifacts/mempalace-recall-query.py). Uses MEMPALACE_PGVECTOR_DSN + search_memories / get_collection / MemoryStack.wake_up.
Patched src/mempalace_recall.py and src/hub_stacks.py:
Fix script path to parents[1]
Prefer venv python (SOVEREIGN_MEMPALACE_PYTHON override) when uv absent
Pass through process env (DSN)
Backups: *.bak.s3b
Created palace wing dirs under /var/lib/sovereign/odysseus/palace/wings/{journal,sovereign,sov_hub}.

Palace namespace note: MemPalace HTTP API on mem (palace_id=/var/lib/sovereign-mempalace/palace, DSN host 127.0.0.1) and Odysseus (palace_id=/var/lib/sovereign/odysseus/palace, DSN host 192.168.1.71) use different pgvector table prefixes. Product write/recall path is Odysseus DSN. :7200 remains ops/smoke only.

J3 — E2E tests¶

Test	Result
Script `add-drawer` canary `ALPHA-S3B-RECALL-7F2A`	OK
Script `search` relevant query	Hit sim ~0.82
In-process `build_recall_context`	Injects canary (sim ≥ 0.28)
Chat auto-recall (log)	`MemPalace auto-recall injected (… chars, 1 hits context)`
Chat answer quotes canary	Model returned `ALPHA-S3B-RECALL-7F2A`
Trivial skip (`thanks` / short)	No recall context
Empty wing filter	No context (`NONE`)
Health after restart	`healthy`
Short-prompt PONG 7B / 24B	Pass (`PONG!` / `PONG!`)

Note: Prompts ≥12 chars that are not in the trivial skip list trigger auto-recall. A long “reply with exactly PONG…” can inject palace context and confuse the 7B. Use a short prompt for pure PONG regression.

Weak-similarity fallback in existing _merge_hits (if no hit ≥0.28, still returns top weak hits) is unchanged upstream behavior; empty-result / skip paths no-op cleanly.

J4 — Docs¶

This section. Artifact script checked into the branch for review. Live Odysseus tree on app is not a git checkout; promote patches into thelast1org/odysseus (or Sovereign overlay) in a follow-up if Ryan wants durable source control beyond the R940 host.