My agents wake up with amnesia. Every session starts as a blank context window, and everything the fleet learned yesterday — a deployment gotcha, a library quirk, a decision I would rather not re-litigate — is gone unless something outside the model carries it forward. That something grew into four layers, a sync fabric, and an auditor that re-checks all of it on a schedule.
Layer one: the vector store
The base layer is a self-hosted retrieval server, GoodMem, backed by Postgres with a vector index, Voyage embeddings, and a reranker in front. It currently holds about 2.9 million embedded chunks across 45 GB, organized into 35 spaces. Spaces are the isolation mechanism. Each project gets its own, cross-project engineering lessons get a shared one, and a few sensitive spaces are walled off from general retrieval entirely.
Retrieval is wired into the session itself. When a prompt arrives, a hook embeds it, pulls the top hits across the relevant spaces, reranks, and injects the survivors into context, labeled as hints rather than truth. The distinction matters. A memory is a claim somebody wrote down once. Treat it as ground truth and stale facts slip straight into fresh decisions.
Layers two through four
Above the store sit three sources that feed it. The knowledge vault holds around 800 curated notes: long-form research, runbooks, postmortems. It lives in Obsidian and syncs every ten minutes; notes opt in through frontmatter, and a scope field routes each one to a project space or the shared one. Code memory lives with each repository and loads when an agent activates that project — symbol-level knowledge about the codebase itself. The third source is auto-memory: facts the sessions write down on their own, typed by frontmatter as user preference, feedback, or engineering learning, and routed by type on the same ten-minute cadence.
Write discipline matters more than write volume. Search before writing. Update instead of duplicating. And one writer at a time: parallel subagents cannot see each other's pending writes, so they return learning candidates in their reports and a single gate dedupes and commits them. Without that rule the store fills with five phrasings of the same lesson, and retrieval quality decays accordingly.
Memory needs decay
Freshness is the part I have not seen described often, and it is the part that changed the most in practice. A memory that names a file, a flag, or a config value is a bet that the world has not changed since it was written. So memories carry verification metadata — paths that should exist, a read-only command that should exit clean — and an auditor re-verifies them on a roughly daily cycle. Failures get marked stale with a reason, and stale memories surface as warnings at session start rather than as facts. The failure this deletes is specific: an agent confidently recommending a flag that was renamed last month.
A memory system without decay is a rumor mill with good uptime.
The strange layer: embedded terminal history
The composition of those 2.9 million chunks deserves honesty: most of them, 98.8%, are machine-written terminal history. Every command and its output, embedded and searchable. That space is deliberately excluded from session retrieval, because it would flood every query. It exists for forensics. When something worked five weeks ago and nobody wrote down how, an agent digs the answer out of there with a scoped query in seconds.
What changes when memory works
Debugging stopped repeating. A session hits an obscure deployment error; retrieval surfaces the fix another session wrote down three weeks earlier, with the incident context attached. Twenty minutes becomes forty seconds.