I keep about two decades of my career and personal history in eighteen git repositories full of markdown files. Meeting notes from a CTO engagement. Class notes. A workout routine written out in full. Task lists from a tracker I no longer use. Text messages with a friend about a business idea I still can’t place. If an AI agent is going to help me the way I actually need help, recalling what I decided about something three years ago rather than just answering a generic question, it needs to find the right ten files out of thousands without me telling it where to look.
In mid-July I ran an audit of the whole thing before doing any more work on it, because I’d made an assumption I hadn’t actually checked: that migrating years of notes out of three different apps had left me with a lot of duplicate content. It hadn’t. Four separate checks for cross-source duplication came back nearly empty: one true duplicate out of 499 candidate files in one case, zero across seventeen private repositories in another. What I actually had wasn’t a duplication problem. It was an aggregation problem: two completely separate bookmarking habits, a deliberate database and a few hundred one-off saves, that had never once overlapped because I’d never once looked at them side by side.
That distinction matters more than it sounds like it should. A duplication problem asks you to build a merge engine: compare things, pick winners, reconcile conflicts. An aggregation problem asks you to build one good home and move things into it. I’d been about to build the wrong tool for the actual job, and the only reason I caught it was that I checked before I started.
Three tiers, not one search box
The harder question wasn’t what to clean up. It was what any of this should look like to an AI agent trying to use it. Everything I have lives at one of three distances from “the thing I need right now”:
Some of it I want in front of the agent every single time, no searching required: who I am, how I write, standing instructions about how to work with me. Some of it is reference material an agent should pull up automatically when a conversation actually touches the topic. And a lot of it, most of it honestly, is material I want preserved but almost never need, the kind of thing you’re glad exists the one time in five years you go looking for it.
I’d been treating all three the same way: one big pile, searchable, done. That’s the mistake practically every serious agent-memory system has already found its way past, though each went past it differently, and the differences are worth more than the similarity. MemGPT frames memory as an operating system managing RAM versus disk, paging context in and out against a fixed window. Zep builds a temporal knowledge graph that tracks not just what’s true but when it became true, so a corrected fact doesn’t quietly compete with the stale one it replaced. Mem0 runs every new memory candidate through an explicit add-or-update-or-discard decision instead of just appending forever. Those are three different architectures solving three different problems: window pressure, fact currency, and write discipline. What I took from them is not a shared design, because there isn’t one. It is the shared premise underneath all three, which is that undifferentiated storage is the thing to get away from. The shape I landed on for my own files is a small always-loaded core, a searchable middle layer, and a cold archive I reach into on purpose, never by default.
Once I saw it that way, the fix was mostly renaming things I already had rather than inventing anything new. My identity and working instructions were already living somewhere an agent reads first, not somewhere it has to search for. That’s the always-loaded tier, already working. My day-to-day reference material was already organized for topic-triggered lookup. The actual gap was the third tier: a large personal archive that no retrieval system touched at all. Not backed up somewhere else, not lost. Just invisible to anything that searches, sitting there the way a filing cabinet sits in a room nobody opens.
Two different agents need two different memories
The other thing the audit forced me to be honest about: the coding agent working directly in my repositories and the chat-based assistant answering questions about my life are not the same kind of reader, and giving them the same retrieval system would be solving the wrong problem for one of them.
A coding agent sitting inside my file tree already has most of what a retrieval index exists to provide: direct access to the current bytes, and exact-text search over all of them. What it does not get for free is semantic search, finding the note whose words you cannot remember, and grep is a poor substitute for that. The trade is still usually wrong for this agent, because an index has to be rebuilt to stay current while the files never go stale, and because the agent needs to open the actual file to edit it regardless of how it found the reference. What I gave up is real, though, and worth naming rather than waving past. The chat assistant has the opposite problem. It never has the actual files. For that one, an index isn’t an optimization. It’s the only way in.
So the file tree stays the source of truth for anyone who can read it directly. The vector index becomes a compiled, disposable artifact that gets regenerated from the files whenever they change, the same way you’d treat a build output rather than something you hand-edit. And the tier that was invisible gets its own lane in that index, reachable on request but excluded from the default search, so an ordinary question doesn’t come back cluttered with five-year-old bookmarks it wasn’t asking for.
The part that actually took the longest
None of the architecture took as long as just checking my own assumptions before acting on them. The duplication I was sure existed mostly didn’t. A “fourteen repositories still need this” count from a project I’d paused a month earlier turned out to be stale in both directions. Some of those repositories had already been emptied out by other work, and a few new ones existed that the count had never seen. Every time I skipped the two minutes it takes to check current reality against what I’d written down earlier, I would have wasted real effort acting on a number that used to be true.
That’s the actual lesson, more than any specific tiering scheme: the thing worth verifying before you scale up a fix isn’t just your facts. It’s the shape of the problem you think you’re solving.