Until very recently, feeding an image to Mnemosyne was not "unsupported." It was unrepresentable. There is exactly one path from content to vector in the codebase, and its entire public surface is text-shaped: embed a string, embed a query, serialize the result. No modality parameter, no dispatch, no second code path.
I have been chipping away at changing that, in the back shed, on nights and weekends, at whatever pace the rest of life allows.
Let me set expectations before anything else, because I would rather disappoint you in the first paragraph than the last one: this is not live, and it is not about to be. There is no version you can install to get it. There is no date. What exists is a set of design documents and some early scaffolding merged behind the scenes, none of which is wired into anything you use today. If you take one thing from this post, take that.
So why write it up at all? Because the decisions are more interesting than the code, and because a design is much easier to argue with before it hardens. I would rather someone tells me now that I have this wrong.
The question we are actually answering
Two questions, precisely:
- "At what point in that video did someone talk about X?"
- "Which of my screenshots shows the dark-mode palette?"
And one hard constraint on how we answer them: without ever becoming an ambient screen recorder.
That constraint is not decoration. There is an entire product category built on continuously capturing your screen and shipping it somewhere for indexing. Mnemosyne exists because I think your memory belongs on your machine. A multimodal feature that quietly inverted that would be worse than no multimodal feature at all.
The core trick: store the reference and the text, never the bytes
The design has two new tables and one line worth memorizing: there is no BLOB column, by design.
media_assets
holds one row per referenced piece of media: what it is (image, video,
audio, document), where it lives (a hash, a URL, a file path, a
YouTube id), its dimensions or duration, when it was actually captured,
and whether we have managed to understand it yet. Its id is
deterministic rather than random, so re-ingesting the same file twice
is idempotent without a lookup round trip.
media_moments
holds the actual content: one row per semantically tagged span. A
caption, a shot description, a transcript segment, an OCR result, a
page summary — each with the span it came from. Video and audio spans
are time offsets, document spans are pages or character ranges, image
spans are either the whole frame or a bounding box.
So what Mnemosyne persists about your video is: a reference to it, and a set of timestamped English sentences about what happens in it. The video itself stays exactly where it was. If you delete it, we hold a dangling reference and some text — not a copy of your life.
The decision I am proudest of: a moment is just a memory
The obvious implementation is a new vector table for moments. We did
not do that. Every retained moment is written to
working_memory
as an ordinary row.
The reasoning is boring and, I think, correct: an ordinary working-memory row already gets the dense vector voice, the lexical FTS voice, recency decay, tier degradation, consolidation into episodic memory during sleep, the veracity multiplier, sync event propagation, and reindex coverage. All of it. For free. A new vector table, by contrast, costs edits in four separate hardcoded lists that must all agree, a rowid-alignment discipline, and a permanently larger blast radius the next time an embedding dimension mismatches.
There is a proof of that cost sitting in the schema already. A table
called vec_facts was
declared long ago and has never once been written to — and the reindex
routine still has to recreate it, empty, forever, so its declared
dimension cannot mismatch a query. That is the tax on a table you
add and then think better of.
The cheapest new subsystem is the one that is not a new subsystem. Because moments are ordinary rows, the media recall voice returns ordinary memory ids, and the existing rank-fusion step needs no translation layer between two incompatible id spaces. Your screenshot caption competes for recall against your text notes on exactly equal terms.
Three clocks that must never be confused
This is the subtle one, and it is the kind of mistake that would have been very expensive to discover in production.
A media offset — "90 seconds in" — is meaningless without knowing which asset it belongs to. It is not comparable across assets. Mnemosyne already has a wall-clock field that is indexed for range queries and consumed by the temporal recall voice, and dumping media offsets into it would corrupt temporal recall for every user, including the ones who never touch a video.
So there are three separate times, and they are never conflated: when the recording was actually made, when Mnemosyne ingested it, and where inside the asset the moment sits. Different axes, different columns, no cross-contamination.
What actually exists so far
Five pieces of scaffolding, in dependency order. None of this is wired into a release, none of it is documented for users yet, and all of it is subject to being torn up if the design turns out to be wrong:
- A reader for the blob store. Mnemosyne has had a content-addressed blob store for a while, and here is the embarrassing part: nothing in the entire tree ever read from it. It was write-only. It now has a resolver.
- An explicit memory type on remember(). A caption should not be classified by the same heuristic that labels your notes, so a moment can now declare itself an artifact rather than being guessed at — with an opt-out for dedupe alongside it.
- The media asset registry and moment index. Two tables, idempotent init, no recall path touched.
- A vendor-neutral provider seam. One small interface for "describe this non-text thing and give me text back," modeled on the existing LLM backend abstraction. Vendor-neutral is the operative word: no provider is privileged, including the one sponsoring our benchmarks.
- A first end-to-end path. Point it at an image, get captions back as recallable memories, with the bytes never entering the database. It works on my machine, on the happy path, on the cases I thought to try. That is a very different claim from "it works."
That last distinction is the whole gap between where this is and where it needs to be. A thing that runs is maybe a third of the way to a thing that ships. The rest is the unglamorous part: the formats I haven't tried, the providers that return something shaped differently than I assumed, the file that is corrupt halfway through, the machine with no network, the database that already has ten thousand rows in it. None of that is interesting to write about and all of it has to be right.
Degrading honestly
Not everyone will have a vision provider configured, and the design treats that as a normal outcome rather than an error. If we cannot describe your file, we register the asset by reference, write zero moments, and raise nothing. You end up with a searchable record that you referenced this file at this time — which is strictly more than you had before, and considerably better than a traceback.
Two more deliberate limits. Shot-level density is opt-in, not the default: a ten-minute video at full shot granularity is dozens of rows, and working memory holds ten thousand items before eviction pressure starts to bite, so retained moments are capped by default. And the ingest path is synchronous on purpose — the core has no access lock, so a helpful background thread would run unserialized against SQLite, which is precisely the shape of a WAL-checkpoint crash we already fixed once.
So when does it land?
I don't know, and I am not going to guess in public. The RFCs are still marked Draft. This targets the next major version, which is not the next release, or the one after that. Image comes first. Video follows. Audio and speaker profiles reuse the same primitives with a speaker dimension, and are deliberately phased last.
What happens between here and there is polish, and polish is slow in a way that is hard to see from outside. It is not one big push; it is a long series of small corrections, most of them prompted by someone hitting an edge I never would have found alone. That is how the rest of Mnemosyne got solid, and it is how this will too — the issue tracker ends up doing most of the work of turning something that runs into something you can rely on.
Which is also to say: none of this is a promise. It is a direction, and a fairly opinionated one. Directions change when the constraints turn out to be different than you thought. If this quietly stops being mentioned for a few months, that is not abandonment — it is a back-shed project competing with the parts of Mnemosyne that people are actually running today, and those win.
If you want the full design, the RFCs are in the repo under
docs/rfc/
— 0002 for the provider seam, 0003 for assets and moments, 0004 for
the archive boundary. They are long, they cite line numbers, and they
argue with themselves in public. That is what an RFC is for.
The privacy-preserving design here — reference hashes plus semantically tagged spans — also happens to be the cheapest design, because text moments reuse the entire retrieval engine we already had. That alignment is not luck. It is what you get when "don't hoard the user's data" is a constraint you accept up front instead of a feature you bolt on later.