How Trove Works

Trove is a Cloudflare Worker that ingests content from your sources, stores it across three storage layers, and makes it searchable through the Trove connection and a GraphQL API.

Sources --> Sync --> Ingest Pipeline --> Storage --> Search/API --> AI Assistants & Apps

Storage Layers

Trove splits storage across three services, each optimized for a different access pattern.

Metadata (D1)

Each user gets their own SQLite database (Cloudflare D1) for document metadata: titles, authors, dates, tags, and preview text. There are no user_id columns. The database itself is the user’s data.

Search results, document listings, and source stats all read from D1 without touching object storage.

Full Text (R2)

Complete document text stored in Cloudflare R2. Keys are prefixed with the user ID ({userId}/docs/{documentId}/{artifact}), so one prefix is a user’s entire footprint. Full text is loaded only when explicitly requested. List views and search results use preview text from D1.

Embeddings (Vectorize)

Semantic embeddings stored in Cloudflare Vectorize. A shared index across all users with a mandatory user_id metadata filter on every query. Documents are represented as 1024-dimensional vectors. Search queries are compared against them using cosine similarity.

Ingest Pipeline

When content arrives — pushed by a sync client through the Sync API, or written by a manual save — it passes through a deterministic pipeline.

1. Dedup. Check if (feed_id, external_id) already exists. Skip duplicates.
2. Preview. Extract the first ~300 words for fast display in search results and list views.
3. Store. Write metadata to D1, full text to R2.
4. Chunk. Split text into overlapping chunks using a content-type-aware splitter.
5. Embed. Generate vector embeddings via Workers AI using the bge-m3 model (1024 dimensions), batched up to 100 chunks per call.
6. Index. Upsert vectors to Vectorize with user_id and other metadata for filtering.

The pipeline processes batches of up to 50 documents and typically completes in 5-10 seconds.

Search Pipeline

When you search:

1. Your query is embedded using the same bge-m3 model.
2. Vectorize performs a cosine similarity search with a mandatory user_id filter plus any additional filters (source, author, date range, content type, tags).
3. Matching document IDs fetch metadata from D1.
4. Results are assembled with snippets (the most relevant text chunk), document metadata, and relevance scores.
5. Optionally, an AI reranker refines the result ordering.

Total search latency is typically 35-55ms without reranking, 55-75ms with reranking.

The Connection

AI assistants reach all of this through one connection, served over MCP. It supports two transports.

• Streamable HTTP (POST /mcp). Stateless Workers with sessions backed by KV (1-hour TTL). Used by most MCP clients including Claude Desktop and Cursor.
• WebSocket. Durable Objects per user with hibernation (zero cost when idle).

Both transports run the same JSON-RPC 2.0 protocol and expose the same 8 Core tools, plus the tools of any toolkits you’ve enabled.

Syncing

Trove does not fetch from your sources on its own. A sync client — the Trove Mac app, or your own code built against the Sync API — owns the sync loop.

1. The sync client reads the source’s stored cursor to know where it left off.
2. It fetches new content from the upstream system (handling its own scheduling, credentials, and retries).
3. It pushes the results to Trove via the Sync API, which runs the ingest pipeline.
4. Trove advances the source’s cursor and sync timestamp, and the client records a SyncRun.