Skip to content
Trove

Example: Hacker News source adapter (collect data in)

The other three examples are toolkits — they serve tools out. This one is a source adapter: it pulls data in. A source is the symmetric authoring pattern — same “user code Trove runs on your behalf,” pointed the other way.

This source syncs a Hacker News user’s favorites into your Trove knowledge base, so your saved HN stories become semantically searchable alongside everything else. It uses the public HN Firebase API and the Algolia HN Search API — both free and unauthenticated.

How a source adapter runs

Section titled “How a source adapter runs”

A source adapter is a folder with two files — manifest.json + index.ts exporting sync(ctx). This one is a public-API source with no browser or local access, so in production it runs in Trove’s cloud, which schedules and syncs it for you; during development you run it locally with the CLI. (Sources that do need a browser or local files run on the Mac app instead.) Either way, sync(ctx) fetches from the upstream and returns documents; Trove stores, indexes, and searches them.

Project layout

Section titled “Project layout”
hn-favorites/
  manifest.json
  index.ts

manifest.json

Section titled “manifest.json”

The manifest declares the source id, a name, a version, the transport, and the document semantics (how Trove should treat the items).

manifest.json
{
  "id": "hacker-news",
  "name": "Hacker News Favorites",
  "version": "1.0.0",
  "transport": "http",
  "document_semantics": "bookmark"
}

index.ts

Section titled “index.ts”

sync(ctx) reads the previous cursor (the newest item synced last time), fetches the user’s current favorites, and returns the new ones as documents. Each document uses a stable id (the HN item id, which maps to externalId on the wire) so re-running the sync is idempotent — Trove skips anything it has already indexed.

index.ts
import type { TroveSource } from "@ontrove/sdk";


interface HnHit {
  objectID: string;
  title: string;
  url?: string;
  author: string;
  story_text?: string;
  created_at: string;
  points?: number;
  num_comments?: number;
}


export default {
  async sync(ctx) {
    // The HN username to sync comes from the source config (set when the source
    // is added). Falls back to a default for local dev.
    const username = (ctx.config.username as string) ?? "pg";


    // ctx.fetch is the source adapter's egress path — routed through the Mac
    // app's per-source timeout/retry/rate-limit handling.
    // Algolia exposes a user's favorited stories, newest first.
    const res = await ctx.fetch(
      `https://hn.algolia.com/api/v1/search?tags=favorite_${username},story&hitsPerPage=100`,
      { headers: { accept: "application/json" } },
    );
    if (!res.ok) {
      throw new Error(`Hacker News search failed: ${res.status}`);
    }
    const body = (await res.json()) as { hits: HnHit[] };


    // Resume from the last cursor: only ingest items newer than we've seen.
    // ctx.cursor is a Watermark tagged union — this feed advances by id set.
    const lastSeen =
      ctx.cursor.type === "idSet" && ctx.cursor.max ? Number(ctx.cursor.max) : 0;
    const fresh = body.hits.filter((h) => Number(h.objectID) > lastSeen);


    ctx.log(`HN favorites for ${username}: ${fresh.length} new of ${body.hits.length}`);


    const documents = fresh.map((h) => ({
      // Stable id from the upstream → dedup + idempotent re-runs. Maps to externalId.
      id: `hn-${h.objectID}`,
      title: h.title,
      // Self-posts carry text; link posts carry a URL. Index whichever exists.
      text:
        h.story_text ??
        `${h.title}\n\n${h.url ?? ""}\n\nvia Hacker News (${h.points ?? 0} points, ` +
          `${h.num_comments ?? 0} comments)`,
      url: h.url ?? `https://news.ycombinator.com/item?id=${h.objectID}`,
      author: h.author,
      date: h.created_at,
      contentType: "bookmark" as const,
      tags: ["hacker-news", "favorite"],
    }));


    // Advance the cursor to the newest item id we just processed.
    const newest = body.hits.reduce((max, h) => Math.max(max, Number(h.objectID)), lastSeen);


    return { documents, cursor: { type: "idSet", values: [], max: String(newest) } };
  },
} satisfies TroveSource;

Why this shape

Section titled “Why this shape”
  • sync(ctx) → { documents, cursor }. The source adapter returns documents to be stored — the mirror of a toolkit returning a tool result to be read. Trove handles indexing, embedding, and search.
  • Stable id (hn-<id>). The SDK field is id (it maps to externalId on the wire). Re-running the sync is safe: Trove dedups on (feed, id) and skips anything already indexed.
  • Cursor. ctx.cursor is a Watermark tagged union — the position from the previous run. This feed uses an idSet watermark; returning a new { type: "idSet", max } lets the next run pick up only newer items. Production scheduling and cursor persistence are handled by the Mac app.
  • ctx.fetch. All egress goes through ctx.fetch — in the Mac app it routes through per-source timeout, retry, and rate-limit handling.
  • contentType: "bookmark" matches the manifest document_semantics — these are saved links, not full articles. See IngestDocumentInput for every field.

Run it locally

Section titled “Run it locally”

The CLI runs sync(ctx) on your machine for local development; in production this public-API source runs in Trove’s cloud. (trove source init/dev/test/sync are the local SDK toolchain steps for source adapters; see the Sources series and CLI docs.)

Terminal window
# Inner dev loop — run sync(ctx) and print the documents, nothing uploaded:
trove source dev


# The real thing — run sync(ctx) and push results via ingestDocuments:
trove source sync --create

Under the hood, trove source sync is the source adapter’s output fed into the same ingestDocuments mutation documented in the Sources SDK Reference — cursor (CAS) semantics included.

What you get

Section titled “What you get”

Once synced, your HN favorites are documents in your knowledge base. Ask Claude (through any Trove connection):

What have I saved from Hacker News about databases?

Claude calls trove_search and finds your favorited HN stories by meaning — no separate tool, because sources feed the same searchable store the core tools read.

Next steps

Section titled “Next steps”