Skip to content
Trove

Example: Hacker News source adapter (collect data in)

The other three examples are toolkits — they serve tools out. This one is a source adapter: it pulls data in. A source is the symmetric authoring pattern — same “user code Trove runs on your behalf,” pointed the other way.

This source syncs a Hacker News user’s favorites into your Trove knowledge base, so your saved HN stories become semantically searchable alongside everything else. It uses the public HN Firebase API and the Algolia HN Search API — both free and unauthenticated.

How a source adapter runs

Section titled “How a source adapter runs”

A source adapter is one defineSource call in extension.ts — what the source is and what it does, in a single value. This one is a public-API source with no browser or local access, so in production it runs in Trove’s cloud, which schedules and syncs it for you; during development you run it locally with the CLI. (Sources that do need a browser or local files run on the Mac app instead.) Either way, sync(ctx) fetches from the upstream and returns documents; Trove stores, indexes, and searches them.

Project layout

Section titled “Project layout”
hn-favorites/
  extension.ts    ← the whole source
  manifest.json   ← generated from it; never edited by hand

extension.ts

Section titled “extension.ts”

One call declares the source and implements it. The declaration is validated when the module is imported, so a bad cadence or an ineligible runsIn fails at your desk rather than on a schedule. sync(ctx) reads the previous cursor (the newest item synced last time), fetches the user’s current favorites, and returns the new ones as documents. Each document uses a stable id (the HN item id, which maps to externalId on the wire) so re-running the sync is idempotent — Trove skips anything it has already indexed.

extension.ts
import { defineSource } from "@ontrove/extend/source";


interface HnHit {
  objectID: string;
  title: string;
  url?: string;
  author: string;
  story_text?: string;
  created_at: string;
  points?: number;
  num_comments?: number;
}


export default defineSource({
  id: "hacker-news",
  name: "Hacker News Favorites",
  description: "Your favorited Hacker News stories.",
  icon: "🟠",
  version: "1.0.0",
  author: "you",
  kind: "scheduled-sync",
  transport: "api",
  cursor: "idSet",
  ingest: "append",
  runsIn: "cloud",
  schedule: "every 6 hours",
  status: "implemented",
  needsBrowser: false,
  egress: ["hn.algolia.com"],
  config: {
    username: { label: "HN Username", type: "text", placeholder: "pg" },
  },


  async sync(ctx) {
    // The HN username to sync comes from the source config (set when the source
    // is added). Falls back to a default for local dev.
    const username = (ctx.config.username as string) ?? "pg";


    // ctx.fetch is the source adapter's egress path — and the allowlist above
    // is enforced on it, so a host absent from `egress` is unreachable here.
    // Algolia exposes a user's favorited stories, newest first.
    const res = await ctx.fetch(
      `https://hn.algolia.com/api/v1/search?tags=favorite_${username},story&hitsPerPage=100`,
      { headers: { accept: "application/json" } },
    );
    if (!res.ok) {
      throw new Error(`Hacker News search failed: ${res.status}`);
    }
    const body = (await res.json()) as { hits: HnHit[] };


    // Resume from the last cursor. `ctx.cursor` is a Cursor tagged union —
    // and it is ABSENT on the first sync, so it has to be reached optionally.
    // An `idSet` remembers WHICH ids were already stored, in `values`.
    //
    // `max` is not part of that decision — it is the cap on how many ids are
    // retained, so the set stays bounded. Reading it as "the highest id seen"
    // is a mistake worth naming, because the code still runs: you get an empty
    // set, every run looks like a first run, and the feed silently re-ingests.
    const seen = new Set(ctx.cursor?.type === "idSet" ? ctx.cursor.values : []);
    const fresh = body.hits.filter((h) => !seen.has(h.objectID));


    ctx.log(`HN favorites for ${username}: ${fresh.length} new of ${body.hits.length}`);


    const documents = fresh.map((h) => ({
      // Stable id from the upstream → dedup + idempotent re-runs. Maps to externalId.
      id: `hn-${h.objectID}`,
      title: h.title,
      // Self-posts carry text; link posts carry a URL. Index whichever exists.
      text:
        h.story_text ??
        `${h.title}\n\n${h.url ?? ""}\n\nvia Hacker News (${h.points ?? 0} points, ` +
          `${h.num_comments ?? 0} comments)`,
      url: h.url ?? `https://news.ycombinator.com/item?id=${h.objectID}`,
      author: h.author,
      date: h.created_at,
      contentType: "bookmark" as const,
      tags: ["hacker-news", "favorite"],
    }));


    // Carry forward every id this run has now stored, newest first, bounded so
    // the cursor cannot grow without limit. Past the cap the oldest ids are
    // evicted, which at worst re-fetches those items once — `(feed, id)` dedup
    // absorbs them.
    //
    // Advance from what was FETCHED, not from what was kept. If this filtered
    // first and remembered only what it stored, anything skipped would be
    // re-fetched on every future run, forever.
    const values = [...new Set([...body.hits.map((h) => h.objectID), ...seen])].slice(0, 1000);


    // `as const` matters: without it `type` widens to `string` and the object
    // no longer matches the Cursor union.
    return { documents, cursor: { type: "idSet" as const, values, max: 1000 } };
  },
});

Why this shape

Section titled “Why this shape”
  • sync(ctx) → { documents, cursor }. The source adapter returns documents to be stored — the mirror of a toolkit returning a tool result to be read. Trove handles indexing, embedding, and search.
  • Stable id (hn-<id>). The SDK field is id (it maps to externalId on the wire). Re-running the sync is safe: Trove dedups on (feed, id) and skips anything already indexed.
  • Cursor. ctx.cursor is a Cursor tagged union — the position from the previous run. This feed uses an idSet cursor; returning a new { type: "idSet", max } lets the next run pick up only newer items. Production scheduling and cursor persistence are handled by the Mac app.
  • ctx.fetch. All egress goes through ctx.fetch — in the Mac app it routes through per-source timeout, retry, and rate-limit handling.
  • contentType: "bookmark" — these are saved links, not full articles. It is a per-document field, unrelated to the manifest’s ingest, which says only whether a later run may replace a stored document. See IngestDocumentInput for every field.

Run it locally

Section titled “Run it locally”

The CLI runs sync(ctx) on your machine for local development; in production this public-API source runs in Trove’s cloud. (trove source init/dev/test/sync are the local SDK toolchain steps for source adapters; see the Sources series and CLI docs.)

Terminal window
# Inner dev loop — run sync(ctx) and print the documents, nothing uploaded:
trove source dev


# The real thing — run sync(ctx) and push results via ingestDocuments:
trove source sync --create

Under the hood, trove source sync is the source adapter’s output fed into the same ingestDocuments mutation documented in the Sources SDK Reference — cursor (CAS) semantics included.

What you get

Section titled “What you get”

Once synced, your HN favorites are documents in your knowledge base. Ask Claude (through any Trove connection):

What have I saved from Hacker News about databases?

Claude calls trove_search and finds your favorited HN stories by meaning — no separate tool, because sources feed the same searchable store the core tools read.

Next steps

Section titled “Next steps”