@ontrove/extend/source

@ontrove/extend/source — the shared vocabulary for Trove sources: scheduled adapters that fetch content into a knowledge base. A source exports a sync(ctx) that returns documents; this package owns the shapes both ends of that call agree on.

What it owns today

• The invoke contract (@ontrove/extend/contract) — the request/response envelope every runtime speaks. Three of them execute it: Trove’s deployed shim, the CLI’s local shim, and the Mac harness. This is the load-bearing part, and the reason the same sync(ctx) runs unchanged in all three.
• The types — Document, SourceContext, Cursor, SourceManifest. Trove imports several of them directly rather than re-declaring them.
• runSource — the local-run harness the CLI drives, with the same validation and dedup the cloud applies.
• validateSourceManifest — what a manifest must say to be installable, including the vocabulary itself: which kind, transport, cursor and ingest values exist, and which subset is buildable today.
• The cursor writer — dateCursor, idSetCursor and their readers. The type and the code that produces it are now in one place, so the contract test asserts against the writer rather than fixtures.
• The guarded fetch seam — fetchPage / fetchBytes, with the host guard, timeout and size caps a source should never re-implement. It takes the fetch it is given, so a source called with a capability-bearing ctx.fetch uses that one.

What it does not own yet

Feed and HTML parsing — the RSS/Atom reader, HTML to text, the scrape loop — still live alongside the sources themselves. Those are the next candidates; the pieces every source needs in order to be correct rather than merely convenient now live here.

It is the symmetric sibling of @ontrove/extend/toolkit, the toolkit-authoring library (every toolkit runs as a full MCP server on Trove’s cloud): a source returns documents to be stored (defineSource + sync); a toolkit’s tools return results to be read live (defineToolkit). The two are at different stages — a toolkit is written in @ontrove/extend/toolkit, while a source is written against a contract this package defines and helpers it does not yet provide.

Example

import { defineSource } from '@ontrove/extend/source';

export default defineSource({
  async sync(ctx) {
    const res = await ctx.fetch('https://hn.algolia.com/api/v1/search?tags=front_page');
    const { hits } = await res.json();
    return hits.map((hit) => ({
      id: hit.objectID,
      title: hit.title,
      text: hit.story_text ?? hit.title,
      url: hit.url,
      author: hit.author,
      date: new Date(hit.created_at_i * 1000).toISOString(),
      contentType: 'bookmark',
    }));
  },
});

Classes

Class	Description
HttpStatusError	A response that arrived intact but was not OK. Carries the status so retry logic can tell a transient 503 from a permanent 404 without parsing prose.
ResponseTooLargeError	A response rejected by the size cap: a permanent condition (the resource is simply too big), unlike a timeout or connection error that may succeed on retry. Callers branch on isTooLargeError rather than on this class.

Interfaces

Interface	Description
Document	A single document a source returns from sync. The fields map 1:1 onto the GraphQL IngestDocumentInput the Mac app pushes via ingestDocuments:
ExtensionCache	A run-to-run cache, where the host offers one.
ExtensionContext	-
FetchedPage	What a fetch learned about the address itself, alongside the body.
GuardedFetchOptions	Options shared by every helper here.
HistoryReach	How much of an upstream’s history a source can actually reach.
LogChannel	A log channel: callable for the common case, with levels when severity matters.
ManifestConfigField	A single field descriptor inside a manifest config object — describes one preference input shown in the source’s setup wizard.
ManifestValidationOptions	How strictly to read a manifest, and what the caller knows that the manifest cannot say for itself.
ManifestValidationResult	The outcome of validateSourceManifest.
RunOptions	The options runSource builds a ctx from. The CLI passes the source’s stored config and the feed’s current cursor; tests inject fetch, log, and now for determinism.
RunResult	The outcome of runSource: the validated, deduped documents, the resolved cursor, and the captured log lines. Mirrors enough of what the cloud ingest reports for trove source test to print a useful summary.
SourceContext	The single argument to sync — ExtensionContext plus what a scheduled, resumable, fan-out-capable source needs on top of it.
SourceExtension	A complete source: what it is, and what it does.
SourceManifest	-
SourceSyncResult	The result of a source sync: the documents fetched this run and, optionally, the cursor the feed should advance to. A source may also return a bare Document[] for convenience — runSource normalizes that to { documents } with no cursor change.
TroveSource	The type a source’s default export satisfies. A source is an object with a sync method that fetches new content and returns documents to index.

Type Aliases

Type Alias	Description
Cursor	A typed cursor describing how a feed resumes between syncs. The opaque Feed.cursor string is parsed into one of these (see the cursors guide, cursor types).
CursorStrategy	One of CURSOR_STRATEGIES.
DirectoryAuthStrategy	One of DIRECTORY_AUTH_STRATEGIES.
DirectoryMode	One of DIRECTORY_MODES.
FanOutFieldType	One of FAN_OUT_FIELD_TYPES.
FetchLike	The standard fetch signature the SDK exposes on SourceContext.fetch. Matches the platform fetch so existing code ports unchanged.
FormattingPolicy	One of FORMATTING.
HistoryReachKind	One of HISTORY_REACH_KINDS.
IngestMode	One of INGEST_MODES.
RunsIn	One of RUNS_IN.
SourceContentType	The default content type Trove assigns a document when it omits contentType. Mirrors the GraphQL ContentType enum surfaced on IngestDocumentInput.
SourceKind	One of SOURCE_KINDS.
SourceSchedule	One of VALID_SCHEDULES.
SourceTransport	One of TRANSPORTS.

Variables

Variable	Description
CLOUD_ELIGIBLE_TRANSPORTS	The transports whose sync is a pure HTTP pull — the necessary condition for a source to run in the cloud at all. A browser source drives a real browser and a local source reads the user’s disk; neither exists in a hosted runtime, so both are pinned to the client.
CURSOR_STRATEGIES	The resume strategy a source declares; the value itself lives in the feed’s cursor between runs. date, idSet and none are the three the SDK’s Cursor type carries today; the rest are declared shapes for feeds that resume by token, by row, or by whole-snapshot comparison.
DEFAULT_ID_SET_MAX	Default cap on how many entries an idSet cursor retains.
DIRECTORY_AUTH_STRATEGIES	The auth strategies Trove knows how to sign a directory lookup with. A directory provider names one and the platform applies it, so no source author ever handles the credential.
DIRECTORY_MODES	The affordances a directoried config field can ask a client to render: search (type a name, pick from results) or resolve (paste something and have it turned into the real address).
FAN_OUT_FIELD_TYPES	The config field types a fan-out source may explode into one feed per entry — a list of feed URLs, or a list of query strings. A scalar field cannot fan out, so naming one in fanOut is rejected rather than silently producing a single feed.
FETCH_TIMEOUT_MS	Per-request ceiling. Without it a single slow or hung host stalls an entire sync run for minutes. A bounded request fails fast and is retried next run.
FORMATTING	Whether Trove reformats a source’s documents into clean Markdown on ingest, or stores them exactly as received.
HISTORY_REACH_KINDS	How much of an upstream’s history is reachable at all.
INGEST_MODES	What ingest does with the documents a run returns. append adds what is new and leaves what is stored alone; upsert lets a later run replace an earlier document with the same id.
MAX_ID_SET_BYTES	Cap on the SERIALIZED size of an idSet cursor.
MAX_REDIRECTS	Redirect hops followed before giving up. Feeds need far fewer than a browser.
MAX_RESPONSE_BYTES	Default response-size cap: large enough for a long article, small enough that one page cannot exhaust a run.
MVP	The MVP cut: the subset of each vocabulary the runtimes actually build and enforce today.
MVP_DEPLOYED_CURSORS	The cursor strategies a runtime: deployed source may additionally use.
RUNS_IN	Default executor for a source’s sync. cloud = a Trove-hosted runtime; client = the user’s own device.
SOURCE_KINDS	Execution contract — which entrypoint the harness invokes.
SOURCE_TYPE_FIELDS	The four type-system fields with their full vocabularies, keyed by field name. Exported so a catalog can render the taxonomy — a picker, a docs table, a test that asserts every source’s declaration is in range — from the same data the validator uses, rather than a copy that drifts.
TRANSPORTS	The mechanism by which a source reaches its data. This is what decides whether the source can run anywhere but the user’s own machine — see CLOUD_ELIGIBLE_TRANSPORTS.
TROVE_USER_AGENT	Descriptive, attributable User-Agent. A site operator who wants to identify or rate-limit this traffic can, which is the difference between a bot that is welcome and one that gets blocked.
VALID_SCHEDULES	The sync cadences a manifest may name. A schedule is a human-readable phrase rather than a cron expression because it is shown to the person enabling the source; the scheduler maps each phrase to an interval.

Functions

Function	Description
advanceDateCursor	The date cursor to return from a run whose sub-sources (feeds, sections, tickers, channels, meeting types) may have individually failed.
assertPublicHttpUrl	Guard an address before fetching it. Only public web pages are ever wanted, so require http(s) and reject private, loopback, and link-local hosts.
dateCursor	Build a typed date cursor from an ISO-8601 string.
defineSource	Validate and return a source definition unchanged.
defineSync	A source authored inline as a single sync function.
fetchBytes	Binary twin of fetchPage: same guard, User-Agent, deadline, and streamed size cap, but returns the raw bytes — for document downloads (PDFs, images, audio) where decoding to text would corrupt the payload.
fetchPage	Fetch a page and return its text, with the honest bot User-Agent, the SSRF guard, the deadline, and the size cap. Throws on a non-200.
fetchPageWithMeta	fetchPage plus what the fetch learned about the address itself.
idSetCursor	Build a typed idSet cursor: deduped, bounded to max entries, and then bounded again to MAX_ID_SET_BYTES so the cursor the platform stores cannot be refused.
isCredentialConfigKey	Whether a single config key looks like a credential and must be rejected. Exposed on its own so an authoring tool can warn on the field the moment it is typed, instead of only when the whole manifest is validated.
isTooLargeError	Whether an error from these helpers was a size-cap rejection — the one failure worth treating as permanent (skip the document) rather than transient (retry next run).
readDateCursor	Read a date cursor as a Date.
readIdSet	Read an idSet cursor as a string array.
runSource	Run a source’s sync against a built ctx and collect/validate the result.
stringList	Read a config field as a list of strings.
toSourceManifest	The manifest half of a source, as the JSON a catalog commits.
validateSourceManifest	Validate a source manifest.json in full — shape, credential lint, the four type-system fields, location and its cloud-eligibility predicate, and the optional schedule, fanOut, formatting and directory declarations.