Pull in external data

Today’s power mix, the river gauge, the weather for the homepage — the data lives elsewhere, and the frontend should show it anyway. Instead of a proxy with its own cache and error handling, you write a schema with type: "external". From then on it is a collection like any other: same URL shape, same paging, same API client.

A schema instead of a microservice

{
  type: "external",
  name: "weather",
  source: {
    url: "${WEATHER_API_BASE}/current",
    headers: { Authorization: "Bearer ${WEATHER_API_TOKEN}" },
    params: { units: "metric", lang: "en" },
    timeout: "5s",
  },
  cache: { ttl: "30m", staleOnError: true },
  itemsPath: "results[*]",
  slug: { path: "id" },
  fields: {
    city:   { path: "city",        as: "string" },
    temp:   { path: "temp",        as: "f64" },
    region: { path: "meta.region", as: "string", default: "unknown" },
  },
}

Field mapping is a deliberately small JSONPath subset: nested paths, array index, one filter predicate (production_types[?name="Wind onshore"].data) and reducers like last_non_null for series whose latest values are still missing. as enforces the type; on mismatch default takes over.

Secrets stay in the environment: ${VAR} in URL, headers and params resolves at boot, and a missing variable fails the boot — not a request at three in the morning. Add ${time:now-24h|unix} for time windows, ${rand:uuid} for a nonce.

Same shape as your own content

GET /api/content/weather                 # list, from cache
GET /api/content/weather/hamburg         # one entry
POST /api/content/weather/_refresh       # refetch upstream now

Entries arrive as _status: "published" with _external: true; write methods answer with 405. The admin UI shows the collection read-only, with “Refresh” in place of “New entry”.

A cache that absorbs outages

TTL per schema, in-memory LRU, a disk snapshot that loads warm after a restart. If the upstream goes down, RustyCMS keeps serving the last good state and marks the response with x-rustycms-external-stale: 1; repeated failures back off from two seconds to five minutes. A hundred simultaneous visitors trigger one upstream request.

From snapshot to time series

With cache.mode: "history", each successful fetch appends to a JSONL file per collection and month — the raw upstream response, not the projection, so a later mapping change never loses old data. retain: "12mo" prunes daily, granularity: "15min" prevents duplicate rows under a short TTL. GET /api/content/<coll>/_history?from=&to= returns the window; a backfill tool fetches the past in chunks.

Schemas hot-reload — edit the file, the cache clears, no restart. The validator checks paths and types without calling the source.

Deliberately left out

No references to or from external collections: foreign IDs change, and a provider rename would break links. No localized fields, no upstream pagination, no OAuth. External entries are leaves in the content graph — which is why they cache cleanly.