Skip to main content
Version: 0.12.0

Content-source governance

Introduced 0.12.0

Content-source governance is a deny-by-default allow/deny policy over the external URIs the ingest pipeline may fetch content from — s3://, https://, gs://, and azure:// sources resolved during ingest by reference, image tiling, and content-extraction tools.

It is the enforcement control regulated enterprises ask for by name: a single, cluster-wide guardrail against server-side request forgery (SSRF), data exfiltration to unapproved destinations, and data-residency violations — expressed as glob allow/deny lists, optionally scoped per organization, and configurable entirely at runtime with no redeploy.

tip

This feature is off by default. With no configuration, ingest behaves exactly as it did before — so enabling it is always a deliberate, opt-in step, and a rolling image upgrade never changes existing behavior.

Table of contents

How it works​

Every external content fetch in the ingest pipeline funnels through one resolution chokepoint. When enforcement is on, that chokepoint checks the source URI against the active policy before any bytes are fetched or any pre-signed URL is minted, and rejects a disallowed source with 403 Forbidden.

The decision is evaluated against the verified caller organization, resolved from the authenticated identity using the same tenancy convention the memory layer uses (see Memory tenancy) — so a caller maps to the same organization everywhere. A caller with no resolvable organization is evaluated against the cluster-default policy.

Settings​

The policy is expressed entirely as dynamic cluster settings, applied through PUT _cluster/settings and hot-swapped into the live enforcement engine with no restart.

SettingDefaultDescription
plugins.ingest_content.source.enforcefalseMaster switch. When false, no source is ever rejected — the exact pre-feature behavior.
plugins.ingest_content.source.allow[]Cluster-default allow globs. An empty list means "no explicit allowlist"; then deny_by_default governs unmatched sources.
plugins.ingest_content.source.deny[]Cluster-default deny globs. A deny match always rejects, regardless of the allow list.
plugins.ingest_content.source.deny_by_defaulttrueWhen enforcement is on and no allow list applies, whether an unmatched source is denied (true, fail-closed) or permitted (false).
plugins.ingest_content.source.org.<org>.allow[]Per-organization allow globs; override the cluster default for the named organization.
plugins.ingest_content.source.org.<org>.deny[]Per-organization deny globs; override the cluster default for the named organization.

Decision precedence​

For a fetch by a caller whose verified organization is org (or none):

  1. If enforcement is off, the fetch is allowed.
  2. The effective allow/deny lists are the organization's own lists when org has a configured entry, otherwise the cluster-default lists.
  3. Deny wins: if the source matches any deny glob, it is rejected.
  4. Otherwise, if the allow list is non-empty, the source is permitted only if it matches an allow glob.
  5. Otherwise (no allow list), deny_by_default decides: true rejects, false permits.

Pattern matching​

Patterns are globs — * matches any run of characters — matched case-insensitively against a canonical scheme://host/path form of the URI. The host is the parsed host, never the userinfo or port, so a pattern such as https://*.acme.com/* cannot be fooled by a look-alike authority like https://evil@acme.com.attacker.test/.

Examples of valid patterns:

s3://acme-*
https://*.acme.com/reports/*
gs://acme-corp-*
azure://acme-blob/*

Examples​

Restrict the whole cluster to approved sources​

Only permit content from an approved S3 bucket and a partner HTTPS host; deny everything else:

curl -X PUT "localhost:9200/_cluster/settings" \
  -H "Content-Type: application/json" \
  -d '{
    "persistent": {
      "plugins.ingest_content.source.enforce": true,
      "plugins.ingest_content.source.deny_by_default": true,
      "plugins.ingest_content.source.allow": [
        "s3://acme-content/**",
        "https://*.partner.example/assets/*"
      ]
    }
  }'

Block a specific destination while allowing the rest​

Keep an open posture (deny_by_default: false) but forbid a known-bad host:

curl -X PUT "localhost:9200/_cluster/settings" \
  -H "Content-Type: application/json" \
  -d '{
    "persistent": {
      "plugins.ingest_content.source.enforce": true,
      "plugins.ingest_content.source.deny_by_default": false,
      "plugins.ingest_content.source.deny": [ "https://*.untrusted.example/*" ]
    }
  }'

Give each tenant its own allowlist​

Bind acme to its own bucket and globex to theirs; neither can reach the other's source:

curl -X PUT "localhost:9200/_cluster/settings" \
  -H "Content-Type: application/json" \
  -d '{
    "persistent": {
      "plugins.ingest_content.source.enforce": true,
      "plugins.ingest_content.source.org.acme.allow":   [ "s3://acme-*" ],
      "plugins.ingest_content.source.org.globex.allow": [ "s3://globex-*" ]
    }
  }'

Turn enforcement off​

curl -X PUT "localhost:9200/_cluster/settings" \
  -H "Content-Type: application/json" \
  -d '{ "persistent": { "plugins.ingest_content.source.enforce": false } }'

Where it is enforced​

The policy applies to every path that resolves an external content reference:

  • Ingest by reference — a content_extract processor that fetches a document from a URI. See Content processing.
  • Geospatial and image tiling — the raster/COG pipeline that fetches imagery for tiling and embedding.
  • Content tools — the fetch_content tool that retrieves a source over MCP.
note

The cluster-default allow/deny policy applies on all of these paths, because it needs no caller identity. Per-organization overrides apply wherever the caller's verified organization is present on the request — including the fetch_content tool path, which resolves the caller's organization from the request identity and enforces that organization's allow/deny before any bytes are fetched. A caller with no verified organization falls to the deny-by-default posture, so an unauthenticated fetch path can never reach an un-allowlisted source.

Connector Studio integration​

Content-source governance is the enforcement backbone for compliant content extraction and ingestion through Connector Studio. Any connector that pulls content by reference resolves its sources through the same governed chokepoint, so a connector cannot be pointed at an unapproved bucket, host, or region — the allow/deny policy is enforced uniformly whether content arrives through a hand-built pipeline, a governance blueprint, or a Studio connector. Pair a connector with a per-organization allowlist to guarantee a tenant's connectors can only reach that tenant's approved sources.

Rolling-upgrade safety​

Content-source governance adds no new serialized wire format, transport action, or cluster-state custom — it is dynamic cluster settings only, and enforcement defaults to false. A node that predates the feature simply never sees the settings, and the default preserves pre-feature behavior until an operator opts in on a fully upgraded cluster. An image roll on a running cluster — including the geospatial/COG ingest pipeline — is unaffected.

Turn enforcement on last, after the policy is proven:

  1. Enumerate every legitimate source your pipelines and connectors fetch from.
  2. Populate plugins.ingest_content.source.allow (and any per-organization allowlists) to cover them.
  3. Verify in staging that real content URIs resolve and that unapproved ones are denied.
  4. Only then set plugins.ingest_content.source.enforce: true.

Denied fetches are recorded, so monitor for 403 source-policy rejections after enabling — an unexpected denial usually means a legitimate source is missing from the allowlist.