Topic drift ingest processor

The topic_drift ingest processor keeps LLM and VLM extraction on the rails during ingest.

Given a mission goal and the text an analysis step produced for a document — say a vision model's description of a satellite chip — it scores how far that output has drifted from the mission and annotates the document with the verdict. Off-mission or hallucinated extractions are flagged for review or re-run instead of silently polluting the searchable index.

A scene analyzed for "catalog maritime vessels in port imagery" that comes back describing the weather, or hallucinating a vehicle, scores high drift and needs_intervention: true, with the goal returned as corrective context to re-inject on a re-run.

It costs a handful of vector operations, not a second model call

Drift scoring embeds the goal and the analysis text using the same embedding provider the rest of ingest uses, then compares them. There is no second LLM call, which is what makes it affordable to run on every ingested item rather than on a sample.

Request fields

Field	Type	Description	Required
goal	String	The mission statement to score against.	Yes
field	String	Document field holding the analysis text. Defaults to analysis.	No
target_field	String	Field that receives the verdict. Defaults to drift.	No
provider	String	Embedding provider used to embed the goal and the text.	No
model_id	String	Embedding model on that provider.	No
dimensions	Integer	Embedding dimensions. Defaults to 768.	No
nudge_at	Float	Score above which the result is worth a nudge.	No
correct_at	Float	Score above which correction is called for.	No
block_at	Float	Score above which the extraction should be blocked.	No

info

The embedding provider and model must match the ones used elsewhere in your pipeline. Two embeddings from different models are not comparable, so a mismatch produces a drift score that means nothing.

What it writes

{
  "drift": {
    "score": 0.71,
    "verdict": "correct",
    "needs_intervention": true,
    "goal_distance": 0.68,
    "novelty": 0.24,
    "corrective": ["catalog maritime vessels in port imagery"]
  }
}

Field	Description
score	Overall drift score.
verdict	The threshold band the score fell into, lowercased.
needs_intervention	Whether the extraction should be reviewed or re-run.
goal_distance	How far the analysis sits from the mission goal.
novelty	How much new material the analysis introduced.
corrective	The goal, returned as context to re-inject on a re-run. Present only when intervention is needed.

Example

PUT /_ingest/pipeline/vessel-analysis
{
  "description": "Score VLM output against the mission goal",
  "processors": [
    {
      "topic_drift": {
        "goal": "catalog maritime vessels in port imagery",
        "field": "analysis",
        "target_field": "drift",
        "provider": "my-embedding-provider",
        "model_id": "text-embedding-v1"
      }
    }
  ]
}

Because the verdict is an ordinary field, you can route on it — index everything and filter the clean extractions at query time:

GET /imagery/_search
{
  "query": {
    "bool": {
      "must":    [ { "match": { "analysis": "container vessel" } } ],
      "filter":  [ { "term": { "drift.needs_intervention": false } } ]
    }
  }
}

Related

• Embed processor — the embedding path this shares
• Inference memory — drift anchors for agent memory
• Image segment processor