Skip to main content
Version: 0.13.0

Similar imagery tool

Introduced 0.13.0

The SimilarImageryTool, which an agent sees as FindSimilarImagery, finds images whose stored vectors are nearest to a query image. The query is an image: a document id already in the index (like), or a URL (image_uri).

Mapping and embed path​

embedding_field is a knn_vector on each scene document. Map it with the same dimension your embedding model writes. Populate it at ingest with the embed processor (source_uri_field for a whole-image vector, or a chunk field that carries embedding). For on-node CLIP, see Local image embedding.

PUT /scenes
{
  "settings": {
    "index": {
      "knn": true
    }
  },
  "mappings": {
    "properties": {
      "image_uri": { "type": "keyword" },
      "title": { "type": "keyword" },
      "embedding": {
        "type": "knn_vector",
        "dimension": 512,
        "method": {
          "name": "hnsw",
          "space_type": "cosinesimil",
          "engine": "lucene"
        }
      }
    }
  }
}

Point embedding_field at embedding (or another root field your pipeline writes). The register example below uses embedding.

Two ways to name the query image​

Pass like with a document id already in the index. The tool reuses that document's stored vector, so the query and the index are in the same embedding space. Pass image_uri with a URL of a query image. Give exactly one of like or image_uri.

When using image_uri, the cluster fetches the image and embeds it with model_id / provider. Use the same model that produced the field's vectors. Supported URI schemes are https, s3, gs, and azure.

Scores​

A k-nearest-neighbour search returns up to k results, the nearest neighbours in the index. With min_score set, only neighbours at or above that score are returned.

The score is an ordering in the field's space (l2, cosine, inner product). Compare scores only within the same index and model.

Parameters​

ParameterRequiredDescription
indexYesThe index holding the imagery.
embedding_fieldYesThe knn_vector field holding each image's embedding.
likeOne of like or image_uriDocument id whose stored vector is reused.
image_uriOne of like or image_uriURL of the query image, embedded at query time.
model_idWith image_uriThe embedding model that produced the field's vectors.
providerNoEmbedding backend. Default depends on the node; see Local image embedding for onnx.
kNoHow many neighbours to return. Default is 10. Maximum is 50.
min_scoreNoNeighbours at or above this score are returned.
filterNoA phrase that restricts which documents are candidates.
label_fieldNoA field used to name each result in the answer.

Register a flow agent that runs the SimilarImageryTool​

POST /_plugins/_agent/agents/_register
{
  "name": "Imagery_Agent",
  "type": "flow",
  "description": "finds visually similar scenes",
  "tools": [
    {
      "type": "SimilarImageryTool",
      "name": "DemoFindSimilarImagery",
      "parameters": {
        "index": "scenes",
        "embedding_field": "embedding"
      }
    }
  ]
}
POST /_plugins/_agent/agents/<agent_id>/_execute
{
  "parameters": { "like": "chip-441" }
}