Version: 0.13.0

Embed processor

Introduced 0.11.0

The embed processor generates vector embeddings from text, image, or multimodal content. It reads chunk arrays produced by the chunk or image_tiling processors and writes a dense vector into each chunk, making them searchable via kNN.

Six embedding providers are supported: AWS Bedrock, OpenAI, a generic HTTP endpoint for self-hosted models, GCP Vertex AI, Azure OpenAI, and Azure AI Vision (multimodal).

tip

The dimensions you set here must match both your chosen model's output width and the dimension of the knn_vector field in your index mapping. A mismatch causes indexing to fail. See End-to-end content processing for how the embed processor fits into a full extract → chunk → embed pipeline.

warning

chunk (or image_tiling) must run first. The embed processor requires field to already be a list of chunk objects, not a plain string. If it isn't a list, the processor throws field [<field>] must be a list of chunks, got <class> and the document fails (or is skipped, depending on on_failure_action). There is no path that lets embed chunk raw text itself — always place chunk/image_tiling earlier in the same pipeline, as shown in the full pipeline example.

Syntax

{
  "embed": {
    "field": "chunks",
    "model_id": "amazon.titan-embed-text-v2:0",
    "provider": "bedrock",
    "dimensions": 1024,
    "provider_config": {
      "region": "us-east-2"
    }
  }
}

Architecture

                        ┌──────────────────────┐
                        │    embed processor    │
                        │                      │
chunks[] ──────────────►│  For each chunk:     │──────────► chunks[] + embedding
 [text, image_data]     │  1. Detect content   │            [text, embedding: [...]]
                        │     type (text/img/  │
                        │     multimodal)      │
                        │  2. Build input      │
                        │  3. Call provider    │
                        │  4. Store vector     │
                        └──────────┬───────────┘
                                   │
                    ┌──────────────┼──────────────┐
                    ▼              ▼              ▼
              ┌──────────┐  ┌──────────┐  ┌──────────┐
              │ Bedrock  │  │  OpenAI  │  │   HTTP   │
              │ Provider │  │ Provider │  │ Provider │
              └────┬─────┘  └────┬─────┘  └────┬─────┘
                   │             │             │
              Titan Text    text-embedding   Your own
              Titan Multi   -3-small/large   model
              Cohere Embed  ada-002          endpoint

Providers

AWS Bedrock

Calls Bedrock InvokeModel in your AWS account. Data never leaves your VPC.

Model	Model ID	Dimensions	Content types
Titan Text Embeddings V2	`amazon.titan-embed-text-v2:0`	256, 512, 1024	Text
Titan Multimodal Embeddings G1	`amazon.titan-embed-image-v1`	1024 (fixed)	Text, image, multimodal
Cohere Embed English v3	`cohere.embed-english-v3`	1024	Text
Cohere Embed Multilingual v3	`cohere.embed-multilingual-v3`	1024	Text

note

Bedrock processes one input per API call (no batch API). For large document sets, consider using Titan Text v2 which is optimized for throughput.

Provider config:

Key	Required/Optional	Description
`region`	Required	AWS region for the model (e.g., `us-east-1`, `us-west-2`). Set per pipeline and independent of the cluster's region and the source bucket's region.
`access_key`	Optional	AWS access key from keystore. When omitted, credentials resolve through the default AWS chain — including IRSA / STS web-identity (`AssumeRoleWithWebIdentity`) on EKS, ECS task roles, and EC2 instance profiles — so a workload identity needs no static keys.
`secret_key`	Optional	AWS secret key from keystore.
`session_token`	Optional	STS session token for temporary credentials.

note

Bring your own model. The model_id is passed straight to Bedrock InvokeModel, so any model your account can invoke works — the Titan and Cohere models above, plus custom or fine-tuned models imported via Bedrock Custom Model Import (referenced by ARN) and Bedrock Marketplace models. The table lists common examples, not an allowlist.

Cross-region and keyless. Because region is per pipeline and STS/IRSA resolves independently of it, you can embed with a model in one region while your data, bucket, and cluster live in others — with no credentials in the pipeline.

OpenAI

Calls the OpenAI Embeddings API. Supports batch processing (up to 100 inputs per call).

Model	Model ID	Dimensions
text-embedding-3-small	`text-embedding-3-small`	1536
text-embedding-3-large	`text-embedding-3-large`	3072
text-embedding-ada-002	`text-embedding-ada-002`	1536

note

OpenAI provider supports text-only embeddings. For multimodal content, use Bedrock or HTTP.

Provider config:

Key	Required/Optional	Description
`api_key_setting`	Required	Keystore setting name (e.g., `ingest.content.openai.api_key`). Raw API keys are not allowed in pipeline configuration.
`endpoint`	Optional	Custom endpoint for Azure OpenAI or proxies. Default is `https://api.openai.com/v1/embeddings`.

HTTP (self-hosted)

Calls any HTTP embedding service. Use this for self-hosted models where data sovereignty requires all inference to stay within your network.

warning

There is no top-level endpoint or url parameter on the embed processor itself. Every HTTP-specific setting -- endpoint, response_embedding_path, auth_header, max_batch_size -- must be nested under provider_config, exactly as shown in the syntax examples on this page. A top-level endpoint is silently ignored, not an error, so a typo here fails as a confusing "endpoint required" error rather than an obvious schema rejection.

Provider config:

Key	Required/Optional	Description
`endpoint`	Required	POST endpoint URL (e.g., `http://embedding-svc.internal:8080/embed`).
`response_embedding_path`	Optional	JSONPath to extract embeddings from response. Default is `$.embeddings`.
`auth_header`	Optional	Authorization header value (e.g., `Bearer <token>`).
`max_batch_size`	Optional	Inputs per HTTP POST. Default is `50`. Cannot raise the processor's HTTP cap of 50; set it lower if your endpoint requires smaller POSTs.

Batching and throughput:

batch_size and max_batch_size both apply, and neither can push a single POST above 50. They compose by nesting rather than by taking the smaller value: chunks are grouped by batch_size, then any group exceeding max_batch_size is split again. batch_size: 5 with max_batch_size: 2 therefore sends 12 chunks as seven requests sized 2, 2, 1, 2, 2, 1, 2 — each five-chunk batch split into 2 + 2 + 1 — not as six requests of 2.

Requests are issued sequentially. A batch is never in flight while another is being sent, and the processor exposes no concurrency control, so parallelism has to come from your endpoint or from splitting the ingest across several bulk requests.

When diagnosing slow ingest through this provider, work through it in this order:

Count chunks, not documents. The processor batches the chunk array named by field (default chunks), so one document can contribute many inputs.
Check whether batching is even in play. With both defaults at 50, a document of fewer than 50 chunks is already a single request. If that is slow, the time is going into your endpoint, not into request overhead, and no batch setting will move it.
Remember that batches do not span documents. Two documents of four chunks each are sent as two POSTs of four, never one of eight, so ingesting many small documents costs one round trip per document no matter how the batch settings are configured.
Measure the endpoint directly. Time a POST to the same URL with the same number of inputs, using the request shape below. That separates your model's inference latency from anything the processor controls.
Only then lower the batch settings — to keep a large batch from pushing your endpoint into timeouts. There is no raising them: 50 is the ceiling.

Request format:

The processor sends one of two shapes, depending on whether any chunk in the batch carries image data. This is not configurable -- your endpoint must handle both.

Text-only batch (the common case -- every chunk in the batch has text and no image_data) uses a flat array of strings:

{
  "texts": ["hello world", "second chunk of text"],
  "model": "your-model-id"
}

As soon as any chunk in the batch has image data, the whole batch switches to structured objects instead, one per input, so text and image inputs can be told apart:

{
  "inputs": [
    {"text": "hello world"},
    {"image": "<base64>", "image_mime_type": "image/jpeg"},
    {"text": "caption", "image": "<base64>", "image_mime_type": "image/png"}
  ],
  "model": "your-model-id"
}

Expected response:

Both request shapes expect the same response -- an array of embedding vectors, in the same order as the input:

{
  "embeddings": [[0.1, 0.2, ...], [0.3, 0.4, ...]]
}

GCP Vertex AI

Embeds via Google Cloud Vertex AI — the model runs in your GCP project. Set model_id to the Vertex model (for example text-embedding-005, or multimodalembedding@001 for text+image). Prefer Workload Identity / Application Default Credentials (no key files): when gcp_access_token is omitted, the provider uses ADC; when it is set, the provider uses that static token.

Provider config:

Key	Required/Optional	Description
`gcp_project`	Required	GCP project ID.
`gcp_location`	Required	Vertex region (for example `us-central1`).
`endpoint`	Optional	Base-URL override (for a private/VPC endpoint).
`gcp_access_token`	Optional	A pre-fetched OAuth token — selects static credential mode. Omit it to use ADC / Workload Identity (recommended).

Azure OpenAI

Embeds via an Azure OpenAI deployment in your Azure resource. Set model_id (or deployment) to the deployment name. Prefer Microsoft Entra (managed identity / AKS workload identity): when api_key is omitted, the provider authenticates with Entra; when it is set, the provider uses the API key.

Provider config:

Key	Required/Optional	Description
`endpoint`	Required	Azure OpenAI resource base URL (for example `https://my-resource.openai.azure.com`).
`deployment`	Optional	The Azure deployment name (falls back to `model_id`).
`api_version`	Optional	Azure REST API version.
`api_key`	Optional	Azure OpenAI API key — selects key credential mode. Omit it to use Entra (recommended).

Azure AI Vision (multimodal)

Embeds text and images into the same vector space via Azure AI Vision 4.0 (dimension 1024). Credentials work like Azure OpenAI (api_key → key mode; omit → Entra).

Provider config:

Key	Required/Optional	Description
`endpoint`	Required	Azure AI Vision resource base URL (for example `https://my-resource.cognitiveservices.azure.com`).
`api_version`	Optional	Azure REST API version.
`model_version`	Optional	Azure AI Vision multimodal model version.
`api_key`	Optional	API key — selects key mode. Omit it to use Entra.

Configuration parameters

Parameter	Data type	Required/Optional	Description
`field`	String	Optional	Source field containing the chunk array. Default is `chunks`.
`model_id`	String	Required	Model identifier (provider-specific).
`provider`	String	Required	Provider name: `bedrock`, `openai`, `http`, `vertex`, `azure`, or `azure_vision`.
`dimensions`	Integer	Optional	Embedding vector dimensions. Default is `1536`. Must be exactly `1024` for Titan Multimodal.
`content_type`	String	Optional	Provider validation hint: `text`, `image`, or `multimodal`. Actual content type is auto-detected per chunk. Default is `text`.
`batch_size`	Integer	Optional	Chunks from one document per provider call. Default is `50`. Does not batch across documents. Effective size is `min(batch_size, provider max)`. Bedrock's max is 1. HTTP's max is 50; `provider_config.max_batch_size` can only lower the HTTP POST size. For how the two interact on the HTTP provider, see Batching and throughput.
`on_failure_action`	String	Optional	`skip` (log warning, continue without embedding -- enables BM25 fallback) or `fail` (reject document). Default is `skip`.
`block_types`	Array	Optional	Chunk types to embed. Non-matching chunks are skipped. Default embeds all types.
`source_uri_field`	String	Optional	For direct image embedding: document field containing image URI. Image is fetched transiently and not stored.
`max_image_embed_bytes`	Integer	Optional	Maximum image size for transient fetch. Default is `26214400` (25 MB).
`reference_config`	Object	Optional	Reference resolver configuration for `source_uri_field`.
`provider_config`	Object	Optional	Provider-specific configuration. See provider sections above.
`description`	String	Optional	A brief description of the processor.
`tag`	String	Optional	An identifier tag for the processor.

Content type detection

The processor auto-detects the content type of each chunk:

Chunk has text + image_data?  →  multimodal embedding
Chunk has text only?          →  text embedding
Chunk has image_data only?    →  image embedding
Chunk has neither?            →  skipped

Inline image data (image_data field in chunks) is limited to 5 MB after base64 decoding. For larger images, use source_uri_field for transient fetch.

Output structure

The processor adds an embedding field to each eligible chunk:

{
  "chunks": [
    {
      "text": "First chunk of text...",
      "chunk_index": 0,
      "embedding": [0.0123, -0.0456, 0.0789, ...]
    },
    {
      "text": "Second chunk...",
      "chunk_index": 1,
      "embedding": [0.0234, -0.0567, 0.0891, ...]
    }
  ]
}

When using source_uri_field, the embedding is stored at the document root:

{
  "source_uri": "s3://bucket/image.jpg",
  "embedding": [0.0123, -0.0456, 0.0789, ...]
}

Security

warning

Raw API keys are not allowed in pipeline configuration. Use the Lucenia keystore to store credentials securely.

# Store an OpenAI API key
bin/lucenia-keystore add ingest.content.openai.api_key

# Store AWS credentials (alternative to instance profile)
bin/lucenia-keystore add ingest.content.bedrock.access_key
bin/lucenia-keystore add ingest.content.bedrock.secret_key

Then reference the keystore setting in your pipeline:

{
  "embed": {
    "provider_config": {
      "api_key_setting": "ingest.content.openai.api_key"
    }
  }
}

Using the processor

Example 1: Text embeddings with Bedrock Titan

PUT _ingest/pipeline/text-embed
{
  "processors": [
    {
      "embed": {
        "field": "chunks",
        "model_id": "amazon.titan-embed-text-v2:0",
        "provider": "bedrock",
        "dimensions": 1024,
        "provider_config": {
          "region": "us-east-2"
        }
      }
    }
  ]
}

Example 2: Multimodal embeddings with Titan Multimodal

PUT _ingest/pipeline/multimodal-embed
{
  "processors": [
    {
      "embed": {
        "field": "chunks",
        "model_id": "amazon.titan-embed-image-v1",
        "provider": "bedrock",
        "dimensions": 1024,
        "content_type": "multimodal",
        "provider_config": {
          "region": "us-east-1"
        }
      }
    }
  ]
}

Example 3: Self-hosted model via HTTP

PUT _ingest/pipeline/self-hosted-embed
{
  "processors": [
    {
      "embed": {
        "field": "chunks",
        "model_id": "sentence-transformers/all-MiniLM-L6-v2",
        "provider": "http",
        "dimensions": 384,
        "provider_config": {
          "endpoint": "http://embedding-service.internal:8080/embed",
          "max_batch_size": 32
        }
      }
    }
  ]
}

Example 4: Direct image embedding from S3

Embed an image without storing it in the document -- the image is fetched transiently from S3, embedded, and discarded:

PUT _ingest/pipeline/image-embed
{
  "processors": [
    {
      "embed": {
        "source_uri_field": "image_uri",
        "model_id": "amazon.titan-embed-image-v1",
        "provider": "bedrock",
        "dimensions": 1024,
        "provider_config": {
          "region": "us-east-1"
        }
      }
    }
  ]
}

PUT /images/_doc/1?pipeline=image-embed
{
  "image_uri": "s3://my-bucket/photos/landscape.jpg",
  "title": "Mountain landscape"
}

Syntax​

Architecture​

Providers​

AWS Bedrock​

OpenAI​

HTTP (self-hosted)​

GCP Vertex AI​

Azure OpenAI​

Azure AI Vision (multimodal)​

Configuration parameters​

Content type detection​

Output structure​

Security​

Using the processor​

Example 1: Text embeddings with Bedrock Titan​

Example 2: Multimodal embeddings with Titan Multimodal​

Example 3: Self-hosted model via HTTP​

Example 4: Direct image embedding from S3​

Syntax

Architecture

Providers

AWS Bedrock

OpenAI

HTTP (self-hosted)

GCP Vertex AI

Azure OpenAI

Azure AI Vision (multimodal)

Configuration parameters

Content type detection

Output structure

Security

Using the processor

Example 1: Text embeddings with Bedrock Titan

Example 2: Multimodal embeddings with Titan Multimodal

Example 3: Self-hosted model via HTTP

Example 4: Direct image embedding from S3