Content extraction processor

Introduced 0.11.0

The content_extract processor extracts structured content blocks from documents in various formats, including PDF, DOCX, HTML, plain text, and images. It uses Apache Tika under the hood and supports multiple input modes: inline text, S3/HTTPS references, multipart streams, and base64-encoded attachments.

The processor outputs a normalized array of ContentBlock objects containing extracted text and, for image/* sources, optional image bytes, along with document-level metadata (language, page count, MIME type). Downstream processors such as chunk, ocr, and embed consume these blocks.

Syntax

The following is the syntax for the content_extract processor:

{
  "content_extract": {
    "field": "content",
    "target_field": "extracted",
    "input_mode": "reference",
    "source_uri_field": "source_uri"
  }
}

Input modes

The processor supports the following input modes. Size caps apply per document.

Mode	Description	Max size (default)	Use case
inline	Text provided directly in the document field	1 MB	Small text payloads via API
reference	URI (S3, Google Cloud Storage gs://, Azure Blob, or HTTPS) pointing to the source content -- the cluster fetches the document directly	250 MB	PDFs, DOCX, images in cloud storage or on the web
stream	Multipart binary upload (POST /{index}/_ingest). The HTTP body is fully buffered on the coordinating node.	100 MB	Direct upload from client applications
attachment	Base64-encoded content in the document field (deprecated -- use stream or reference instead)	10 MB	Legacy compatibility

tip

Use reference mode for most use cases. It lets the cluster fetch content directly from cloud storage (Amazon S3, Google Cloud Storage, Azure Blob) or HTTPS without uploading data through the API. This is the most efficient approach for large documents and supports both private and public (anonymous) buckets.

max_inline_bytes, max_stream_bytes, max_reference_bytes, and max_attachment_bytes are optional processor settings (integers, in bytes) that override the defaults above. Stream mode is also bounded by the cluster setting http.max_content_length (default 100 MB in lucenia.yml). Raising max_stream_bytes without raising http.max_content_length still rejects the request.

See Content ingest API for the stream request format. After extraction, stream-mode bytes are dropped and are not stored in _source.

Configuration parameters

The following table lists the required and optional parameters for the content_extract processor.

Parameter	Data type	Required/Optional	Description
field	String	Optional	The source field containing content or used as a placeholder for reference mode. Default is content.
target_field	String	Optional	The field where extracted content blocks are stored. Default is extracted.
input_mode	String	Optional	The input mode. Valid values are inline, reference, stream, attachment. Default is inline.
source_uri_field	String	Optional	The document field containing the source URI — S3 (s3://), Google Cloud Storage (gs://), Azure Blob (https://<account>.blob.core.windows.net/...), or HTTPS. Required when input_mode is reference.
region_field	String	Optional	A document field that overrides the S3 region per document. Enables multi-region ingestion in a single pipeline.
mime_type_field	String	Optional	The document field containing the MIME type. If not specified, the MIME type is taken from the stream part's Content-Type, the reference resolver, or mime_type, and otherwise defaults to text/plain.
mime_type	String	Optional	A static MIME type override for all documents processed by this pipeline.
preserve_image_data	Boolean	Optional	Whether to keep base64-encoded image data in extracted image blocks. Required if a downstream ocr processor needs those bytes. Default is false. Applies to image/* extractors only.
max_inline_bytes	Integer	Optional	Maximum size for inline content. Default is 1048576 (1 MB).
max_reference_bytes	Integer	Optional	Maximum size for reference content. Default is 262144000 (250 MB).
max_stream_bytes	Integer	Optional	Maximum size for stream content. Default is 104857600 (100 MB). Also limited by http.max_content_length.
max_attachment_bytes	Integer	Optional	Maximum decoded size for deprecated attachment mode. Default is 10485760 (10 MB).
s3_client	String	Optional	Named S3 client for reference mode (same key names as repository-s3). Default is default. Set this on the processor, not inside reference_config.
reference_config	Object	Optional	Configuration for reference resolution. See Reference configuration.
description	String	Optional	A brief description of the processor.
tag	String	Optional	An identifier tag for the processor.

Reference configuration

When using reference input mode, the reference_config object controls how URIs are resolved. Values are strings.

Parameter	Data type	Required/Optional	Description
region	String	Optional	The AWS region for S3 URIs.
anonymous	String	Optional	Set to "true" for public S3 buckets that don't require credentials (e.g., NASA open data). Default is "false".
presign_duration_minutes	String	Optional	Duration for presigned S3 URLs. Default is "60".
endpoint	String	Optional	S3-compatible endpoint override (MinIO, LocalStack). Forces path-style access.
access_key / secret_key / session_token	String	Optional	Per-request credential override. Prefer the Lucenia keystore.

The keys above apply to s3:// URIs. Google Cloud Storage and Azure Blob URIs use the following reference_config keys.

HTTP / HTTPS:

Parameter	Description
allow_http	Allow plaintext http:// URIs. Default "false" (HTTPS only).
username / password	HTTP basic auth.
header.<name>	Extra request header. Example: "header.X-Custom": "value".
connect_timeout_seconds	Connect timeout. Default 30.
socket_timeout_seconds	Socket / response timeout. Default 60.

Reference mode probes metadata first (HTTP HEAD / S3 HeadObject), then opens a stream when the extractor runs. The raw object is never written to _source. Extracted text is held in memory up to max_reference_bytes.

Google Cloud Storage (gs://bucket/object or gcs://bucket/object):

Parameter	Data type	Required/Optional	Description
anonymous	String	Optional	Set to "true" for public buckets. Default is "false".
credentials_json	String	Optional	Inline service-account JSON (from keystore). When omitted and not anonymous, credentials resolve through Application Default Credentials — including GKE Workload Identity — so no static key is needed.
project_id	String	Optional	The GCP project id.

Azure Blob Storage (https://<account>.blob.core.windows.net/<container>/<blob>):

Parameter	Data type	Required/Optional	Description
connection_string	String	Optional	A full Azure Storage connection string (from keystore).
account_key	String	Optional	A storage account shared key (from keystore).
sas_token	String	Optional	A shared access signature appended to requests.
anonymous	String	Optional	Public containers require no credentials (the default when no key is configured).

S3 credentials resolve in order: explicit keys in reference_config → s3.client.<name>.* keystore settings (same names as repository-s3) → the AWS default chain (instance profile, env, ~/.aws/credentials).

Output structure

The processor writes the following structure to the target_field. A PDF:

{
  "extracted": {
    "blocks": [
      {
        "type": "text",
        "text": "Extracted text content from the document..."
      }
    ],
    "metadata": {
      "content_type": "application/pdf",
      "language": "en",
      "page_count": 12
    }
  }
}

An image/* source with preserve_image_data: true yields an image block (image_data, image_mime_type, width/height) instead of a text block. When input_mode is reference, extraction output also includes a reference object with uri and, when known, content_length.

Supported formats

The processor supports the following document formats:

Format	MIME type	Extraction
PDF	application/pdf	Text only, via Tika. Scanned PDFs often yield empty text. See Scanned PDFs.
Word (DOCX)	application/vnd.openxmlformats-officedocument.wordprocessingml.document	Full text extraction including tables.
Excel (XLSX)	application/vnd.openxmlformats-officedocument.spreadsheetml.sheet	Cell values extracted as text.
PowerPoint (PPTX)	application/vnd.openxmlformats-officedocument.presentationml.presentation	Slide text extraction.
HTML	text/html	Text extracted from rendered HTML.
Plain text	text/plain	Passed through directly.
Markdown	text/markdown	Passed through directly.
JSON	application/json	Passed through directly.
Images	image/*	EXIF metadata extracted (GPS, camera info, dimensions). Optionally preserves image data for embedding or OCR.
GeoTIFF	image/tiff	Dimensions and EXIF. CRS, geo-transform, and COG range reads are handled by image_tiling.

Scanned PDFs

Tika reads a PDF's text layer. A scan with no text layer produces an empty or near-empty text block. This processor does not emit a page image per PDF page.

Rasterize each page to PNG or JPEG outside Lucenia, then OCR those images:

1. Index each page image with content_extract (input_mode reference or stream, MIME image/*, preserve_image_data: true) and an ocr processor on extracted.blocks.
2. Or index a page-image URI and run ocr with source_uri_field.

Then chunk and embed the OCR text.

Using the processor

Example 1: Extract text from an S3 document

Step 1: Create a pipeline

PUT _ingest/pipeline/doc-extract
{
  "description": "Extract content from S3 documents",
  "processors": [
    {
      "content_extract": {
        "field": "content",
        "target_field": "extracted",
        "input_mode": "reference",
        "source_uri_field": "source_uri",
        "reference_config": {
          "region": "us-east-2"
        }
      }
    }
  ]
}

Step 2: Ingest a document

Point at an S3 object -- the cluster fetches it directly:

PUT /my-documents/_doc/1?pipeline=doc-extract
{
  "source_uri": "s3://my-bucket/reports/annual-report.pdf",
  "title": "2024 Annual Report"
}

Example 2: Extract from a public HTTPS URL

PUT /my-documents/_doc/2?pipeline=doc-extract
{
  "source_uri": "https://example.com/research-paper.pdf",
  "title": "Research Paper"
}

Plaintext http:// URIs fail unless reference_config.allow_http is "true".

Example 3: Extract from a public S3 bucket (anonymous access)

PUT _ingest/pipeline/nasa-extract
{
  "description": "Extract from NASA public S3 bucket",
  "processors": [
    {
      "content_extract": {
        "field": "content",
        "target_field": "extracted",
        "input_mode": "reference",
        "source_uri_field": "source_uri",
        "reference_config": {
          "region": "us-west-2",
          "anonymous": "true"
        }
      }
    }
  ]
}

Example 4: Inline text extraction

PUT /my-documents/_doc/3?pipeline=inline-extract
{
  "content": "This is inline text content that will be extracted and split into content blocks."
}

Example 5: Multi-region ingestion with per-document region override

If your documents span multiple S3 regions, use region_field to override the region per document:

PUT _ingest/pipeline/multi-region
{
  "processors": [
    {
      "content_extract": {
        "input_mode": "reference",
        "source_uri_field": "source_uri",
        "region_field": "doc_region",
        "reference_config": {
          "region": "us-east-1"
        }
      }
    }
  ]
}

Then each document can specify its own region:

PUT /docs/_doc/1?pipeline=multi-region
{
  "source_uri": "s3://east-bucket/report.pdf",
  "doc_region": "us-east-1"
}

PUT /docs/_doc/2?pipeline=multi-region
{
  "source_uri": "s3://west-bucket/report.pdf",
  "doc_region": "us-west-2"
}

Example 6: Multipart upload (stream mode)

PUT _ingest/pipeline/stream-extract
{
  "processors": [
    {
      "content_extract": {
        "input_mode": "stream",
        "target_field": "extracted",
        "max_stream_bytes": 104857600
      }
    }
  ]
}

Then POST /my-documents/_ingest?pipeline=stream-extract with Content-Type: multipart/related. See Content ingest API.

Combining with other processors

The content_extract processor is typically the first step in a pipeline chain. Common patterns:

PDF / text pipeline (extract + chunk + embed):

PUT _ingest/pipeline/full-document-pipeline
{
  "processors": [
    { "content_extract": { "input_mode": "reference", "source_uri_field": "source_uri" } },
    { "chunk": { "field": "extracted.blocks", "target_field": "chunks", "algorithm": "recursive", "chunk_size": 2000 } },
    { "embed": { "field": "chunks", "model_id": "amazon.titan-embed-text-v2:0", "provider": "bedrock", "dimensions": 1024, "provider_config": { "region": "us-east-2" } } }
  ]
}

Image OCR pipeline:

PUT _ingest/pipeline/image-ocr-pipeline
{
  "processors": [
    { "content_extract": { "input_mode": "reference", "source_uri_field": "source_uri", "preserve_image_data": true } },
    { "ocr": { "field": "extracted.blocks", "model_id": "anthropic.claude-3-haiku-20240307-v1:0", "provider": "bedrock", "block_types": ["image"], "provider_config": { "region": "us-east-2" } } },
    { "chunk": { "field": "extracted.blocks", "target_field": "chunks", "algorithm": "recursive", "chunk_size": 2000 } },
    { "embed": { "field": "chunks", "model_id": "amazon.titan-embed-text-v2:0", "provider": "bedrock", "dimensions": 1024, "provider_config": { "region": "us-east-2" } } }
  ]
}

GeoTIFF pipeline (extract + tile + embed):

PUT _ingest/pipeline/geotiff-pipeline
{
  "processors": [
    { "content_extract": { "input_mode": "reference", "source_uri_field": "source_uri" } },
    { "image_tiling": { "field": "extracted.blocks", "target_field": "chunks", "profile": "geo_search" } },
    { "embed": { "field": "chunks", "model_id": "amazon.titan-embed-image-v1", "provider": "bedrock", "dimensions": 1024, "provider_config": { "region": "us-east-2" } } }
  ]
}

Custom extractors

Developers can implement custom content extractors by implementing the ContentExtractor SPI interface and registering it via META-INF/services. Custom extractors are selected by MIME type and priority, allowing you to override the default extraction behavior for specific formats. See Extending ingest-content for details.