End-to-end content processing

Lucenia provides a complete content processing pipeline built directly into its ingest framework. Raw documents — PDFs, Word files, HTML pages, images, and even satellite imagery — are transformed into vector-searchable content through a series of composable processors that you configure as a single ingest pipeline.

Pipeline overview

A typical content processing pipeline chains the following processors:

Document ──► content_extract ──► compliance ──► chunk ──► embed ──► Index
                  │              (redact PII)
                  ├──► ocr (for images/charts)
                  └──► image_tiling (for GeoTIFF/satellite imagery)

Each processor reads the output of the previous one. You define the entire pipeline in a single API call, and Lucenia handles the orchestration.

Content extraction

The content_extract processor extracts structured content blocks from documents in a wide range of formats.

Supported formats:

Category	Formats
Documents	PDF, Word (DOCX), Excel (XLSX), PowerPoint (PPTX)
Web	HTML
Text	Plain text, Markdown, JSON
Images	JPEG, PNG, TIFF, GeoTIFF (with geospatial metadata)

Input modes:

Mode	Description	Default cap	Use case
reference	Stream from S3 or HTTPS URI	250 MB	Large documents in cloud storage (recommended)
inline	Text embedded directly in the request	1 MB	Small text payloads via API
stream	Multipart binary upload (POST /{index}/_ingest)	100 MB	Direct upload from client applications
attachment	Base64-encoded content	10 MB	Legacy compatibility

Caps apply per document. The reference mode is recommended for most use cases — it lets the cluster fetch content directly from S3 or HTTPS without uploading data through the API, supporting both private and public S3 buckets with multi-region support.

PDF and Office extraction is text only. For scanned PDFs, see Scanned PDFs.

Redaction and compliance

The compliance processor detects and redacts sensitive data — PII, PHI, payment data, and secrets — from extracted text before it is chunked, embedded, or indexed, so regulated or classified values never enter an index in the clear. Apply a named profile (GDPR, HIPAA, PCI-DSS, SOC 2, FedRAMP, CCPA), override the redaction mode per entity, or run in audit-only mode to discover what a source contains without altering it.

Place it right after content_extract so both the stored fields and the vectors are built from clean text:

{ "compliance": { "profile": "hipaa", "fields": ["content"] } }

Redaction runs in-process, in your cluster — nothing leaves your trust boundary. See the compliance processor for the full profile, entity, and redaction-mode reference.

Chunking

The chunk processor splits extracted content into smaller, overlapping segments optimized for vector search and RAG. Four algorithms are available:

Algorithm	Strategy	Best for
recursive (default)	Splits by paragraphs, then sentences, then words	General-purpose documents
fixed	Splits at fixed character intervals with overlap	Uniform chunk sizes
semantic	Splits where embedding similarity drops below threshold	Topic-coherent chunks
topic_shift	Splits where vocabulary overlap between windows drops	Detecting topic boundaries

The semantic and topic_shift algorithms use embedding similarity and vocabulary analysis respectively to find natural breakpoints in the text, producing chunks that are more meaningful for retrieval.

Embedding

The embed processor generates vector embeddings from text, image, or multimodal content. Six embedding providers are supported:

Provider	Models	Data privacy
AWS Bedrock	Titan Text v2, Titan Multimodal G1, Cohere Embed v3	Data stays in your VPC
OpenAI	text-embedding-3-small, text-embedding-3-large, ada-002	Sent to OpenAI API
HTTP (self-hosted)	Any model with a REST endpoint	Fully private
GCP Vertex AI	text-embedding-005, multimodalembedding@001	Stays in your GCP project
Azure OpenAI	Your Azure deployment	Stays in your Azure resource
Azure AI Vision	Vision 4.0 (multimodal, 1024-dim)	Stays in your Azure resource

For maximum privacy, use a self-hosted model via the HTTP provider (fully private) or an in-account cloud provider. See the embed processor for each provider's configuration.

OCR

The ocr processor extracts text from images, charts, diagrams, and tables using LLM-powered inference (Bedrock Claude) or the HTTP inference provider. Unlike traditional OCR engines, the Bedrock path understands the semantic structure of visual content — extracting table data as structured text, reading diagram labels, and interpreting chart values.

Image tiling

The image_tiling processor splits large images into a grid of tiles for multimodal search. For geospatial imagery (Cloud Optimized GeoTIFFs), it extracts spatial metadata and indexes each tile with its geographic bounding box, enabling spatially-aware image search.

Rerank preparation

The rerank_prepare processor annotates chunks with metadata and position scores needed by the downstream multimodal rerank search processor. This enables the search pipeline to rerank results using the full context of the original document.

Example: Full pipeline from PDF to searchable vectors

PUT _ingest/pipeline/ai-retrieval
{
  "description": "Extract, chunk, and embed PDF documents",
  "processors": [
    {
      "content_extract": {
        "field": "content",
        "target_field": "extracted",
        "input_mode": "reference",
        "source_uri_field": "source_uri"
      }
    },
    {
      "chunk": {
        "field": "extracted.blocks",
        "target_field": "chunks",
        "algorithm": "recursive",
        "chunk_size": 2000,
        "chunk_overlap": 200
      }
    },
    {
      "embed": {
        "field": "chunks",
        "model_id": "amazon.titan-embed-text-v2:0",
        "provider": "bedrock",
        "dimensions": 1024,
        "provider_config": {
          "region": "us-east-2"
        }
      }
    }
  ]
}

Then ingest a document:

PUT /knowledge-base/_doc/1?pipeline=ai-retrieval
{
  "title": "Quarterly Report Q4 2024",
  "source_uri": "s3://my-docs-bucket/reports/q4-2024.pdf"
}

Lucenia fetches the PDF from S3, extracts the text, chunks the content into overlapping segments, generates vector embeddings for each chunk, and indexes everything — all in a single request. Scanned PDFs with no text layer need page images sent through ocr separately.