Content ingest

Introduced 0.11.0

POST /{index}/_ingest indexes one document as multipart/related: JSON metadata plus a raw binary part. The binary is attached as a transient field, the ingest pipeline runs on the coordinating node that received the request, and the raw bytes are not written to _source.

Use this with a content_extract processor whose input_mode is stream.

Path and HTTP methods

POST /{index}/_ingest?pipeline={pipeline}
POST /{index}/_ingest/{id}?pipeline={pipeline}

Item	Required	Description
index	Yes	Target index.
id	No	Document ID. Omitted: auto-generated.
pipeline	Yes	Ingest pipeline. Must include content_extract with input_mode: "stream".

Other query parameters match index-document: routing, timeout, refresh.

Content-Type

The request Content-Type must start with multipart/related and include a boundary.

RFC 2387 multipart/related: part 0 is JSON metadata, part 1 is the binary. Extra parts are ignored.

Request body

POST /my-index/_ingest/doc-1?pipeline=stream-extract
Content-Type: multipart/related; boundary=boundary123

--boundary123
Content-Type: application/json

{"title": "My Document"}
--boundary123
Content-Type: application/pdf

<raw PDF bytes>
--boundary123--

Part	Content-Type	Body
First	Must contain json (typically application/json)	Document _source metadata.
Second	The file's MIME type (application/pdf, image/png, …)	Raw bytes. content_extract uses this header as the MIME type when present. Base64 in this part is stored as encoded bytes and extraction fails.

Pipeline

Create the pipeline first:

PUT _ingest/pipeline/stream-extract
{
  "processors": [
    {
      "content_extract": {
        "input_mode": "stream",
        "target_field": "extracted",
        "max_stream_bytes": 104857600
      }
    }
  ]
}

Ingest fails if the pipeline is missing or input_mode is not stream. The pipeline runs on the coordinating node that received the request.

Size limits

Two independent ceilings, both default 100 MB:

Limit	Where	Default
max_stream_bytes	content_extract processor on the pipeline	104857600
http.max_content_length	Cluster setting (lucenia.yml)	100mb

The HTTP body is fully buffered, so both limits apply. To accept a 200 MB PDF over multipart, raise both. The coordinating node holds that body on heap until extraction finishes. For files that large, reference mode (index an S3 or HTTPS URI) is the intended path.

curl example

Build the full multipart body (JSON part, PDF bytes, closing boundary), then POST it. A heredoc that ends before the file contents sends an empty binary part.

{
  printf '%s\r\n' '--boundary123' 'Content-Type: application/json' '' '{"title": "Q4 report"}'
  printf '%s\r\n' '--boundary123' 'Content-Type: application/pdf' ''
  cat report.pdf
  printf '\r\n%s\r\n' '--boundary123--'
} | curl -sS -X POST "https://localhost:9200/my-index/_ingest/doc-1?pipeline=stream-extract" \
  -H "Content-Type: multipart/related; boundary=boundary123" \
  --data-binary @-

Response

Success looks like a normal index response (_index, _id, _version, result). Failures return 4xx/5xx with the processor or size-limit message (max_stream_bytes, missing pipeline, wrong Content-Type).