Query Parquet files

Introduced 0.12.0

The Parquet module mounts a Parquet file as a read-only Lucenia index and lets you query it in place with the normal search, aggregation, and GET APIs — no ingest, no copy. The Parquet file is the source of truth; the index is a thin, read-only view over it, served by a custom Lucene codec that reads the file's data pages directly at query time.

important

This is an experimental feature and is not recommended for use in a production environment. For updates on its progress, see the Lucenia version history.

The module ships with the Lucenia distribution and is enabled by default. To confirm that it is loaded, check the installed plugins:

GET _cat/plugins?v

The response includes an entry for ParquetCodecPlugin.

note

A mounted index is read-only and exposes a deliberately limited query surface: doc-values range queries, term and _id queries, and numeric aggregations. To get the full query and aggregation surface — indexed points, keyword aggregations, and sort — promote the file to a regular index with _reindex (see Promoting to a regular index).

How it works

The module maps a Parquet file onto Lucene's read APIs with a fixed, one-to-one model:

• One row → one document (docID = the Parquet row index)
• One column → one field
• One file → one segment → one shard

Because the file is the segment, a mounted index is always single-shard and cannot be written to. The schema is inferred from the Parquet footer and validated before the index is created, so a bad file fails the mount cleanly (a 400) instead of creating a broken index.

Mount a Parquet file

Mount a file by sending a PUT request to /_plugins/parquet/<index>, where <index> is the name of the index to create:

PUT /_plugins/parquet/mtcars
{
  "path": "data/cars.parquet"
}

The request body accepts only the following two fields. All index settings (codec, path, single shard, and — for local files — zero replicas) are derived server-side and cannot be set through the request.

Field	Data type	Required	Description
path	String	Yes	Path or URI to the Parquet file. See Path sources.
id_column	String	No	A column whose values become each document's _id. See Using an ID column. If omitted, the row index is the _id.

Before creating the index, Lucenia validates that the file exists, is readable, and is a parseable Parquet file, and then infers a field mapping from its schema. If any check fails, the request returns a 400 and no index is created. See Mount errors.

Mount response

The response reports the row count, the inferred queryable fields (with their mapped types), and any columns that were skipped:

{
  "acknowledged": true,
  "index": "mtcars",
  "row_count": 32,
  "fields": {
    "model": "keyword",
    "mpg": "double",
    "cyl": "integer",
    "hp": "integer"
  },
  "skipped_columns": []
}

A skipped column is reported as an object with its name, Parquet type, and the reason it was not mapped:

{
  "skipped_columns": [
    { "name": "geometry", "parquet_type": "BINARY", "reason": "non-string binary columns are not mapped" }
  ]
}

See Field-type mapping for which columns are mapped and which are skipped.

Path sources

The path field accepts local paths and URIs. The path form determines where the file is read from and how the index is configured.

Path form	Example	Backed by	Notes
Relative	data/cars.parquet	Node config directory	Resolved against the node's config directory; always allowed.
Absolute local	/srv/parquet/cars.parquet	Node filesystem	Must be under a parquet.allowed_paths entry; replicas are forced to 0.
file:// URI	file:///srv/parquet/cars.parquet	Node filesystem	Treated as a local path (allowlist applies; replicas forced to 0).
Hadoop URI	hdfs://nn/path/file.parquet, s3a://bucket/file.parquet	Hadoop FileSystem	Requires the connector on the classpath. See Object storage and Hadoop filesystems.

Local paths (both bare paths and file:// URIs) are read through java.nio. Because only the mounting node holds a local file, a local mount forces number_of_replicas: 0. A Hadoop URI is reachable from every node, so its replica count is left to the cluster default.

tip

The file:// scheme is matched case-insensitively, so FILE:///... is also treated as a local path and is subject to the allowlist.

Object storage and Hadoop filesystems

Any Hadoop-filesystem URI is opened through Hadoop's FileSystem abstraction. The scheme resolves only if its connector is on the plugin classpath. Out of the box, only the filesystems bundled with hadoop-common resolve (file, http, https, har, and viewfs). To mount from a real object store, add the connector and its Hadoop configuration:

• HDFS (hdfs://): add hadoop-client-api and hadoop-hdfs; configure hdfs-site.xml.
• Amazon S3 (s3a://): add hadoop-aws and the AWS SDK; configure core-site.xml.
• GCS / Azure: add the respective connector JAR.

A URI whose connector is not installed fails the mount with a 400 (no filesystem for the scheme), and no index is created.

warning

Credentials for object storage come exclusively from the Hadoop configuration (core-site.xml / hdfs-site.xml on the classpath or under HADOOP_CONF_DIR). A credential is never accepted in the mount request or stored in index settings.

Node settings

Set the following node-scoped setting in opensearch.yml.

Setting	Description
parquet.allowed_paths	A list of directories under which absolute local Parquet files may be mounted, in addition to the always-allowed node config directory. Prevents a mount request from reading arbitrary files off the node. Does not apply to Hadoop URIs.

For example, to allow mounting files stored under /srv/parquet:

parquet.allowed_paths: ["/srv/parquet"]

A local absolute path that resolves (after symlinks are followed) outside the node config directory and every parquet.allowed_paths entry is rejected with a 400.

Field-type mapping

Each top-level primitive column is mapped to an index field. The mapping keys on the column's physical type; a logical-type annotation (DECIMAL, DATE, TIMESTAMP, or unsigned integer) refines it.

Parquet column	Mapped field type	Field settings	Notes
INT32	integer	index: false	Range, term, sort, and aggregations run on doc values.
INT64	long	index: false	Same as above.
FLOAT	float	index: false
DOUBLE	double	index: false
INT32 / INT64 with DECIMAL	double	index: false	Value is decoded with the column's scale (for example, unscaled 1234 at scale 2 becomes 12.34).
INT32 with DATE	long	index: false	Stored as epoch milliseconds, not as the date type.
INT32 / INT64 with TIMESTAMP	long	index: false	Epoch milliseconds; microsecond and nanosecond units are truncated to milliseconds.
Unsigned integer ≤ 32 bits	long	index: false	Zero-extended into a signed long.
BINARY (UTF-8 string)	keyword	doc_values: false	Term and match queries run on codec postings. Aggregations and sort are not available.
BOOLEAN	boolean	index: false, doc_values: false	Preserved in _source; not directly queryable on the mount (see Query a mounted index).

warning

Dates and timestamps are mapped as long epoch milliseconds, not as the date type. The codec writes no Lucene points, and an index: false date field has no doc-values range path, so it would silently match nothing. Query a temporal column with a numeric range over epoch-millisecond values — for example, { "range": { "ts": { "gte": 1609459200000 } } }.

Skipped columns

Columns that cannot be faithfully served as a queryable field are omitted from the mapping and reported in skipped_columns:

Column	Reason
Nested columns (struct or list leaves, such as grp.a)	nested columns are not supported
Repeated (array) columns	repeated (array) columns are not supported
Non-string BINARY (for example, WKB geometry)	non-string binary columns are not mapped
FIXED_LEN_BYTE_ARRAY (including fixed-length DECIMAL and UUID), INT96	unsupported parquet type
Unsigned 64-bit integer	unsupported logical type [...]

note

A skipped column is not queryable, but it is still emitted into _source (binary columns as base64 strings). This is deliberate: it lets values such as GeoParquet WKB geometry survive a _reindex so that you can remap or decode them on the destination. Null cells are omitted from _source rather than rendered as null.

Using an ID column

By default, a document's _id is its Parquet row index (0, 1, 2, and so on), so you retrieve a row by its position:

GET /mtcars/_doc/0

Row indices shift if the file is regenerated in a different order. To get stable IDs, set id_column at mount time to a column whose values become each row's _id:

PUT /_plugins/parquet/mtcars
{
  "path": "data/cars.parquet",
  "id_column": "model"
}

You then retrieve a document by that column's value, and the bare row index is no longer a valid ID:

GET /mtcars/_doc/Valiant

The id_column must:

• Name an inferred, top-level primitive column (one that appears in the mount response's fields).
• Be non-null for every row. A column with any null value is rejected at mount time.

Uniqueness is not enforced. If the column has duplicate values, a GET resolves to the last matching row, and a _reindex is last-write-wins on the destination (so the destination may hold fewer documents than the source). For a non-string column, the value's string form is used as the ID (for example, an id_column value of 20 is retrieved with GET /mtcars/_doc/20). Like index.parquet.path, the ID column is fixed at creation time and cannot be changed.

Query a mounted index

A mounted index answers _search, _count, and GET requests. Which clauses work depends on how each column is mapped:

• Numeric columns are mapped index: false, so no Lucene points are written. Range and term queries run against the codec's sortable-encoded doc values instead, and numeric aggregations and sort work because doc values are present.
• Keyword (string) columns are mapped doc_values: false. Term and match queries run against the codec's postings, but aggregations and sort on a keyword column are not available.
• Boolean columns are neither indexed nor doc-valued, so they are not directly queryable on the mount. Their values are preserved in _source and become queryable after a _reindex.

Examples

Range query on a numeric column (runs on doc values):

POST /mtcars/_search
{
  "query": { "range": { "mpg": { "gte": 31.0, "lte": 35.0 } } }
}

Term query on a keyword column (runs on postings):

POST /mtcars/_search
{
  "query": { "term": { "model": "Valiant" } }
}

Numeric aggregation (runs on doc values):

POST /mtcars/_search
{
  "size": 0,
  "aggs": { "avg_mpg": { "avg": { "field": "mpg" } } }
}

Because null cells are stored sparsely rather than zero-filled, exists (and its negation) and numeric aggregations count only the rows that actually have a value:

POST /mtcars/_search
{
  "query": { "exists": { "field": "hp" } }
}

_count, and _search pagination with from, size, and sort on _id, all work against the mounted index.

Read-only semantics

A parquet-backed index rejects every document write — index, update, delete, and _bulk — with a 400 that names the index as a read-only parquet-backed index. The index.parquet.path setting is final and cannot be changed after creation.

Supported compression codecs

Only the UNCOMPRESSED and SNAPPY Parquet compression codecs are supported. A file whose column chunks use any other codec — such as ZSTD, GZIP, LZ4, LZO, or BROTLI — is rejected at mount time with a 400, before the index is created.

warning

Some Parquet writers default to a compression codec other than Snappy (for example, ZSTD). When you generate a file for mounting, set the compression explicitly to snappy or to no compression.

Promoting to a regular index with _reindex

When a mounted file is worth keeping, promote it with a standard _reindex — there is no special API:

POST /_reindex
{
  "source": { "index": "mtcars" },
  "dest":   { "index": "mtcars_ingested" }
}

Reindex scrolls the parquet-backed source (reading each row's _source) and bulk-indexes into the destination. The destination is a regular index that gets its own mapping — dynamic, or one you create first — and full indexing: Lucene points, keyword field data, aggregations, and sort that the read-only parquet view does not provide. If the source was mounted with an id_column, the reindex carries that column's value into the destination as the document _id.

After reindexing, you can delete the mounted index. Deleting it never touches the underlying Parquet file.

Limitations

• Read-only. No index, update, delete, or _bulk writes.
• Single shard. One file maps to one segment and one shard. A local-file mount runs on a single node (number_of_replicas: 0); mount through a Hadoop or object-storage URI to allow replicas across nodes.
• Limited query surface. Numeric range and term queries and numeric aggregations; keyword term and match queries only (no keyword aggregations or sort); boolean columns are not directly queryable. Promote with _reindex for the full surface.
• Schema constraints. Nested, array, non-string-binary, INT96, fixed-length, and unsigned 64-bit columns are skipped (but still appear in _source).
• Compression. Only Snappy and uncompressed files are supported.
• Memory. Per-column term dictionaries and the synthetic _id sort are built in heap on first term or GET access and cached for the segment's lifetime. A very large file can put pressure on heap, so size the node accordingly for large mounts.

Mount errors

All of the following return a 400 and create no index.

Condition	Message contains
Missing or empty path	missing required field [path]
Unknown field in the body	the parquet mount request accepts only [path, id_column]
File does not exist, is not a regular file, or is not readable	Invalid path [...]
Local path outside the config directory and every parquet.allowed_paths entry	not under the node config directory or any parquet.allowed_paths entry
URI scheme with no installed connector	Invalid path [...]
Not a valid Parquet file	not a readable parquet file: ...
Column named like a reserved field (_id, _source, _version, _seq_no, _primary_term, parquet_seed)	reserved column named [...]
Unsupported compression codec	unsupported compression codec [...]
File has zero rows	has no rows; there is nothing to mount
id_column is not an inferred primitive column	is not a primitive column of the parquet file
id_column has one or more null values	an id column must be non-null