Skip to main content
Version: 0.12.0

Distributed Analytics

Introduced 0.12.0

Lucenia can run distributed analytics compute co-located with your data. Instead of exporting index data to a separate analytics cluster, you run a parallel compute job whose workers sit on the same nodes as your Lucenia shards and read those shards in place — analyzing cluster data without moving it across the network.

warning

Distributed Analytics is a Lucenia enterprise capability. Co-locating analytics compute with Lucenia data nodes — pod affinity, read-only data mounts, and the surrounding deployment topology — is orchestrated by the Lucenia Kubernetes operator, which is available with a commercial Lucenia subscription and is not published publicly. There is no license key or in-code license check — the gate is that the operator is not publicly distributed.

Table of contents

What it enables​

The distributed-analytics connector lets a general-purpose distributed compute engine read from and write to Lucenia with true parallelism:

  • Parallel reads and writes across compute partitions and Lucenia shards, so a large index is processed by many workers at once.
  • Co-located, zero-network reads. When a compute worker runs on the same node as the shard it needs, the connector reads the shard's on-disk Lucene index directly, bypassing the network and the REST layer entirely.
  • Analytics in place. Data never has to leave the cluster to be analyzed — the compute engine's DataFrame, SQL, and RDD-style APIs operate against live index data.
  • Spatial analytics — spatial joins and predicates over geo_point / geo_shape fields, evaluated using Lucenia's BKD spatial index rather than brute-force scans.
  • Materialized acceleration structures — co-located covering indexes, materialized views, and skipping indexes to speed up repeated queries.
note

The compute layer is a distributed analytics engine that runs alongside Lucenia; this page describes the capability generically. The connector targets an Apache Spark-compatible engine (Spark 3.4, 3.5, and 4.x; Scala 2.12/2.13, with Spark 4.x on 2.13), so the concrete APIs referenced below are Spark's DataFrame, SQL, and RDD APIs.

How data locality works​

The connector's core idea is shard-locality-aware partitioning: it maps compute partitions 1:1 onto Lucenia shards and schedules each partition to run where its shard's data physically lives.

  1. Partition discovery. The connector asks Lucenia for the physical layout of an index's shards — the index UUID, the on-disk data-path roots, and the node locations that host each shard.
  2. Locality hints. Each partition is annotated with the hostnames of the nodes holding its shard, so the compute scheduler prefers to place that partition's worker on a co-located node.
  3. Local read fast path. When a worker runs on a node that has the shard on its local filesystem, the connector resolves the shard's Lucene index directory and reads it directly on disk — no network hop, no REST call.
  4. Transparent REST fallback. If a shard is not local to the worker (for example, it relocated, or the deployment is only partially co-located), the connector transparently falls back to reading over REST. Partial co-location is safe by design.

The connector also deprioritizes nodes that are excluded from allocation (that is, nodes being drained), which keeps analytics work off nodes the autoscaler is trying to empty.

Coordination with autoscaling​

Co-located analytics and autoscaling have to cooperate: the autoscaler must not drain and remove a data node while an analytics job is actively reading shards from it. The connector solves this with the autoscale engine's drain leases.

When lease coordination is enabled (the default), an analytics job:

  • acquires a short-TTL drain lease that pins the data nodes hosting the shards it will read (via POST /_autoscale/lease),
  • renews the lease periodically from the job driver for as long as the job runs (via POST /_autoscale/lease/{token}/renew), and
  • lets the lease expire — or releases it — when the job finishes.

Because leases have a bounded time-to-live and a capped total lifetime, a hung or crashed job cannot pin nodes indefinitely; the lease simply expires and the autoscaler regains freedom to scale the tier. On clusters that don't have the autoscale module, lease coordination is best-effort: the connector logs a warning and proceeds without leases.

tip

Grant the analytics service account the reserved autoscale_lease action group (lease acquire/renew/release only). This lets the job hold drain leases without granting it the ability to drain or scale the cluster. See Autoscaling → Security and RBAC.

Deployment shape​

Co-located distributed analytics is a deployment topology as much as a connector. In a Kubernetes deployment driven by the Lucenia operator, the pieces that make co-location work are:

  • Pod affinity — schedule compute workers onto the same nodes as the Lucenia data pods so shard reads stay local.
  • Read-only data mounts — mount the Lucenia data volumes into the compute workers read-only, so they can open the on-disk Lucene indexes without risk of mutating them.
  • Allocation awareness — tag data nodes (for example by zone) and constrain shard placement so the shards a job needs are predictably located.

Partial co-location is always safe: any shard a worker cannot read locally is served over REST instead.

What is verified vs. what is not​

This page is written against the connector source and its user guide. The following were confirmed in source: shard-locality-aware partitioning (partition-per-shard), the local on-disk Lucene read fast path with REST fallback, drain-lease coordination with the autoscale engine (acquire/renew from the driver, TTL and max-lifetime caps), spatial joins over geo_point/geo_shape using the BKD index, and Spark 3.4/3.5/4.x support.

Not independently verified here: performance/latency figures comparing local reads to REST reads, and the exact server-side implementation of the shard-path discovery endpoint the connector consumes. The specific pod-affinity and mount wiring is a property of the (non-public) operator deployment and is described here at the level the connector's deployment guidance documents.

  • Autoscaling — the autoscale engine and drain leases this capability coordinates with.
  • Geographic field types — the geo_point and geo_shape fields spatial analytics operate on.