Skip to main content
Version: 0.13.0

Distributed Analytics

Introduced 0.12.0

Lucenia can run distributed analytics compute co-located with your data. Instead of exporting index data to a separate analytics cluster, you run a parallel compute job whose workers sit on the same nodes as your Lucenia shards and read those shards in place — analyzing cluster data without moving it across the network.

warning

Distributed Analytics is a Lucenia enterprise capability. Co-locating analytics compute with Lucenia data nodes — pod affinity, read-only data mounts, and the surrounding deployment topology — is orchestrated by the Lucenia Kubernetes operator, which is available with a commercial Lucenia subscription and is not published publicly. There is no license key or in-code license check — the gate is that the operator is not publicly distributed.

Table of contents

What it enables​

The distributed-analytics connector lets a general-purpose distributed compute engine read from and write to Lucenia with true parallelism:

  • Parallel reads and writes across compute partitions and Lucenia shards, so a large index is processed by many workers at once.
  • Co-located, zero-network reads. When a compute worker runs on the same node as the shard it needs, the connector reads the shard's on-disk Lucene index directly, bypassing the network and the REST layer entirely.
  • Analytics in place. Data never has to leave the cluster to be analyzed — the compute engine's DataFrame, SQL, and RDD-style APIs operate against live index data.
  • Spatial analytics — spatial joins and predicates over geo_point / geo_shape fields, evaluated using Lucenia's BKD spatial index rather than brute-force scans. A join can ask for any of twelve spatial relations, matching PostGIS rather than a four-predicate subset.
  • Materialized acceleration structures — co-located covering indexes, materialized views, and skipping indexes to speed up repeated queries.
note

The compute layer is a distributed analytics engine that runs alongside Lucenia; this page describes the capability generically. The connector targets an Apache Spark-compatible engine (Spark 3.4, 3.5, and 4.x; Scala 2.12/2.13, with Spark 4.x on 2.13), so the concrete APIs referenced below are Spark's DataFrame, SQL, and RDD APIs.

How data locality works​

The connector's core idea is shard-locality-aware partitioning: it maps compute partitions 1:1 onto Lucenia shards and schedules each partition to run where its shard's data physically lives.

  1. Partition discovery. The connector asks Lucenia for the physical layout of an index's shards — the index UUID, the on-disk data-path roots, and the node locations that host each shard.
  2. Locality hints. Each partition is annotated with the hostnames of the nodes holding its shard, so the compute scheduler prefers to place that partition's worker on a co-located node.
  3. Local read fast path. When a worker runs on a node that has the shard on its local filesystem, the connector resolves the shard's Lucene index directory and reads it directly on disk — no network hop, no REST call.
  4. Transparent REST fallback. If a shard is not local to the worker (for example, it relocated, or the deployment is only partially co-located), the connector transparently falls back to reading over REST. Partial co-location is safe by design.

The connector also deprioritizes nodes that are excluded from allocation (that is, nodes being drained), which keeps analytics work off nodes the autoscaler is trying to empty.

Coordination with autoscaling​

Co-located analytics and autoscaling have to cooperate: the autoscaler must not drain and remove a data node while an analytics job is actively reading shards from it. The connector solves this with the autoscale engine's drain leases.

When lease coordination is enabled (the default), an analytics job:

  • acquires a short-TTL drain lease that pins the data nodes hosting the shards it will read (via POST /_autoscale/lease),
  • renews the lease periodically from the job driver for as long as the job runs (via POST /_autoscale/lease/{token}/renew), and
  • lets the lease expire — or releases it — when the job finishes.

Because leases have a bounded time-to-live and a capped total lifetime, a hung or crashed job cannot pin nodes indefinitely; the lease simply expires and the autoscaler regains freedom to scale the tier. On clusters that don't have the autoscale module, lease coordination is best-effort: the connector logs a warning and proceeds without leases.

tip

Grant the analytics service account the reserved autoscale_lease action group (lease acquire/renew/release only). This lets the job hold drain leases without granting it the ability to drain or scale the cluster. See Autoscaling → Security and RBAC.

Deployment shape​

Co-located distributed analytics is a deployment topology as much as a connector. In a Kubernetes deployment driven by the Lucenia operator, the pieces that make co-location work are:

  • Pod affinity — schedule compute workers onto the same nodes as the Lucenia data pods so shard reads stay local.
  • Read-only data mounts — mount the Lucenia data volumes into the compute workers read-only, so they can open the on-disk Lucene indexes without risk of mutating them.
  • Allocation awareness — tag data nodes (for example by zone) and constrain shard placement so the shards a job needs are predictably located.

Partial co-location is always safe: any shard a worker cannot read locally is served over REST instead.

What is verified vs. what is not​

This page is written against the connector source and its user guide. The following were confirmed in source: shard-locality-aware partitioning (partition-per-shard), the local on-disk Lucene read fast path with REST fallback, drain-lease coordination with the autoscale engine (acquire/renew from the driver, TTL and max-lifetime caps), spatial joins over geo_point/geo_shape using the BKD index, the twelve spatial relations and their PostGIS equivalences (read from SpatialQueryRelation at the v0.13.0 tag), and Spark 3.4/3.5/4.x support.

The connector-side ST_Contains/ST_Within mapping described in the warning above is read from the connector's own declaration, not from this repository.

Not independently verified here: performance/latency figures comparing local reads to REST reads, and the exact server-side implementation of the shard-path discovery endpoint the connector consumes. The specific pod-affinity and mount wiring is a property of the (non-public) operator deployment and is described here at the level the connector's deployment guidance documents.

Spatial join relations​

Expanded in 0.13.0

A spatial join names the relation it wants. Until 0.13.0 a join could ask for four predicates — intersects, disjoint, within, contains — while the engine underneath had twelve. That is why "these two parcels touch but neither contains the other", which is what a border is, could not be expressed at all.

All twelve are now available, and a join and a search route through the same factory, so they answer the same question rather than being two implementations that agree until they do not.

RelationEquivalentWhat it means
intersectsST_IntersectsAny point in common. Identical in PostGIS, Lucene and here.
disjointST_DisjointNo point in common. Identical everywhere.
st_containsST_ContainsThe query is inside the document, and the interiors meet.
st_withinST_WithinInside, and the interiors meet.
coversST_CoversNo part of the query lies outside the document. True of a shape and itself.
covered_byST_CoveredByNo part of the document lies outside the query.
equalsST_EqualsThe same points, however either shape is written.
st_touchesST_TouchesThey meet only along their boundaries. What a border is.
st_overlapsST_OverlapsThe interiors meet and neither shape contains the other.
st_crossesST_CrossesThey meet in something of lower dimension — a road through a district.
contains(legacy)Lucene's contains: covers minus the touching case.
within(legacy)Lucene's within, which is coveredBy.

PostGIS is the reference for what each name means​

The people most likely to write these queries are arriving from PostGIS, and a predicate that quietly means something else is worse than one that is missing. So st_contains and st_within resolve to the strict relations that PostGIS agrees with — not to Lucene's contains and within, which are really covers minus touching and coveredBy.

The four legacy spellings keep their old meanings exactly, so joins already written answer as they did.

warning

Migrating a PostGIS query: check which spelling you get. The connector declares ST_Contains and ST_Within and maps them to the legacy contains and within relations, not to the strict ones. A query moved across from PostGIS will therefore differ precisely on the boundary cases — two shapes that touch — which are the cases least likely to be checked.

Where the exact PostGIS semantics matter, name st_contains or st_within explicitly rather than relying on the ST_* spelling to resolve the way it does in PostGIS.