Skip to main content
Version: 0.12.0

Autoscaling

Introduced 0.12.0

Lucenia can grow and shrink to match real demand. An in-cluster autoscale engine continuously evaluates each node tier and produces safe, per-tier scaling recommendations, and the Lucenia Kubernetes operator turns those recommendations into real changes to your cluster — adding, resizing, and removing nodes without risking your data.

warning

Autoscaling is a Lucenia enterprise capability. The scaling actions are driven by the Lucenia Kubernetes operator, which is available with a commercial Lucenia subscription and is not published publicly. The in-cluster autoscale engine (the capacity, safety, and drain APIs described below) ships with Lucenia and can be called directly, but adding or removing nodes in response to its recommendations requires the operator. There is no license key or in-code license check — the gate is that the operator is not publicly distributed.

Table of contents

Mental model: brain and hands​

Autoscaling in Lucenia is split into two cooperating components:

  • The autoscale engine (the autoscale module, in-cluster) is the brain. It answers three questions on demand: Does this tier need more or fewer resources? (capacity), Is it safe to take this action right now? (safety), and Move all shards off this node so it can be removed (drain). It has no background threads, no timers, and no scheduled tasks — every evaluation is triggered by an explicit REST call or a cluster-state change. It never scales anything itself.
  • The Kubernetes operator (commercial) is the hands. It runs your cluster as Kubernetes StatefulSets, polls the engine's REST APIs, and executes the recommendations: scaling node pools, and draining then removing nodes when scaling down.

This separation is deliberate: the cluster owns the decisions that require deep knowledge of shards, snapshots, and quorum; the operator owns the Kubernetes-level mechanics.

The autoscale engine​

The engine is organized around policies, capacity deciders, safety checks, and drains.

Policies​

A policy maps a set of node roles to scaling constraints and per-decider configuration. You create one policy per tier you want to autoscale (for example, a data tier and an ingest tier):

curl -X PUT "localhost:9200/_autoscale/policy/data_tier" \
  -H "Content-Type: application/json" \
  -d '{
    "roles": ["data"],
    "deciders": {
      "storage":      { "headroom_percent": 20, "high_watermark": 85 },
      "search_load":  { "queue_threshold": 100, "idle_threshold": 0.3 },
      "shard_balance":{ "max_shards_per_node": 1000 }
    },
    "min_nodes": 3,
    "max_nodes": 20,
    "scale_down_enabled": true
  }'
FieldDefaultDescription
roles(required)Node roles this policy governs, for example ["data"] or ["data","ingest"].
deciders{}Per-decider configuration. Omitted deciders use their defaults.
min_nodes1Floor — capacity evaluation never recommends fewer nodes.
max_nodes2147483647Ceiling.
scale_down_enabledtrueSet to false to forbid any DOWN recommendation for the tier.

Policies are stored in cluster state (as cluster metadata), so they survive restarts.

Capacity deciders​

When you evaluate a policy, the engine runs each applicable decider and aggregates their verdicts. Each decider looks at one dimension of load and returns a direction (UP, DOWN, or STEADY), a priority (CRITICAL, NORMAL, or OPTIMIZATION), a human-readable reason, and a recommended resource shape — or abstains when it lacks the data to decide.

DeciderApplies toWhat it decides
storagedata tiersDisk usage. Scales up when reactive need (used bytes + watermark buffer) exceeds allocated storage, or CRITICAL when shards are unassigned due to disk pressure. Scales down when total capacity exceeds need plus headroom_percent.
search_loadsearch-serving tiersSearch thread-pool pressure. Up on queue depth over queue_threshold (or CRITICAL on search rejections); down when utilization falls below idle_threshold.
ingest_pressuredata/ingest tiersWrite/bulk thread-pool and indexing-memory pressure. Up on write queue depth over queue_threshold (or CRITICAL on write/indexing-pressure rejections); down when write utilization falls below idle_threshold. Its defaults differ from search_load: queue_threshold 200, idle_threshold 0.2 (vs 100/0.3).
shard_balancedata tiersEnsures node count is sufficient for shard placement. Up (CRITICAL) when node count is below ceil(total_shards / max_shards_per_node); down when count exceeds the minimum required plus node_buffer. Abstains when max_shards_per_node is 0.
idle_indexdata tiersDetects fully-idle, over-replicated indexes and recommends replica reduction (scale-down only). Never recommends scale-up. Skips indexes matching protected_pattern (default .lucenia-*,system-*), and never reduces below min_replicas (default 0).
cluster_managercluster-manager tierVertical heap sizing for cluster-manager nodes. Manager load scales with cluster-state size (indices, shards, nodes), not query volume, so this decider recommends growing per-node heap/memory rather than node count.

Decider thresholds come from the policy's deciders block; the cluster_manager decider's tunables (for example heap_per_indices, heap_per_shards, heap_per_nodes, min_heap_gb, max_heap_gb, heap_high_watermark, scale_down_buffer_percent) are node-scoped settings with industry-norm defaults (roughly 1 GB of manager heap per ~3,000 indices, capped at 31 GB to stay under the compressed-oops boundary). The cluster_manager decider reports its recommendation on the per-node memory dimension (memory ≈ 2× heap, matching how the operator derives -Xmx from the memory request).

Aggregation rule: the most demanding decider wins. Within the same priority level, UP beats DOWN beats STEADY; resource dimensions (storage, memory, processors, node count) are aggregated by taking the maximum across deciders. If every decider abstains, the tier evaluates to STEADY.

To evaluate:

# all policies
curl "localhost:9200/_autoscale/capacity"

# a single policy
curl "localhost:9200/_autoscale/capacity/data_tier"

The response reports required_capacity (with total and per-node resource shapes), the aggregate direction/priority/reason, and a per-decider breakdown showing each decider's individual recommendation or abstention.

Safety checks​

Before any potentially destructive action, an orchestrator asks the safety service, which has veto power. A check returns one of three verdicts:

  • SAFE — proceed.
  • UNSAFE — do not proceed; the condition will not clear on its own (read reason).
  • WAIT — a transient condition is blocking (a snapshot, an in-flight relocation). The response includes a retry_after hint, but the recommended pattern is to re-check on the next cluster-state change rather than sleeping.
# can this node be drained?
curl "localhost:9200/_autoscale/safety/check?action=drain_node&node=node-5"

# can this index's replicas be reduced?
curl "localhost:9200/_autoscale/safety/check?action=reduce_replicas&index=my-index&target=1"

# can this index be closed?
curl "localhost:9200/_autoscale/safety/check?action=close_index&index=old-logs"

For a drain_node check the service refuses to drain the active cluster-manager node, refuses any drain that would leave a primary shard with no copy elsewhere, waits while a snapshot is in progress on the node's shards, and waits when recovery pressure (concurrent relocations) is already high. Replica-reduction and index-close checks protect system indexes and wait on in-progress snapshots or active shard operations.

Drains​

A drain relocates all shards off a node so it can be safely terminated. It uses Lucenia's existing allocation-exclusion primitives (it sets cluster.routing.allocation.exclude._name for the target) rather than any custom shard-movement mechanics, and all phase transitions are event-driven (no timers).

# start a drain (returns a drain_id)
curl -X POST "localhost:9200/_autoscale/drain/node-5"

# poll status (computed on demand from the live routing table)
curl "localhost:9200/_autoscale/drain/node-5/status"

# cancel a drain (removes the allocation exclusion)
curl -X POST "localhost:9200/_autoscale/drain/node-5/_cancel"

The drain lifecycle moves through EXCLUDING → RELOCATING → DRAINED → COMPLETED (with CANCELLED/FAILED as terminal off-ramps). Status reports the current phase, shards_remaining/shards_total, and bytes_remaining. When phase reaches DRAINED, the node holds no shards and the orchestrator can terminate it.

Drain leases​

A drain lease lets a client pin one or more nodes against being drained for a bounded time-to-live. This is the coordination point between the operator and any workload co-located with the data (see Distributed Analytics): a long-running job can hold a lease so the operator does not drain a node out from under it. Leases are stored in cluster state and expire on their TTL.

# acquire a lease pinning nodes for a TTL (millis); returns a token
curl -X POST "localhost:9200/_autoscale/lease" -H 'Content-Type: application/json' \
  -d '{ "nodes": ["<nodeId>"], "ttl_millis": 300000, "holder": "spark-job-1" }'

# renew before expiry (ttl_millis is required and must be > 0)
curl -X POST "localhost:9200/_autoscale/lease/<token>/renew" -H 'Content-Type: application/json' \
  -d '{ "ttl_millis": 300000 }'

# release early
curl -X DELETE "localhost:9200/_autoscale/lease/<token>"

The acquire request takes nodes (node IDs to pin), ttl_millis (lease lifetime), and an optional holder (a human-readable description for observability). The lease token is returned on acquire and used to renew or release.

REST API reference​

All endpoints are under /_autoscale. These paths are verified in the module source.

MethodPathPurpose
PUT/_autoscale/policy/{name}Create or update a policy.
GET/_autoscale/policy/{name}Get one policy.
GET/_autoscale/policyList all policies.
DELETE/_autoscale/policy/{name}Delete a policy.
GET/_autoscale/capacityEvaluate all policies.
GET/_autoscale/capacity/{policy}Evaluate one policy.
GET/_autoscale/safety/checkSafety verdict for a proposed action (action, node/index/target query params).
POST/_autoscale/drain/{node_id}Start draining a node.
GET/_autoscale/drain/{node_id}/statusDrain progress.
POST/_autoscale/drain/{node_id}/_cancelCancel a drain.
POST/_autoscale/leaseAcquire a drain lease.
POST/_autoscale/lease/{token}/renewRenew a drain lease.
DELETE/_autoscale/lease/{token}Release a drain lease.

Security and RBAC​

Each endpoint maps to a namespaced transport action that the Security plugin authorizes by action name (default-deny). Read endpoints use cluster:monitor/autoscale/* privileges; mutating endpoints (policy writes, drain start/cancel, lease acquire/renew/release) use cluster:admin/autoscale/*. Two built-in reserved action groups make least-privilege grants easy:

  • autoscale_manage — every autoscale admin + monitor action. Grant this to the Kubernetes operator service account.
  • autoscale_lease — lease acquire/renew/release only. Grant this to co-located readers (for example a distributed-analytics job) that must hold drain leases but must never be able to drain or scale the cluster.

The Kubernetes operator​

The Lucenia Kubernetes operator deploys and manages Lucenia clusters (and Lucenia Dashboards) as native Kubernetes resources. You describe your cluster declaratively with a custom resource, and the operator reconciles that spec into StatefulSets, services, and configuration. Per its published feature set, the operator supports multiple node pools with all node roles, per-pool resource scaling, rolling version upgrades with quorum-safe restarts, and online volume expansion for disk scaling.

For autoscaling, the operator is both a producer and a consumer of the engine:

  • Producer. For each node pool you opt into autoscaling, the operator registers the desired policy with the engine (PUT /_autoscale/policy/{name}) — including which deciders apply. The cluster-manager pool auto-registers with the cluster_manager (vertical heap) decider.
  • Consumer. On a reconcile loop (default ~30 s), the operator polls GET /_autoscale/capacity and acts on the winning recommendation. It only polls when the cluster is initialized, healthy (not RED), and free of crash-looping pods.

The operator then translates recommendations into Kubernetes changes along two paths:

  • Horizontal (data/search/ingest tiers): it changes the node pool's replica count on the StatefulSet.
  • Vertical (cluster-manager tier): it resizes the pool's memory request/limit (heap is derived as roughly half the memory request) and rolls the manager pods one at a time to preserve quorum.

Safe scale-down. When removing a node, the operator follows the safe path this page describes: run a drain_node safety check, start a drain, poll drain status until the node reaches DRAINED, and only then reduce the replica count and terminate the pod. Data safety on scale-down comes from the drain relocating every shard off the node before it is removed — the emptied node's StatefulSet PVC is then cleaned up as an orphan (standard StatefulSet scale-down semantics). Across a rolling upgrade or restart, by contrast, a pod is recreated at the same StatefulSet ordinal and reuses its existing PVC, so persistent data is retained in place.

note

The autoscale engine deliberately does not track cooldowns or act on a schedule — it returns point-in-time recommendations only. Cooldown windows, reconciliation cadence, and the decision to bypass cooldowns for a CRITICAL recommendation are the operator's responsibility.

note

The operator scales Kubernetes pods/replicas and pod resources — it does not provision underlying cluster nodes. To make room for added pods (or reclaim nodes after scale-down), pair it with a Kubernetes node autoscaler such as Cluster Autoscaler or Karpenter.