Tenant capacity and admission control

Introduced 0.13.0

Cluster tenant isolation keeps tenants from seeing each other's data. It says nothing about them competing for the same CPU. One customer running an expensive aggregation in a loop can still make the cluster slow for everybody on it.

This page covers the other half: knowing whose load is whose, bounding what any one tenant may occupy, and keeping autoscaling from growing the cluster on behalf of the tenant it is supposed to be holding back.

note

Off by default, end to end. With no tenant data — every single-tenant and self-hosted cluster — every decision described here comes out exactly as it did before. Nothing needs to be turned off.

Attribution: whose load is this

The tenant is known at the coordinator, but the CPU is spent on data nodes. To connect the two, the tenant identity travels as a request header rather than a thread-context transient, because a transient stops at the coordinator.

Per-task CPU time and allocated bytes were already measured — task_resource_tracking.enabled defaults to true, and both search tasks opt in. Attribution needed a header and a listener, not new measurement, so it costs essentially nothing.

The tenant header cannot be sent by a caller

The header is _lucenia_tenant_id, and the cluster trusts it. That makes forging it the obvious attack: a tenant sending _lucenia_tenant_id: globex would poison every counter and slip any limit built on top of them.

It is refused at the REST edge. The guard that already rejects the security plugin's internal headers now covers the whole _lucenia_ prefix, so a client cannot set it at all. It is stamped internally, idempotently — the stamp is reached twice in ordinary operation, once when an auto-create re-enters the filter and once on a data node where the header already arrived over the wire.

Admission control

Admission control bounds how many requests one tenant may have in flight at a coordinator. That is the only place the tenant is known before work fans out, and the cheapest place to refuse.

Excess requests take a 429 rather than a queue slot. A queued request still occupies something, and telling a caller to retry is kinder than making its neighbours wait behind it.

Settings

Both are dynamic and node-scoped, so they can be changed on a running cluster.

Setting	Default	Description
multitenancy.admission_control.mode	disabled	How firmly the limit is applied. One of disabled, monitor or enforced.
multitenancy.admission_control.max_concurrent_per_tenant	0	The most requests one tenant may have in flight at a coordinator. 0 means no limit.

The three modes

Mode	What it does
disabled	Nothing is counted and nothing is refused. The default.
monitor	In-flight work is counted and a breach is recorded, but the work still runs. Nothing is refused.
enforced	A tenant over its limit receives a 429.

warning

Use monitor before enforced. Turning an untested concurrency bound straight on is how you discover in production that it was set too low — and a tenant locked out of its own cluster reads as a fault, not as policy. In monitor mode the cluster counts what a limit would have refused while still running the work, so a number can be chosen from evidence.

max_concurrent_per_tenant defaults to 0 for the same reason: it keeps monitor honest before anybody has picked a number, counting in-flight work and never reporting a breach.

PUT _cluster/settings
{
  "persistent": {
    "multitenancy.admission_control.mode": "monitor",
    "multitenancy.admission_control.max_concurrent_per_tenant": 32
  }
}

Watch for breaches, then change mode to enforced when the number looks right.

Telling a deliberate refusal from a cluster in trouble

A 429 now means two things, and they call for opposite responses: the cluster is drowning, or the cluster is deliberately holding one tenant back and is working exactly as configured.

Every refusal admission control causes carries the marker tenant_admission_control. Anything that reads refusals to decide whether to add capacity must exclude them — otherwise successful containment looks like evidence the tier is too small, and the cluster grows on behalf of the very tenant being contained. The Lucenia Kubernetes operator already excludes it.

note

The marker string is a cross-process contract, matched on by other software. Treat it as part of the API rather than as log text.

Thread pool rejections are counted separately

A request can also be rejected because a thread pool overflowed. Those rejections are attributed to a tenant too — recorded on the rejecting thread, while the submitting request's context is still current, because by the time a rejection reaches a caller it is an exception with no identity left on it.

They are kept apart from admission refusals and deliberately do not feed the scaling rules below. Pool overflow means the cluster is failing, which is a reason to add capacity, not to withhold it. Merging the two counts would invert both decisions.

What this changes about autoscaling

Autoscaling gains two rules when tenant data is present. Both concern the search tier.

It does not scale down while admission control is refusing work. Scale-down reads active threads against total threads on the search pool, which is only honest while everything a tenant asked for actually arrives there. Work held back earlier makes a saturated tier read as idle, in proportion to how hard it is being held back.

It does not scale up when one tenant owns most of the tier's load and is already being throttled. Both halves are required. Concentration alone says who caused the load, not that anything is stopping them — withholding capacity on concentration alone would leave the tenant unthrottled and the tier undersized, punishing the neighbours twice.

Limitations

• Admission control is per coordinator, not cluster-wide. The limit bounds in-flight requests at each coordinating node, so a tenant spreading requests across several coordinators can exceed the nominal figure in aggregate.
• The tenant-aware scaling rules apply to the search tier. Other tiers are unaffected.
• Attribution depends on task resource tracking. With task_resource_tracking.enabled set to false, per-tenant CPU and allocated bytes are not measured.
• A limit of 0 is no limit, not a refusal of everything — worth knowing before setting it from a template.