Skip to main content
Version: 0.13.0

Release Notes

0.13.0​

Highlights​

  • Dedicated Agent module: The agent, tool and MCP subsystem moves out of ML Commons into its own module, with preferred APIs under /_plugins/_agent, including POST /_plugins/_agent/agents/{id}/_execute. The legacy /_plugins/_ml agent, tool and MCP routes still work and emit a deprecation warning.
  • Shared-cluster multitenancy: Several customers can share one cluster, each addressing indices by its own names without knowing the others exist. Requests are rewritten on the way in and responses on the way out, so the physical prefix never reaches a tenant. Paired with per-tenant load attribution, admission control, and autoscaling that will not grow a tier on behalf of the tenant it is holding back.
  • Concept expansion: Index a vocabulary and retrieve every document about any kind of a thing — including documents that never print the term — with a set whose completeness can be shown. Vocabulary import, a concept_link ingest processor, two new queries, and an agent tool.
  • OGC API - Features: QGIS, ArcGIS Pro, OpenLayers and GDAL/OGR read the cluster directly, rather than through a translating service.
  • Local image embedding: A module that embeds imagery on the node, on ONNX Runtime, for the case where the imagery cannot leave. It is the first implementation of a new node-level extension point that lets an installed plugin contribute embedding providers.
  • Geospatial reach: Vector tiles (_mvt) render the cluster on a map, road routing answers "within ten minutes' drive" as network travel cost rather than a radius, and geo agent tools expose concentrations, tracks, features and routes to an agent.
  • Binary quantization: One bit per dimension — 32x smaller than the float32 vector it stands in for — with the raw vectors retained so the final ranking is still exact-scored. Recall@10 of 0.987 at 20x oversampling and 1.000 at 40x.
  • Substantial k-NN performance work: Twenty-five changes across the graph walk, the rescore pass and quantized decoding, including reaching native float32 kernels from the JVM engine. Measured against Elasticsearch at iso-recall, Lucenia is 1.10x to 1.22x faster from 0.90 through 0.97 recall, outside a noise floor measured at 6–8% across three runs.

New Features​

  • Cluster tenant isolation: hidden index namespacing for shared clusters, with tenant setup, personal access tokens, ISM, snapshot and restore, shared indices, and break glass. Off by default (multitenancy.enabled).
  • Per-tenant attribution and admission control: load is attributed to the tenant that caused it, and a tenant's in-flight requests can be bounded at the coordinator. Three modes (disabled, monitor, enforced), disabled by default. Refusals carry a marker so an operator can tell deliberate containment from a cluster in trouble.
  • Tenant-aware autoscaling: the search-load decider no longer scales down while admission control is refusing work, nor up when one tenant owns most of the load and is already throttled.
  • Concept expansion queries: graph_traversal for a provably complete set and concept_similarity for graded relatedness, with five similarity measures and information-content recording.
  • Vocabulary import: read a published SKOS/RDF vocabulary, or a domain graph such as a road network, into concepts and relations.
  • concept_link ingest processor: link documents to the concepts they name, and record what could not be linked.
  • ConceptSearchTool (FindByMeaning): finds every document about a subject and states the share of the index it could ever have searched.
  • CheckAnalysisTool (CheckAnalysis): checks analysis code a model authored against an allowlist before anything runs it, and reports how much of it was actually examined. Executes nothing.
  • Geo agent tools: ContourTool (FindConcentrations) and GeoLineTool (FindTracks), each stating what its answer does and does not cover.
  • More agent tools: routing, compliance checking, change detection, temporal coverage, similar imagery, dissemination, reporting and image regions.
  • neural_sparse query: learned sparse retrieval, including expansion directly from a vocabulary.
  • vectorize ingest processor: turns categorical rasters into geometry, with no model.
  • image_segment ingest processor: segmentation masks into mask_xy_shape.
  • convex_hull aggregation: the region a bucket actually occupies.
  • Graph edges on vector tiles: draw a network's edges into an _mvt response.
  • Vector tiles: GET|POST /{index}/_mvt/{field}/{z}/{x}/{y} returns Mapbox Vector Tiles straight from the cluster.
  • Road routing: network-based travel cost over a graph_edge field, so a catchment is what can actually be driven rather than a circle.
  • OCR and content extraction: extract text and structure from documents and imagery at ingest.
  • topic_drift processor: detect where a document's subject moves.
  • ml_inference ingest and search processors, and system-generated late-interaction search processors.
  • circle ingest processor, and the contour and line aggregations alongside convex_hull.
  • Twelve spatial join relations: a distributed-analytics spatial join can ask for any OGC relation, matching PostGIS, rather than four Lucene predicates.
  • Binary quantization (bbq encoder): one-bit-per-dimension quantization for lucene-engine knn_vector fields, built on the RaBitQ construction with no per-segment training pass. The graph is built on the raw vectors and only scored on the quantized ones, and a 10x rescore window is applied by default.
  • Embedding provider extension point: EmbeddingProviderPlugin lets an installed plugin contribute providers to the whole node, including the semantic query.
  • 14-day evaluation: a cluster runs for 14 days before a license is required, and a node joining an unlicensed cluster is no longer stopped.

Improvements​

  • k-NN rescoring no longer compounds ef_search into the oversample factor; the factor multiplies k alone, with ef_search kept as a floor. The upper bound on the factor moved from 100 to 10,000.

  • A cosine k-NN field is indexed as a dot product over unit vectors, removing roughly two-thirds of the arithmetic per comparison for an identical answer.

  • Vectors are reconstructed from the vector data rather than stored a second time in _source, for indices created at 0.13.0 or later.

  • Local inference runs on its own bounded thread pool rather than competing with search, and registers as heavy work so a drain waits for it.

  • Autoscaling notices a data node running out of heap before it dies, and an inference pool refusing work.

  • Every built-in agent tool is discoverable over MCP, served from the code rather than from a snapshot taken at registration; one unbuildable tool no longer takes the whole MCP surface down.

  • GeoTIFF window reads return the rows they asked for, and crop without paying for it twice.

  • The quickstart is a single docker run, with a 14-day evaluation and no certificate needed to start.

  • Official Python, JavaScript and Java clients are documented against lucenia-py, @lucenia/client and lucenia-java, including the JavaScript bulk helper.

Deprecations​

  • Agent, tool and MCP routes under /_plugins/_ml/... are deprecated in favor of /_plugins/_agent/.... They still work for backward compatibility but emit a deprecation warning.

Known issues​

  • Reduced recall on float32 k-NN fields that have no rescore. index.knn.hnsw_early_termination.enabled defaults to true in this release and is applied to every query shape, including plain float32, which has no rescore pass to recover the tail of the walk it gives up. Measured recall@100 is 0.858 on SIFT and 0.776 on GIST, against 0.984 and 0.929 on 0.12.1. Affected indices are knn_vector on the lucene engine with no compression_level of 8x or higher and no bbq encoder. Recover per index without a reindex by setting index.knn.hnsw_early_termination.enabled to false; the setting is dynamic and takes effect on the next query. Do not apply it cluster-wide — on a quantized field the same setting is buying the top of the recall curve.

0.12.1​

Highlights​

  • Hybrid search score normalization: concurrent collector-spec handling no longer bypasses score combination, so hybrid queries return consistent ranks under concurrent segment search.
  • Generative QA: context-aware async execution now routes to the real RAG implementation, so generative-QA agents run instead of silently doing nothing.
  • MCP initialize: unknown client capability fields are ignored rather than failing the handshake, so current MCP clients can connect.

Improvements​

  • Container images include the k-NN native libraries, so k-NN works in Docker without a separate native-library install.
  • The Docker distribution base image moved to Chainguard Wolfi.
  • The S3 client cache is keyed by the full client configuration, not only the endpoint.
  • PyTorch, DJL and ONNX runtime upgrades for local model inference.

Bug Fixes​

  • Parquet Hadoop compatibility fix for in-place Parquet queries.
  • Windows k-NN / Faiss build script patches against the current Faiss version.

0.12.0​

Highlights​

  • Persistent agent memory: A new inference-memory layer (_plugins/_memory) gives agents, chat loops, and ingest pipelines a durable place to remember what happened and later recall it by meaning, with hybrid (lexical and vector), recency-aware recall. Includes an organization-tenant-scoped Conversation management API.
  • Compliance & content governance (enterprise): A productized, standards-based layer to detect, redact, and mask sensitive data (PII, PHI, payment data, and secrets) and govern which external sources content may be ingested from — enforced at runtime with no rebuild or redeploy. Ships compliance profiles for GDPR, HIPAA, PCI-DSS, SOC 2, FedRAMP, CCPA, ISO 27001, and NIST 800-53, and supports CMMC / NIST 800-171 alignment.
  • Query Parquet files in place (experimental): Mount a Parquet file as a read-only Lucenia index and query it with the normal search, aggregation, and GET APIs — no ingest, no copy.
  • Distributed analytics (enterprise): Run analytics compute co-located with your data, reading shards in place instead of exporting index data to a separate analytics cluster.
  • Autoscaling (enterprise): An in-cluster autoscale engine produces safe, per-tier scaling recommendations that the Lucenia Kubernetes operator turns into real cluster changes.

New Features​

  • Inference memory (_plugins/_memory): durable remember/recall built on Lucenia's own vector and lexical search, with recency-aware hybrid recall, tenancy, and configurable embedding providers.
  • Conversation management API (/_plugins/_memory/conversations): organization-tenant-scoped conversation and message storage with compliance and ABAC enforcement.
  • API keys (personal access tokens): native, revocable PATs issued and verified by Lucenia — an id plus a high-entropy secret (stored only as a bcrypt hash), bound to an identity, carrying a bounded set of roles, and expiring on a schedule.
  • Compliance ingest processor: detects and redacts PII, PHI, payment data, financial identifiers, and secrets before documents are indexed, applying a named compliance profile and recording compliance evidence.
  • Content-source governance (enterprise): a deny-by-default allow/deny policy over the external URIs (s3://, https://, gs://, azure://) the ingest pipeline may fetch content from.
  • Multi-vector fields: store multiple embeddings in a single k-NN vector field within a single document.
  • MCP memory and content tools: expose the memory verbs and content extraction as native MCP tools, callable by any MCP client over JSON-RPC.
  • Bayesian score combination: a new bayesian combination technique for the hybrid-search normalization processor.

Improvements​

  • Refreshed the agents-and-tools catalog: added a connector tool and a list-index tool, updated the cat-index, search-index, and visualization tools, and revised the register, get, and search agent APIs.
  • Expanded snapshot and restore documentation.
  • Concurrent segment search documentation updates.
  • Document-level security and access-control APIs now recognize API-key-authenticated callers.

Deprecations​

  • The legacy /_plugins/_ml/memory/* conversation-CRUD API is deprecated in favor of the organization-tenant-scoped Conversation management API (/_plugins/_memory/conversations). The old routes still work for backward compatibility but emit a deprecation warning.

0.11.1​

Highlights​

  • Anonymous S3 support
  • Expanded GeoTIFF support
  • Per-document region overrides

Improvements​

  • ImageTilingProcessor outputs bbox_xy_shape envelope for geo images

0.11.0​

Highlights​

  • Late Interaction Models: Introduced Late Interaction Models for more nuanced user interactions.
  • Security Migration: Security is now a module and is no longer separated from Skylite

0.10.0​

Highlights​

  • Lucene 10.4 Upgrade: Lucenia now includes Apache Lucene 10.4, delivering the latest search and indexing improvements to the core engine.
  • Hybrid Search & Re-ranking: Enhanced search capabilities with support for hybrid search queries and re-ranking modules for improved result accuracy.
  • MMR Normalization: Support for Maximal Marginal Relevance (MMR) normalization to provide more diverse search results.
  • JPMS Compatibility: Full JPMS module and classpath compatibility, including backwards compatibility support.
  • Shard Stabilization: Resolved a shard stabilization and replica promotion issue where primary terms were only being bumped for initializing shards, ensuring reliable replica promotion during cluster operations.

New Features​

  • Hybrid search library and module for score normalization and combination.
  • System-generated search pipeline support for automated query optimization.
  • SecureEventBus annotation processor for enhanced event-driven communication.
  • PipelineProcessingContext for cross-processor state sharing.

Improvements​

  • Upgrade Notifications to AWS SDK v2.
  • JPMS-compatible plugin loader architecture and class path resolution fixes.
  • Add module-info.java to 28 modules and refactor for full JPMS compliance.

Bug Fixes​

  • Fixed replica-to-primary promotion during cluster recovery without primary term increment.
  • Fixed concurrent segment search aggregation failure by ensuring postCollection is called.
  • Fixed slow log implementation for coordinator-level search requests.

0.9.1​

Highlights​

  • Fixed segfaults in KNN caused by bad pointer management

0.9.0​

Highlights​

  • Index State Management: Automate index lifecycle operations with policy-driven index state management, including rollover, deletion, and replica management
  • OpenTelemetry Configuration: Configure OpenTelemetry endpoints directly in the otel plugin for seamless observability integration

New Features​

  • Index State Management module with core abstractions for policy-driven index lifecycle automation
  • OpenTelemetry plugin endpoint configuration for customizable telemetry collection
  • AWS Marketplace Metering Plugin for usage-based billing integration
  • Notifications module with supporting infrastructure for alerting and monitoring workflows
  • JKS keystore and truststore support in SecureRestClientBuilder for enhanced SSL configuration

Improvements​

  • KnnVectorsFormat reload support for dynamic knn plugin updates
  • Parent filter support for nested search in knn queries
  • Notification and replication libraries integrated into core
  • MetadataClient improvements for direct response object returns
  • ContextIndexSearcher optimization to prevent duplicate query execution

Bug Fixes​

  • Fixed ML model controller deployment under Java SecurityManager
  • Fixed static SIS Provider with doPrivileged for proper security context
  • Fixed job-scheduler and notifications integration tests for external test clusters
  • Fixed ml-commons integration tests
  • Fixed race condition in continuous transform integration test

0.8.0​

Highlights​

  • Lucene 10.3.1 Upgrade: Lucenia now includes Apache Lucene 10.3.1, bringing significant performance improvements and new capabilities for search and vector operations

Performance Improvements​

  • 40% Faster Lexical Search: Vectorized search implementation leverages SIMD instructions for dramatically improved query performance on disjunctive and conjunctive queries
  • 20% Faster Vector Search: Enhanced parallelization for fetching vectors into CPU cache, significantly improving kNN and vector similarity operations
  • 30% Faster Primary Key Lookups: Optimized terms dictionary performance benefits indexing operations and TermInSet queries

0.7.1​

Highlights​

  • Model Context Protocol Streamable HTTP Support
  • SSL improvements
  • ML model deployment stability improvements

0.7.0​

Highlights​

  • MCP Server: Lucenia now supports the Model Context Protocol (MCP), making it easier to integrate and manage AI models within your search infrastructure
  • Windows Support: Lucenia can now be installed and run on Windows operating systems, expanding deployment options

New Features​

  • MCP Server Support: Now includes MCP server functionality with tool management capabilities for ML workflows and seamless integration
  • SSE for MCP over REST: Real-time streaming support for model interactions using Server-Sent Events
  • Windows OS Support: Full compatibility with Windows environments for installation and operation. This rounds out our OS support to include Linux, macOS, Windows, and Docker containers.

0.6.1​

Highlights​

  • A fix for lucenia-security install_demo_configuration.sh.

0.6.0​

Highlights​

  • Ingest Attachment included as a module.
  • Remote backed indexing improvements.
  • Darwin Tar Artifacts Released.
  • Gradle updated to 8.14.

0.5.1​

Highlights​

  • Add support for lucenia.yml configuration and _cluster/settings API to set version and distribution override.

0.5.0​

Highlights​

  • Secure ML Processing: Process organization data without third-party preprocessing; attribute-based access controls (ABAC) prevent PII/sensitive data leakage.
  • Model Flexibility: Compatible with HuggingFace, Bedrock, LLaMA, PyTorch (public/private); No vendor lock-in.
  • Simplified RAG: Built-in Retrieval-Augmented Generation processors; Query/response augmentation via standard API calls.
  • ConversationalSearch: Contextual and nuanced search with natural language queries.
  • Performance Improvements: 80% smaller vector storage footprint than Lucene HNSW/Elastic KNN; 90% cost reduction vs Elastic Cloud/AWS OpenSearch.
  • Technical Updates: First to implement Lucene 10.2.1 in production.

0.4.0​

Highlights​

  • KNN Vector Search Enhancements - Whether its Facebook Faiss, NMSLIB, or vanilla Lucene vector search Lucenia has you covered for hybrid search. knn_vector field types continue to improve through Lucenia's ongoing indexing and search performance improvements. Start your Retrieval Augmented Generation Hybrid Search journey by indexing vectors and performing knn without requiring AWS infrastructure.
  • Lucene 10.1 Upgrade Improvements - As the first to market with Lucene 10 five months ago, we continue with our rapid updates with Lucene 10, including merge policy updates and overall bug fixes and speed improvements above and beyond the vanilla lucene releases.
  • Geospatial Coordinate Reference System enhancements - Lucenia continues to deliver on rapid improvements and performance enhancements to spatial reference system support at both index and query time. Use custom projection strings, EPSG data sets, or bring your own reprojection logic through an extensible reprojection provider interface.

0.3.1​

Highlights​

The addition of Enterprise, Product and Local License Types

0.3.0​

Highlights​

Lucenia 0.3.0 adds Developer Licenses and Spatial Reprojection Support

New Features​

  • Multi-cloud repository support installed by default for seamless backup, snapshot, and restore functionality across cloud providers
  • Spatial reprojection processor and improved search/ingest pipeline processing with 5-10% performance boost in certain use cases
  • Simplified migrations from OpenSearch with improved compatibility
  • Developer License support for local development and testing
  • Updated search request and response processors for coordinate transformations

0.2.1​

Highlights​

Lucenia 0.2.1 is now available on the AWS Marketplace

New Features​

  • AWS marketplace integration with license verification improvements - added agreement ID verification against AWS and implemented different license types based on running plugins
  • Support for both "bring your own license" and yearly agreement options in the AWS marketplace
  • Released with support for multiple deployment options including Docker containers and Helm charts

0.2.0​

Highlights​

Lucenia 0.2.0 upgrades to Lucene 10 and added to AWS Marketplace

Improvements​

  • Upgrade to Lucene 10
  • Update to min JDK 21
  • Lucenia License checker updates

New Features​

  • AWS Annual Purchase Offering
  • Circle Ingest Processor w/ Circle Queries
  • Ellipse Ingest Processor w/ Ellipse Queries’

0.1.1​

Highlights​

Lucenia 0.1.1 bug fixes for Lucenia License Manager

Fixes​

  • Fixes for the Lucenia license
  • License checks now include checking a cluster ID against a license to ensure a license is used for 1 cluster
  • License checks confirm whether a license is active, and shutdown nodes tied to an expired license

0.1.0​

Highlights​

Lucenia 0.1.0 is the first GA release of Lucenia. It includes several new features designed to help you save on your cloud search deployments while providing a fully supported on-prem option that is 17% faster than OpenSearch.

New Features​

  • Fips 140-2/3 Security compliance
  • Role Based Access (RBAC) and Attribute Based Access (ABAC) Security Controls
  • Dynamic Fields API (e.g., Runtime Fields)
  • K8s Helm Chart Deployment Support
  • Mixed Cluster Compatibility with OpenSearch and Elasticsearch for easy migration
  • 17% performance boost over OpenSearch
  • Native Zstd compression for stored fields
  • Remote Backed Indexes with cloud storage
  • Segment Replication and Time Series index optimizations