System-generated search processors

Introduced 0.10.0

System-generated search processors are processors that Lucenia creates automatically based on the search request. Unlike user-defined processors that you manually configure in pipelines, system-generated processors are triggered automatically when certain features are used, eliminating the need for manual processor configuration.

Enabling system-generated search processors

To enable system-generated search processor creation, set the cluster.search.enabled_system_generated_factories cluster setting to * (all factories) or explicitly list the factories you want to enable. The following example enables the MMR and late-interaction factories along with semantic-highlighter:

PUT _cluster/settings
{
  "persistent": {
    "cluster.search.enabled_system_generated_factories": [
      "mmr_over_sample_factory",
      "mmr_rerank_factory",
      "late_interaction_oversample_factory",
      "late_interaction_rerank_factory",
      "semantic-highlighter"
    ]
  }
}

Processor types

Lucenia supports the following types of system-generated processors:

• Search request processors
• Search response processors

Each system-generated processor runs at a fixed execution stage, either before or after user-defined processors of the same type.

System-generated search request processors

The following table lists the available system-generated search request processors.

Processor name	Processor factory name	Execution stage	Trigger condition	Description
mmr_over_sample	mmr_over_sample_factory	Runs after any user-defined request processors.	Triggered when a search request includes the mmr parameter in the ext object. See Vector search with MMR reranking.	Adjusts the query size and k value of the knn or neural query to oversample candidates for maximal marginal relevance (MMR) reranking.
late_interaction_oversample	late_interaction_oversample_factory	Runs after any user-defined request processors.	Triggered when a search request includes the late_interaction parameter in the ext object. See Late interaction reranking.	Raises the number of candidates the knn query retrieves, so that the reranking pass has a deeper pool to score with MaxSim.

System-generated search response processors

The following table lists the available system-generated search response processors.

Processor name	Processor factory name	Execution stage	Trigger condition	Description
mmr_rerank	mmr_rerank_factory	Runs before any user-defined response processors.	Triggered when a search request includes the mmr parameter in the ext object. See Vector search with MMR reranking.	Reranks the oversampled results using MMR and reduces them to the original query size.
late_interaction_rerank	late_interaction_rerank_factory	Runs before any user-defined response processors.	Triggered when a search request includes the late_interaction parameter in the ext object. See Late interaction reranking.	Rescores the oversampled candidates against the query's multi-vector representation and reduces them to the original query size.

Late interaction reranking

Late interaction keeps a multi-vector representation of a document — one vector per token rather than a single pooled vector — and scores a query against it with MaxSim: each query vector takes its best match among the document's vectors, and those maxima are summed. That is more faithful than a single dot product, and far too expensive to run over a whole index.

The two processors above exist to make it affordable, and they always work as a pair:

1. late_interaction_oversample widens the first pass, so the knn query returns a deeper candidate pool than the caller asked for.
2. late_interaction_rerank rescores just that pool with MaxSim and trims the result back to the requested size.

Neither is configured in a pipeline. Both are generated automatically when a search request carries a late_interaction object in ext, and both must be enabled through cluster.search.enabled_system_generated_factories.

GET /my-index/_search
{
  "query": {
    "knn": {
      "embedding": { "vector": [0.1, 0.2, "..."], "k": 10 }
    }
  },
  "ext": {
    "late_interaction": {
      "candidates": 100,
      "vector_field_path": "token_embeddings",
      "vectors": [ [0.11, 0.02, "..."], [0.31, 0.27, "..."] ]
    }
  }
}

Field	Type	Required/Optional	Description
candidates	Integer	Optional	How many candidates the first pass should retrieve before reranking. Larger values improve recall and cost more.
vector_field_path	String	Optional	Path to the document field holding the multi-vector representation.
index_to_vector_field_path	Object	Optional	Per-index override of vector_field_path, for searches spanning indexes that store their vectors under different names.
vector_field_space_type	String	Optional	Space type used when comparing vectors.
vectors	Array	Optional	The query's multi-vector representation — the set of vectors MaxSim scores against each document.

Because the reranking pass only ever sees the oversampled pool, candidates sets the ceiling on what reranking can recover. A document outside the first pass cannot be rescued by the second.

Limitations

The following limitations apply to system-generated processors:

• Lucenia supports only one system-generated processor per processor type and execution stage for a given search request. Since each processor type (request and response) can run at two execution stages (before or after user-defined processors), a single search request can include multiple system-generated processors, as long as they are of different types or run at different execution stages. This limitation ensures deterministic execution order and predictable behavior.

Related documentation

• Vector search with MMR reranking