Search by meaning

Someone asks for "beta blockers" and expects the paper that only ever says metoprolol. Keyword search cannot reach it: the phrase is not in the document.

Lucenia answers that in two ways, and they answer genuinely different questions.

Question	Use	What comes back
What is all of it?	Concept expansion	Every document about any kind of the thing asked for — a set whose completeness can be shown
What is like it?	The semantic query, below	The closest matches, ranked

Retrieve every document about a subject

Ask for a drug class, a product category, a corporate subsidiary — and get every document about any kind of it. Including the ones that never print the term you asked for.

This is the part a nearest-neighbour search cannot do. A vector can tell you what looks similar; it cannot tell you that you have seen all of it, and it cannot say why a document was returned. Here the relationships are a vocabulary you control, indexed alongside your data and expanded inside the query, so the answer has a reason attached to it.

A worked query

A slice of MeSH, the medical vocabulary published by the NIH. 319 is Adrenergic beta-Antagonists — beta blockers — and 8790, 1262 and 11433 are metoprolol, atenolol and propranolol beneath it:

GET /articles/_search
{
  "query": {
    "graph_traversal": {
      "field": "subclass_of",
      "source": 319,
      "graph_index": "concepts",
      "node_field": "concept"
    }
  }
}

That recovers every paper about a beta blocker, including the papers that name only metoprolol or atenolol and never print the phrase. Those are precisely the papers a keyword search loses and an embedding search finds only by luck.

Ranking by how closely concepts relate, rather than taking the whole set, is the concept_similarity query instead.

The claim is completeness, and it was measured

Against ground truth from human indexers — 936 MeSH concepts over a curated corpus:

Approach	Share of the true set returned
Concept expansion	the complete set, every time
Embedding search	18.8%
BM25	14.5%
Embedding search, on documents that never state the term	11.3%

note

Complete with respect to your annotations. If a document is tagged with a leaf concept, expansion from any ancestor provably finds it. It cannot find a document nobody annotated — annotation coverage is a separate, prior problem, and no retrieval strategy solves it. The FindByMeaning agent tool states its coverage in every answer for exactly this reason.

Those numbers are about membership — did the whole set come back. They are not a claim that concept scoring ranks better than an embedding, which was looked for and not found.

What it needs

A vocabulary, and documents that carry concepts from it.

• Importing a vocabulary — turn a published SKOS or RDF vocabulary into concepts and relations
• Concept link processor — link your documents to those concepts as they are indexed, and record what could not be linked
• Concept expansion and vocabularies — the full reference: both queries, the five similarity measures, and the modelling traps

Good fit: medicine, biology, law, finance taxonomies, product catalogs, internal ontologies — places where annotations often already exist and you need an answer you can defend. Poor fit: general prose with no curated hierarchy. There is nothing to expand, and the semantic query below is the better tool.

Nearest matches: the semantic query

Where there is no vocabulary to lean on, nearest-neighbour similarity is the right tool. This is ordinary embedding search — send text, get the closest matches back — and it behaves the way it does in any other engine.

Send plain text; Lucenia understands what it means and finds the closest matches — no embeddings, no vector math on your side. The semantic query embeds your text at search time and runs a k-NN search for you, so a caller never has to produce a vector or even know one is involved.

warning

The semantic query embeds through the k-NN plugin's registry. A stock node registers only hashing — a token-hash, not a trained model. provider: "http", "bedrock", or "openai" on this query fails with Unknown embedding provider [...]. Available: [hashing].

Since 0.13.0 an installed plugin can contribute a provider to that registry, so a node is not limited to hashing forever — local image embedding registers onnx this way. The cloud providers are still not among them.

For Bedrock, OpenAI, HTTP, Vertex, or Azure at search time, use the query_embedding search request processor. It uses the same providers as the ingest embed processor. The examples below use hashing-v1 only to show the semantic query shape.

The query

Wrap your text in a semantic query, naming the knn_vector field to search:

GET /knowledge-base/_search
{
  "query": {
    "semantic": {
      "chunks.embedding": {
        "query_text": "airfield near a river",
        "model_id": "hashing-v1",
        "k": 10,
        "filter": { "term": { "tenant": "acme" } }
      }
    }
  }
}

The outer field name (chunks.embedding above) is the knn_vector field the query searches against — the same nesting the raw knn query uses.

Parameters

Parameter	Type	Required/Optional	Description
query_text	String	Required	The plain text to search with. Lucenia embeds it for you.
model_id	String	Required	The embedding model (space) to embed the text with. It must match the model the index was embedded with at ingest time.
provider	String	Optional	Embedding provider name looked up on the k-NN registry. Default is hashing. A stock node has no other names; see the warning above.
k	Integer	Optional	The number of nearest neighbors to retrieve. Default is 10.
filter	Object	Optional	A query domain-specific language (DSL) object applied alongside the k-NN search (for example, to scope results by tenant or metadata).

match vs. semantic

The two queries answer different questions, and reading them side by side is the fastest way to understand semantic:

• match finds documents that contain your words.
• semantic finds documents that mean the same thing, even when they use different words.

A match query for "airfield near a river" only finds documents with those tokens. With a trained embedding (via query_embedding), a meaning search for the same text can also surface a document that says "airstrip beside a stream". hashing-v1 will not reliably do that.

Hybrid: keyword and meaning together

You rarely have to choose. The hero pattern combines an exact-keyword match with a meaning-based semantic subquery inside a hybrid query, so precise term hits and semantic matches are ranked together:

GET /knowledge-base/_search?search_pipeline=hybrid-pipeline
{
  "query": {
    "hybrid": {
      "queries": [
        {
          "match": {
            "chunks.text": "airfield near a river"
          }
        },
        {
          "semantic": {
            "chunks.embedding": {
              "query_text": "airfield near a river",
              "model_id": "hashing-v1",
              "k": 10
            }
          }
        }
      ]
    }
  }
}

This hybrid shape uses semantic on the vector side, so it is hashing-only on a stock node. For http or Bedrock query embedding plus BM25, use the hybrid example on the query_embedding page (request processor plus normalization).

Because the lexical and semantic subqueries score on different scales, pair the hybrid query with a normalization search pipeline:

PUT _search/pipeline/hybrid-pipeline
{
  "phase_results_processors": [
    {
      "normalization-processor": {
        "normalization": { "technique": "min_max" },
        "combination": {
          "technique": "arithmetic_mean",
          "parameters": { "weights": [0.3, 0.7] }
        }
      }
    }
  ]
}

For the full set of normalization and combination options, see Combining the scores.

Under the hood

The semantic query is a thin, well-named front end over machinery Lucenia already ships. At query time it embeds your query_text with the configured provider and model, then rewrites itself into a k-NN search against the named knn_vector field. No vector ever appears in the DSL — that's the whole point.

Correctness note. The provider and model_id in the query must match the provider and model that embedded the documents at ingest time. Vectors produced by different providers or models don't live in the same space and aren't comparable, so a mismatch quietly returns meaningless nearest-neighbor results — no error is raised. If you embedded documents with the ingest embed processor (http, bedrock, and so on), search with query_embedding using that same provider and model. Do not put those provider names on a semantic query.

Security

The semantic query is an ordinary query, so it inherits the caller's index-read authorization — there is no special scope to grant. Restrict who can search an index the same way you would for any other query, and use the filter clause to scope results (for example, by tenant).

Governing which models a caller may embed with

Introduced 0.13.0

Index-read authorization says nothing about which model a caller may embed their text with. On a shared cluster that matters: embedding is a call to a paid provider, and it can carry the caller's query text to a third party. Model access governs the provider and model_id a semantic query is allowed to name, and is enforced when the query is rewritten — before anything is embedded.

note

Off by default. With enforce left at false, nothing is ever rejected — exactly the behavior from before the feature existed, so an unconfigured or mixed-version cluster is unaffected.

Setting	Default	Description
plugins.knn.semantic.model_access.enforce	false	The master switch. Nothing is rejected while this is false.
plugins.knn.semantic.model_access.allow	(empty)	Allow globs. Empty means "no explicit allowlist", in which case deny_by_default decides.
plugins.knn.semantic.model_access.deny	(empty)	Deny globs. A deny match always rejects, whatever the allow list says.
plugins.knn.semantic.model_access.deny_by_default	true	When enforcement is on and no allow list applies, whether an unmatched pair is denied. Fail-closed.
plugins.knn.semantic.model_access.org.<org>.allow	(empty)	Per-organization allow globs. Replaces the cluster allow list for that organization.
plugins.knn.semantic.model_access.org.<org>.deny	(empty)	Per-organization deny globs. Replaces the cluster deny list for that organization.

All six are node-scoped and dynamic, so a policy can be changed on a running cluster.

Patterns

A pattern is a glob matched case-insensitively against the canonical provider:model_id pair, where * matches any run of characters. That lets a policy name a whole provider, a model family across providers, or one exact pair:

Pattern	Matches
bedrock:*	every model on the Bedrock provider
*:titan-embed-*	the Titan embedding family, whichever provider serves it
http:acme-embed-v2	exactly that provider and model

How a decision is reached

In order, stopping at the first that applies:

1. Enforcement off — everything is allowed.
2. Deny wins. A match in the applicable deny list rejects, regardless of any allow entry. There is no allow that overrides a deny.
3. An allow list, if there is one. When the applicable allow list is non-empty, the pair is permitted only if it matches an entry.
4. deny_by_default. With no allow list, true rejects the unmatched pair and false permits it.

A rejected query fails with 403. Nothing is embedded and no provider is called, because the check runs at rewrite time rather than after the fact.

PUT _cluster/settings
{
  "persistent": {
    "plugins.knn.semantic.model_access.enforce": true,
    "plugins.knn.semantic.model_access.allow": ["bedrock:*", "onnx:*"],
    "plugins.knn.semantic.model_access.deny": ["*:*-preview"]
  }
}

Per-organization rules replace, they do not add

warning

Configuring either one replaces both. An organization with any per-organization rule at all stops using the cluster-default lists entirely — its allow list and its deny list are taken from its own settings, and whichever of the two you did not set is empty.

So setting only org.acme.allow silently stops the cluster's deny list from applying to acme. If the cluster denied *:*-preview everywhere, acme can now use preview models, and nothing reports that the denial was dropped.

When you give an organization its own rules, set both keys, and repeat any cluster-wide denial you still want enforced for them.

The caller's organization is resolved through the shared tenancy convention — the same attribute, and optional backend-role prefix, that memory isolation and content-source governance use, so a caller maps to the same organization everywhere. A caller with no organization gets the cluster-default rules.