Sparse retrieval (neural_sparse query)
Vector search finds documents that mean the same thing, but it cannot tell you why one matched. Sparse retrieval gives you most of the same recall while staying legible: a document and a query are each represented as a map of terms to weights, and a match decomposes into the terms that produced it.
That matters whenever a result has to be defended — a regulated review, an audit, a submission — because "the cosine similarity was 0.83" is not an explanation anyone can act on.
The neural_sparse query searches a rank_features field holding such a map.
How a document is represented
The document side is a rank_features field: a JSON object mapping terms to strictly positive weights.
PUT /literature
{
"mappings": {
"properties": {
"title": { "type": "text" },
"ml_tokens": { "type": "rank_features" }
}
}
}
POST /literature/_doc
{
"title": "Cardioselective agents in post-infarction management",
"ml_tokens": {
"metoprolol": 2.9,
"beta": 2.4,
"blocker": 2.2,
"cardioselective": 1.8
}
}
Note that the document never prints the phrase "beta blocker" in its title, but carries beta and blocker as weighted terms. That is the value of a sparse representation: whatever produced the map may add terms the text lacks, and retrieval then finds documents that never state your words.
The query
A document scores as the dot product of the query-side map with the document-side map:
score(d) = Σ q(t) × d(t) for each term t in both maps
The simplest form supplies the query-side map directly:
GET /literature/_search
{
"query": {
"neural_sparse": {
"ml_tokens": {
"query_tokens": { "beta": 2.1, "blocker": 1.9, "cardioselective": 0.8 }
}
}
}
}
The outer field name (ml_tokens) is the rank_features field to search, matching the nesting the semantic query uses.
Three ways to produce the query side
The query-side map is just terms and weights. Nothing requires a model to have produced it, and neural_sparse accepts three sources. Exactly one is required, and they are mutually exclusive.
| Source | Use it when |
|---|---|
query_tokens | You already have weights — from your own encoder, a cached expansion, or an upstream system. |
query_text | You have words and want the cluster to encode them. |
concept_expansion | You want expansion grounded in a published vocabulary rather than inferred by a model. |
query_tokens — supply the map
{
"neural_sparse": {
"ml_tokens": {
"query_tokens": { "cardiac": 1.9, "arrest": 1.4 }
}
}
}
Weights must be strictly positive and finite. A zero contributes nothing to a dot product while still costing a postings lookup, so omit terms you do not want rather than sending them as 0.
query_text — let the cluster encode
{
"neural_sparse": {
"ml_tokens": {
"query_text": "cardiac arrest management",
"provider": "hashing"
}
}
}
The text is encoded once during query rewrite, on the coordinating node. No shard sees the raw text and no shard calls an encoder — what travels to the shards is terms and weights.
provider names a registered sparse encoder and is only valid alongside query_text. Omit it to use the default.
The default encoder is not a learned model. Lucenia ships a dependency-free default (hashing) so that every path works without a model runtime or a network call. It lowercases, splits on non-letters, and weights by saturating frequency. It does not expand a query with terms the text lacks, which is the entire value of a learned sparse encoder. Register a SPLADE-class encoder and name it in provider for real sparse retrieval — see Extensibility.
concept_expansion — expand from a vocabulary
Instead of asking a model what a query is about, ask an indexed vocabulary. Lucenia scores the vocabulary once per request and turns the result into weighted query terms:
{
"neural_sparse": {
"ml_concepts": {
"concept_expansion": {
"graph_index": "vocabulary",
"field": "subclass_of",
"source": 319,
"measure": "wu_palmer",
"min_similarity": 0.6
}
}
}
}
Documents carry their concept annotations in a rank_features field keyed by concept id ({"8790": 1.0}). Every expanded term then traces to an edge in a vocabulary somebody else maintains, rather than to a model's belief — expansion by definition instead of by inference.
This is the option to reach for when a link has to survive being questioned.
The vocabulary itself is an index of graph_edge documents, built from scratch in the walkthrough immediately below. wu_palmer and path score on the shape of the hierarchy alone and need nothing recorded first; resnik, lin and jiang_conrath additionally weigh how informative a shared ancestor is, so they require corpus frequencies to have been recorded for the vocabulary and refuse until they have.
See Concept expansion and vocabularies for building a vocabulary in full, what each measure does, and the single-shard requirement on the index holding the edges.
A worked expansion
The ids above are real MeSH descriptors: 319 is Adrenergic beta-Antagonists — beta blockers — sitting under 959 Antihypertensive Agents, with 8790 Metoprolol, 1262 Atenolol and 11433 Propranolol beneath it, and 2121 Calcium Channel Blockers beside it under the same parent.
Build the vocabulary and annotate a corpus:
PUT /vocabulary
{
"settings": { "number_of_shards": 1, "number_of_replicas": 0 },
"mappings": { "properties": { "subclass_of": { "type": "graph_edge" } } }
}
POST /_bulk?refresh=true
{ "index": { "_index": "vocabulary", "_id": "e1" } }
{ "subclass_of": { "source": 319, "target": 959 } }
{ "index": { "_index": "vocabulary", "_id": "e2" } }
{ "subclass_of": { "source": 2121, "target": 959 } }
{ "index": { "_index": "vocabulary", "_id": "e3" } }
{ "subclass_of": { "source": 8790, "target": 319 } }
{ "index": { "_index": "vocabulary", "_id": "e4" } }
{ "subclass_of": { "source": 1262, "target": 319 } }
{ "index": { "_index": "vocabulary", "_id": "e5" } }
{ "subclass_of": { "source": 11433, "target": 319 } }
{ "index": { "_index": "vocabulary", "_id": "e6" } }
{ "subclass_of": { "source": 999001, "target": 11433 } }
PUT /literature
{
"settings": { "number_of_shards": 1, "number_of_replicas": 0 },
"mappings": {
"properties": {
"title": { "type": "text" },
"ml_concepts": { "type": "rank_features" }
}
}
}
Not one of these titles contains the phrase "beta blocker":
POST /_bulk?refresh=true
{ "index": { "_index": "literature", "_id": "class-overview" } }
{ "title": "Mechanisms of the drug class", "ml_concepts": { "319": 1.0 } }
{ "index": { "_index": "literature", "_id": "metoprolol-trial" } }
{ "title": "Cardioselective agent in post-infarction management", "ml_concepts": { "8790": 1.0 } }
{ "index": { "_index": "literature", "_id": "atenolol-review" } }
{ "title": "Once-daily dosing and adherence", "ml_concepts": { "1262": 1.0 } }
{ "index": { "_index": "literature", "_id": "propranolol-la-study" } }
{ "title": "Extended release preparation pharmacokinetics", "ml_concepts": { "999001": 1.0 } }
{ "index": { "_index": "literature", "_id": "amlodipine-study" } }
{ "title": "Peripheral oedema in long-term therapy", "ml_concepts": { "2121": 1.0 } }
{ "index": { "_index": "literature", "_id": "unrelated" } }
{ "title": "Orthopedic fracture fixation", "ml_concepts": { "77777": 1.0 } }
Expand from the class:
GET /literature/_search
{
"query": {
"neural_sparse": {
"ml_concepts": {
"concept_expansion": {
"graph_index": "vocabulary",
"field": "subclass_of",
"source": 319,
"measure": "wu_palmer"
}
}
}
}
}
Four hits, class-overview first and propranolol-la-study last. Read what that ordering means:
| Document | Concept | Why it placed there |
|---|---|---|
class-overview | 319 — the class itself | Scores highest; it is the concept asked for |
metoprolol-trial | 8790 — direct child | A kind of the class, one hop down |
atenolol-review | 1262 — direct child | Likewise |
propranolol-la-study | 999001 — grandchild | Still beneath the class, but furthest away |
amlodipine-study | 2121 — sibling class | No match. Shares only the root, which scores zero |
unrelated | 77777 — not in the vocabulary | No match. Nothing to expand from |
Three of the four recovered papers name only a specific drug. A keyword search for "beta blocker" returns none of them.
This is the difference from graph_traversal. A traversal would return the same four documents as equals — set membership, no ordering. The expansion ranks them: the class above a direct child, a direct child above a grandchild. Completeness and a ranking out of the same pass.
Tighten the floor to drop the most distant concept:
{
"neural_sparse": {
"ml_concepts": {
"concept_expansion": {
"graph_index": "vocabulary",
"field": "subclass_of",
"source": 319,
"measure": "wu_palmer",
"min_similarity": 0.6
}
}
}
}
Three hits — the grandchild falls below the floor; the class and its direct children stay.
And because it is an ordinary query, it intersects with a keyword requirement in one request:
GET /literature/_search
{
"query": {
"bool": {
"must": [
{
"neural_sparse": {
"ml_concepts": {
"concept_expansion": {
"graph_index": "vocabulary",
"field": "subclass_of",
"source": 319,
"measure": "wu_palmer"
}
}
}
},
{ "match": { "title": "adherence" } }
]
}
}
}
One hit: atenolol-review. The expansion leg admits everything beneath the drug class; the keyword leg narrows to the paper about adherence.
concept_expansion parameters
| Parameter | Type | Required/Optional | Description |
|---|---|---|---|
graph_index | String | Required | The index holding the graph_edge vocabulary. |
field | String | Required | The graph_edge field describing the vocabulary. |
source | Long | Required | The concept id to expand from. |
measure | String | Optional | Similarity measure: wu_palmer, path, resnik, lin, jiang_conrath. Default is lin. |
min_similarity | Float | Optional | Concepts scoring below this are not included. Default is 0. |
max_nodes | Integer | Optional | Ceiling on resolved concepts. Default is 65536. |
Query parameters
| Parameter | Type | Required/Optional | Description |
|---|---|---|---|
query_tokens | Object | One of three | Term-to-weight map. Weights must be strictly positive and finite. |
query_text | String | One of three | Text for a sparse encoder to encode. |
concept_expansion | Object | One of three | Vocabulary expansion settings, as above. |
provider | String | Optional | Named sparse encoder. Only valid with query_text. |
prune | Boolean | Optional | Enable lossy query-token pruning. Default is false. |
prune_ratio | Float | Optional | With pruning enabled, drop terms weighing less than prune_ratio × max(weight). Must be in [0, 1). Default is 0.4. |
boost | Float | Optional | Standard query boost. Default is 1.0. |
_name | String | Optional | Standard named-query label. |
Exact by default
A real sparse query is long — a learned encoder emits a hundred or more terms, most of them faint. Evaluating all of them costs something, and the common mitigation is to discard the faint ones before searching.
Lucenia implements that, but leaves it off:
{
"neural_sparse": {
"ml_tokens": {
"query_tokens": { "...": 1.0 },
"prune": true,
"prune_ratio": 0.4
}
}
}
On a 200,000-document benchmark with 100-term queries, prune_ratio: 0.4 discards 80 of the 100 terms and returns 8 of the 10 documents the complete query would have returned. Two of your top ten disappear — not reordered, absent — and the ones that vanish are those that matched only on subtle terms.
So pruning is something you opt into, not something you inherit.
With pruning off, results are exact. Underneath, the dot product is evaluated as a disjunction over feature impacts, which lets Lucene skip blocks of the index that provably cannot reach the top of the result list. The returned top k is identical to an exhaustive scan; the skipping is a speed optimization, not an approximation.
Worked example
Paste-able end to end. It shows both what sparse retrieval finds and what pruning costs.
1. Create an index with a sparse field.
PUT /literature
{
"settings": { "number_of_shards": 1, "number_of_replicas": 0 },
"mappings": {
"properties": {
"title": { "type": "text" },
"ml_tokens": { "type": "rank_features" }
}
}
}
2. Index three documents. Note that none of them contains the word "cardiac" in its title — the terms live in the sparse map.
POST /_bulk?refresh=true
{ "index": { "_index": "literature", "_id": "1" } }
{ "title": "resuscitation protocol", "ml_tokens": { "cardiac": 3.0, "arrest": 2.5, "resuscitation": 1.2 } }
{ "index": { "_index": "literature", "_id": "2" } }
{ "title": "imaging review", "ml_tokens": { "cardiac": 0.4, "imaging": 2.0 } }
{ "index": { "_index": "literature", "_id": "3" } }
{ "title": "fracture fixation", "ml_tokens": { "orthopedic": 4.0 } }
3. Search with text. The cluster encodes it for you.
GET /literature/_search
{
"query": {
"neural_sparse": {
"ml_tokens": { "query_text": "cardiac arrest" }
}
}
}
Two hits, document 1 first: it carries both encoded terms with heavy weights, while 2 shares only a faint cardiac. Document 3 shares no term and does not match.
4. Watch what pruning costs. Add a faint term that only one document carries.
GET /literature/_search
{
"query": {
"neural_sparse": {
"ml_tokens": {
"query_tokens": { "cardiac": 2.0, "resuscitation": 0.05 }
}
}
}
}
Two hits — the exact query keeps document 1 on the strength of cardiac, and resuscitation adds a little more.
Now enable pruning:
GET /literature/_search
{
"query": {
"neural_sparse": {
"ml_tokens": {
"query_tokens": { "cardiac": 2.0, "resuscitation": 0.05 },
"prune": true,
"prune_ratio": 0.4
}
}
}
}
resuscitation weighs 0.05, which is below 2.0 × 0.4, so it is discarded before the search runs. On this corpus the hit count is unchanged, but the contribution that distinguished document 1 is gone — and a document matching only on a pruned term disappears entirely. That is the trade, and it is why pruning is off unless you ask for it.
5. Combine with a keyword requirement.
GET /literature/_search
{
"query": {
"bool": {
"must": [
{ "neural_sparse": { "ml_tokens": { "query_text": "cardiac" } } },
{ "match": { "title": "imaging" } }
]
}
}
}
One hit: document 2. The sparse leg admits both cardiac documents, and the keyword leg narrows to the one whose title mentions imaging.
Composing with other queries
neural_sparse is an ordinary query, not a search-pipeline processor, so it nests anywhere a query can appear — including inside bool and hybrid, alongside match, knn, semantic and graph_traversal:
GET /literature/_search?search_pipeline=hybrid-pipeline
{
"query": {
"hybrid": {
"queries": [
{ "match": { "title": "post-infarction beta blockade" } },
{
"neural_sparse": {
"ml_tokens": { "query_text": "cardioselective agents after MI" }
}
},
{
"semantic": {
"abstract_vector": { "query_text": "cardioselective agents after MI", "k": 10 }
}
}
]
}
}
}
Because the subqueries score on different scales, pair hybrid with a normalization pipeline as described in Combining the scores.
Errors you may hit
| Message | Cause |
|---|---|
only works on [rank_features] fields | The named field is a different type. Sparse retrieval reads a rank_features field. |
requires a [rank_features] field with [positive_score_impact] set to true | The field was mapped with positive_score_impact: false. Sparse scores are a dot product of non-negative weights, so an inverted field is not meaningful here. |
requires one of [query_tokens], [concept_expansion] or [query_text] | No query side was supplied. |
accepts only one of [query_tokens], [concept_expansion] and [query_text] | More than one query side was supplied. |
[provider] only applies with [query_text] | A provider was named without query_text to encode. |
no sparse encoding provider registered under [...] | The named provider is not registered on this node. The message lists what is available. |
query token [...] must have a strictly positive finite weight | A weight was zero, negative, or not a number. |
An unmapped field matches nothing rather than raising an error, matching the behavior of other queries. Text that encodes to no terms likewise matches nothing.
Security
neural_sparse is an ordinary query and inherits the caller's index-read authorization — there is no additional scope to grant. When concept_expansion is used, the caller must also be able to read the vocabulary index named in graph_index.
Under the hood
The document side is stored as Lucene feature fields, one per term, with the weight carried in the term frequency. The query is assembled as a disjunction of linear feature clauses, one per query term. Because those postings expose per-block score bounds, evaluation skips blocks that cannot contribute to the top k while still returning exact results.
When query_text or concept_expansion is used, resolution happens once during query rewrite on the coordinating node, and the resolved terms and weights are what reach the shards. Neither the raw text nor the vocabulary is ever sent to a shard.