Concept expansion and vocabularies

Vector search answers what is similar. Concept expansion answers a different question: what is all of it.

If you ask for documents about a drug class, a beta blocker, a product category, or a corporate subsidiary, you usually do not want the ten nearest neighbors. You want every document about any kind of the thing you asked for — including the documents that never print the term — and you want to be able to show that the set is complete.

Lucenia indexes a vocabulary as first-class data and expands it inside a query. Two queries read it:

Query	Answers	Returns
graph_traversal	Is this concept beneath that one?	A complete set, unranked by relatedness
concept_similarity	How close are these concepts?	A graded ranking

They compose with match, knn, semantic and neural_sparse in a single request.

Indexing a vocabulary

A vocabulary is an index whose edges are stored in a graph_edge field. Each edge document names a source and a target.

PUT /concepts
{
  "settings": { "number_of_shards": 1 },
  "mappings": {
    "properties": {
      "subclass_of": { "type": "graph_edge" }
    }
  }
}

Index edges child to parent, so that traversing from a concept walks up toward its ancestors:

POST /_bulk
{ "index": { "_index": "concepts", "_id": "a" } }
{ "subclass_of": { "source": 2, "target": 1 } }
{ "index": { "_index": "concepts", "_id": "b" } }
{ "subclass_of": { "source": 3, "target": 2 } }
{ "index": { "_index": "concepts", "_id": "c" } }
{ "subclass_of": { "source": 4, "target": 3 } }

Concept identifiers are longs. Use whatever identifier your vocabulary already publishes — a MeSH descriptor number, a GO term, an internal product-category id.

The vocabulary index must have exactly one shard

This is enforced, not advisory. Mapping a graph_edge field on a multi-shard index fails outright:

[graph_edge] requires a single-shard index, but this index has 5

A traversal unions the segments a single shard can see, so a vocabulary spread across shards would answer from whichever part it landed on and report that as the whole thing. A four-hop route could come back as one edge. Rather than return a confidently partial answer, the mapping is refused.

The restriction applies only to the index holding the edges, and vocabularies are small enough that one shard is not a real constraint. The corpus you actually search has no such limit — see Searching a corpus on another index.

A graph_edge field is traversed rather than searched. Running an ordinary term query against it returns an error directing you to a traversal query.

You can hold several independent relations in one index by mapping several graph_edge fields (subclass_of, part_of, and so on). A traversal over one never leaks into the other.

Annotating a corpus

The corpus you actually search is a separate index whose documents carry the concepts they are about:

PUT /articles
{
  "mappings": {
    "properties": {
      "concept": { "type": "long" },
      "title":   { "type": "text" }
    }
  }
}

POST /_bulk
{ "index": { "_index": "articles", "_id": "dogs" } }
{ "concept": 4, "title": "dog training" }
{ "index": { "_index": "articles", "_id": "eagles" } }
{ "concept": 7, "title": "eagle migration" }

The corpus index has no shard restriction — give it as many shards as the data needs. It holds no graph. When a query names graph_index, the vocabulary is walked once during rewrite on its single shard, and what reaches the corpus shards is a set of concept ids they filter locally, knowing nothing about graphs.

concept may be multi-valued. A document is scored on its best concept, so annotating thoroughly is never penalized by concept_similarity.

graph_traversal — completeness

GET /articles/_search
{
  "query": {
    "graph_traversal": {
      "field": "subclass_of",
      "source": 2,
      "graph_index": "concepts",
      "node_field": "concept"
    }
  }
}

This returns every article annotated with a concept beneath 2, at any depth.

Parameters

Parameter	Type	Required/Optional	Description
field	String	Required	The graph_edge field describing the vocabulary.
source	Long	Required	The concept to traverse from.
graph_index	String	Optional	The index holding the vocabulary, when it is not the index being searched.
node_field	String	Optional	The field on the searched corpus carrying concept ids.
hops	Integer	Optional	Limit the traversal depth. Omit for the full closure.
max_nodes	Integer	Optional	Ceiling on resolved concepts. Default is 65536.

graph_index and node_field travel together. Naming graph_index without node_field is refused rather than guessed. With both, the traversal resolves concepts in the vocabulary index and matches them against node_field on the corpus.

Without node_field, the traversal matches the edge documents themselves by their endpoints. That is the self-contained form, useful when the vocabulary and the data live in the same index — not the corpus-filtering form.

concept_similarity — graded relatedness

A traversal tells you a concept is beneath another. It cannot tell you that a dog is closer to a cat than to an eagle. concept_similarity scores the vocabulary and ranks the corpus by it:

GET /articles/_search
{
  "query": {
    "concept_similarity": {
      "field": "subclass_of",
      "source": 4,
      "measure": "wu_palmer",
      "min_similarity": 0.0001,
      "graph_index": "concepts",
      "node_field": "concept"
    }
  }
}

Parameters

Parameter	Type	Required/Optional	Description
field	String	Required	The graph_edge field describing the vocabulary.
source	Long	Required	The concept to score against.
measure	String	Optional	One of the five measures below. Default is lin.
min_similarity	Double	Optional	Concepts scoring below this are not resolved. Default is 0.
graph_index	String	Optional	The index holding the vocabulary.
node_field	String	Optional	The field on the corpus carrying concept ids.
max_nodes	Integer	Optional	Ceiling on resolved concepts. Default is 65536.

The five measures

Measure	Needs a corpus	What it uses
wu_palmer	No	Depth of the shared ancestor over the depths of both concepts.
path	No	Reciprocal of the path length through the nearest shared ancestor.
resnik	Yes	How informative the shared ancestor is. Unbounded above.
lin	Yes	Resnik normalized into [0, 1].
jiang_conrath	Yes	Information distance turned into a similarity.

The structural measures need only the hierarchy. The corpus-aware measures rest on information content — a concept that annotates nearly everything carries little information, and a rare one carries a lot — so they need frequencies recorded first.

They are worth the extra step. In a corpus where nearly every annotation is a mammal, Animal distinguishes nothing, so lin will drop a bird that wu_palmer happily returns on tree geometry alone.

A corpus measure refuses until frequencies are recorded rather than silently falling back to a structural one. A caller who asked for lin and silently got wu_palmer would be ranking on tree geometry while believing they were ranking on usage.

min_similarity is a prune, not a filter. It is not applied after scoring the vocabulary; it bounds the search itself, which stops expanding once nothing beneath the current concept could still reach the threshold. On a Gene Ontology-sized vocabulary that is the difference between 0.1 ms and 104 ms — roughly 1000×. Set it as high as your use case tolerates.

Querying several concepts at once — use dis_max

One concept_similarity clause names one source. When a query is about several concepts, how you combine the clauses matters more than any other knob measured:

{
  "dis_max": {
    "queries": [
      { "concept_similarity": { "field": "subclass_of", "source": 278,  "measure": "lin", "min_similarity": 0.4, "graph_index": "concepts", "node_field": "concept" } },
      { "concept_similarity": { "field": "subclass_of", "source": 7049, "measure": "lin", "min_similarity": 0.4, "graph_index": "concepts", "node_field": "concept" } }
    ]
  }
}

A bool/should sums its clauses, which quietly ranks by how thoroughly a document was annotated — a document carrying forty concepts gets forty chances to match each clause. Measured on NFCorpus, the summed form scored 0.1327 nDCG@10 against dis_max at 0.2125, a 60% difference, and the summed form's top results averaged 24.2 concepts against a corpus mean of 17.6.

Reach for bool/should only when you genuinely want "matches more of the things I asked about" to outrank "matches one thing closely".

Recording information content

Counts come from your corpus, which is usually a different index from the vocabulary, so you supply them. The field query parameter is required and names the graph_edge field the counts apply to:

POST /concepts/_graph/information_content?field=subclass_of
{
  "counts": {
    "4": 500,
    "5": 480,
    "7": 2
  }
}

{
  "_shards": { "successful": 1 },
  "concepts_supplied": 3,
  "concepts_covered": 7,
  "corpus_size": 982
}

concepts_covered exceeds concepts_supplied because counts roll up: annotating a dog also counts toward mammal, animal and thing. That rollup is what makes the measure meaningful, and it is why you supply leaf counts rather than pre-aggregated ones.

corpus_size is the count that accumulated at the most general concept — not the sum of the counts you supplied. A document annotated with three concepts is one document, and counting it three times would distort every score derived from it.

Re-record after a material change in your corpus. The values are derived state, not source data.

How a vocabulary is built

An edge is a document. There is no separate store to keep in step with the index — you index a vocabulary the way you index anything else, and the adjacency is derived from the segments Lucene already wrote. Deleting an edge document is retracting the relationship, and it takes effect immediately with no rebuild.

The derived adjacency is written to a scratch directory belonging to the shard rather than into the Lucene commit, and read back memory-mapped rather than held on the heap. Three consequences worth knowing:

• A snapshot contains exactly what Lucene wrote, and CheckIndex finds nothing it cannot account for.
• A replica derives its own copy rather than receiving one.
• The structure costs essentially no heap.

The first query needing a vocabulary builds it; later queries reuse it, and rebuilding happens naturally as segments change. No explicit refresh step is required beyond the usual index refresh.

What behaves as you would hope

• Multiple inheritance is followed down every branch, and a concept reachable by two routes is reported once.
• Cycles terminate. They arrive in real vocabularies through modelling errors, sameAs pairs, and merges of disagreeing sources.
• Depth is the longest path to a root. Real vocabularies are DAGs, so a concept has many depths; taking the shortest would make a concept look more general the more parents somebody gave it.
• The lowest common subsumer is a set. In a tree there is exactly one; in a DAG several can be mutually incomparable. The measures use the deepest — or, for corpus-aware measures, the most informative, which is not always the same concept.
• Undefined stays undefined. A concept your corpus never used has no information content, so corpus-aware measures return NaN rather than infinity — −log(0) would make an unused concept look maximally specific, which is backwards.

Use graph_traversal, not road_distance

Lucenia's road_distance query also walks a graph_edge field, but it is backed by a contraction hierarchy, which is quadratic in node degree and therefore catastrophic on hub-and-spoke topology. A road junction has degree three; an upper-level concept has thousands of children. A 50,000-concept taxonomy will not finish contracting, while a 929,000-junction road network contracts in about 1.3 seconds. That accelerator is for road networks; this walk is for everything else.

_graph/build is a different feature. Lucenia also uses graph_edge fields for weighted road networks, where a contraction hierarchy is built ahead of time via POST /{index}/_graph/build?field=...&weight_field=.... That endpoint requires a numeric weight field and belongs to the routing path — it is not how a concept vocabulary is prepared, and a weightless vocabulary has nothing to give it.

Scale

Resolving is cheap; holding and shipping the answer is what scales against you. Every resolved concept is sent to every shard.

Measured on NCBI Taxonomy (2,917,213 concepts):

Operation	Time
Build the vocabulary from segments	827 ms
Expand 862,034 descendants	~150 ms
Expand 4,337 descendants	< 2 ms

The max_nodes default of 65536 is reached by ordinary queries at that scale — 105 of those 2.9 million concepts exceed it. A vocabulary the size of MeSH never comes close; its largest subtree is 7,735. If you hit the ceiling, the error names the field, the source, and the trade-off.

Worked example

A complete walkthrough you can paste into a cluster. It uses a deliberately tiny vocabulary so the difference between a structural and a corpus-aware measure is visible by eye.

Thing (1)
└── Animal (2)
    ├── Mammal (3)
    │   ├── Dog (4)
    │   └── Cat (5)
    └── Bird (6)
        └── Eagle (7)

Mineral (50)          ← a separate root, sharing nothing with the above
└── Quartz (51)

1. Create the vocabulary and the corpus.

PUT /concepts
{
  "settings": { "number_of_shards": 1, "number_of_replicas": 0 },
  "mappings": { "properties": { "subclass_of": { "type": "graph_edge" } } }
}

PUT /articles
{
  "settings": { "number_of_shards": 1, "number_of_replicas": 0 },
  "mappings": {
    "properties": {
      "concept": { "type": "long" },
      "title":   { "type": "text" }
    }
  }
}

2. Load the edges, child to parent.

POST /_bulk?refresh=true
{ "index": { "_index": "concepts", "_id": "a" } }
{ "subclass_of": { "source": 2,  "target": 1  } }
{ "index": { "_index": "concepts", "_id": "b" } }
{ "subclass_of": { "source": 3,  "target": 2  } }
{ "index": { "_index": "concepts", "_id": "c" } }
{ "subclass_of": { "source": 4,  "target": 3  } }
{ "index": { "_index": "concepts", "_id": "d" } }
{ "subclass_of": { "source": 5,  "target": 3  } }
{ "index": { "_index": "concepts", "_id": "e" } }
{ "subclass_of": { "source": 6,  "target": 2  } }
{ "index": { "_index": "concepts", "_id": "f" } }
{ "subclass_of": { "source": 7,  "target": 6  } }
{ "index": { "_index": "concepts", "_id": "g" } }
{ "subclass_of": { "source": 51, "target": 50 } }

3. Annotate the corpus.

POST /_bulk?refresh=true
{ "index": { "_index": "articles", "_id": "dogs" } }
{ "concept": 4,  "title": "dog training" }
{ "index": { "_index": "articles", "_id": "cats" } }
{ "concept": 5,  "title": "cat behaviour" }
{ "index": { "_index": "articles", "_id": "eagles" } }
{ "concept": 7,  "title": "eagle migration" }
{ "index": { "_index": "articles", "_id": "quartz" } }
{ "concept": 51, "title": "quartz crystals" }

4. Rank by a structural measure. Nothing needs to be recorded first.

GET /articles/_search
{
  "query": {
    "concept_similarity": {
      "field": "subclass_of",
      "source": 4,
      "measure": "wu_palmer",
      "min_similarity": 0.0001,
      "graph_index": "concepts",
      "node_field": "concept"
    }
  }
}

Three hits, dogs first. Tree geometry reaches the sibling (cats) and the cousin (eagles) alike. quartz sits under a separate root and shares nothing, so it does not match at all.

5. Record information content. Counts come from your corpus, so you supply them.

POST /concepts/_graph/information_content?field=subclass_of
{
  "counts": { "4": 500, "5": 480, "7": 2 }
}

{
  "_shards": { "successful": 1 },
  "concepts_supplied": 3,
  "concepts_covered": 7,
  "corpus_size": 982
}

Seven concepts are covered though only three were supplied. Each supplied count rolls up through every ancestor of the concept it was given for: Dog and Cat carry theirs up through Mammal, Eagle carries its own up through Bird, and all three reach Animal and Thing. The covered set is the union — Dog, Cat, Eagle, Mammal, Bird, Animal, Thing.

corpus_size is 982 — the count that accumulated at the most general concept, not the sum of every entry in the table. A document annotated with three concepts must not be counted three times.

6. Rank by a corpus-aware measure.

GET /articles/_search
{
  "query": {
    "concept_similarity": {
      "field": "subclass_of",
      "source": 4,
      "measure": "lin",
      "min_similarity": 0.0001,
      "graph_index": "concepts",
      "node_field": "concept"
    }
  }
}

Now two hits, still dogs first. eagles has dropped out, and lin is right to drop it: in this corpus 980 of 982 annotations are mammals, so Animal carries almost no information and is a near-worthless thing to share. Mammal still separates dog from cat, so cats survives.

That disagreement is the whole reason the corpus-aware measures exist. Structural measures see the shape of the tree; corpus measures see how much a shared ancestor actually tells you.

Run step 6 before step 5 and the query is refused with an error naming information_content, rather than quietly falling back to tree geometry.

A real vocabulary

The tree above is small enough to reason about by eye, but the ids you will actually index are the ones your vocabulary already publishes. The examples in the rest of this page use a slice of MeSH (Medical Subject Headings, published by the NIH), with the numeric part of each descriptor code as the node id — D000319 becomes 319:

Id	Descriptor	Concept	Sits under
959	D000959	Antihypertensive Agents	—
319	D000319	Adrenergic beta-Antagonists (beta blockers)	959
2121	D002121	Calcium Channel Blockers	959
8790	D008790	Metoprolol	319
1262	D001262	Atenolol	319
11433	D011433	Propranolol	319

POST /_bulk?refresh=true
{ "index": { "_index": "vocabulary", "_id": "e1" } }
{ "subclass_of": { "source": 319,   "target": 959 } }
{ "index": { "_index": "vocabulary", "_id": "e2" } }
{ "subclass_of": { "source": 2121,  "target": 959 } }
{ "index": { "_index": "vocabulary", "_id": "e3" } }
{ "subclass_of": { "source": 8790,  "target": 319 } }
{ "index": { "_index": "vocabulary", "_id": "e4" } }
{ "subclass_of": { "source": 1262,  "target": 319 } }
{ "index": { "_index": "vocabulary", "_id": "e5" } }
{ "subclass_of": { "source": 11433, "target": 319 } }

Traversing from 319 recovers every paper about a beta blocker — including papers that name only metoprolol or atenolol and never print the phrase "beta blocker". Those are precisely the papers a keyword search loses.

A sibling class is not a near miss. 2121 (calcium channel blockers) sits beside 319 under the same parent, so it is tempting to expect it to score as "related". Under wu_palmer it scores zero and does not match at all: Wu-Palmer is 2 × depth(shared ancestor) / (depth(a) + depth(b)), the only ancestor the two share is the root 959, and a root has depth 0. Concepts scoring zero are dropped rather than returned faintly.

That is the correct answer for a drug-class query — a calcium channel blocker is not a kind of beta blocker — but it is worth knowing before you assume a sibling will appear in the ranking.

Modelling: two traps worth knowing before you index

1. Not every hierarchy composes transitively

A transitive traversal is only correct when the edges mean strict subsumption.

• Composes: is_a in Gene Ontology, subClassOf in OWL. If A is a B and B is a C, then A is a C.
• Does not compose: "broader term" relations in thesauri — MeSH, SNOMED-style vocabularies, most human-indexer schemes. "Broader than" does not chain.

Indexing MeSH with one node per descriptor and walking transitively disagreed with MeSH's own tree numbers on 592 of 936 test concepts, always over-reaching: Acids expanded to 158 descendants against a true 40, sweeping in Alum Compounds, which are salts.

The fix is to make the tree position the node rather than the descriptor. Each tree number becomes its own node, its parent is that number minus its last segment, and a document annotated with a descriptor is indexed under every position it occupies. Closure then matches the vocabulary's own "explode" semantics exactly.

Check which kind of hierarchy you have before you index it.

2. Similarity is not compatibility

Sibling leaves under a shared parent score as closely related. That is correct when they are alternatives, and wrong when they are mutually exclusive.

Consider a vocabulary of electrical socket types. wu_palmer scores a BS 1363 socket against a NEMA 5-15 socket at 0.667 — they sit side by side under "socket type". But a single wall cannot hold both, so observing one where you expected the other is evidence against a match, not for it.

Worse, the genuine cross-branch signal often scores lower: a Type G socket and a 230 V rating plate score only 0.333, because their nearest shared ancestor is the whole electrical system. Ranked on similarity alone, the strongest contradictions outrank the true matches, and raising min_similarity removes the true matches first.

If your vocabulary's leaves are mutually exclusive, a similarity measure alone is the wrong tool. Model the exclusivity explicitly — treat same-parent leaf pairs as contradictory rather than related — and carry cross-branch evidence some other way.

Composing with keyword and vector search

Both concept queries are ordinary queries, so they nest wherever a query can appear.

Intersect a traversal with a keyword requirement:

GET /articles/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "graph_traversal": {
            "field": "subclass_of",
            "source": 319,
            "graph_index": "concepts",
            "node_field": "concept"
          }
        },
        { "match": { "title": "mortality" } }
      ]
    }
  }
}

Or fuse concept, keyword and vector into one ranking with hybrid:

GET /articles/_search?search_pipeline=hybrid-pipeline
{
  "query": {
    "hybrid": {
      "queries": [
        {
          "graph_traversal": {
            "field": "subclass_of",
            "source": 319,
            "graph_index": "concepts",
            "node_field": "concept"
          }
        },
        { "match": { "title": "post-infarction management" } },
        { "semantic": { "abstract_vector": { "query_text": "post-infarction management", "k": 10 } } }
      ]
    }
  }
}

See Combining the scores for normalization options.

Vocabulary-grounded sparse retrieval

A concept_similarity result is a set of concepts with weights — the same shape a sparse query takes. The neural_sparse query can therefore expand from the vocabulary directly:

{
  "neural_sparse": {
    "ml_concepts": {
      "concept_expansion": {
        "graph_index": "concepts",
        "field": "subclass_of",
        "source": 319,
        "measure": "wu_palmer"
      }
    }
  }
}

The difference from graph_traversal is ranking. A traversal returns everything beneath a concept as equals; this ranks the concept itself above a direct child, and a direct child above a grandchild — completeness and an ordering from the same pass.

What the completeness claim covers

Concept expansion is complete with respect to your annotations. If a document is tagged with a leaf concept, expansion from any ancestor provably finds it.

It cannot find a document nobody annotated. Annotation coverage is a separate, prior problem, and no retrieval strategy solves it.

Measured against ground truth from human indexers — 936 MeSH concepts over a curated corpus — concept expansion returned the complete set every time. Embedding search returned 18.8% of it, and BM25 14.5%. On documents that never state the term, embedding search returned 11.3%. Neither baseline returned a complete set for any of the 936 concepts.

It is completeness that was demonstrated, not better ranking

Those numbers are about membership — did the complete set come back. They are not evidence that concept scoring ranks better, and it is worth being precise about that, because the two get conflated.

Measured on NFCorpus, adding concept_similarity to a BM25 + embedding hybrid produced no improvement distinguishable from noise on either the test or the dev split, and was consistently harmful on queries whose terminology is common. Repeating it with the curated MeSH headings a human indexer assigned in PubMed — rather than dictionary-matched ones — was marginally worse, so this is not a problem better annotations fix.

The honest comparison is against embeddings, not BM25. Embeddings need no curated hierarchy, cover open-domain text, and where they apply they are the stronger ranker.

What concept expansion offers that an embedding cannot is exactness, auditability, and typed relations. is_a and part_of are different questions and a vector cannot tell them apart; "the Gene Ontology says these share a parent" is a defensible reason to have surfaced a document, and "the vector said so" is not. Those are the reasons to reach for this. A general ranking improvement is not one — it was looked for and not found.

Treat any token-saving argument ("6 relevant chunks in context instead of 20") as a hypothesis to test on your own corpus, not a property of the feature.

When not to reach for this

You need a vocabulary, and your documents must carry concepts.

• Good fit — medicine (SNOMED, MeSH, ICD), biology (GO), law, finance taxonomies, product catalogs, internal ontologies: places where annotations often already exist and you need an answer you can defend.
• Poor fit — general prose with no curated hierarchy. There is nothing to traverse, and embeddings are the better tool.
• If you must generate annotations with an LLM, price that work against exactness and auditability, which it does buy. Do not assume it buys ranking quality.

Security

Both queries are ordinary queries and inherit the caller's index-read authorization. When graph_index names a separate vocabulary index, the caller must be able to read that index too.