Skip to main content
Version: 0.13.0

Concept link processor

Introduced 0.13.0

The concept_link processor reads a document's text, finds the names of concepts from a vocabulary in it, and writes the concept identifiers onto the document. It is what connects concept expansion to your actual content: without it, the vocabulary knows that a beta-blocker is a cardiac drug but nothing knows which of your documents are about one.

It also records what it could not link, which turns out to matter more than it sounds.

Why a dictionary, and not a model​

A dictionary linker is exhaustive over what it knows, needs no model, runs air-gapped, and — above all — can explain itself. A document is about dogs because the word "hound" appears at character 412 and that is a label of the concept. A model that scores 0.83 cannot say that, and a linker whose decisions cannot be argued with is one nobody will trust with a compliance question.

Parameters​

ParameterRequiredDefaultDescription
fieldYesThe text field to read.
vocabularyYesThe index holding the vocabulary's concepts and their labels.
target_fieldNoconceptThe field the matched concept identifiers are written to. This is the field graph_traversal uses as its node_field.
coverage_fieldNolinkedThe boolean field written to every document, saying whether anything was linked.
include_mentionsNofalseAlso write _mentions, recording each match with its text, start, end and concept.
ignore_missingNofalseSkip documents that do not have field instead of failing them.

Example​

PUT _ingest/pipeline/link-concepts
{
  "processors": [
    {
      "concept_link": {
        "field": "abstract",
        "vocabulary": "mesh",
        "target_field": "concept",
        "include_mentions": true
      }
    }
  ]
}

A document whose abstract mentions a hound comes out carrying the concept identifier for that concept, linked: true, and — because include_mentions was set — the evidence:

{
  "abstract": "The hound was examined ...",
  "concept": [ 4285301 ],
  "linked": true,
  "_mentions": [
    { "text": "hound", "start": 4, "end": 9, "concept": 4285301 }
  ]
}

How matching works​

By whole tokens, and this is not a detail​

Substring matching would link every document containing "category" to the concept cat. The failure is silent and it compounds: those documents then answer questions about animals, and a recall claim becomes a precision embarrassment. A mention must begin and end where a word does.

Longest match wins, overlaps are kept apart​

Where "New York City" and "New York" both match at the same position, the longer one is what the author meant. But a shorter name starting elsewhere is a separate mention and is kept — text saying "New York City and York" names two things.

Ambiguous names report every candidate​

When one name belongs to several concepts, all of them are recorded. The processor does not guess which sense was meant; disambiguation is a separate decision, and silently picking one would be unexplainable.

The linked field, and why it exists​

The processor writes linked to every document it sees — false as readily as true.

That field exists so that coverage is an ordinary aggregation. You can ask what share of an index is reachable by concept search with a plain query, and the FindByMeaning tool reads exactly this field to state its coverage in every answer.

GET articles/_search
{
  "size": 0,
  "aggs": { "coverage": { "terms": { "field": "linked" } } }
}
warning

Documents that are not linked cannot be found by concept search at all. This is the honest boundary of the completeness claim: concept expansion returns every linked document about a concept, not every document. Measure your coverage before you rely on it.

Order in the pipeline​

concept_link reads text, so it belongs after anything that produces text — content extraction, for example — and before indexing. It does not need embeddings and does not call any external service.

Next​