Multi-Vector Fields

Multi-vector fields allow you to store multiple embeddings in a single field within a single document. This is particularly useful for representing documents that have multiple parts, such as long text broken into chunks, images with multiple objects, or products with multiple images. Each document can have a different number of embeddings, providing flexibility for complex data structures.

Standard k-NN fields support only one vector per document. Multi-vector fields, however, allow for an array of vectors for each document, and the k-NN plugin will efficiently find the best match among all the vectors in the field.

Getting started with multi-vector fields

To use multi-vector search, you must first create a k-NN index with index.knn set to true.

Next, you must add one or more fields of the knn_multi_vector data type. The following example creates an index with a knn_multi_vector field using the faiss engine and hnsw algorithm:

PUT my-knn-index-1
{
  "settings": {
    "index": {
      "knn": true
    }
  },
  "mappings": {
    "properties": {
      "my_vectors": {
        "type": "knn_multi_vector",
        "dimension": 3,
        "method": {
          "name": "hnsw",
          "space_type": "l2",
          "engine": "faiss",
          "parameters": {
            "ef_construction": 128,
            "m": 24
          }
        }
      }
    }
  }
}

The knn_multi_vector data type supports the same engines (faiss, lucene) and algorithms as the standard knn_vector type.

Vector data types

Like knn_vector, the knn_multi_vector type supports the float (default), byte, and binary vector data types through the data_type mapping parameter. For example, to store binary embeddings:

"my_vectors": {
  "type": "knn_multi_vector",
  "data_type": "binary",
  "dimension": 16,
  "method": {
    "name": "hnsw",
    "space_type": "hamming",
    "engine": "faiss"
  }
}

Quantization (for example, the scalar sq encoder), and compression_level are also supported, just as for the standard knn_vector type.

Indexing documents with multiple embeddings

When indexing documents, you provide an array of arrays for the knn_multi_vector field. Each inner array represents a single embedding. You can provide a variable number of embeddings for each document, including an empty array.

POST _bulk
{ "index": { "_index": "my-knn-index-1", "_id": "1" } }
{ "my_vectors": [[1.5, 2.5, 3.5], [8.1, 2.7, 3.9]], "price": 12.2 }
{ "index": { "_index": "my-knn-index-1", "_id": "2" } }
{ "my_vectors": [[2.5, 3.5, 1.2], [2.2, 6.3, 1.1], [0.7, 0.1, 0.0]], "price": 7.1 }
{ "index": { "_index": "my-knn-index-1", "_id": "3" } }
{ "my_vectors": [[3.5, 4.5, 2.1], [4.5, 5.5, 3.3]], "price": 12.9 }
{ "index": { "_index": "my-knn-index-1", "_id": "4" } }
{ "my_vectors": [[5.5, 6.5, 7.7], [4.5, 5.5, 1.7]], "price": 1.2 }
{ "index": { "_index": "my-knn-index-1", "_id": "5" } }
{ "my_vectors": [], "price": 3.7 }

In this example:

• Document 1 has 2 embeddings.
• Document 2 has 3 embeddings.
• Document 5 has no embeddings.

Binary wire format (base64)

In addition to JSON arrays, you can index multi-vector fields using a compact binary wire format encoded as a base64 string. This is particularly useful for reducing the size of the request payload when indexing a large number of embeddings.

The base64 string must represent a byte array with the following layout:

• num_vectors: 4-byte big-endian integer.
• num_dimensions: 4-byte big-endian integer.
• vector_data: Raw vector data. The size depends on the data_type:
  • float: 4 bytes per dimension (big-endian float).
  • byte: 1 byte per dimension (signed byte).
  • binary: 1 bit per dimension (packed into bytes, (dimension + 7) / 8 bytes per vector).

For example, if you have a knn_multi_vector field with dimension: 3 and data_type: float, and you want to index two vectors [1.5, 2.5, 3.5] and [8.1, 2.7, 3.9], the binary payload would be 32 bytes:

• 00 00 00 02 (2 vectors)
• 00 00 00 03 (3 dimensions)
• 3F C0 00 00 (1.5)
• 40 20 00 00 (2.5)
• 40 60 00 00 (3.5)
• 41 01 99 9A (8.1)
• 40 2C CC CD (2.7)
• 40 79 99 9A (3.9)

Indexing the base64-encoded string:

POST _bulk
{ "index": { "_index": "my-knn-index-1", "_id": "1" } }
{ "my_vectors": "AAAAAgAAAAM/wAAAQCAAAEBgAABBAZmaQCzMzUB5mZo=" }

This makes the payload considerably smaller compared to the array-of-arrays encoding, allowing better transfer rates and larger batches of vectors to be processed at a time.

note

The binary wire format is currently supported only for indexing knn_multi_vector fields. Standard knn_vector fields and the vector parameter in knn queries do not support base64 encoding.

note

The base64 data will be placed in _source unless the field is configured to not be indexed in _source. Thus, if returning _source in a query, the client may need to parse the base64 data.

Searching multi-vector fields

To search multi-vector fields, use the knn query. This query type is designed to work with knn_multi_vector fields and finds the k-nearest neighbors among all vectors stored in those fields.

GET my-knn-index-1/_search
{
  "size": 2,
  "query": {
    "knn": {
      "my_vectors": {
        "vector": [2.0, 3.0, 1.0],
        "k": 2
      }
    }
  }
}

Scoring and results

When you perform a knn query, the plugin calculates the distance between the query vector and every vector within the specified field for each document. The score for a document is determined by its closest vector (the one with the minimum distance or maximum similarity).

For example, if a document has three vectors and their distances to the query vector are 0.5, 1.2, and 0.8, the distance used for that document's score will be 0.5.

The Lucenia score is then calculated based on this minimum distance, following the standard distance-to-score conversion: score = 1 / (1 + distance).

Query parameters

• vector: (Required) The query vector you want to find neighbors for.
• k: (Required) The number of nearest neighbors to return.
• min_score: (Optional) Minimum score for a document to be included in the results.
• max_distance: (Optional) Maximum distance for a document to be included in the results.

For more details on radial search using min_score and max_distance, see k-NN radial search.

warning

Radial search (using min_score or max_distance) against multi-vector fields is experimental. To use it, you must first enable the cluster setting knn.feature.multi_vector.experimental_queries.enabled:

PUT _cluster/settings
{
  "persistent": {
    "knn.feature.multi_vector.experimental_queries.enabled": true
  }
}

If this setting is disabled, a radial knn query is rejected with an error.

Even with the experimental setting enabled, two additional restrictions apply regardless:

• binary data type: Radial search is not supported for binary vectors, even with the experimental flag enabled.
• Quantization: Radial search is not supported for indexes that use quantization (for example, the sq encoder or on_disk / compression_level settings), even with the experimental flag enabled.

Limitations and considerations

Size limits

The total size of all vectors for a single document in a knn_multi_vector field must not exceed 2,621,440 bytes. This limit includes an 8-byte header (num_vectors and num_dimensions) and the serialized binary representation of all vectors in the field for that document.

Calculating maximum vectors per document

To calculate the maximum number of vectors ($N$) you can store in a single document for a given dimension ($D$) and data_type, use the following formulas:

• float: $N = \lfloor \frac{2,621,432}{D \times 4} \rfloor$
• byte: $N = \lfloor \frac{2,621,432}{D} \rfloor$
• binary: $N = \lfloor \frac{2,621,432}{\lceil D / 8 \rceil} \rfloor$

For example, with 128 dimensions:

• float: A maximum of 5,119 vectors per document.
• byte: A maximum of 20,479 vectors per document.
• binary: A maximum of 163,839 vectors per document.

Excluding multi-vector fields from _source

Because multi-vector fields can store a large amount of data per document, they can significantly increase the size of the _source field. This can waste disk space and network bandwidth on a field that typically is not needed for search results. For this reason, it is usually recommended to exclude knn_multi_vector fields from the _source.

Here is how to exclude a multi-vector field named my_vectors from the _source:

PUT my-knn-index-1
{
  "mappings": {
    "_source": {
      "excludes": ["my_vectors"]
    },
    "properties": {
      "my_vectors": {
        "type": "knn_multi_vector",
        "dimension": 128,
        "method": {
          "name": "hnsw",
          "engine": "faiss",
          "space_type": "l2"
        }
      }
    }
  }
}

• Engines: Multi-vector fields are supported for both Lucene and Faiss engines.
• Filters: Like standard approximate k-NN, filters are applied to the results of the initial vector search. For more information, see k-NN search with filters.
• Nested Fields: Multi-vector fields are not yet supported for use in nested fields, as of version 0.12.0.