Radial search
Preview
Radial search enhances the k-NN plugin’s capabilities beyond approximate top-k searches. With radial search, you can search all points within a vector space that reside within a specified maximum distance or minimum score threshold from a query point. This provides increased flexibility and utility in search operations.
Parameter type
max_distance allows users to specify a physical distance within the vector space, identifying all points that are within this distance from the query point. This approach is particularly useful for applications requiring spatial proximity or absolute distance measurements.
min_score enables the specification of a similarity score, facilitating the retrieval of points that meet or exceed this score in relation to the query point. This method is ideal in scenarios where relative similarity, based on a specific metric, is more critical than physical proximity.
Only one query variable, either k, max_distance, or min_score, is required to be specified during radial search. For more information about the vector spaces, see Spaces.
Supported cases
You can perform radial search with either the Lucene or Faiss engines. The following table summarizes radial search use cases by engine.
| Engine supported | Filter supported | Nested field supported | Search type |
|---|---|---|---|
| Lucene | true | false | approximate |
| Faiss | true | true | approximate |
Spaces
A space corresponds to the function used to measure the distance between two points in order to determine the k-nearest neighbors. When using k-NN, a lower score equates to a closer and better result. This is the opposite of how Lucenia scores results, where a greater score equates to a better result. To convert distances to Lucenia scores, radial search uses the following formula: 1 / (1 + distance). The k-NN plugin supports the following spaces. Not every method supports each of these spaces. Be sure to refer to the method documentation to verify that the space you want to use is supported.
| Space type | Distance function (d) | Lucenia score |
|---|---|---|
| l1 | \[ d(\mathbf{x}, \mathbf{y}) = \sum_{i=1}^n |x_i - y_i| \] | \[ score = {1 \over 1 + d } \] |
| l2 | \[ d(\mathbf{x}, \mathbf{y}) = \sum_{i=1}^n (x_i - y_i)^2 \] | \[ score = {1 \over 1 + d } \] |
| \[ d(\mathbf{x}, \mathbf{y}) = max(|x_i - y_i|) \] | \[ score = {1 \over 1 + d } \] | |
| cosinesimil | \[ d(\mathbf{x}, \mathbf{y}) = 1 - cos { \theta } = 1 - {\mathbf{x} · \mathbf{y} \over \|\mathbf{x}\| · \|\mathbf{y}\|}\]\[ = 1 - {\sum_{i=1}^n x_i y_i \over \sqrt{\sum_{i=1}^n x_i^2} · \sqrt{\sum_{i=1}^n y_i^2}}\] where \(\|\mathbf{x}\|\) and \(\|\mathbf{y}\|\) represent the norms of vectors x and y respectively. | nmslib and Faiss:\[ score = {1 \over 1 + d } \] Lucene:\[ score = {2 - d \over 2}\] |
| innerproduct | \[ d(\mathbf{x}, \mathbf{y}) = - {\mathbf{x} · \mathbf{y}} = - \sum_{i=1}^n x_i y_i \] Lucene: \[ d(\mathbf{x}, \mathbf{y}) = {\mathbf{x} · \mathbf{y}} = \sum_{i=1}^n x_i y_i \] | \[ \text{If} d \ge 0, \] \[score = {1 \over 1 + d }\] \[\text{If} d < 0, score = −d + 1\] Lucene: \[ \text{If} d > 0, score = d + 1 \] \[\text{If} d \le 0\] \[score = {1 \over 1 + (-1 · d) }\] |
The cosine similarity formula does not include the 1 - prefix. However, because similarity search libraries equate lower scores with closer results, they return 1 - cosineSimilarity for the cosine similarity space. This is why 1 - is included in the distance function.
With cosine similarity, it is not valid to pass a zero vector ([0, 0, ...]) as an input. This is because the magnitude of such a vector is 0, which raises a divide by 0 exception in the corresponding formula. Requests containing a zero vector will be rejected, and a corresponding exception will be thrown.
Examples
The following examples can help you to get started with radial search.
Prerequisites
To use a k-NN index with radial search, create a k-NN index by setting index.knn to true. Specify one or more fields of the knn_vector data type, as shown in the following example:
PUT knn-index-test
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"index.knn": true
},
"mappings": {
"properties": {
"my_vector": {
"type": "knn_vector",
"dimension": 2,
"method": {
"name": "hnsw",
"space_type": "l2",
"engine": "faiss",
"parameters": {
"ef_construction": 100,
"m": 16,
"ef_search": 100
}
}
}
}
}
}
After you create the index, add some data similar to the following:
PUT _bulk?refresh=true
{"index": {"_index": "knn-index-test", "_id": "1"}}
{"my_vector": [7.0, 8.2], "price": 4.4}
{"index": {"_index": "knn-index-test", "_id": "2"}}
{"my_vector": [7.1, 7.4], "price": 14.2}
{"index": {"_index": "knn-index-test", "_id": "3"}}
{"my_vector": [7.3, 8.3], "price": 19.1}
{"index": {"_index": "knn-index-test", "_id": "4"}}
{"my_vector": [6.5, 8.8], "price": 1.2}
{"index": {"_index": "knn-index-test", "_id": "5"}}
{"my_vector": [5.7, 7.9], "price": 16.5}
Example: Radial search with max_distance
The following example shows a radial search performed with max_distance:
GET knn-index-test/_search
{
"query": {
"knn": {
"my_vector": {
"vector": [
7.1,
8.3
],
"max_distance": 2
}
}
}
}
All documents that fall within the squared Euclidean distance (l2^2) of 2 are returned, as shown in the following response:
Results
{
"took": 6,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 4,
"relation": "eq"
},
"max_score": 0.98039204,
"hits": [
{
"_index": "knn-index-test",
"_id": "1",
"_score": 0.98039204,
"_source": {
"my_vector": [
7.0,
8.2
],
"price": 4.4
}
},
{
"_index": "knn-index-test",
"_id": "3",
"_score": 0.9615384,
"_source": {
"my_vector": [
7.3,
8.3
],
"price": 19.1
}
},
{
"_index": "knn-index-test",
"_id": "4",
"_score": 0.62111807,
"_source": {
"my_vector": [
6.5,
8.8
],
"price": 1.2
}
},
{
"_index": "knn-index-test",
"_id": "2",
"_score": 0.5524861,
"_source": {
"my_vector": [
7.1,
7.4
],
"price": 14.2
}
}
]
}
}
Example: Radial search with max_distance and a filter
The following example shows a radial search performed with max_distance and a response filter:
GET knn-index-test/_search
{
"query": {
"knn": {
"my_vector": {
"vector": [7.1, 8.3],
"max_distance": 2,
"filter": {
"range": {
"price": {
"gte": 1,
"lte": 5
}
}
}
}
}
}
}
All documents that fall within the squared Euclidean distance (l2^2) of 2 and have a price within the range of 1 to 5 are returned, as shown in the following response:
Results
{
"took": 4,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 2,
"relation": "eq"
},
"max_score": 0.98039204,
"hits": [
{
"_index": "knn-index-test",
"_id": "1",
"_score": 0.98039204,
"_source": {
"my_vector": [
7.0,
8.2
],
"price": 4.4
}
},
{
"_index": "knn-index-test",
"_id": "4",
"_score": 0.62111807,
"_source": {
"my_vector": [
6.5,
8.8
],
"price": 1.2
}
}
]
}
}
Example: Radial search with min_score
The following example shows a radial search performed with min_score:
GET knn-index-test/_search
{
"query": {
"knn": {
"my_vector": {
"vector": [7.1, 8.3],
"min_score": 0.95
}
}
}
}
All documents with a score of 0.9 or higher are returned, as shown in the following response:
Results
{
"took": 3,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 2,
"relation": "eq"
},
"max_score": 0.98039204,
"hits": [
{
"_index": "knn-index-test",
"_id": "1",
"_score": 0.98039204,
"_source": {
"my_vector": [
7.0,
8.2
],
"price": 4.4
}
},
{
"_index": "knn-index-test",
"_id": "3",
"_score": 0.9615384,
"_source": {
"my_vector": [
7.3,
8.3
],
"price": 19.1
}
}
]
}
}
Example: Radial search with min_score and a filter
The following example shows a radial search performed with min_score and a response filter:
GET knn-index-test/_search
{
"query": {
"knn": {
"my_vector": {
"vector": [
7.1,
8.3
],
"min_score": 0.95,
"filter": {
"range": {
"price": {
"gte": 1,
"lte": 5
}
}
}
}
}
}
}
All documents that have a score of 0.9 or higher and a price within the range of 1 to 5 are returned, as shown in the following example:
Results
{
"took": 4,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": 0.98039204,
"hits": [
{
"_index": "knn-index-test",
"_id": "1",
"_score": 0.98039204,
"_source": {
"my_vector": [
7.0,
8.2
],
"price": 4.4
}
}
]
}
}