Dense vector field type
editDense vector field type
editThe dense_vector
field type stores dense vectors of float values. Dense
vector fields can be used in the following ways:

In
script_score
queries, to score documents matching a filter  In the kNN search API, to find the k most similar vectors to a query vector
The dense_vector
type does not support aggregations or sorting.
You add a dense_vector
field as an array of floats:
PUT myindex { "mappings": { "properties": { "my_vector": { "type": "dense_vector", "dims": 3 }, "my_text" : { "type" : "keyword" } } } } PUT myindex/_doc/1 { "my_text" : "text1", "my_vector" : [0.5, 10, 6] } PUT myindex/_doc/2 { "my_text" : "text2", "my_vector" : [0.5, 10, 10] }
Unlike most other data types, dense vectors are always singlevalued.
It is not possible to store multiple values in one dense_vector
field.
Index vectors for kNN search
editThis functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features.
A knearest neighbor (kNN) search finds the k nearest vectors to a query vector, as measured by a similarity metric.
Dense vector fields can be used to rank documents in
script_score
queries. This lets you perform
a bruteforce kNN search by scanning all documents and ranking them by
similarity.
In many cases, a bruteforce kNN search is not efficient enough. For this
reason, the dense_vector
type supports indexing vectors into a specialized
data structure to support fast kNN retrieval through the
kNN search API. You can enable indexing by setting the
index
parameter:
PUT myindex2 { "mappings": { "properties": { "my_vector": { "type": "dense_vector", "dims": 3, "index": true, "similarity": "dot_product" } } } }
Elasticsearch uses the HNSW algorithm to support efficient kNN search. Like most kNN algorithms, HNSW is an approximate method that sacrifices result accuracy for improved speed.
Indexing vectors for approximate kNN search is an expensive process. It
can take substantial time to ingest documents that contain vector fields with
index
enabled.
Dense vector fields cannot be indexed if they are within
nested
mappings.
Parameters for dense vector fields
editThe following mapping parameters are accepted:

dims

(Required, integer)
Number of vector dimensions. Can’t exceed
1024
for indexed vectors ("index": true
), or2048
for nonindexed vectors. 
index

(Optional, Boolean)
If
true
, you can search this field using the kNN search API. Defaults tofalse
.

similarity

(Required^{*}, string) The vector similarity metric to use in kNN search. Documents are ranked by their vector field’s similarity to the query vector. The
_score
of each document will be derived from the similarity, in a way that ensures scores are positive and that a larger score corresponds to a higher ranking.^{*} If
index
istrue
, this parameter is required.Valid values for
similarity

l2_norm

Computes similarity based on the L^{2} distance (also known as Euclidean
distance) between the vectors. The document
_score
is computed as1 / (1 + l2_norm(query, vector)^2)
. 
dot_product

Computes the dot product of two vectors. This option provides an optimized way
to perform cosine similarity. In order to use it, all vectors must be of unit
length, including both document and query vectors. The document
_score
is computed as(1 + dot_product(query, vector)) / 2
. 
cosine

Computes the cosine similarity. Note that the most efficient way to perform
cosine similarity is to normalize all vectors to unit length, and instead use
dot_product
. You should only usecosine
if you need to preserve the original vectors and cannot normalize them in advance. The document_score
is computed as(1 + cosine(query, vector)) / 2
. Thecosine
similarity does not allow vectors with zero magnitude, since cosine is not defined in this case.

Although they are conceptually related, the similarity
parameter is
different from text
field similarity
and accepts
a distinct set of options.

index_options

(Optional, object) An optional section that configures the kNN indexing algorithm. The HNSW algorithm has two internal parameters that influence how the data structure is built. These can be adjusted to improve the accuracy of results, at the expense of slower indexing speed. When
index_options
is provided, all of its properties must be defined.Properties of
index_options

type

(Required, string)
The type of kNN algorithm to use. Currently only
hnsw
is supported. 
m

(Required, integer)
The number of neighbors each node will be connected to in the HNSW graph.
Defaults to
16
. 
ef_construction

(Required, integer)
The number of candidates to track while assembling the list of nearest
neighbors for each new node. Defaults to
100
.
