Definition

What is vector search?

Vector search leverages machine learning (ML) to capture the meaning and context of unstructured data, including text and images, transforming it into a numeric representation. Frequently used for semantic search, vector search finds similar data using approximate nearing neighbor (ANN) algorithms. Compared to traditional keyword search, vector search yields more relevant results and executes faster.

Why IT leaders need vector search
videoImage

Why is vector search important?

How often have you looked for something, but you're not sure what it’s called? You may know what it does or have a description. But without the keywords, you're left searching.

Vector search overcomes this limitation, allowing you to search by what you mean. It can quickly deliver answers to queries based on context. That's because vector embeddings capture synonyms and associations — essentially the meaning behind your search. You can enhance the search experience by combining vector search with filtering and aggregations to optimize relevance by implementing a hybrid search and combining it with traditional scoring.

videoImage

How does a vector search engine work?

Vector search engines — known as vector databases, semantic, or cosine search — find the nearest neighbors to a given (vectorized) query.

Where traditional search relies on mentions of keywords, lexical similarity, and the frequency of word occurrences, vector search engines use distances in the embedding space to represent similarity. Finding related data becomes searching for nearest neighbors of your query.

Technical process of how a vector search engine works

  • Vector embedding

    Vector embeddings are the numeric representation of data and related context, stored in high dimensional (dense) vectors. Models that generate embeddings can be trained on millions of examples to deliver more relevant and accurate results. For some cases, numeric data that you collected or engineered to represent key features of your documents can be used as embeddings. All you need is the ability to search efficiently.

    Read more on text embeddings

  • Similarity score

    At the core of a vector search engine is the idea that if data and documents are alike, their vectors will be similar. By indexing both queries and documents with vector embeddings, you find similar documents as the nearest neighbors of your query.

    Read more on text similarity

  • ANN algorithm

    Traditional nearest neighbor algorithms, like k-Nearest Neighbor Algorithm (kNN), lead to excessive execution times and zaps computational resources. ANN sacrifices perfect accuracy in exchange for executing efficiently in high dimensional embedding spaces, at scale.

    Read more on ANN

Vector search use cases

Vector search not only powers the next generation of search experiences, it opens the door to a range of new possibilities.

Power search with ML

  • Semantic search

    Vector search powers semantic or similarity search. Since the meaning and context is captured in the embedding, vector search finds what users mean, without requiring an exact keyword match. It works with textual data (documents), images, and audio. Easily and quickly find products that are similar or related to their query.

    Watch video how vector search enables semantic search

  • Recommendations

    The model that generates the embedding learns to recognize similar documents and their vectors in the embedding space. For example, an application may recommend movies or products that others who purchased the one you bought also liked. But, make sure that the embeddings are derived with some measure of popularity or likability as the target metric.

    Vector distances can be combined with other metrics so that recommendations achieve multiple goals. For example, rank product recommendations by satisfaction scores and revenue potential.

  • Question answering

    Converting documents to text embeddings can be combined with modern Natural Language Processing (NLP) to deliver full text answers to questions. This approach spares users from studying lengthy manuals and empowers your teams to provide answers more quickly.

    A "question answering" transformer model can take the text embedding representation for both the knowledge base of documents and your current question to deliver the closest match as an "answer."

    How to apply question answering in Elastic

More you can do with vector search

Don't stop at just semantic search!

  • Browse unstructured data

    Search any unstructured data. You can create embeddings for text, images, audio, or sensor measurements.

    How to implement image similarity search

  • Filter on metadata

    Filter vector search results using metadata. Maintain recall without sacrificing speed by applying a filter in line with Approximate Nearest Neighbor (ANN) search.

    Learn how to filter queries

  • Re-rank search results

    Vector similarities can be interpreted as similarity scores that you can re-rank with other data. That includes static fields already in your vector search database, and new properties by applying machine learning models.

    Learn-to-rank in Elastic

  • Hybrid scoring

    To further optimize, combine vector similarities with BM25F scores, known as hybrid scoring. This allows you to have images ranked by vector similarity while implementing BM25F, which can provide better rankings for text.

    Use hybrid scoring in Elastic

How to get started

Vector search and NLP made easy with Elastic

You don't have to move mountains to implement vector search and apply NLP models. Elastic Search Platform covers all the bases for you: import pretrained models from public repositories, apply them on the fly while ingesting your data, and use model predictions to improve your search experience and automated system monitoring.

videoImage

  • Access NLP models

    The Eland library lets you load NLP models from PyTorch or public model repositories like Hugging Face. Use models off-the-shelf without worrying about needing the expertise to train models yourself.

    How to import PyTorch NLP models

  • Add NLP inference

    Leverage Elastic machine learning features to analyze natural language and make predictions.

    How to apply NLP in Elastic

  • Implement transformers

    Use transformers as a deep neural network that learns context in sequential data by tracking the relationships and relevance of each element in a sequence. With Elastic, you can use transformers to convert text into their vector representation, infer sentiment, and answer questions in full text natural language.

    Integrate NLP models

  • Text embedding and more

    Learn how to assign sentiments and other categories to your data with Elastic. Apply Named Entity Recognition (NER) to improve search experiences with additional metadata.

    Modern NLP in the Elastic Stack

Engage with us on vector search and NLP

Get an introduction to vector search and NLP models from the experts at Elastic.

Watch now