Learn how Elasticsearch engineer and Apache Lucene contributor Julie Tibshirani came to join the open source search community, and about its friendly and accepting culture.

<p>Open source software development can have a reputation for abrasive behavior. The search community is a clear counterexample for me, with a culture that emphasizes respect and acceptance. This culture played an important part in my own path to open source development.
</p><p>A little over six years ago, I was a wide-eyed software engineer settling into my first full-time job. My company made data analysis software for governments and large organizations, and I had just joined the team that built and maintained the search infrastructure.
</p><p>Until then, when I thought of ‘search’, what came to mind was a web search engine or online shopping site. It soon became clear that search was a far more general and fundamental problem — our users needed to find the relevant parts of their data, and find them quickly. Search formed a core component of each organization’s critical work, which included anti-fraud investigations, combating human trafficking, tracking epidemics, and more.
</p><p>Search technology brings together work from often disparate fields: systems and databases, linguistics, and even mathematical fields like statistics and information theory. I found it so engaging to be able to work across these different areas —&nbsp;in a given week I might be designing a data schema, pondering the rules of plural nouns, or debugging why certain queries had high tail latencies. Needless to say, it didn’t take long for me to fall in love with search.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt2d56347a807d1cea/5e8cba1575c4742aa0e609dc/Google_search_sign.jpg" data-sys-asset-uid="blt2d56347a807d1cea" alt="Google_search_sign.jpg" style="display: block; margin: auto;">
</p><figcaption>Seen outside a favorite bookstore in Mountain View.</figcaption><p>Our search infrastructure was built on the open source search library Apache Lucene, and we were working on writing a&nbsp;new search service&nbsp;using&nbsp;Elasticsearch (which is also based on Lucene under the hood). Using an open source library made it possible to deliver features to our users that would have taken years for our tiny search team to develop ourselves. In addition to being packed full of capabilities, Lucene was so fast and seemed to constantly be getting faster. During my studies, I took a deep interest in data structures and foundational algorithms like prefix trees, skip lists, and sorting techniques. In Lucene I saw these topics come to life in code and play a role in powering hundreds of critical applications.
</p><p>A few months into my new job, I was scratching my head at why a new search highlighting feature we were building wasn’t working as expected. In debugging the problem, I discovered an issue in Elasticsearch where <a href="https://github.com/elastic/elasticsearch/issues/6420">nested fields weren’t highlighted</a>. I continued to dig deeper and <a href="https://issues.apache.org/jira/browse/LUCENE-5929">traced the problem to Lucene</a>. I had a sense for how to fix the issue, and it seemed like a good opportunity to contribute to a project that had benefited my team so much.
</p><p>Although it was a small change, I was quite nervous once it came time to submit the patch. One of the "Mythical Lucene Developers" would be reviewing it, those search experts who built this wonderful&nbsp;package. And the code review would be open for the whole internet to see — in other open source projects I had seen examples of developers being <a href="https://marc.info/?l=linux-kernel&m=135628421403144&w=2">publicly berated</a> for a bad patch. From my experience in university and my first job, I had developed a working picture of an accomplished software engineer: they were ‘prickly’ and forceful with their opinions, and were always self-assured, sometimes to the point of overconfidence. I assumed these talented open source developers would fit this mold.
</p><p>I closed my eyes and hit ‘submit’, and after a friendly and straightforward review, the patch was merged! Encouraged by this experience, I made some <a href="https://github.com/elastic/elasticsearch/pull/10359">small</a> <a href="https://github.com/elastic/elasticsearch/issues/10356">contributions</a> to Elasticsearch in my rare free time at work. A couple years after my first open source contributions, I started <a href="https://github.com/grf-labs/grf">my own open source project</a> and became the one giving friendly and straightforward reviews.
</p><p>I now work for Elastic as a core developer on Elasticsearch, where I contribute full-time to these projects I once admired from afar. As I’ve spent more time in the open source search community, I’ve seen that these initial helpful reviews are part of a broader pattern. By and large, the developers I interact with treat each other with respect and kindness. Despite their considerable accomplishments, most contributors don’t have ‘big heads’ — the ratio of ego to talent remains surprisingly low. Of course there are arguments and heated discussions, but engineers seem motivated by what they think is technically ‘right’, by what they feel is best for the project.
</p><p>In previous jobs, design discussions were always centered around a whiteboard. A group of engineers would huddle in a room, sometimes for hours, and debate the details of a new feature or architecture. Although this process was engaging, I preferred having the time to think deeply about the problem on my own, and organize my thoughts before discussing with others. As a stereotypically polite Canadian, I sometimes found it difficult to seize the floor in these lively discussions filled with interruptions.
</p><p>In open source development, it’s much more common to rely on written, asynchronous communication. I appreciate having the space to carefully ponder a problem, then draft up an email or GitHub post laying out my thoughts. Even the most soft-spoken engineers are able to weigh in and bring their opinions to the discussion.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt492850bd1d24272d/5e8e2fcbbe1a31776ec326de/polite_canadians.jpg" data-sys-asset-uid="blt492850bd1d24272d" alt="polite_canadians.jpg" style="display: block; margin: auto;"></p><figcaption>After you! No, after you. Credit: Enzo Comics<br></figcaption><p>The open source search community spans many cultures and languages, which contributes to there being a diversity of communication styles. Contributors tend to be quite flexible when someone has a style that doesn’t match their own. I found that I could ‘speak Canadian’ and still be understood, like when I deliver important but gentle feedback in a code review, or apologize in the middle of a bug report.
</p><p>Lucene turned 20 years old this year, which is a big accomplishment for such a widely-used open source project. In terms of both its technology and its people, the open source search community is one to admire. I am so happy to be part of it, and hope it continues to grow in the years to come.
</p><p><em>Interested in joining a company with <a href="https://www.elastic.co/about/our-source-code?baymax=web&elektra=culture-finding-a-home-and-career-in-the-open-source-community">a Source Code</a> to live by? We’re hiring. Check out <a href="https://www.elastic.co/about/teams?baymax=web&elektra=culture-finding-a-home-and-career-in-the-open-source-community">our teams</a> and <a href="https://www.elastic.co/about/careers?baymax=web&elektra=culture-finding-a-home-and-career-in-the-open-source-community">find your fit!</a> Want to read more about life at Elastic? Read more <a href="https://www.elastic.co/blog/category/culture/?baymax=web&elektra=culture-finding-a-home-and-career-in-the-open-source-community">on our blog!</a></em>
</p>

Elastic_SeriousAnalytics_Hero.jpg

Elastic_SeriousAnalytics_Thumbnail_.jpg

Finding a home (and career) in the open source community

This post explores how text embeddings and Elasticsearch’s new dense_vector type could be used to support similarity search.

<p>From its beginnings as a <a href="/about/history-of-elasticsearch">recipe search engine</a>, Elasticsearch was designed to provide fast and powerful full-text search. Given these roots, improving text search has been an important motivation for our ongoing work with vectors. In Elasticsearch 7.0, we introduced experimental field types for high-dimensional vectors, and now the 7.3 release brings support for using these vectors in document scoring.
</p><p>This post focuses on a particular technique called text similarity search. In this type of search, a user enters a short free-text query, and documents are ranked based on their similarity to the query. Text similarity can be useful in a variety of use cases:
</p><ul>
	<li><strong>Question-answering:</strong> Given a collection of frequently asked questions, find questions that are similar to the one the user has entered.</li>
	<li><strong>Article search:</strong> In a collection of research articles, return articles with a title that’s closely related to the user’s query.</li>
	<li><strong>Image search:</strong> In a dataset of captioned images, find images whose caption is similar to the user’s description.</li>
</ul><p>A straightforward approach to similarity search would be to rank documents based on how many words they share with the query. But a document may be similar to the query even if they have very few words in common — a more robust notion of similarity would take into account its syntactic and <a href="https://en.wikipedia.org/wiki/Semantic_similarity">semantic</a> content as well.
</p><p>The natural language processing (NLP) community has developed a technique called text embedding that encodes words and sentences as numeric vectors. These vector representations are designed to capture the linguistic content of the text, and can be used to assess similarity between a query and a document.
</p><p>This post explores how text embeddings and Elasticsearch’s <code>dense_vector</code> type could be used to support similarity search. We’ll first give an overview of embedding techniques, then step through a simple prototype of similarity search using Elasticsearch.
</p><table style="background: #FFFFD2;">
<tbody>
<tr>
	<td><strong>Note:</strong> Using text embeddings in search is a <em>complex and evolving</em> area. We hope this blog post gives a jumping off point for exploration, but it is not a recommendation for a particular search architecture or implementation.
	</td>
</tr>
</tbody>
</table><h2>What are text embeddings?</h2><p>Let's take a closer look at different types of text embeddings, and how they compare to traditional search approaches.
</p><h3>Word embeddings</h3><p>A <a href="https://en.wikipedia.org/wiki/Word_embedding">word embedding</a> model represents a word as a dense numeric vector. These vectors aim to capture semantic properties of the word — words whose vectors are close together should be similar in terms of semantic meaning. In a good embedding, directions in the vector space are tied to different aspects of the word’s meaning. As an example, the vector for "Canada" might be close to "France" in one direction, and close to "Toronto" in another.
</p><p>The NLP and search communities have been interested in vector representations of words for quite some time. There was a resurgence of interest in word embeddings in the past few years, when many traditional tasks were being revisited using neural networks. Some successful word embedding algorithms were developed, including <a href="https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf">word2vec</a> and <a href="https://nlp.stanford.edu/pubs/glove.pdf">GloVe</a>. These approaches make use of large text collections, and examine the context each word appears in to determine its vector representation:
</p><ul>
	<li>The word2vec Skip-gram model trains a neural network to predict the context words around a word in a sentence. The internal weights of the network give the word embeddings.</li>
	<li>In GloVe, the similarity of words depends on how frequently they appear with other context words. The algorithm trains a simple linear model on word co-occurrence counts.</li>
</ul><p>Many research groups distribute models that have been pre-trained on large text corpora like Wikipedia or Common Crawl, making them convenient to download and plug into downstream tasks. Although pre-trained versions are sometimes used directly, it can be helpful to adjust the model to fit the specific target dataset and task. This is often accomplished by running a 'fine-tuning' step on the pre-trained model.
</p><p>Word embeddings have proven quite robust and effective, and it is now common practice to use embeddings in place of individual tokens in NLP tasks like machine translation and sentiment classification.
</p><h3>Sentence embeddings</h3><p>More recently, researchers have started to focus on embedding techniques that represent not only words, but longer sections of text. Most current approaches are based on complex neural network architectures, and sometimes incorporate labelled data during training to aid in capturing semantic information.
</p><p>Once trained, the models are able to take a sentence and produce a vector for each word in context, as well as a vector for the entire sentence. Similarly to word embedding, pre-trained versions of many models are available, allowing users to skip the expensive training process. While the training process can be very resource-intensive, invoking the model is much more lightweight — sentence embedding models are typically fast enough to be used as part of real-time applications.
</p><p>Some common sentence embedding techniques include <a href="https://arxiv.org/abs/1705.02364">InferSent</a>, <a href="https://arxiv.org/abs/1803.11175">Universal Sentence Encoder</a>, <a href="https://arxiv.org/abs/1802.05365">ELMo</a>, and <a href="https://arxiv.org/abs/1810.04805">BERT</a>. Improving word and sentence embeddings is an active area of research, and it’s likely that additional strong models will be introduced.
</p><h3>Comparison to traditional search approaches</h3><p>In traditional information retrieval, a common way to represent text as a numeric vector is to assign one dimension for each word in the vocabulary. The vector for a piece of text is then based on the number of times each term in the vocabulary appears. This way of representing text is often referred to as "bag of words," because we simply count word occurrences without regard to sentence structure.
</p><p>Text embeddings differ from traditional vector representations in some important ways:
</p><ul>
	<li>The encoded vectors are dense and relatively low-dimensional, often ranging from 100 to 1,000 dimensions. In contrast, bag of words vectors are sparse and can comprise 50,000+ dimensions. Embedding algorithms encode the text into a lower-dimensional space as part of modeling its semantic meaning. Ideally, synonymous words and phrases end up with a similar representation in the new vector space.</li>
	<li>Sentence embeddings can take the order of words into account when determining the vector representation. For example the phrase "tune in" may be mapped as a very different vector than "in tune".</li>
	<li>In practice, sentence embeddings often don’t generalize well to large sections of text. They are not commonly used to represent text longer than a short paragraph.</li>
</ul><h2>Using embeddings for similarity search</h2><p>Let’s suppose we had a large collection of questions and answers. A user can ask a question, and we want to retrieve the most similar question in our collection to help them find an answer.
</p><p>We could use text embeddings to allow for retrieving similar questions:
</p><ul>
	<li>During indexing, each question is run through a sentence embedding model to produce a numeric vector.</li>
	<li>When a user enters a query, it is run through the same sentence embedding model to produce a vector. To rank the responses, we calculate the vector similarity between each question and the query vector. When comparing embedding vectors, it is common to use <a href="https://en.wikipedia.org/wiki/Cosine_similarity">cosine similarity</a>.</li>
</ul><p><a href="https://github.com/jtibshirani/text-embeddings">This repository</a> gives a simple example of how this could be accomplished in Elasticsearch. The main script indexes ~20,000 questions from the <a href="https://github.com/elastic/rally-tracks/tree/master/so">StackOverflow dataset</a>, then allows the user to enter free-text queries against the dataset.
</p><p>We’ll soon walk through each part of the script in detail, but first let’s look at some example results. In many cases, the method is able to capture similarity even when there was not strong word overlap between the query and indexed question:
</p><ul>
	<li>"zipping up files" returns "Compressing / Decompressing Folders & Files"</li>
	<li>"determine if something is an IP" returns "How do you tell whether a string is an IP or a hostname"</li>
	<li>"translate bytes to doubles" returns "Convert Bytes to Floating Point Numbers in Python"</li>
</ul><h3>Implementation details</h3><p><a href="https://github.com/jtibshirani/text-embeddings/blob/blog/src/main.py">The script</a> begins by downloading and creating the embedding model in TensorFlow. We chose Google’s Universal Sentence Encoder, but it’s possible to use many other embedding methods. The script uses the embedding model as-is, without any additional training or fine-tuning.
</p><p>Next, we create the Elasticsearch index, which includes mappings for the question title, tags, and also the question title encoded as a vector:
</p><pre class="prettyprint">"mappings": {
  "properties": {
    "title": {
      "type": "text"
    },
    "title_vector": {
      "type": "dense_vector",
      "dims": 512
    }
    "tags": {
      "type": "keyword"
    },
    ...
  }
}
</pre><p>In the mapping for <code>dense_vector</code>, we’re required to specify the number of dimensions the vectors will contain. When indexing a <code>title_vector</code> field, Elasticsearch will check that it has the same number of dimensions as specified in the mapping.
</p><p>To index documents, we run the question title through the embedding model to obtain a numeric array. This array is added to the document in the <code>title_vector</code> field.
</p><p>When a user enters a query, the text is first run through the same embedding model and stored in the parameter <code>query_vector</code>. As of 7.3, Elasticsearch provides a <a href="/guide/en/elasticsearch/reference/7.6/query-dsl-script-score-query.html#vector-functions"><code>cosineSimilarity</code> function</a> in its native scripting language. So to rank questions based on their similarity to the user’s query, we use a <code>script_score</code> query:
</p><pre class="prettyprint">{
  "script_score": {
    "query": {"match_all": {}},
    "script": {
      "source": "cosineSimilarity(params.query_vector, 'title_vector') + 1.0",
      "params": {"query_vector": query_vector}
    }
  }
}
</pre><p>We make sure to pass the query vector as a script parameter to <a href="/guide/en/elasticsearch/reference/7.6/modules-scripting-using.html#prefer-params">avoid recompiling the script</a>() on every new query. Since Elasticsearch does not allow negative scores, it's necessary to add one to the cosine similarity.
</p><table style="background: #FFFFD2;">
<tbody>
<tr>
	<td><strong>Note:</strong> this blog post originally used a <a href="/guide/en/elasticsearch/reference/7.3/query-dsl-script-score-query.html#vector-functions">different syntax for vector functions</a> that was available in Elasticsearch 7.3, but was deprecated in 7.6.
	</td>
</tr>
</tbody>
</table><h3>Important limitations</h3><p>The <code>script_score</code> query is designed to wrap a restrictive query, and modify the scores of the documents it returns. However, we’ve provided a <code>match_all</code> query, which means the script will be run over all documents in the index. This is a current limitation of vector similarity in Elasticsearch — vectors can be used for scoring documents, but not in the initial retrieval step. Support for retrieval based on vector similarity is an important area of <a href="https://github.com/elastic/elasticsearch/issues/42326">ongoing work</a>.
</p><p>To avoid scanning over all documents and to maintain fast performance, the <code>match_all</code> query can be replaced with a more selective query. The right query to use for retrieval is likely to depend on the specific use case.
</p><p>While we saw some encouraging examples above, it’s important to note that the results can also be noisy and unintuitive. For example, "zipping up files" also assigns high scores to "Partial .csproj Files" and "How to avoid .pyc files?". And when the method returns surprising results, it is not always clear how to debug the issue — the meaning of each vector component is often opaque and doesn’t correspond to an interpretable concept. With traditional scoring techniques based on word overlap, it is often easier to answer the question "why is this document ranked highly?"
</p><p>As mentioned earlier, this prototype is meant as an example of how embedding models could be used with vector fields, and not as a production-ready solution. When developing a new search strategy, it is critical to test how the approach performs on your own data, making sure to compare against a strong baseline like a <code>match</code> query. It may be necessary to make major changes to the strategy before it achieves solid results, including fine-tuning the embedding model for the target dataset, or trying different ways of incorporating embeddings such as word-level query expansion.
</p><h2>Conclusions</h2><p>Embedding techniques provide a powerful way to capture the linguistic content of a piece of text. By indexing embeddings and scoring based on vector distance, we can compare documents using a notion of similarity that goes beyond their word-level overlap.
</p><p>We’re looking forward to introducing more functionality based around the vector field type. Using vectors for search is a nuanced and developing area — as always, we would love to hear about your use cases and experiences on <a href="https://github.com/elastic/elasticsearch">Github</a> and the <a href="https://discuss.elastic.co/">Discuss forums</a>!
</p>

books.jpg

books_small.jpg

Text similarity search with vector fields

Learn about the new 'alias' datatype in Elasticsearch, including how it helps with field renaming and multi-index search.

earth-from-international-space-station.jpg

international-space-station.jpg

Introducing Field Aliases in Elasticsearch

This Week in Elasticsearch and Apache Lucene - 2019-05-05

Julie Tibshirani

Articles By

Register to Attend

Sign In to Attend

Register to Watch

You'll also receive an email with related content

Watch Now

Learn More

Contact Info

See All Posts

What to explore next...

Load more news

Load more press releases

Load More

View more learning opportunities

More

Date

Location

Agenda

Hosted by

Related workshops

View next

Translate Content

Articles by Julie Tibshirani

Finding a home (and career) in the open source community

Text similarity search with vector fields

Introducing Field Aliases in Elasticsearch

This Week in Elasticsearch and Apache Lucene - 2019-05-05

Products & Solutions

Company

Resources

Featured topics

News

Articles by Julie Tibshirani

Finding a home (and career) in the open source community

Text similarity search with vector fields

Introducing Field Aliases in Elasticsearch

This Week in Elasticsearch and Apache Lucene - 2019-05-05