This post shares strategies for incorporating academic papers in a software application, drawing our experiences with Elasticsearch and Lucene.

<p>While developing Elasticsearch, we occasionally come across an important problem with no simple or established approach to solving it. It's natural to ask “hmm, is there an academic paper that addresses this?” Other times, academic work is a source of inspiration. We'll encounter a paper proposing a new algorithm or data structure and think “this would be so useful!” Here are just a few examples of how Elasticsearch and Apache Lucene incorporate academic work:
</p><ul>
	<li dir="ltr" aria-level="1"><a href="https://research.google/pubs/pub40671/">HyperLogLog++</a> for <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.15/search-aggregations-metrics-cardinality-aggregation.html">cardinality aggregations</a></li>
	<li dir="ltr" aria-level="1"><a href="https://www.usenix.org/system/files/conference/nsdi15/nsdi15-paper-suresh.pdf">C3 algorithm</a> for <a href="https://www.elastic.co/blog/improving-response-latency-in-elasticsearch-with-adaptive-replica-selection">adaptive replica selection</a></li>
	<li dir="ltr" aria-level="1"><a href="https://arxiv.org/abs/1603.09320">Hierarchical Navigable Small World Graphs (HNSW)</a> for nearest vector search in Lucene</li>
	<li dir="ltr" aria-level="1"><a href="https://jmlr.csail.mit.edu/papers/volume17/15-308/15-308.pdf">MIC statistic</a> to <a href="https://github.com/elastic/ml-cpp/pull/488">improve machine learning classification</a></li>
	<li dir="ltr" aria-level="1"><a href="http://engineering.nyu.edu/~suel/papers/bmw.pdf">Block-max WAND</a> for <a href="https://www.elastic.co/blog/faster-retrieval-of-top-hits-in-elasticsearch-with-block-max-wand">faster top-hits retrieval in Lucene</a></li>
	<li dir="ltr" aria-level="1">... and <a href="https://www.elastic.co/guide/en/elasticsearch///reference/7.15/query-dsl-combined-fields-query.html">many</a> <a href="https://github.com/elastic/elasticsearch/blob/b2a9328890b23e7ccf6c66a3b13d6d65e453a3dd/server/src/main/java/org/elasticsearch/search/sort/BucketedSort.java#L302-L323">more</a><span></span></li>
</ul><p>Academic papers are an invaluable resource for engineers developing data-intensive systems. But implementing them can be intimidating and error-prone — algorithm descriptions are often complex, with important practical details omitted. And testing is a real challenge: for example, how can we thoroughly test a machine learning algorithm whose output depends closely on the dataset?
</p><p>This post shares strategies for implementing academic papers in a software application. It draws on examples from Elasticsearch and Lucene in hopes of helping other engineers learn from our experiences. You might read these strategies and think “but this is just software development!” And that would indeed be true: as engineers we already have the right practices and tools, they just need to be adapted to a new challenge.
</p><h2>Evaluate the paper as you would a software dependency</h2><p>Adding a new software dependency requires careful evaluation: if the other package is incorrect, slow, or insecure, our project could be too. Before pulling in a dependency, developers make sure to evaluate its quality.
</p><p>The same applies to academic papers you're considering implementing. It may seem that because an algorithm was published in a paper, it must be correct and perform well. But even though it passed a review process, an academic paper can have issues. Maybe the correctness proof relies on assumptions that aren't realistic. Or perhaps the “experiments” section shows much better performance than the baseline, but this only holds on a specific dataset. Even if the paper is of great quality, its approach may not be a good fit for your project.
</p><p>When thinking about whether to take a “dependency” on an academic paper, it's helpful to ask the same questions we would of a software package:
</p><ul>
	<li dir="ltr" aria-level="1">Is the library widely-used and “battle tested”? → Have other packages implemented this paper, and has it worked well for them?</li>
	<li dir="ltr" aria-level="1">Are performance benchmarks available? Do these seem accurate and fair? → Does the paper include realistic experiments? Are they well designed?</li>
	<li dir="ltr" aria-level="1">Is a performance improvement big enough to justify the complexity? → Does the paper compare to a strong baseline approach? How much does it outperform this baseline?</li>
	<li dir="ltr" aria-level="1">Will the approach integrate well with our system? → Do the algorithm's assumptions and trade-offs fit our use case?</li>
</ul><p>Somehow when a software package publishes a performance comparison against their competitors, the package always comes out fastest! If a third party designed the benchmarks, they may be more balanced. The same phenomenon applies to academic papers. If an algorithm performs well not only in the original paper, but also appears in other papers as a strong baseline, then it is very likely to be solid.
</p><h2>Get creative with testing</h2><p>Algorithms from academic papers often have more sophisticated behavior than the types of algorithms we routinely encounter. Perhaps it's an approximation algorithm that trades off accuracy for better speed. Or maybe it's a machine learning method that takes in a large dataset, and produces (sometimes unexpected) outputs. How can we write tests for these algorithms if we can't characterize their behavior in a simple way?
</p><h3>Focus on invariants</h3><p>When designing unit tests, it's common to think in terms of examples: if we give the algorithm this example input, it should have that output. Unfortunately for most mathematical algorithms, example-based testing doesn't sufficiently cover their behavior.
</p><p>Let's consider the C3 algorithm, which Elasticsearch uses to figure out what node should handle a search request. It ranks each node using a nuanced formula that incorporates the node's previous service and response times, and its queue size. Testing a couple examples doesn't really verify we understood the formula correctly. It helps to step back and think about testing invariants: if service time increases, does the node's rank decrease? If the queue size is 0, is the rank determined by response time, as the paper claims?
</p><p>Focusing on invariants can help in a number of common cases:
</p><ul>
	<li dir="ltr" aria-level="1">Is the method supposed to be order-agnostic? If so, passing the input data in a different order should result in the same output.</li>
	<li dir="ltr" aria-level="1">Does some step in the algorithm produce class probabilities? If so, these probabilities should sum to 1.</li>
	<li dir="ltr" aria-level="1">Is the function symmetric around the origin? If so, flipping the sign of the input should simply flip the sign of the output.</li>
</ul><p>When we first implemented C3, we had a bug in the formula where we accidentally used the inverse of response time in place of response time. This meant slower nodes could be ranked higher! When fixing the issue, we <a href="https://github.com/elastic/elasticsearch/pull/70283">made sure to add invariant checks</a> to guard against future mistakes.
</p><h3>Compare to a reference implementation</h3><p>Alongside the paper, the authors hopefully published an implementation of the algorithm. (This is especially likely if the paper contains experiments, as many journals require authors to post code for reproducing the results.) You can test your approach against this reference implementation to make sure you haven't missed important details of the algorithm.
</p><p>While developing Lucene's HNSW implementation for nearest-neighbor search, we <a href="https://issues.apache.org/jira/browse/LUCENE-9937">tested against a reference library</a> by the paper's authors. We ran both Lucene and the library against the same dataset, comparing the accuracy of their results and the number of computations they performed. When these numbers match closely, we know that Lucene faithfully implements the algorithm.
</p><p>When incorporating an algorithm into a system, you often need to make modifications or extensions, like scaling it to multiple cores, or adding heuristics to improve performance. It's best to first implement a "vanilla" version, test it against the reference, then make incremental changes. That way you can be confident you've captured all the key parts before making customizations.
</p><h3>Duel against an existing algorithm</h3><p>The last section raises another idea for a test invariant: comparing the algorithm's output to a simpler and better-understood algorithm's output. As an example, consider the block-max WAND algorithm in Lucene, which speeds up document retrieval by skipping over documents that can't appear in the top results. It is difficult to describe exactly how block-max WAND should behave in every case, but we do know that applying it shouldn't change the top results! So our tests can generate several random search queries, then <a href="https://github.com/apache/lucene/blob/main/lucene/core/src/test/org/apache/lucene/search/TestWANDScorer.java#L669">run them both with and without the WAND optimization</a> and check that their results always match.
</p><p>An important aspect of these tests is that they <a href="https://www.elastic.co/blog/elasticsearch-testing-qa-increasing-coverage-randomizing-test-runs">generate random inputs</a> on which to run the comparison. This can help exercise cases you wouldn't have thought of, and surface unexpected issues. As an example, Lucene's randomized comparison test for BM25F scoring has helped<a href="https://issues.apache.org/jira/browse/LUCENE-10039"> catch bugs in subtle edge cases</a>. The idea of feeding an algorithm random inputs is closely related to the concept of<a href="https://en.wikipedia.org/wiki/Fuzzing"> fuzzing</a>, a common testing technique in computer security.
</p><p>Elasticsearch and Lucene frequently use this testing approach. If you see a test that mentions a "duel" between two algorithms (TestDuelingAnalyzers, testDuelTermsQuery...), then you know this strategy is in action.
</p><h2>Use the paper's terminology</h2><p>When another developer works with your code, they'll need to consult the paper to follow its details. The<a href="https://github.com/elastic/elasticsearch/blob/4f22f437ee50cacb94b37b457be1da0b8ba0e8ce/server/src/main/java/org/elasticsearch/search/aggregations/metrics/HyperLogLogPlusPlus.java#L24-L39"> comment on Elasticsearch's HyperLogLog++ implementation</a> says it well: “Trying to understand what this class does without having read the paper is considered adventurous.” This method comment also sets a good example. It includes a link to the academic paper, and highlights what modifications were made to the algorithm as it was originally described.
</p><p>Since developers will base their understanding of the code on the paper, it's helpful to use the exact same terminology. Since mathematical notation is terse, this can result in names that would not usually be considered “good style”, but are very clear in the context of the paper. Formulas from academic papers are one of the few times you'll encounter cryptic variable names in Elasticsearch like<a href="https://github.com/elastic/elasticsearch/blob/4f22f437ee50cacb94b37b457be1da0b8ba0e8ce/server/src/main/java/org/elasticsearch/node/ResponseCollectorService.java#L151"> rS and muBarSInverse</a>.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt83d9ad71c909c603/6154b0ac58e1f54c5cbc48f2/elastic-blog-academicpaper.jpg" data-sys-asset-uid="blt83d9ad71c909c603" alt="elastic-blog-academicpaper.jpg"><span id="docs-internal-guid-0dab3aab-7fff-4f7f-8b92-e1500e9e915e"><br><em>The author's recommended way of reading a paper: with a large coffee.</em></span></p><h2>You can email the author</h2><p>When working through a tough paper, you may spend hours puzzling over a formula, unsure if you're misunderstanding or if there's just a typo. If this were an open source project, you could ask a question on GitHub or StackOverflow. But where can you turn for an academic paper? The authors seem busy and might be annoyed by your emails.
</p><p>On the contrary, many academics love hearing that their ideas are being put into practice and are happy to answer questions over email. If you work on a product they're familiar with, they might even list the application on their website!
</p><p>There's also a growing trend for academics to discuss papers in the open, using many of the same tools from software development. If a paper has an accompanying software package, you might find answers to<a href="https://github.com/facebookresearch/faiss/issues/1928"> common questions on Github</a>. Stack Exchange communities like “Theoretical Computer Science” and “Cross Validated” also contain<a href="https://cstheory.stackexchange.com/questions/49296/problem-in-the-paper-stable-minimum-space-partitioning-in-linear-time"> detailed discussions about popular papers</a>. Some conferences have begun to publish all paper reviews online. These reviews contain<a href="https://openreview.net/forum?id=H1eA7AEtvS"> back-and-forth discussions</a> with the authors that can surface helpful insights about the approach.
</p><h2>To be continued</h2><p>This post focuses on the basics of choosing an academic paper and implementing it correctly, but doesn't cover all aspects of actually deploying the algorithm. For example, if the algorithm is just one component in a complex system, how do we ensure that changes to the component lead to end-to-end improvements? And what if integrating the algorithm requires substantial modifications or extensions that the original paper doesn't cover? These are important topics we hope to share more about in future posts.
</p>

blog-banner-education-books.png

blog-thumb-education-books.png

Implementing academic papers: lessons learned from Elasticsearch and Lucene

Learn how Elasticsearch engineer and Apache Lucene contributor Julie Tibshirani came to join the open source search community, and about its friendly and accepting culture.

<p>Open source software development can have a reputation for abrasive behavior. The search community is a clear counterexample for me, with a culture that emphasizes respect and acceptance. This culture played an important part in my own path to open source development.
</p><p>A little over six years ago, I was a wide-eyed software engineer settling into my first full-time job. My company made data analysis software for governments and large organizations, and I had just joined the team that built and maintained the search infrastructure.
</p><p>Until then, when I thought of ‘search’, what came to mind was a web search engine or online shopping site. It soon became clear that search was a far more general and fundamental problem — our users needed to find the relevant parts of their data, and find them quickly. Search formed a core component of each organization’s critical work, which included anti-fraud investigations, combating human trafficking, tracking epidemics, and more.
</p><p>Search technology brings together work from often disparate fields: systems and databases, linguistics, and even mathematical fields like statistics and information theory. I found it so engaging to be able to work across these different areas —&nbsp;in a given week I might be designing a data schema, pondering the rules of plural nouns, or debugging why certain queries had high tail latencies. Needless to say, it didn’t take long for me to fall in love with search.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt2d56347a807d1cea/5e8cba1575c4742aa0e609dc/Google_search_sign.jpg" data-sys-asset-uid="blt2d56347a807d1cea" alt="Google_search_sign.jpg" style="display: block; margin: auto;">
</p><figcaption>Seen outside a favorite bookstore in Mountain View.</figcaption><p>Our search infrastructure was built on the open source search library Apache Lucene, and we were working on writing a&nbsp;new search service&nbsp;using&nbsp;Elasticsearch (which is also based on Lucene under the hood). Using an open source library made it possible to deliver features to our users that would have taken years for our tiny search team to develop ourselves. In addition to being packed full of capabilities, Lucene was so fast and seemed to constantly be getting faster. During my studies, I took a deep interest in data structures and foundational algorithms like prefix trees, skip lists, and sorting techniques. In Lucene I saw these topics come to life in code and play a role in powering hundreds of critical applications.
</p><p>A few months into my new job, I was scratching my head at why a new search highlighting feature we were building wasn’t working as expected. In debugging the problem, I discovered an issue in Elasticsearch where <a href="https://github.com/elastic/elasticsearch/issues/6420">nested fields weren’t highlighted</a>. I continued to dig deeper and <a href="https://issues.apache.org/jira/browse/LUCENE-5929">traced the problem to Lucene</a>. I had a sense for how to fix the issue, and it seemed like a good opportunity to contribute to a project that had benefited my team so much.
</p><p>Although it was a small change, I was quite nervous once it came time to submit the patch. One of the "Mythical Lucene Developers" would be reviewing it, those search experts who built this wonderful&nbsp;package. And the code review would be open for the whole internet to see — in other open source projects I had seen examples of developers being <a href="https://marc.info/?l=linux-kernel&m=135628421403144&w=2">publicly berated</a> for a bad patch. From my experience in university and my first job, I had developed a working picture of an accomplished software engineer: they were ‘prickly’ and forceful with their opinions, and were always self-assured, sometimes to the point of overconfidence. I assumed these talented open source developers would fit this mold.
</p><p>I closed my eyes and hit ‘submit’, and after a friendly and straightforward review, the patch was merged! Encouraged by this experience, I made some <a href="https://github.com/elastic/elasticsearch/pull/10359">small</a> <a href="https://github.com/elastic/elasticsearch/issues/10356">contributions</a> to Elasticsearch in my rare free time at work. A couple years after my first open source contributions, I started <a href="https://github.com/grf-labs/grf">my own open source project</a> and became the one giving friendly and straightforward reviews.
</p><p>I now work for Elastic as a core developer on Elasticsearch, where I contribute full-time to these projects I once admired from afar. As I’ve spent more time in the open source search community, I’ve seen that these initial helpful reviews are part of a broader pattern. By and large, the developers I interact with treat each other with respect and kindness. Despite their considerable accomplishments, most contributors don’t have ‘big heads’ — the ratio of ego to talent remains surprisingly low. Of course there are arguments and heated discussions, but engineers seem motivated by what they think is technically ‘right’, by what they feel is best for the project.
</p><p>In previous jobs, design discussions were always centered around a whiteboard. A group of engineers would huddle in a room, sometimes for hours, and debate the details of a new feature or architecture. Although this process was engaging, I preferred having the time to think deeply about the problem on my own, and organize my thoughts before discussing with others. As a stereotypically polite Canadian, I sometimes found it difficult to seize the floor in these lively discussions filled with interruptions.
</p><p>In open source development, it’s much more common to rely on written, asynchronous communication. I appreciate having the space to carefully ponder a problem, then draft up an email or GitHub post laying out my thoughts. Even the most soft-spoken engineers are able to weigh in and bring their opinions to the discussion.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt492850bd1d24272d/5e8e2fcbbe1a31776ec326de/polite_canadians.jpg" data-sys-asset-uid="blt492850bd1d24272d" alt="polite_canadians.jpg" style="display: block; margin: auto;"></p><figcaption>After you! No, after you. Credit: Enzo Comics<br></figcaption><p>The open source search community spans many cultures and languages, which contributes to there being a diversity of communication styles. Contributors tend to be quite flexible when someone has a style that doesn’t match their own. I found that I could ‘speak Canadian’ and still be understood, like when I deliver important but gentle feedback in a code review, or apologize in the middle of a bug report.
</p><p>Lucene turned 20 years old this year, which is a big accomplishment for such a widely-used open source project. In terms of both its technology and its people, the open source search community is one to admire. I am so happy to be part of it, and hope it continues to grow in the years to come.
</p><p><em>Interested in joining a company with <a href="https://www.elastic.co/about/our-source-code?baymax=web&elektra=culture-finding-a-home-and-career-in-the-open-source-community">a Source Code</a> to live by? We’re hiring. Check out <a href="https://www.elastic.co/about/teams?baymax=web&elektra=culture-finding-a-home-and-career-in-the-open-source-community">our teams</a> and <a href="https://www.elastic.co/about/careers?baymax=web&elektra=culture-finding-a-home-and-career-in-the-open-source-community">find your fit!</a> Want to read more about life at Elastic? Read more <a href="https://www.elastic.co/blog/category/culture/?baymax=web&elektra=culture-finding-a-home-and-career-in-the-open-source-community">on our blog!</a></em>
</p>

Elastic_SeriousAnalytics_Hero.jpg

Elastic_SeriousAnalytics_Thumbnail_.jpg

Finding a home (and career) in the open source community

This post explores how text embeddings and Elasticsearch’s new dense_vector type could be used to support similarity search.

<p>From its beginnings as a <a href="/about/history-of-elasticsearch">recipe search engine</a>, Elasticsearch was designed to provide fast and powerful full-text search. Given these roots, improving text search has been an important motivation for our ongoing work with vectors. In Elasticsearch 7.0, we introduced experimental field types for high-dimensional vectors, and now the 7.3 release brings support for using these vectors in document scoring.
</p><p>This post focuses on a particular technique called text similarity search. In this type of search, a user enters a short free-text query, and documents are ranked based on their similarity to the query. Text similarity can be useful in a variety of use cases:
</p><ul>
	<li><strong>Question-answering:</strong> Given a collection of frequently asked questions, find questions that are similar to the one the user has entered.</li>
	<li><strong>Article search:</strong> In a collection of research articles, return articles with a title that’s closely related to the user’s query.</li>
	<li><strong>Image search:</strong> In a dataset of captioned images, find images whose caption is similar to the user’s description.</li>
</ul><p>A straightforward approach to similarity search would be to rank documents based on how many words they share with the query. But a document may be similar to the query even if they have very few words in common — a more robust notion of similarity would take into account its syntactic and <a href="https://en.wikipedia.org/wiki/Semantic_similarity">semantic</a> content as well.
</p><p>The natural language processing (NLP) community has developed a technique called text embedding that encodes words and sentences as numeric vectors. These vector representations are designed to capture the linguistic content of the text, and can be used to assess similarity between a query and a document.
</p><p>This post explores how text embeddings and Elasticsearch’s <code>dense_vector</code> type could be used to support similarity search. We’ll first give an overview of embedding techniques, then step through a simple prototype of similarity search using Elasticsearch.
</p><table style="background: #FFFFD2;">
<tbody>
<tr>
	<td><strong>Note:</strong> Using text embeddings in search is a <em>complex and evolving</em> area. We hope this blog post gives a jumping off point for exploration, but it is not a recommendation for a particular search architecture or implementation.
	</td>
</tr>
</tbody>
</table><h2>What are text embeddings?</h2><p>Let's take a closer look at different types of text embeddings, and how they compare to traditional search approaches.
</p><h3>Word embeddings</h3><p>A <a href="https://en.wikipedia.org/wiki/Word_embedding">word embedding</a> model represents a word as a dense numeric vector. These vectors aim to capture semantic properties of the word — words whose vectors are close together should be similar in terms of semantic meaning. In a good embedding, directions in the vector space are tied to different aspects of the word’s meaning. As an example, the vector for "Canada" might be close to "France" in one direction, and close to "Toronto" in another.
</p><p>The NLP and search communities have been interested in vector representations of words for quite some time. There was a resurgence of interest in word embeddings in the past few years, when many traditional tasks were being revisited using neural networks. Some successful word embedding algorithms were developed, including <a href="https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf">word2vec</a> and <a href="https://nlp.stanford.edu/pubs/glove.pdf">GloVe</a>. These approaches make use of large text collections, and examine the context each word appears in to determine its vector representation:
</p><ul>
	<li>The word2vec Skip-gram model trains a neural network to predict the context words around a word in a sentence. The internal weights of the network give the word embeddings.</li>
	<li>In GloVe, the similarity of words depends on how frequently they appear with other context words. The algorithm trains a simple linear model on word co-occurrence counts.</li>
</ul><p>Many research groups distribute models that have been pre-trained on large text corpora like Wikipedia or Common Crawl, making them convenient to download and plug into downstream tasks. Although pre-trained versions are sometimes used directly, it can be helpful to adjust the model to fit the specific target dataset and task. This is often accomplished by running a 'fine-tuning' step on the pre-trained model.
</p><p>Word embeddings have proven quite robust and effective, and it is now common practice to use embeddings in place of individual tokens in NLP tasks like machine translation and sentiment classification.
</p><h3>Sentence embeddings</h3><p>More recently, researchers have started to focus on embedding techniques that represent not only words, but longer sections of text. Most current approaches are based on complex neural network architectures, and sometimes incorporate labelled data during training to aid in capturing semantic information.
</p><p>Once trained, the models are able to take a sentence and produce a vector for each word in context, as well as a vector for the entire sentence. Similarly to word embedding, pre-trained versions of many models are available, allowing users to skip the expensive training process. While the training process can be very resource-intensive, invoking the model is much more lightweight — sentence embedding models are typically fast enough to be used as part of real-time applications.
</p><p>Some common sentence embedding techniques include <a href="https://arxiv.org/abs/1705.02364">InferSent</a>, <a href="https://arxiv.org/abs/1803.11175">Universal Sentence Encoder</a>, <a href="https://arxiv.org/abs/1802.05365">ELMo</a>, and <a href="https://arxiv.org/abs/1810.04805">BERT</a>. Improving word and sentence embeddings is an active area of research, and it’s likely that additional strong models will be introduced.
</p><h3>Comparison to traditional search approaches</h3><p>In traditional information retrieval, a common way to represent text as a numeric vector is to assign one dimension for each word in the vocabulary. The vector for a piece of text is then based on the number of times each term in the vocabulary appears. This way of representing text is often referred to as "bag of words," because we simply count word occurrences without regard to sentence structure.
</p><p>Text embeddings differ from traditional vector representations in some important ways:
</p><ul>
	<li>The encoded vectors are dense and relatively low-dimensional, often ranging from 100 to 1,000 dimensions. In contrast, bag of words vectors are sparse and can comprise 50,000+ dimensions. Embedding algorithms encode the text into a lower-dimensional space as part of modeling its semantic meaning. Ideally, synonymous words and phrases end up with a similar representation in the new vector space.</li>
	<li>Sentence embeddings can take the order of words into account when determining the vector representation. For example the phrase "tune in" may be mapped as a very different vector than "in tune".</li>
	<li>In practice, sentence embeddings often don’t generalize well to large sections of text. They are not commonly used to represent text longer than a short paragraph.</li>
</ul><h2>Using embeddings for similarity search</h2><p>Let’s suppose we had a large collection of questions and answers. A user can ask a question, and we want to retrieve the most similar question in our collection to help them find an answer.
</p><p>We could use text embeddings to allow for retrieving similar questions:
</p><ul>
	<li>During indexing, each question is run through a sentence embedding model to produce a numeric vector.</li>
	<li>When a user enters a query, it is run through the same sentence embedding model to produce a vector. To rank the responses, we calculate the vector similarity between each question and the query vector. When comparing embedding vectors, it is common to use <a href="https://en.wikipedia.org/wiki/Cosine_similarity">cosine similarity</a>.</li>
</ul><p><a href="https://github.com/jtibshirani/text-embeddings">This repository</a> gives a simple example of how this could be accomplished in Elasticsearch. The main script indexes ~20,000 questions from the <a href="https://github.com/elastic/rally-tracks/tree/master/so">StackOverflow dataset</a>, then allows the user to enter free-text queries against the dataset.
</p><p>We’ll soon walk through each part of the script in detail, but first let’s look at some example results. In many cases, the method is able to capture similarity even when there was not strong word overlap between the query and indexed question:
</p><ul>
	<li>"zipping up files" returns "Compressing / Decompressing Folders & Files"</li>
	<li>"determine if something is an IP" returns "How do you tell whether a string is an IP or a hostname"</li>
	<li>"translate bytes to doubles" returns "Convert Bytes to Floating Point Numbers in Python"</li>
</ul><h3>Implementation details</h3><p><a href="https://github.com/jtibshirani/text-embeddings/blob/blog/src/main.py">The script</a> begins by downloading and creating the embedding model in TensorFlow. We chose Google’s Universal Sentence Encoder, but it’s possible to use many other embedding methods. The script uses the embedding model as-is, without any additional training or fine-tuning.
</p><p>Next, we create the Elasticsearch index, which includes mappings for the question title, tags, and also the question title encoded as a vector:
</p><pre class="prettyprint">"mappings": {
  "properties": {
    "title": {
      "type": "text"
    },
    "title_vector": {
      "type": "dense_vector",
      "dims": 512
    }
    "tags": {
      "type": "keyword"
    },
    ...
  }
}
</pre><p>In the mapping for <code>dense_vector</code>, we’re required to specify the number of dimensions the vectors will contain. When indexing a <code>title_vector</code> field, Elasticsearch will check that it has the same number of dimensions as specified in the mapping.
</p><p>To index documents, we run the question title through the embedding model to obtain a numeric array. This array is added to the document in the <code>title_vector</code> field.
</p><p>When a user enters a query, the text is first run through the same embedding model and stored in the parameter <code>query_vector</code>. As of 7.3, Elasticsearch provides a <a href="/guide/en/elasticsearch/reference/7.6/query-dsl-script-score-query.html#vector-functions"><code>cosineSimilarity</code> function</a> in its native scripting language. So to rank questions based on their similarity to the user’s query, we use a <code>script_score</code> query:
</p><pre class="prettyprint">{
  "script_score": {
    "query": {"match_all": {}},
    "script": {
      "source": "cosineSimilarity(params.query_vector, 'title_vector') + 1.0",
      "params": {"query_vector": query_vector}
    }
  }
}
</pre><p>We make sure to pass the query vector as a script parameter to <a href="/guide/en/elasticsearch/reference/7.6/modules-scripting-using.html#prefer-params">avoid recompiling the script</a>() on every new query. Since Elasticsearch does not allow negative scores, it's necessary to add one to the cosine similarity.
</p><table style="background: #FFFFD2;">
<tbody>
<tr>
	<td><strong>Note:</strong> this blog post originally used a <a href="/guide/en/elasticsearch/reference/7.3/query-dsl-script-score-query.html#vector-functions">different syntax for vector functions</a> that was available in Elasticsearch 7.3, but was deprecated in 7.6.
	</td>
</tr>
</tbody>
</table><h3>Important limitations</h3><p>The <code>script_score</code> query is designed to wrap a restrictive query, and modify the scores of the documents it returns. However, we’ve provided a <code>match_all</code> query, which means the script will be run over all documents in the index. This is a current limitation of vector similarity in Elasticsearch — vectors can be used for scoring documents, but not in the initial retrieval step. Support for retrieval based on vector similarity is an important area of <a href="https://github.com/elastic/elasticsearch/issues/42326">ongoing work</a>.
</p><p>To avoid scanning over all documents and to maintain fast performance, the <code>match_all</code> query can be replaced with a more selective query. The right query to use for retrieval is likely to depend on the specific use case.
</p><p>While we saw some encouraging examples above, it’s important to note that the results can also be noisy and unintuitive. For example, "zipping up files" also assigns high scores to "Partial .csproj Files" and "How to avoid .pyc files?". And when the method returns surprising results, it is not always clear how to debug the issue — the meaning of each vector component is often opaque and doesn’t correspond to an interpretable concept. With traditional scoring techniques based on word overlap, it is often easier to answer the question "why is this document ranked highly?"
</p><p>As mentioned earlier, this prototype is meant as an example of how embedding models could be used with vector fields, and not as a production-ready solution. When developing a new search strategy, it is critical to test how the approach performs on your own data, making sure to compare against a strong baseline like a <code>match</code> query. It may be necessary to make major changes to the strategy before it achieves solid results, including fine-tuning the embedding model for the target dataset, or trying different ways of incorporating embeddings such as word-level query expansion.
</p><h2>Conclusions</h2><p>Embedding techniques provide a powerful way to capture the linguistic content of a piece of text. By indexing embeddings and scoring based on vector distance, we can compare documents using a notion of similarity that goes beyond their word-level overlap.
</p><p>We’re looking forward to introducing more functionality based around the vector field type. Using vectors for search is a nuanced and developing area — as always, we would love to hear about your use cases and experiences on <a href="https://github.com/elastic/elasticsearch">Github</a> and the <a href="https://discuss.elastic.co/">Discuss forums</a>!
</p>

books.jpg

books_small.jpg

Text similarity search with vector fields

Learn about the new 'alias' datatype in Elasticsearch, including how it helps with field renaming and multi-index search.

earth-from-international-space-station.jpg

international-space-station.jpg

Introducing Field Aliases in Elasticsearch

This Week in Elasticsearch and Apache Lucene - 2019-05-05

Get started

pattern-footer-cta-generic-bottom-left-172x90-blue-bg.svg

pattern-footer-cta-generic-bottom-right-145x250-blue-bg.svg

pattern-footer-cta-generic-top-left-145x250-blue-bg.svg

pattern-footer-cta-generic-top-right-172x90-blue-bg.svg

Generic - Blue - Color

Blog Footer CTA

Sign up for Elastic Cloud free trial

Blog Footer

Julie Tibshirani

Articles By

Register to Attend

Sign In to Attend

Register to Watch

You'll also receive an email with related content

Watch Now

Learn More

Contact Info

See All Posts

What to explore next...

Load more news

Load more press releases

Load More

View more learning opportunities

More

Date

Location

Agenda

Hosted by

Related workshops

See when this webinar starts in my time zone

Highlights

Featured webinar

On-demand webinar

Upcoming webinar

View next

Register now

Explore similar demos

Video for

Small image for

Share

Continue reading

Share by email

Print

View more posts

Global Virtual Event

Translate Content

AUTHOR

Articles by Julie Tibshirani

Implementing academic papers: lessons learned from Elasticsearch and Lucene

Finding a home (and career) in the open source community

Text similarity search with vector fields

Introducing Field Aliases in Elasticsearch

This Week in Elasticsearch and Apache Lucene - 2019-05-05

Sign up for Elastic Cloud free trial

Products & Solutions

Company

We're hiring

Resources

Solutions

Elastic (ELK) Stack

New

Featured topics

News

AUTHOR

Articles by Julie Tibshirani

Implementing academic papers: lessons learned from Elasticsearch and Lucene

Finding a home (and career) in the open source community

Text similarity search with vector fields

Introducing Field Aliases in Elasticsearch

This Week in Elasticsearch and Apache Lucene - 2019-05-05

Sign up for Elastic Cloud free trial

We're hiring