Mixing exact search with stemmingedit

When building a search application, stemming is often a must as it is desirable for a query on skiing to match documents that contain ski or skis. But what if a user wants to search for skiing specifically? The typical way to do this would be to use a multi-field in order to have the same content indexed in two different ways:

PUT index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "english_exact": {
          "tokenizer": "standard",
          "filter": [
            "lowercase"
          ]
        }
      }
    }
  },
  "mappings": {
    "type": {
      "properties": {
        "body": {
          "type": "text",
          "analyzer": "english",
          "fields": {
            "exact": {
              "type": "text",
              "analyzer": "english_exact"
            }
          }
        }
      }
    }
  }
}

PUT index/type/1
{
  "body": "Ski resort"
}

PUT index/type/2
{
  "body": "A pair of skis"
}

POST index/_refresh

With such a setup, searching for ski on body would return both documents:

GET index/_search
{
  "query": {
    "simple_query_string": {
      "fields": [ "body" ],
      "query": "ski"
    }
  }
}

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 2,
    "max_score": 0.2876821,
    "hits": [
      {
        "_index": "index",
        "_type": "type",
        "_id": "2",
        "_score": 0.2876821,
        "_source": {
          "body": "A pair of skis"
        }
      },
      {
        "_index": "index",
        "_type": "type",
        "_id": "1",
        "_score": 0.2876821,
        "_source": {
          "body": "Ski resort"
        }
      }
    ]
  }
}

On the other hand, searching for ski on body.exact would only return document 1 since the analysis chain of body.exact does not perform stemming.

GET index/_search
{
  "query": {
    "simple_query_string": {
      "fields": [ "body.exact" ],
      "query": "ski"
    }
  }
}

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 0.2876821,
    "hits": [
      {
        "_index": "index",
        "_type": "type",
        "_id": "1",
        "_score": 0.2876821,
        "_source": {
          "body": "Ski resort"
        }
      }
    ]
  }
}

This is not something that is easy to expose to end users, as we would need to have a way to figure out whether they are looking for an exact match or not and redirect to the appropriate field accordingly. Also what to do if only parts of the query need to be matched exactly while other parts should still take stemming into account?

Fortunately, the query_string and simple_query_string queries have a feature that solve this exact problem: quote_field_suffix. This tell Elasticsearch that the words that appear in between quotes are to be redirected to a different field, see below:

GET index/_search
{
  "query": {
    "simple_query_string": {
      "fields": [ "body" ],
      "quote_field_suffix": ".exact",
      "query": "\"ski\""
    }
  }
}

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 0.2876821,
    "hits": [
      {
        "_index": "index",
        "_type": "type",
        "_id": "1",
        "_score": 0.2876821,
        "_source": {
          "body": "Ski resort"
        }
      }
    ]
  }
}

In the above case, since ski was in-between quotes, it was searched on the body.exact field due to the quote_field_suffix parameter, so only document 1 matched. This allows users to mix exact search with stemmed search as they like.

Getting consistent scoringedit

The fact that Elasticsearch operates with shards and replicas adds challenges when it comes to having good scoring.

Scores are not reproducibleedit

Say the same user runs the same request twice in a row and documents do not come back in the same order both times, this is a pretty bad experience isn’t it? Unfortunately this is something that can happen if you have replicas (index.number_of_replicas is greater than 0). The reason is that Elasticsearch selects the shards that the query should go to in a round-robin fashion, so it is quite likely if you run the same query twice in a row that it will go to different copies of the same shard.

Now why is it a problem? Index statistics are an important part of the score. And these index statistics may be different across copies of the same shard due to deleted documents. As you may know when documents are deleted or updated, the old document is not immediately removed from the index, it is just marked as deleted and it will only be removed from disk on the next time that the segment this old document belongs to is merged. However for practical reasons, those deleted documents are taken into account for index statistics. So imagine that the primary shard just finished a large merge that removed lots of deleted documents, then it might have index statistics that are sufficiently different from the replica (which still have plenty of deleted documents) so that scores are different too.

The recommended way to work around this issue is to use a string that identifies the user that is logged is (a user id or session id for instance) as a preference. This ensures that all queries of a given user are always going to hit the same shards, so scores remain more consistent across queries.

This work around has another benefit: when two documents have the same score, they will be sorted by their internal Lucene doc id (which is unrelated to the _id or _uid) by default. However these doc ids could be different across copies of the same shard. So by always hitting the same shard, we would get more consistent ordering of documents that have the same scores.

Relevancy looks wrongedit

If you notice that two documents with the same content get different scores or that an exact match is not ranked first, then the issue might be related to sharding. By default, Elasticsearch makes each shard responsible for producing its own scores. However since index statistics are an important contributor to the scores, this only works well if shards have similar index statistics. The assumption is that since documents are routed evenly to shards by default, then index statistics should be very similar and scoring would work as expected. However in the event that you either - use routing at index time, - query multiple indices, - or have too little data in your index then there are good chances that all shards that are involved in the search request do not have similar index statistics and relevancy could be bad.

If you have a small dataset, the easiest way to work around this issue is to index everything into an index that has a single shard (index.number_of_shards: 1). Then index statistics will be the same for all documents and scores will be consistent.

Otherwise the recommended way to work around this issue is to use the dfs_query_then_fetch search type. This will make Elasticsearch perform an inital round trip to all involved shards, asking them for their index statistics relatively to the query, then the coordinating node will merge those statistics and send the merged statistics alongside the request when asking shards to perform the query phase, so that shards can use these global statistics rather than their own statistics in order to do the scoring.

In most cases, this additional round trip should be very cheap. However in the event that your query contains a very large number of fields/terms or fuzzy queries, beware that gathering statistics alone might not be cheap since all terms have to be looked up in the terms dictionaries in order to look up statistics.