How to build scoped search suggestions and search query corrections

By

Lionel Palacin

blog-search-results-dark-720x420.png

You get one shot to keep shoppers on your ecommerce website with relevant search results. According to Harris Poll, 76% of online shoppers abandon a retail website after an unsuccessful search.

Therefore, it’s critical to optimize your search experience so buyers can find what they need fast. That’s the theory behind a modern search experience: Nowadays it’s not enough to simply provide a search bar that returns matching products. Your ecommerce site must include a complete set of search tools that personally guide users to the products they want.

In this blog, you’ll learn how to start improving ecommerce search by adding two common features: scoped suggestions and query corrections, also known as “Did you mean?”

Note the following examples have been tested using Elastic 8.5. 

Scoped suggestions 

One common challenge among ecommerce retailers is large product catalogs with many different categories. In this scenario, it’s difficult to parse a simple query to understand what domain or category a user might be interested in. For example, if a user searches for “flat screen” on an electronic retail store, they may see results under TVs, TV stands, computer monitors, and so on. Presenting flat screen TV stands while the user is looking for a TV might not be relevant at this stage in their buyer journey.

Scoped suggestions help your users narrow down their searches on topics they are most interested in by suggesting queries within specific categories or brands.

This is a powerful feature to help users see a refined result set quickly. Implementing scoped search suggestions using Search UI and Elastic is simple, too. The next section walks you through how to build it. 

Gather data for suggestions

As described in the example above, as users type, they’ll be presented with suggested queries and associated scopes. To build those suggestions, it’s a common practice to use analytics data to gather the most common queries and scopes they’re commonly associated with by customers.

For example, let’s say that upon reviewing the analytics data for our ecommerce website, we found that the top query is “lcd tv” and products that users click on most often are found in the category “TV & Home Theater.”

Using this data we can build a document that represents a corresponding scoped suggestion, like this: 

{
   "name": "lcd tv",
   "weight": 112,
   "category": {
       "name": "TV & Home Theater",
       "value": 111
   }
}

This document contains a query and an associated category. This can be made more complex  if required; for example, we can have a list of categories or add another scope, like “brands.”

We then create a dedicated index in Elasticsearch, in our case we name it “suggest”, to store a list of scoped suggestions that we built using the analytics data. This can be done using data transforms or using custom code, such as a Python script. The index mapping example for our index “suggest”  can be found here. 

Autocomplete suggestions

Now that we have our list of suggestions ready to be consumed, we need to add it to our search experience.

Using Search UI, it takes only a few minutes to build a search experience on top of Elastic. We can then use our newly built interface as a basis for our scoped suggestions.

Search UI uses a configuration object to tailor search to your needs. Below is a snippet of the configuration object that corresponds to the suggestion configuration. In this case, we specify the index and fields to query and the fields to return as part of the results:

suggestions: {
     types: {
       popularQueries: {
         search_fields: {
           "name.suggest": {} // fields used to query
         },
         result_fields: {
           name: {
             raw: {}
           },
           "category.name": {
             raw: {}
           }
         },
         index: "suggest",
         queryType: "results"
       }
     },
     size: 5
   }

Then we configure the SearchBox component to pass the category as part of the query when navigating to the search result page:

// Search bar component
<SearchBox
   onSelectAutocomplete={(suggestion, config, defaultHandler) => {
       // User selects a scoped suggestion - Category
       if (suggestion.name && suggestion.category) {
           const params = { q: suggestion.name.raw, category: suggestion.category.raw.name };
           // Navigate to search result page with category passed as parameters
           navigate({
               pathname: '/search',
               search: `?${createSearchParams(params)}`,
           });
       // User selects normal suggestion
       } else if (suggestion) {
           // Navigate to search result page
           window.location.href = "/search?q=" + suggestion.suggestion;
       }
       defaultHandler(suggestion);
   }}
   autocompleteSuggestions={{
       popularQueries: {
           sectionTitle: "Popular queries",
           queryType: "results",
           displayField: "name",
           categoryField: "category"
       }
   }}
   autocompleteView={AutocompleteView}
/>

Note that we pass an AutocompleteView function to customize the autocomplete view so we can display the category alongside the suggested query. 

Below is a code snippet of the AutocompleteView function that shows how to display a query with the associated scope: 

{suggestions.slice(0,1).map((suggestion) => {
   index++;
   const suggestionValue = getDisplayField(suggestion)
   const suggestionScope = getCategoryField(suggestion)
   return (
     <li
       {...getItemProps({
         key: suggestionValue,
         index: index - 1,
         item: {
           suggestion: suggestionValue,
           ...suggestion.result
         }
       })}
     >
       <span>{suggestionValue}</span>
       <ul><span style={{marginLeft: "20px"}}>in {suggestionScope}</span></ul>
     </li>
   );
})}

Then on the search result page, we simply need to process the query parameters and filter the result set: 

 const [searchParams] = useSearchParams();
   useEffect(() => {
       if (searchParams.get('category')) addFilter("department", [searchParams.get('category')], "all")
   }, [searchParams]);

Once we put everything together, the result looks like this:

Video thumbnail

Query corrections, aka “Did you mean?”

Shoppers can get frustrated when they miss out on seeing relevant search results due to a poor choice of query terms or a typo. To avoid returning no search results or very few, it’s a best practice to suggest a better query, commonly referred to as “Did you mean?”

There are many different ways of implementing this feature, but this is the most straightforward way to build it using Search UI and Elastic. 

Analyzing the data

The dataset we’ll use to build the “Did you mean?” feature is similar to the one we used for scoped suggestions, so you can suggest a popular query to the users in case their query didn’t return results.  

Use a similar document structure for “Did you mean?”

{
   "name": "lcd tv",
   "weight": 112,
   "category": {
       "name": "TV & Home Theater",
       "value": 111
   }
}

The key to this step is the way data is indexed in Elasticsearch. To provide relevant suggestions, you’ll need to use specific features offered by Elasticsearch such as custom analyzers.

Below are the index settings and mappings used in our example: 

{
    "settings":
    {
        "index":
        {
            "number_of_shards": 1,
            "analysis":
            {
                "analyzer":
                {
                    "trigram":
                    {
                        "type": "custom",
                        "tokenizer": "standard",
                        "filter":
                        [
                            "lowercase",
                            "shingle"
                        ]
                    },
                    "reverse":
                    {
                        "type": "custom",
                        "tokenizer": "standard",
                        "filter":
                        [
                            "lowercase",
                            "reverse"
                        ]
                    }
                },
                "filter":
                {
                    "shingle":
                    {
                        "type": "shingle",
                        "min_shingle_size": 2,
                        "max_shingle_size": 3
                    }
                }
            }
        }
    },
    "mappings":
    {
        "properties":
        {
            "category":
            {
                "properties":
                {
                    "name":
                    {
                        "type": "text",
                        "fields":
                        {
                            "keyword":
                            {
                                "type": "keyword",
                                "ignore_above": 256
                            }
                        }
                    },
                    "value":
                    {
                        "type": "long"
                    }
                }
            },
            "name":
            {
                "type": "search_as_you_type",
                "doc_values": "false",
                "max_shingle_size": 3,
                "fields":
                {
                    "reverse":
                    {
                        "type": "text",
                        "analyzer": "reverse"
                    },
                    "suggest":
                    {
                        "type": "text",
                        "analyzer": "trigram"
                    }
                }
            },
            "weight":
            {
                "type": "rank_feature",
                "fields":
                {
                    "numeric":
                    {
                        "type": "integer"
                    }
                }
            }
        }
    }
}

Refer to the suggesters documentation for more depth on why to use specific settings and mappings to retrieve relevant suggestions. 

Preparing the search

Now that your index is ready, you can work on the search experience. To retrieve suggestions based on a query, leverage the Elasticsearch API. If you don’t want to add too much complexity on the client side, create a search template in Elasticsearch that will contain the query. Then from the client side, you just have to execute that search template:

PUT _scripts/did-you-mean-template
{
   "script": {
       "lang": "mustache",
       "source": {
           "suggest": {
               "text": "{{query_string}}",
               "simple_phrase": {
                   "phrase": {
                       "field": "name.suggest",
                       "size": 1,
                       "direct_generator": [
                           {
                               "field": "name.suggest",
                               "suggest_mode": "always"
                           },
                           {
                               "field": "name.reverse",
                               "suggest_mode": "always",
                               "pre_filter": "reverse",
                               "post_filter": "reverse"
                           }
                       ]
                   }
               }
           }
       }
   }
}

Getting suggestions from the application

In order to get suggestions from Elasticsearch, we’re adding a backend API that the client can consume on /api/suggest. This new API calls Elasticsearch, executes the search template, and passes along the query you want suggestions for, then returns a query suggestion. The backend API allows us to reduce complexity on the front end part.

client.searchTemplate({
       index: "suggest",
       id: 'did-you-mean-template',
       params: {
           query_string: req.body.query
       }
   })

For example, if the query is “spakers,” the API will return the suggestion “speakers.” This is based on the data previously ingested that contains the most popular queries. The suggest API will return a term that is the closest syntactically to the query. 

Adding “Did you mean?” to the result page

Now we can add the “Did you mean?” feature on the front end application. For that we’re going to continue working on the same React application we used for the scoped suggestions part. 

The new API we added can be consumed from the React application and display a suggestion in case there are no results for the current query. 

The idea here is to get a suggestion for each query entered by the user. If there is no result, then the user will see a suggestion. Alternative implementations are also possible: You can display suggestions in case there are few results, or you can execute the suggestion automatically in place of a user’s query.

In our application, we have a React component named SearchResults that displays our search results. There we can add a function that fetches the suggestion from our backend API /api/suggest.

const fetchDidYouMeanSuggestion = async (query) => {
   const response = await fetch('/api/did_you_mean', {
       method: 'POST',
       headers: {
           Accept: 'application/json, text/plain, */*',
           'Content-Type': 'application/json',
       },
       body: JSON.stringify(query)
   })
   const body = await response.json();
   if (response.status !== 200) throw Error(body.message);
   return body;
}

Then as the query changes, according to user refinement, you can refresh the suggestion by calling the API:

   // Get search params from dom router
   const [searchParams] = useSearchParams();
 
   useEffect(() => {
       // When the searchParams contains a query
       if (searchParams.get('q')) {
           // Set query for Search UI - Run the search
           setSearchTerm(searchParams.get('q'))
           // Fetch suggestion from backend API
           fetchDidYouMeanSuggestion({ query: searchParams.get('q') }).then(res => {
               setSuggestion(res.body?.suggest?.simple_phrase[0]?.options[0]?.text)
           })
               .catch(err => console.log(err));
       }
  }, [searchParams]);

Finally, display the suggestion in case there is no result for the user query:

{wasSearched && totalResults == 0 && <span>No results to show{suggestion ? <>, did you mean <span style={{ cursor: "pointer", color: "blue" }} onClick={() => navigateSuggest(suggestion)}>{suggestion}</span>?</> : "."}</span>}

The final result looks like this:

Video thumbnail

Conclusion

In this blog post, you learned how to easily build “Did you mean?” and scoped suggestions features using Elastic Enterprise Search. These features can be easily integrated into your existing search experience so that your users can find what they are looking for faster. If you want to review the code used to build these examples, find it in this GitHub repo.

Delivering a modern search experience is critical for ecommerce websites and for many other use cases, such as customer support, website search, internal workplace search, and custom search applications. The steps in this blog post can be followed to accelerate building a better search experience for all of your end-users.

Sign up for Elastic Cloud free trial

Spin up a fully loaded deployment on the cloud provider you choose. As the company behind Elasticsearch, we bring our features and support to your Elastic clusters in the cloud.

Start free trial