Building software today is a commitment to a lifetime of learning. As you will have seen from the earlier blogs in this series, Carly recently started playing with Go.
Search has undergone an evolution of different practices. It can be difficult to decide between your own search use case. Building on the keyword and vector search examples covered in part one and two of this series. In this part, we'll share examples of how you can combine both vector and keyword search using Elasticsearch and the Elasticsearch Go client.
Prerequisites
Just like part one in this series, the following prerequisites are required for this example:
- Installation of Go version 1.13 or later
- Create your own Go repo using the recommended structure and package management covered in the Go documentation
- Creating your own Elasticsearch cluster, populated with a set of rodent-based pages, including for our friendly Gopher, from Wikipedia:
Connecting to Elasticsearch
As a reminder, in our examples, we will make use of the Typed API offered by the Go client. Establishing a secure connection for any query requires configuring the client using either:
- Cloud ID and API key if making use of Elastic Cloud
- Cluster URL, username, password and the certificate
Connecting to our cluster located on Elastic Cloud would look like this:
func GetElasticsearchClient() (*elasticsearch.TypedClient, error) {
var cloudID = os.Getenv("ELASTIC_CLOUD_ID")
var apiKey = os.Getenv("ELASTIC_API_KEY")
var es, err = elasticsearch.NewTypedClient(elasticsearch.Config{
CloudID: cloudID,
APIKey: apiKey,
Logger: &elastictransport.ColorLogger{os.Stdout, true, true},
})
if err != nil {
return nil, fmt.Errorf("unable to connect: %w", err)
}
return es, nil
}The client connection can then be used for searching, as demonstrated in the subsequent sections.
Manual boosting
When combining any set of search algorithms, the traditional approach has been to manually configure constants to boost each query type. Specifically, a factor is specified for each query, and the combined results set is compared to the expected set to determine the recall of the query. Then we repeat for several sets of factors and pick the one closest to our desired state.
For example, combining a single text search query boosted by a factor of 0.8 with a knn query with a lower factor of 0.2 can be done by specifying the Boost field in both query types, as shown in the below example:
func HybridSearchWithBoost(client *elasticsearch.TypedClient, term string) ([]Rodent, error) {
var knnBoost float32 = 0.2
var queryBoost float32 = 0.8
res, err := client.Search().
Index("vector-search-rodents").
Knn(types.KnnQuery{
Field: "text_embedding.predicted_value",
Boost: &knnBoost,
K: 10,
NumCandidates: 10,
QueryVectorBuilder: &types.QueryVectorBuilder{
TextEmbedding: &types.TextEmbedding{
ModelId: "sentence-transformers__msmarco-minilm-l-12-v3",
ModelText: term,
},
}}).
Query(&types.Query{
Match: map[string]types.MatchQuery{
"title": {
Query: term,
Boost: &queryBoost,
},
},
}).
Do(context.Background())
if err != nil {
return nil, err
}
return getRodents(res.Hits.Hits)
}The factor specified in the Boost option for each query is added to the document score. By increasing the score of our match query by a larger factor than the knn query, results from the keyword query are more heavily weighted.
The challenge of manual boosting, particularly if you're not a search expert, is that it requires tuning to figure out the factors that will lead to the desired result set. It's simply a case of trying out random values to see what gets you closer to your desired result set.
Reciprocal Rank Fusion
Reciprocal Rank Fusion, or RRF, was released under technical preview for hybrid search in Elasticsearch 8.9. It aims to reduce the learning curve associated with tuning and reduce the amount of time experimenting with factors to optimize the result set.
With RRF, the document score is recalculated by blending the scores by the below algorithm:
score := 0.0
// q is a query in the set of queries (vector and keyword search)
for _, q := range queries {
// result(q) is the results
if document in result(q) {
// k is a ranking constant (default 60)
// rank(result(q), d) is the document's rank within result(q)
// range from 1 to the window_size (default 100)
score += 1.0 / (k + rank(result(q), d))
}
}
return scoreThe advantage of using RRF is that we can make use of the sensible default values within Elasticsearch. The ranking constant k defaults to 60. To provide a tradeoff between the relevancy of returned documents and the query performance when searching over large data sets, the size of the result set for each considered query is limited to the value of window_size, which defaults to 100 as outlined in the documentation.
k and windows_size can also be configured within the Rrf configuration within the Rank method in the Go client, as per the below example:
func HybridSearchWithRRF(client *elasticsearch.TypedClient, term string) ([]Rodent, error) {
// Minimum required window size for the default result size of 10
var windowSize int64 = 10
var rankConstant int64 = 42
res, err := client.Search().
Index("vector-search-rodents").
Knn(types.KnnQuery{
Field: "text_embedding.predicted_value",
K: 10,
NumCandidates: 10,
QueryVectorBuilder: &types.QueryVectorBuilder{
TextEmbedding: &types.TextEmbedding{
ModelId: "sentence-transformers__msmarco-minilm-l-12-v3",
ModelText: term,
},
}}).
Query(&types.Query{
Match: map[string]types.MatchQuery{
"title": {Query: term},
},
}).
Rank(&types.RankContainer{
Rrf: &types.RrfRank{
WindowSize: &windowSize,
RankConstant: &rankConstant,
},
}).
Do(context.Background())
if err != nil {
return nil, err
}
return getRodents(res.Hits.Hits)
}Conclusion
Here we've discussed how to combine vector and keyword search in Elasticsearch using the Elasticsearch Go client.
Check out the GitHub repo for all the code in this series. If you haven't already, check out part 1 and part 2 for all the code in this series.
Happy gopher hunting!
