Semantic Queries

With the index now equipped with ELSER embeddings, the handle_search() function in app.py can be changed to search these embeddings. For now, you'll see how to search only through ELSER, later the previous search methods will be incorporated back to create a combined solution.

To use ELSER inferences when searching, the text_expansion query type is used. Below you can see an updated handle_search() function with this query:

@app.post('/')
def handle_search():
    query = request.form.get('query', '')
    filters, parsed_query = extract_filters(query)
    from_ = request.form.get('from_', type=int, default=0)

    results = es.search(
        query={
            'text_expansion': {
                'elser_embedding': {
                    'model_id': '.elser_model_2',
                    'model_text': parsed_query,
                }
            },
        },
        size=5,
        from_=from_,
    )
    return render_template('index.html', results=results['hits']['hits'],
                           query=query, from_=from_,
                           total=results['hits']['total']['value'])

The text_expansion query receives a key with the name of the field to be searched. Under this key, model_id configures which model to use in the search, and model_text defines what to search for. Note how in this case there is no need to generate an embedding for the search text, as Elasticsearch manages the model and can take care of that.

In the above version of handle_search() the filters have been left unused, and the aggregations have been omitted. These can be added back in the same way they were incorporated into the full-text search solution. Below is an updated handle_search() function that moves the text_expansion query inside a bool.must section, with filters included in bool.filter and aggregations added as before.

@app.post('/')
def handle_search():
    query = request.form.get('query', '')
    filters, parsed_query = extract_filters(query)
    from_ = request.form.get('from_', type=int, default=0)

    results = es.search(
        query={
            'bool': {
                'must': [
                    {
                        'text_expansion': {
                            'elser_embedding': {
                                'model_id': '.elser_model_2',
                                'model_text': parsed_query,
                            }
                        },
                    }
                ],
                **filters,
            }
        },
        aggs={
            'category-agg': {
                'terms': {
                    'field': 'category.keyword',
                }
            },
            'year-agg': {
                'date_histogram': {
                    'field': 'updated_at',
                    'calendar_interval': 'year',
                    'format': 'yyyy',
                },
            },
        },
        size=5,
        from_=from_,
    )
    aggs = {
        'Category': {
            bucket['key']: bucket['doc_count']
            for bucket in results['aggregations']['category-agg']['buckets']
        },
        'Year': {
            bucket['key_as_string']: bucket['doc_count']
            for bucket in results['aggregations']['year-agg']['buckets']
            if bucket['doc_count'] > 0
        },
    }
    return render_template('index.html', results=results['hits']['hits'],
                           query=query, from_=from_,
                           total=results['hits']['total']['value'], aggs=aggs)

Spend some time experimenting with different searches. You will notice that as with dense vector embeddings, searches driven by the ELSER model work better than full-text search when the exact words do not appear in the indexed documents.