Set Up the Elasticsearch Python Client

In this section you will install the Elasticsearch client library for Python and use it to connect to the Elasticsearch service.

Installation

The Elasticsearch client library is a Python package that is installed with pip. Make sure the virtual environment you created earlier is activated, and then run the following command to install the client:

pip install elasticsearch

To avoid any potential incompatibilities, make sure the version of the Elasticsearch client library you install matches the version of the Elasticsearch stack that you are using.

It is always recommended to keep a requirements.txt file updated with all your dependencies, so this is a good time to update this file to include the newly installed package. Run the following command from your terminal:

pip freeze > requirements.txt

Connect to Elasticsearch

To create a connection to your Elasticsearch service, an Elasticsearch object must be created with the appropriate connection options.

Create a new search.py file in your code editor, located in the search-tutorial directory. The search.py file is going to be where all the search functions will be defined. The idea of having a separate file for the search functionality is that this will make it easy for you to extract this file and add it into your own projects later on.

Enter the following code in search.py to add a Search class:

import json
from pprint import pprint
import os
import time

from dotenv import load_dotenv
from elasticsearch import Elasticsearch

load_dotenv()


class Search:
    def __init__(self):
        self.es = Elasticsearch()  # <-- connection options need to be added here
        client_info = self.es.info()
        print('Connected to Elasticsearch!')
        pprint(client_info.body)

There is a lot to unpack here. The load_dotenv() function that is called right after the imports comes from the python-dotenv package. This package knows how to work with .env files, which are used to store configuration variables such as passwords and keys. The load_dotenv() function reads the variables that are stored in the .env file and imports them into the Python process as environment variables.

The Search class has a constructor that creates an instance of the Elasticsearch client class. This is where all the client logic to communicate with the Elasticsearch service lives. Note that this line is currently incomplete, as connection options appropriate to your service need to be included. You will learn what options apply in your case below. Once created, the Elasticsearch object is then stored in an instance variable named self.es.

To ensure that the client object can communicate with your Elastic Cloud deployment, the info() method is invoked. This method makes a call to the service requesting basic information. If this call succeeds, then you can assume that you have a valid connection to the service.

The method then prints a status message indicating that the connection has been established, and then uses the pprint function from Python to display the information that the service returned in an easy to read format.

To complete the constructor of the Search class, the Elasticsearch object needs to be given appropriate connection options. The following sub-sections will tell you what options you need for the Elastic Cloud and Docker methods of installation.

Connect to an Elastic Cloud Deployment

If you followed the instructions to create an Elastic Cloud deployment, you will need to know the deployment's Cloud ID and your API Key. Since these are sensitive values, it is not a good idea to include them directly in the application code. Instead, create a .env (pronounced dot-env) file in which these secrets can be safely stored.

Open your favorite code editor and create a new file in the search-tutorial project directory with the name .env (don't forget the leading dot). Enter the following contents into this file:

ELASTIC_CLOUD_ID="paste your Cloud ID here"
ELASTIC_API_KEY="paste your API Key here"

.env

After you have entered your credentials in the .env file, go back to the Search class constructor in search.py and edit the first line as follows:

class Search:
    def __init__(self):
        self.es = Elasticsearch(cloud_id=os.environ['ELASTIC_CLOUD_ID'],
                                api_key=os.environ['ELASTIC_API_KEY'])
        # ...

The values for the cloud_id and api_key arguments are extracted from the environment, which Python maintains in the os.environ dictionary. These variables are read from the .env file and stored in this dictionary by the load_dotenv() function.

Connect to a Self-Hosted Elasticsearch Docker Container

If you opted to run a locally hosted Elasticsearch service using Docker, then the only connection option that is needed is the connection endpoint. Edit the first line of the Search class constructor in search.py as follows:

class Search:
    def __init__(self):
        self.es = Elasticsearch('http://localhost:9200')
        # ...

In this version of the constructor, the Elasticsearch object is instantiated with the URL to the top-level endpoint of the Elasticsearch service, which is normally http://localhost:9200. Recall that the Docker instructions explicitly disable encryption and authentication, so for that reason it is not necessary to provide any credentials.

Test the Connection

At this point you are ready to make a connection to your Elasticsearch service. To do this, make sure that your Python virtual environment is activated, and then type python to start a Python interactive session. You should see the familiar >>> prompt, in which you can enter Python statements.

Import the Search class as follows:

from search import Search

Next, instantiate the new class:

es = Search()

You should see a connected message, followed by the information returned by the info() method of the client. Except for differences in identifiers and version numbers, the output should look like the following:

Connected to Elasticsearch!
{'cluster_name': 'ad552eba959043158be67dfc021e91cdc',
 'cluster_uuid': 'ks_HfcCdSf2adeKQEsk9gL',
 'name': 'instance-0000000000',
 'tagline': 'You Know, for Search',
 'version': {'build_date': '2023-11-04T10:04:57.184859352Z',
             'build_flavor': 'default',
             'build_hash': 'b4a62ac808e886ff032700c391f45f1408b2538c',
             'build_snapshot': False,
             'build_type': 'docker',
             'lucene_version': '9.7.0',
             'minimum_index_compatibility_version': '7.0.0',
             'minimum_wire_compatibility_version': '7.17.0',
             'number': '8.13.0'}}

If you receive errors, make sure that you have entered correct credentials in the .env file if you are using an Elastic Cloud deployment, or in the case of a self-hosted deployment, that you are running the Elasticsearch Docker container in your computer as instructed.

Integrate Elasticsearch with the Flask Application

The final step in this section is to integrate the work done so far into the little Flask application that you installed earlier. The goal is for the application to automatically create a connection to Elasticsearch when it starts.

To achieve this, open app.py in your code editor. Add an import statement for the search.py module below the only existing imports:

from search import Search

Then find the line in which the app variable is created, and right after create the instance of the new Search class:

es = Search()

That's it! Now the application has an es object to use when needed. If you are still running the Flask application in a terminal, you should see the application reload as soon as you save the file. As a result of the reload, the connection information printed by the Search class constructor should appear, and will continue to appear every time the application restarts from now on.

If you were not running the Flask application, now it is a good time to start it. Change to the project directory in a terminal window, activate the Python virtual environment, and then start the application with:

flask run

To help you troubleshoot in case you receive errors, here is a complete copy of app.py with the integrated Elasticsearch client:

import re
from flask import Flask, render_template, request
from search import Search

app = Flask(__name__)
es = Search()


@app.get('/')
def index():
    return render_template('index.html')


@app.post('/')
def handle_search():
    query = request.form.get('query', '')
    return render_template(
        'index.html', query=query, results=[], from_=0, total=0)


@app.get('/document/<id>')
def get_document(id):
    return 'Document not found'