In this article we'll take a look at the new code generation feature in the Kibana Dev Console, and how it can significantly reduce your development effort by generating ready to use code for you.

This feature is available in our Serverless platform and in Elastic Cloud and self-hosted releases 8.16 and up.

The Kibana Dev Console

This section provides a quick introduction to the Kibana Dev Console, in case you have never used it before. Skip to the next section if you are already familiar with it.

While you are in any part of the Search section in Kibana, you will notice a "Console" link at the bottom of your browser's page:

When you click this link, the Console expands to cover the page. Click it again to collapse it.

In the left-side panel of the Dev Console, you can enter Elasticsearch requests, with the help of an interactive editor that provides auto-completion and checks your syntax. Some example requests are already pre-populated so that you have something to start experimenting with.

When the cursor is on a request, a "play" button appears to its right. You can click this button to send the request to your Elasticsearch server.

After you execute a request, the response from the server appears in the panel on the right.

Code Export feature in Kibana Dev Console

The Dev Console makes it easy to prototype your requests or queries until you get exactly what you want. But what happens next? If you need to convert the request to code so that you can incorporate it into your application, then you can save time using the new code export feature.

Next to the Play button you will find the three dot or "kebab" button, which opens a menu of options. The first option provides access to the code export feature. If you've never used this feature before, it will appear with a "Copy as curl" label.

If you select this option, your clipboard will be loaded with a curl command that is equivalent to the selected request.

Now, things get more interesting when you click the "Change" link, which allows you to switch to a different target language. In this initial release, the code export adds support for Python and JavaScript. More languages are expected to be added in future releases.

You can now select your desired language and click "Copy code" to put the exported code in your clipboard. You can also change the default language that is offered in the menu.

The exported code is a complete script in the selected language, using the official Elasticsearch client for that language. Here is an example of how the PUT /my-index request shown above looks when exported to the Python language:

To use the exported code follow these steps:

• Paste the code from the clipboard to a new file with the correct extension (.py for Python, or .js for JavaScript).

• In your terminal, add an environment variable called ELASTIC_API_KEY with a valid API Key for your Elasticsearch cluster. You can create an API key right in Kibana if you don't have one yet.

• Execute the script with the python or node commands depending on your language, making sure the official Elasticsearch client is installed.

Now you are ready to adapt the exported code as needed to integrate it into your application!

Conclusion

In this article you have learned about the new Code Export feature in the Kibana Dev Console. We hope this feature will streamline your development process with Elasticsearch!

First, we'll load the data using Node.js and ensure it remains current through regular updates. Then, we'll leverage the capabilities of Azure Function Apps to automate these updates, thereby ensuring your index is always fresh and reliable.

For this blog post, we will be using the Near Earth Object Web Service (NeoWs), a RESTful web service offering detailed information about near-earth asteroids. By integrating NeoWs with Node.js services integrated as Azure serverless functions, this example will provide you with a robust framework to handle the complexities of managing dynamic data effectively. This approach will help you minimize the risks of working with outdated information and maximize the accuracy and usefulness of your data.

Prerequisites

• This example uses Elasticsearch version 8.13; if you are new to Elasticsearch, check out our Quick Start on Elasticsearch. Any 8.0 version should work for this blog post.

• Download the latest NPM and Node.js version. This tutorial uses Node v21.6.1 and npm 10.5.0.

• An API key for NASA's APIs.

• An active Azure account with access to create a Function App.

• Access to the Azure portal or Azure CLI

Setting up locally

Before you begin indexing and loading your data locally, setting up your environment is essential. First, create a directory and initialize it. Then, download the necessary packages and create a .env file to store your configuration settings. This preliminary setup ensures your local environment is prepared to handle the data efficiently.

You will be using the Elasticsearch node client to connect to Elastic, Axios to connect to the NASA APIs and dotenv to parse your secrets. You will want to download the required packages running the following commands:

After downloading the required packages, you can create a .env file at the root of the project directory. The .env file allows you to keep your credentials secure locally. Check out the example .env file to learn more. To learn more about connecting to Elasticsearch, be sure to take a look at the documentation on the subject.

To create a .env file, you can use this command at the root of your project:

In your .env , be sure to have the following entered in. Be sure to add your complete endpoint:

You will also want to create a new JavaScript file as well:

Creating your index and loading your data in

Now that you have set up the proper file structure and downloaded the required packages, you are ready to create a script that creates an index and loads data into the index. If you get stuck along the way be sure to check out the full version of the file you are creating in this section.

In the file loading_data_into_a_index.js, configure the dotenv package to use the keys and tokens stored in your .env file. You should also import the Elasticsearch client to connect to Elasticsearch and Axios and make HTTP requests.

Since your keys and tokens are currently stored as environment variables, you will want to retrieve them and create a client to authenticate to Elasticsearch.

You can develop a function to retrieve data from NASA's NEO (Near Earth Object) Web Service asynchronously. You will first configure the base URL for the NASA API request and create date objects for today and the previous week to establish the query period. After you format these dates in the YYYY-MM-DD format required for the API request, set up the dates as query parameters and execute the GET request to the NASA API. Additionally, the function includes error-handling mechanisms to aid debugging should any issues arise.

Now, you can create a function to transform the raw data from the NASA API into a structured format. Since the data you get back is currently nested in a complex JSON response. A more straightforward array of objects makes handling data easier.

You will want to create an index to store the data from the API. An index inside Elasticsearch is where you can store your data in documents. In this function, you will check to see if an index exists and create a new one if needed. You will also specify the proper mapping of fields for your index. This function also loads the data into the index as documents and maps the id field from the NASA data to the _id field in Elasticsearch.

You will want to create a main function to fetch, structure, and index the data. This function will also print out the number of records being uploaded and log whether the data is indexed, whether there is no data to index, or whether it failed to get data back from the NASA API. After creating the run function, you will want to call the function and catch any errors that may come up.

You can now run the file from your command line by running the following:

To confirm that your index has been successfully loaded, you can check in the Elastic Dev Tools by executing the following API call:

Keeping your index updated with an Azure Function App

Now that you've successfully loaded your data into your index locally, this data can quickly become outdated. To ensure your information remains current, you can set up an Azure Function App to automatically fetch new data daily and upload it to your Elasticsearch index.

The first step is to configure your Function app in Azure Portal. A helpful resource for getting started is the Azure quick start guide.

After you've set up your function, you can ensure that you have environment variables set up for ELASTICSEARCH_ENDPOINT, ELASTICSEARCH_API_KEY, and NASA_API_KEY. In Function Apps, environment variables are called Application settings. Inside your function app, click on the "Configuration" option in the left panel under "Settings." Under" the "Application settings" tab, click on "+ New application setting."

You will want to make sure the required libraries are installed as well. If you go to your terminal on the Azure Portal, you can install the necessary packages by entering the following:

The packages you are installing should look very similar to the previous install, except you will be using the moment to parse dates, and you no longer need to load an env file since you just set your secrets to be Application settings.

You can click where it says create to create a new function inside your Function App select the template entitled “Timer trigger”. You will now have a file called function.json set for you. You will want to adjust it to look as follows to run this application every day at 10 am.

You'll also want to upload your package.json file and ensure it appears as follows:

The next step is to create a index.js file. This script is designed to automatically update the data daily. It accomplishes this by systematically fetching and parsing new data each day and then seamlessly updating the dataset accordingly. Elasticsearch can use the same method to ingest time series or immutable data, such as webhook responses. This method ensures the information remains current and accurate, reflecting the latest available data.You can can check out the full code as well.

The main differences between the script you run locally and this one are as follows:

• You will no longer need to load a .env file, since you have already set your environment variables

• There is also different logging designed more towards creating a more sustainable script

• You keep your index updated based on the most recent close approach date

• There is an entry point for an Azure Function App

You will first want to set up your libraries and authenticate to Elasticsearch as follows:

Afterward, you will want to obtain the last date update date from Elasticsearch and configure a backup method to get data from the past day if anything goes wrong.

The following function connects to NASA's NEO (Near Earth Object) Web Service to get the data to keep your index updated. There is also some additional error handling that can capture any API errors that might come up.

Now, you will want to create a function to organize your data by iterating over the objects of each date.

Now, you will want to load your data into Elasticsearch using the bulk indexing operation. This function should look similar to the one in the previous section.

Finally, you will want to create an entry point for the function that will run according to the timer you set. This function is similar to a main function, as it calls the functions created previously in the file. There is also some additional logging, such as printing the number of records and informing you if the data was indexed correctly.

Conclusion

Using Node.js and Azure's Function App, you should be able to ensure that your Elasticsearch index is updated regularly. By utilizing Node.js's capabilities in conjunction with Azure's Function App, you can efficiently maintain your index's regular updates. This powerful combination offers a streamlined, automated process, reducing the manual effort involved in keeping your index regularly updated. Full code for this example can be found on Search Labs GitHub. Let us know if you built anything based on this blog or if you have questions on our forums and the community Slack channel.