Ingest data with Node.js on Elastic Cloudedit

This guide tells you how to get started with:

  • Securely connecting to Elasticsearch Service with Node.js
  • Ingesting data into your deployment from your application
  • Searching and modifying your data on Elasticsearch Service

If you are an Node.js application programmer who is new to the Elastic Stack, this content can help you get started more easily.

Time required: 45 minutes

Prerequisitesedit

These steps are applicable to your existing application. If you don’t have one, you can use the example included here to create one.

Create the npm package.json fileedit

npm init

Get the elasticsearch and config packagesedit

npm install @elastic/elasticsearch
npm install config

Note that the config package is not required if you have your own method to keep your configuration details private.

Create a configuration fileedit

mkdir config
vi config/default.json

The example here shows what the config package expects. You need to update config/default.json with your deployment details in the following sections:

{
  "elastic": {
    "cloudID": "deploymentname:deploymentconnectiondetails",
    "username": "elastic",
    "password": "longpassword"
  }
}

Get Elasticsearch Serviceedit

  1. Get a free trial.
  2. Log into Elastic Cloud.
  3. Click Create deployment.
  4. Select Elastic Stack, leave it at the I/O optimized default, and give your deployment a name.
  5. Click Create deployment and save your Elastic deployment credentials. You will need these credentials later on.
  6. You also need the Cloud ID later on, as it simplifies sending data to Elastic Cloud. Click on the deployment name from the Elastic Cloud portal or the Deployments page and copy down the information under Cloud ID:

    A picture highlighting the Cloud ID information available for the deployment

Prefer not to subscribe to yet another service? You can also get Elasticsearch Service through AWS, Azure, and GCP marketplaces.

Connect securelyedit

If you are using Elastic Cloud, a language client offers an easy way to connect with the cloud_id (Cloud ID) parameter. You must pass the Cloud ID that you can find in the cloud console, and then your authentication details inside the auth option.

To connect and use a language client with Elasticsearch Service, you need to think about authentication. Two authentication mechanisms are supported, API key and basic authentication. Here we’ll show you how to use basic authentication to get started more quickly, but you can also generate API keys as shown later on.

Basic authenticationedit

For basic authentication, use the same deployment credentials (username and password parameters) and Cloud ID you copied down earlier when you created your deployment. (If you did not save the password, you can reset it first.)

You first need to edit your config/default.json file with these deployment details:

{
  "elastic": {
    "cloudID": '<CloudID>',
    "username": '<Username>',
    "password": '<Password>'
  }
}

Next, you need to use these configs with the Node.js Elasticsearch client:

const { Client } = require('@elastic/elasticsearch')
const config = require('config');
const elasticConfig = config.get('elastic');

const client = new Client({
  cloud: {
    id: elasticConfig.cloudID
  },
  auth: {
    username: elasticConfig.username,
    password: elasticConfig.password
  }
})

You can now confirm that you have connected to the deployment by returning some information about the deployment:

client.info()
  .then(response => console.log(response))
  .catch(error => console.error(error))

Ingest dataedit

After connecting to your deployment, you are ready to index and search data. Let us create a new index, insert some quotes from our favorite characters, and refresh the index so that it is ready to be searched. A refresh makes all operations performed on an index since the last refresh available for search.

async function run() {
  await client.index({
    index: 'game-of-thrones',
    body: {
      character: 'Ned Stark',
    quote: 'Winter is coming.'
    }
  })

  await client.index({
    index: 'game-of-thrones',
    body: {
      character: 'Daenerys Targaryen',
    quote: 'I am the blood of the dragon.'
    }
  })

  await client.index({
    index: 'game-of-thrones',
    body: {
      character: 'Tyrion Lannister',
    quote: 'A mind needs books like a sword needs whetstone.'
    }
  })

  await client.indices.refresh({index: 'game-of-thrones'})
}

run().catch(console.log)

When using the client.index API, the request automatically creates the game-of-thrones index if it doesn’t already exist, as well as document IDs for each indexed document if they are not explicitly specified.

Search and modify dataedit

After creating a new index and ingesting some data, you are now ready to search. Let’s find what characters have said things about winter:

async function read() {
  const { body } = await client.search({
    index: 'game-of-thrones',
    body: {
      query: {
        match: { quote: 'winter' }
      }
    }
  })
  console.log(body.hits.hits)
}

read().catch(console.log)

The search request returns content of documents containing 'winter' in the quote field, including document IDs that were automatically generated. You can make updates to specific documents using document IDs. Let’s add a birthplace for our character:

async function update() {
  await client.update({
    index: 'game-of-thrones',
    id: <ID>,
    body: {
      script: {
        source: "ctx._source.birthplace = 'Winterfell'"
      }
    }
  })
  const { body } = await client.get({
    index: 'game-of-thrones',
    id: <ID>
  })

  console.log(body)
}

update().catch(console.log)

This more comprehensive list of API examples includes bulk operations, checking the existence of documents, updating by query, deleting, scrolling, and SQL queries. To learn more, see the complete API reference.

Switch to API key authenticationedit

To get started, authentication to Elasticsearch used the elastic superuser and password, but an API key is much safer and a best practice for production.

In the example that follows, an API key is created with the cluster monitor privilege which gives read-only access for determining the cluster state. Some additional privileges also allow create_index, write, read, and manage operations for the specified index. The index manage privilege is added to enable index refreshes.

The security.createApiKey function returns an id and api_key value which can then be concatenated and encoded in base64:

async function generateApiKeys (opts) {
  const { body } = await client.security.createApiKey({
    body: {
      name: 'nodejs_example',
      role_descriptors: {
        'nodejs_example_writer': {
          'cluster': ['monitor'],
          'index': [
            {
              'names': ['game-of-thrones'],
              'privileges': ['create_index', 'write', 'read', 'manage']
            }
          ]
        }
      }
    }
  })

  return Buffer.from(`${body.id}:${body.api_key}`).toString('base64')
}

generateApiKeys()
  .then(console.log)
  .catch(err => {
  console.error(err)
  process.exit(1)
})

The base64 encoded output is as follows and is ready to be added to the configuration file:

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX==

Edit the config/default.json configuration file you created earlier and add this API key:

{
  "elastic-cloud": {
    "cloudID": "deploymentname:deploymentconnectiondetails",
    "username": "elastic",
    "password": "longpassword",
    "apiKey": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=="
  }
}

Now the API key can be used in place of the username and password. The client connection becomes:

const elasticConfig = config.get('elastic-cloud');

const client = new Client({
  cloud: {
    id: elasticConfig.cloudID
  },
  auth: {
    apiKey: elasticConfig.apiKey
  }
})

See Create API key API to learn more about API Keys and Security privileges to understand which privileges are needed. If you are not sure what the right combination of privileges for your custom application is, you can enable audit logging on Elasticsearch to find out what privileges are being used. To learn more about how logging works on Elastic Cloud, see Monitoring Elastic Cloud deployment logs and metrics.

Best practicesedit

Security

When connecting to Elastic Cloud, the client automatically enables both request and response compression by default, since it yields significant throughput improvements. Moreover, the client also sets the SSL option secureProtocol to TLSv1_2_method unless specified otherwise. You can still override this option by configuring it.

Do not enable sniffing when using Elastic Cloud, since the nodes are behind a load balancer. Elastic Cloud takes care of everything for you. Take a look at Elasticsearch sniffing best practices: What, when, why, how if you want to know more.

Connections
If your application connecting to Elasticsearch Service runs under the Java security manager, you should at least disable the caching of positive hostname resolutions. To learn more, see Disable DNS caching.
Schema
When the above example code was run an index mapping was created automatically. The field types were selected by Elasticsearch based on the content seen when the first record was ingested, and updated as new fields appeared in the data. It would be more efficient to specify the fields and field types in advance to optimize performance. See the Elastic Common Schema documentation and Field Type documentation when you are designing the schema for your production use cases.
Ingest
For more advanced scenarios, this bulk ingestion reference gives an example of the bulk API that makes it possible to perform multiple operations in a single call. This bulk example also explicitly specifies document IDs. If you have a lot of documents to index, using bulk to batch document operations is significantly faster than submitting requests individually.