Sending data to Elasticsearch Serverless

IMPORTANT: This documentation is no longer updated. Refer to Elastic's version policy and the latest documentation.

Sending data to Elasticsearch Serverless

When you use Elasticsearch on Elastic Cloud Serverless you don’t need to worry about managing the infrastructure that keeps Elasticsearch distributed and available. These resources are automated on the serverless platform and are designed to scale up and down with your workload.

Logstash to Elastic Cloud Serverless

You’ll use the Logstash Elasticsearch output plugin to send data to Elastic Cloud Serverless. Note these differences between Elasticsearch Serverless and both Elastic Cloud Hosted and self-managed Elasticsearch:

Use API keys to access Elastic Cloud Serverless from Logstash as it does not support native user authentication. Any user-based security settings in your Logstash Elasticsearch output plugin configuration are ignored and may cause errors.
Elastic Cloud Serverless uses data streams and data lifecycle management (DLM) instead of index lifecycle management (ILM). Any ILM settings in your Elasticsearch output plugin configuration are ignored and may cause errors.
Logstash monitoring is available through the Logstash Integration in Elastic Observability on Elastic Cloud Serverless.

Known issue for Logstash to Elasticsearch Serverless.
The logstash-output-elasticsearch hosts setting defaults to port :9200.
Set the value to port :443 instead.

Communication between Logstash and Elasticsearch Serverless

edit

Elasticsearch Serverless simplifies safe, secure communication between Logstash and Elasticsearch. To send data to a Serverless project, configure the Logstash Elasticsearch output plugin to connect using the project’s Elasticsearch endpoint URL and an API key.

output {elasticsearch { hosts => "ELASTICSEARCH_ENDPOINT_URL" api_key => "<api key>" } }

The value of the api_key option is in the format id:api_key, where id and api_key are the values returned by the Create API key API.

Elasticsearch endpoint URL

edit

Log in to Elastic Cloud.
Find your Elasticsearch endpoint URL:
- Select Manage next to your project. Then find the Elasticsearch endpoint under Application endpoints, cluster and component IDs.
- Alternatively, open your project, select the help icon, then select Connection details.

API key

edit

Create an API key with the appropriate privileges. Refer to Create API key for detailed steps. For information on the required privileges, refer to Grant access using API keys.

When you create an API key for Logstash, select Logstash from the API key format dropdown. This option formats the API key in the correct id:api_key format required by Logstash.

The Elasticsearch input, Elasticsearch output, and Elasticsearch filter plugins, as well as the Elastic_integration filter plugin, all support the api_key option in their configurations.

Using Logstash Central Pipeline Management with Elasticsearch Serverless

edit

To set up Central Pipeline management in Elasticsearch Serverless, update the logstash.yml config file to provide the API key and set the value for xpack.management.elasticsearch.hosts to your Elasticsearch endpoint URL.

xpack.management.elasticsearch.hosts: "ELASTICSEARCH_ENDPOINT_URL"  
xpack.management.elasticsearch.api_key: "<api_key>"

	Use the `hosts` option with the Elasticsearch endpoint URL to connect to Elasticsearch Serverless
	API key formatted for Logstash: `id:api_key`

« Sending data to Elastic Cloud Hosted Logstash configuration examples »