Logstash

Use Logstash to ship data to Elasticsearch.

Logstash is an open source data collection engine with real-time pipelining capabilities. It supports a wide variety of data sources, and can dynamically unify data from disparate sources and normalize the data into destinations of your choice.

Logstash can collect data using a variety of Logstash input plugins, enrich and transform the data with Logstash filter plugins, and output the data to Elasticsearch using the Logstash Elasticsearch output plugin.

You can use Logstash to extend Beats for advanced use cases, such as data routed to multiple destinations or when you need to make your data persistent.

Logstash for Elasticsearch on serverless

Logstash is a powerful, versatile ETL (Extract, Transform, Load) engine that can play an important role in organizations of all sizes. Some capabilities and features for large, self-managed users aren't appropriate for serverless.

You'll use the Logstash Elasticsearch output plugin to send data to serverless Elasticsearch. Some differences to note between serverless Elasticsearch and self-managed Elasticsearch:

  • Your logstash-output-elasticsearch configuration uses API keys to access Elasticsearch from Logstash. User-based security settings are ignored and may cause errors.
  • serverless Elasticsearch uses data lifecycle management (DLM) instead of index lifecycle management (ILM). If you add ILM settings to your Elasticsearch output configuration, they are ignored and may cause errors.
  • Logstash monitoring for serverless is available through the Logstash Integration in Elastic Observability.

Known issue

  • The logstash-output-elasticsearch hosts setting defaults to port :9200. Set the value to port :443 instead.

Requirements

To use Logstash to send data to serverless Elasticsearch, you must be using:

Secure connection

Serverless Elasticsearch simplifies secure communication between Logstash and Elasticsearch. Configure the Elasticsearch output plugin to use cloud_id and api_key. No additional SSL configuration steps are needed.

API keys for connecting Logstash to serverless Elasticsearch

Use the Security: API key section in the UI to create an API key for securely connecting the Logstash Elasticsearch output to serverless Elasticsearch. We recommend creating a unique API key per Logstash instance. You can create as many API keys as necessary.

When you set up your API keys, use the metadata option to tag each API key with details that are meaningful to you. This step makes managing multiple API keys easier.

After you generate an API key, add it to your Logstash Elasticsearch output plugin config file's api_key setting. Here's an example:

output {
  elasticsearch {
    api_key => "TiNAGG4BaaMdaH1tRfuU:KnR6yE41RrSowb0kQ0HWoA"
  }
}

Migrating Elasticsearch data using Logstash

You can use Logstash to migrate data from self-managed Elasticsearch or Elasticsearch Service to serverless Elasticsearch, or to migrate data from one serverless Elasticsearch deployment to another.

Create a Logstash pipeline that includes the Elasticsearch input plugin and output plugin.

Configure the Elasticsearch input to point to your source deployment or instance, and configure the Elasticsearch output with the cloud_id and api_key settings for your target serverless Elasticsearch instance.

If your origin index is using settings that aren't supported in Serverless, then you might need to adjust your index settings.

Next steps

Check out the Logstash product page to see what Logstash can do for you. When you're ready, dive into the Logstash documentation.

On this page