To send data from Elastic Agent to Logstash securely, you need to configure Transport Layer Security (TLS). Using TLS ensures that your Elastic Agents send encrypted data to trusted Logstash servers, and that your Logstash servers receive data from trusted Elastic Agent clients.

Prerequisitesedit

Generate custom certificates and private keysedit

You can use whatever process you typically use to generate PEM-formatted certificates. The examples shown here use the certutil tool provided by Elasticsearch.

Store these files in a secure location.

Configure the Logstash pipelineedit

In your Logstash configuration directory, open the pipelines.yml file and add the following configuration. Replace the path to your file.

- pipeline.id: elastic-agent-pipeline
  path.config: "/etc/path/to/elastic-agent-pipeline.conf"

In the elastic-agent-pipeline.conf file, add the pipeline configuration. Note that the configuration needed for Elasticsearch Service on Elastic Cloud is different from self-managed Elasticsearch clusters. If you copied the configuration shown in Fleet, adjust it as needed.

Elasticsearch Service example:

input {
  elastic_agent {
    port => 5044
    ssl => true
    ssl_certificate_authorities => ["/path/to/ca.crt"]
    ssl_certificate => "/path/to/logstash.crt"
    ssl_key => "/path/to/logstash.pkcs8.key"
    ssl_verify_mode => "force_peer"
  }
}

output {
  elasticsearch {
    cloud_id => "xxxx:xxxxxxxxxxxxxxxxxxxxxxxxxxxxx=" 
    api_key => "xxxx:xxxx" 
    data_stream => true
    ssl => true 
  }
}

Self-managed Elasticsearch cluster example:

input {
  elastic_agent {
    port => 5044
    ssl => true
    ssl_certificate_authorities => ["/path/to/ca.crt"]
    ssl_certificate => "/path/to/logstash.crt"
    ssl_key => "/path/to/logstash.pkcs8.key"
    ssl_verify_mode => "force_peer"
  }
}

output {
  elasticsearch {
    hosts => "https://xxxx:9200"
    api_key => "xxxx:xxxx"
    data_stream => true
    ssl => true
    cacert => "/path/to/http_ca.crt" 
  }
}

To learn more about the Logstash configuration, refer to:

When you’re done configuring the pipeline, restart Logstash:

bin/logstash

Add a Logstash output to Fleetedit

This section describes how to add a Logstash output and configure SSL settings in Fleet. If you’re running Elastic Agent standalone, refer to the Logstash output configuration docs.

When you’re done, save and apply the settings.

Select the Logstash output in an agent policyedit

Logstash is now listening for events from Elastic Agent, but events are not streaming into Elasticsearch yet. You need to select the Logstash output in an agent policy. You can edit an existing policy or create a new one:

Any Elastic Agents enrolled in the agent policy will begin sending data to Elasticsearch via Logstash. If you don’t have any installed Elastic Agents enrolled in the agent policy, do that now.

There might be a slight delay while the Elastic Agents update to the new policy and connect to Logstash over a secure connection.

Test the connectionedit

To make sure Logstash is sending data, run the following command from the host where Logstash is running:

curl -XGET localhost:9600/_node/stats/events

The request should return stats on the number of events in and out. If these values are 0, check the Elastic Agent logs for problems.

When data is streaming to Elasticsearch, go to Observability and click Metrics to view metrics about your system.