Elastic Docs ›Logstash and Kubernetes Reference ›Quick start
« Quick start Logstash configuration files in Kubernetes »

Sample configuration filesedit

This documentation is still in development and may be changed or removed in a future release.

These configuration files are used in the Logstash and Kubernetes quick start. You can use them as templates when you configure Logstash together with the rest of the Elastic Stack in a Kubernetes environment.

You can download the files together as a zip archive.

Setup filesedit

These files are used to create certificates and keys required for secure communication between Beats and Logstash. They are included for illustration purposes only. For production environments, supply your own keys and certificates as appropriate.

cert/generate_cert.sh
Generates the ca.crt, client.key, client.crt, server.key, and server.pkcs8.key used to establish a secure connection between Filebeat and Logstash. The certificates and keys are all contained in the 001-secret.yaml file that is generated when you run generate_cert.sh.
cert/openssl.conf
The OpenSSL Generated Server Certificate used for TLS communication between resources.

This config file creates a secrets file 001-secret.yaml. We will install the secrets file as we set up the Elastic Stack.

Logstash configuration filesedit

001-configmap.yaml

This file contains the Logstash settings and pipeline configuration: 

  ---
# ConfigMap for logstash pipeline definition
data:
  logstash.conf: | 
    input {
      beats {
        port => "5044"
        ssl => true
        ssl_certificate_authorities => ["/usr/share/logstash/config/ca.crt"]
        ssl_certificate => "/usr/share/logstash/config/server.crt"
        ssl_key => "/usr/share/logstash/config/server.pkcs8.key"
        ssl_verify_mode => "force_peer"
      }
    }
    output {
      elasticsearch {
        hosts => ["https://demo-es-http:9200"]
        index => "kube-apiserver-%{+YYYY.MM.dd}"
        cacert => "/usr/share/logstash/config/es_ca.crt"
        user => 'elastic'
        password => '${ELASTICSEARCH_PASSWORD}'
      }
    }
---
# ConfigMap for logstash.yml definition
data:
  logstash.yml: | 
    api.http.host: "0.0.0.0"

Definition of Logstash configuration file. We will refer to this definition later in the deployment file, where we will define volumes.

Definition of logstash.yml file Define each key/value pair to override defaults. We will refer to this definition later in the deployment file.
001-secrets.yaml

This secrets file includes certificates and key files required for secure communication between Logstash and the rest of the Elastic Stack. This example was generated by the supplied script, but for your own configuration it should contain the base64 encoded representations of your own certificates and keys.

You can generate this file for your own certs and keys by using the kubectl create secret generic command:

kubectl create secret generic logstash-beats-tls --from-file=ca.crt --from-file=client.crt --from-file=client.key --from-file=server.crt --from-file=server.pkcs8.key --dry-run=client -o yaml | kubectl label -f- --dry-run=client -o yaml --local app=logstash-demo  > ../001-secret.yaml

The command generates a secrets file that looks resembles this.

apiVersion: v1
data:
  ca.crt: <BASE64 representation of ca cert, used in beats client and logstash beats input>
  client.crt: <BASE64 representation of beats client cert>
  client.key: <BASE64 representation of beats client key>
  server.crt: <BASE64 representation of server certificate, used in beats input>
  server.pkcs8.key: <BASE64 representation of PKCS8 server key, used in beats input>
kind: Secret
metadata:
  creationTimestamp: null
  labels:
    app: logstash-demo
  name: logstash-beats-tls
002-deployment.yaml

Contains the configuration definition for Logstash. 

spec:
  replicas: 1
  selector:
    matchLabels:
      app: logstash-demo
  template:
    metadata:
      labels:
        app: logstash-demo
    spec:
      containers:
        - name: logstash
          securityContext:
            runAsNonRoot: true
            runAsUser: 1000
          image: {docker-image} 
          env:
            - name: LS_JAVA_OPTS 
              value: "-Xmx1g -Xms1g"
            - name: ELASTICSEARCH_PASSWORD 
              valueFrom:
                secretKeyRef:
                  name: demo-es-elastic-user
                  key: elastic
          resources:
            limits: 
              cpu: 2000m
              memory: 2Gi
            requests:
              cpu: 1000m
              memory: 2Gi
          ports: 
            - containerPort: 9600
              name: stats
            - containerPort: 5044
              name: beats
          livenessProbe: 
            httpGet:
              path: /
              port: 9600
            initialDelaySeconds: 60
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3
          readinessProbe: 
            httpGet:
              path: /
              port: 9600
            initialDelaySeconds: 30
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3
          volumeMounts: 
            - name: logstash-pipeline
              mountPath: /usr/share/logstash/pipeline
            - name: logstash-config 
              mountPath: /usr/share/logstash/config/logstash.yml
              subPath: logstash.yml
            - name: es-certs 
              mountPath: /usr/share/logstash/config/es_ca.crt
              subPath: ca.crt
            - name: logstash-beats-tls
              mountPath: /usr/share/logstash/config/ca.crt
              subPath: ca.crt
            - name: logstash-beats-tls
              mountPath: /usr/share/logstash/config/server.pkcs8.key
              subPath: server.pkcs8.key
            - name: logstash-beats-tls
              mountPath: /usr/share/logstash/config/server.crt
              subPath: server.crt
      volumes:
        - name: logstash-pipeline 
          configMap:
            name: logstash-pipeline
        - name: logstash-config 
          configMap:
            name: logstash-config
        - name: es-certs 
          secret:
            secretName: demo-es-http-certs-public
        - name: logstash-beats-tls 
          secret:
            secretName: logstash-beats-tls
        - name: es-user 
          secret:
            secretName: demo-es-elastic-user

Logstash docker image

Set non-default JVM settings, such as memory allocation, here in the LS_JAVA_OPTS env variable to avoid the need to add a whole jvm.options file in a ConfigMap

Resource/memory limits for the pod. Refer to Kubernetes documentation to set resources appropriately for each pod. Ensure that each pod has sufficient memory to handle the heap specified in <2>, allowing enough memory to deal with direct memory. Check out Logstash JVM settings for details.

Expose the necessary ports on the container. Here we are exposing port 5044 for the beats input, and 9600 for the metricbeat instance to query the logstash metrics API for stack monitoring purposes.

Liveness probe to determine whether Logstash is running. Here we point to the Logstash Metrics API, an HTTP based API that will be ready shortly after logstash starts. Note that the endpoint shows no indication that Logstash is active, only that the API is available.

Readiness probe to determine whether Logstash is running. Here we point to the Logstash Metrics API, an HTTP based API that will be ready shortly after Logstash starts. Note that the endpoint shows no indication that Logstash is active, only that the API is available.

The pipeline configuration that we created in the ConfigMap declaration needs a volume and a volumeMount. The volume refers to the created config map and the volumeMount refers to the created volume and mounts in a location that logstash will read. Unless a separate pipeline.yml file is created by a further ConfigMap definition, the expected location of pipeline configurations is /usr/share/logstash/pipelines and the mountPath should be set accordingly.

Name of the Logstash configuration we created earlier. This file should contain key/value pairs intended to override the default values in logstash.yml, using the flat key syntax described in that document. To setup, this needs a volume and a volumeMount. The volume refers to the created config map and the volumeMount refers to the created volume and mounts in a location that Logstash will read. The mountPath should be set to ` /usr/share/logstash/logstash.yml.

Volume and VolumeMount definitions for certificates to use with Elasticsearch. This contains the CA certificate to output data to Elasticsearch. Refer to TLS certificates in the Elastic Cloud on Kubernetes Guide for details.

Volume and VolumeMount definitions for certificates to use with Beats.

The Elasticsearch password is taken from demo-es-elastic-user and passed to the Logstash pipeline as an ELASTICSEARCH_PASSWORD environment variable. Refer to Access the Elasticsearch endpoint in the Elastic Cloud on Kubernetes Guide for details.
003-service.yaml

This file contains the Service definition, opening up ports on the logstash pods to the internal metricbeat (for stack monitoring) and filebeat in this instance.

spec:
  type: ClusterIP
  ports:
    - port: 9600 
      name: "stats"
      protocol: TCP
      targetPort: 9600 
    - port: 5044 
      name: "beats"
      protocol: TCP
      targetPort: 5044 
  selector:
    app: logstash-demo

Opens port 9600 for Metricbeat to connect to the Logstash metrics API.

Opens port 5044 for Filebeat to connect to the Beats input defined in the ConfigMap.
004-hpa.yml

This file sets up a horizontal pod autoscaler to scale Logstash instances up and down, depending on the load on the Logstash instance(s). See kubernetes autoscaler docs for more details.

apiVersion: autoscaling/v2 
kind: HorizontalPodAutoscaler
metadata:
  name: logstash
  labels:
    app: logstash-demo
spec:
  minReplicas: 1 
  maxReplicas: 2
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60 
    scaleDown:
      stabilizationWindowSeconds: 180
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: logstash 
  metrics:
    - type: Resource 
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 80
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80

Requires Kubernetes 1.23 and higher.

Specifies the maximum and minimum number of Logstashes desired for the cluster.

Specifies stabilization windows to avoid rapidly scaling nodes up and down unnecessarily.

Deployment created earlier
006-metricbeat.yaml

Enables the Metricbeat Logstash module and sets it to collect metrics data from logstash:9600: 

  - module: logstash 
    metricsets:
      - node
      - node_stats
    period: 10s
    hosts:
      - logstash:9600
    xpack.enabled: true

Definition for logstash module, defined under spec.config.metricbeat.modules
005-filebeat.yaml

This file includes the configuration required for a beat to communicate with Logstash. It includes the Logstash output definition, and makes the generated certs and key files from the secrets file available to the beat to enable secure communication with Logstash. 

volumes: 
  - name: logstash-beats-tls
    secret:
      secretName: logstash-beats-tls

Volume definition for certs/keys defined under deployment.podTemplate.spec.

 
volumeMounts: 
  - name: logstash-beats-tls
    mountPath: /usr/share/filebeat/ca.crt
    subPath: ca.crt
  - name: logstash-beats-tls
    mountPath: /usr/share/filebeat/client.key
    subPath: client.key
  - name: logstash-beats-tls
    mountPath: /usr/share/filebeat/client.crt
    subPath: client.crt

Volume mount definition for certs/keys defined under deployment.podTemplate.spec.containers.

 
output.logstash: 
  hosts:
    - "logstash:5044"
  ssl.certificate_authorities: ["/usr/share/filebeat/ca.crt"]
  ssl.certificate: "/usr/share/filebeat/client.crt"
  ssl.key: "/usr/share/filebeat/client.key"

Logstash output definition defined under spec.config.
000-elasticsearch.yaml
Configures a single Elasticsearch instance to receive output data from Logstash.
007-kibana.yaml
Configures a single Kibana instance to visualize the logs and metrics data.
« Quick start Logstash configuration files in Kubernetes »