Logstash configuration files in Kubernetes | Logstash and Kubernetes Reference

› ›

« Sample configuration files Setting up Logstash and Kubernetes »

Logstash configuration files in Kubernetesedit

This documentation is still in development. This feature may be changed or removed in a future release.

This guide walks you through configuring Logstash and setting up Logstash pipelines in Kubernetes.

Logstash uses two types of configuration files:

pipeline configuration files, which define the Logstash processing pipeline
settings files which specify options that control Logstash startup and execution. Logstash configuration files topic contains information on these files. This guide explains how these map to a Kubernetes configuration.

Pipeline configurationedit

This section explains how to configure single and multiple pipeline Logstash configurations. Note that this section does not cover using Centralized Pipeline Management.

Each of these configurations requires creating one or more ConfigMap definitions to define the pipeline, creating a volume to be made available to the Logstash container, and then mounting the definition in these volumes

Single pipelineedit

The Logstash existing docker image contains a default pipeline.yml, which expects a single pipeline, with the definition of that pipeline present in /usr/share/logstash/pipeline, as either a single file or collection of files, typically defined as a ConfigMap or series of ConfigMaps - note that a single Kubernetes ConfigMap has a size limit of 1MB.

This example contains a simple pipeline definition, with the inputs and outputs split into separate configuration files:

apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-pipeline 
  labels:
    app: logstash-demo
data:
  logstash-input.conf: | 
    input {
      beats {
        port => "5044"
      }
    }
  logstash-output.conf: | 
    output {
      elasticsearch {
        hosts => ["https://demo-es-http:9200"]
      }
    }

	Name of `ConfigMap` to be referenced in `Deployment`.
	Creates a `ConfigMap` representing the inputs for a pipeline.
	Creates a `CongigMap` representing the outputs for a pipeline.

Next, define your Volume in your Deployment template:

volumes:
  - name: logstash-pipeline
    configMap:
      name: logstash-pipeline

and mount the volume in your container:

volumeMounts:
    - name: logstash-pipeline
      mountPath: /usr/share/logstash/pipeline

Multiple pipelinesedit

Logstash uses the pipelines.yml file to define multiple pipelines. Logstash in Kubernetes requires a ConfigMap to represent the content that would otherwise be in pipelines.yml. You can create pipeline configurations inline, or in separate configMap files or folders.

Example: Pipelines.yml ConfigMap with an inline pipeline definition

apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-pipeline-yaml 
  labels:
    app: logstash-demo
data:
  pipelines.yml: | 
    - pipeline.id: test 
      pipeline.workers: 1
      pipeline.batch.size: 1
      config.string: "input { generator {} } filter { sleep { time => 1 } } output { stdout { codec => dots } }"
    - pipeline.id: pipeline2 
      pipeline.workers: 8
      path.config: "/usr/share/logstash/pipeline2"

	Name of `ConfigMap` to be referenced in `Deployment`.
	Defines a `pipelines.yml` `ConfigMap`.
	Defines a pipeline inside the `pipelines.yml`.
	Defines a pipeline, and a location where the pipeline definitions are stored. See below for these pipeline definitions.

Example: Pipelines defined in separate files

apiVersion: v1
kind: ConfigMap
metadata:
  name: pipeline2
  labels:
    app: logstash-demo
data:
  logstash-input.conf: |
    input {
      beats {
        port => "5044"
      }
    }
  logstash-output.conf: |
    output {
      elasticsearch {
        hosts => ["https://demo-es-http:9200"]
        index => "kube-apiserver-%{+YYYY.MM.dd}"
        cacert => "/usr/share/logstash/config/es_ca.crt"
        user => 'elastic'
        password => '${ELASTICSEARCH_PASSWORD}'
      }
    }

Make pipelines available to Logstashedit

Create the volume(s) in your Deployment/StatefulSet

volumes:
  - name: logstash-pipelines-yaml
    configMap:
      name: logstash-pipelines-yaml
  - name: pipeline2
    configMap:
      name: pipeline2

and mount the volume(s) in your container spec

#
volumeMounts:
    - name: pipeline2
      mountPath: /usr/share/logstash/pipeline2
    - name: logstash-pipelines-yaml
      mountPath: /usr/share/logstash/config/pipelines.yml
      subPath: pipelines.yml

Settings configurationedit

The logstash.yml fileedit

Unless you specify a configuration file, default values for the logstash.yml file are used. To override the default values, create a ConfigMap with the settings that you want to override:

apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-config
  labels:
    app: logstash-demo
data:
  logstash.yml: |
    api.http.host: "0.0.0.0"
    log.level: info
    pipeline.workers: 2

In your Deployment/StatefulSet, create the Volume:

volumes:
  - name: logstash-config
    configMap:
      name: logstash-config

Create the volumeMount in the container:

  volumeMounts:
    - name: logstash-config
      mountPath: /usr/share/logstash/config/logstash.yml
      subPath: logstash.yml

JVM optionsedit

JVM settings are best set using environment variables to override the default settings in jvm.options. This approach ensures that the expected settings from jvm.options are set, and only those options that explicitly need to be overridden are.

The JVM settings should be added in the LS_JAVA_OPTS environment variable in the container definition of your Deployment/StatefulSet:

spec:
  containers:
    - name: logstash
      env:
        - name: LS_JAVA_OPTS
          value: "-Xmx2g -Xms2g"

Logging configurationedit

By default, we use the log4j2.properties from the logstash docker image, that will log to stdout only. To change the log level, to use debug logging, use the log.level option in logstash.yml

You can apply temporary logging changes using the Logging APIs. If you require broader changes that persist across container restarts, you need to create a full and correct log4j2.properties file, and ensure that it is visible to the Logstash container.

This example uses a configMap and the base log4j2.properties file from the Docker container, adding debug logging for elasticsearch output plugins:

apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-log4j
  labels:
    app: logstash-demo
data:
  log4j2.properties: |
    status = error
    name = LogstashPropertiesConfig

    appender.console.type = Console
    appender.console.name = plain_console
    appender.console.layout.type = PatternLayout
    appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c]%notEmpty{[%X{pipeline.id}]}%notEmpty{[%X{plugin.id}]} %m%n

    appender.json_console.type = Console
    appender.json_console.name = json_console
    appender.json_console.layout.type = JSONLayout
    appender.json_console.layout.compact = true
    appender.json_console.layout.eventEol = true

    rootLogger.level = ${sys:ls.log.level}
    rootLogger.appenderRef.console.ref = ${sys:ls.log.format}_console
    logger.elasticsearchoutput.name = logstash.outputs.elasticsearch
    logger.elasticsearchoutput.level = debug

In your Deployment/StatefulSet, create the Volume:

volumes:
        - name: logstash-log4j
          configMap:
            name: logstash-log4j

Create the volumeMount in the container:

  volumeMounts:
    - name: logstash-log4j
      mountPath: /usr/share/logstash/config/log4j.properties
      subPath: log4j.properties

« Sample configuration files Setting up Logstash and Kubernetes »