Autodiscoveredit

When you run applications on containers, they become moving targets to the monitoring system. Autodiscover allows you to track them and adapt settings as changes happen. By defining configuration templates, the autodiscover subsystem can monitor services as they start running.

You define autodiscover settings in the filebeat.autodiscover section of the filebeat.yml config file. To enable autodiscover, you specify a list of providers.

Providersedit

Autodiscover providers work by watching for events on the system and translating those events into internal autodiscover events with a common format. When you configure the provider, you can use fields from the autodiscover event to set conditions that, when met, launch specific configurations.

On start, Filebeat will scan existing containers and launch the proper configs for them. Then it will watch for new start/stop events. This ensures you don’t need to worry about state, but only define your desired configs.

Dockeredit

The Docker autodiscover provider watches for Docker containers to start and stop. These are the available fields on every event:

  • host
  • port
  • docker.container.id
  • docker.container.image
  • docker.container.name
  • docker.container.labels

For example:

{
  "host": "10.4.15.9",
  "port": 6379,
  "docker": {
    "container": {
      "id": "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51"
      "name": "redis",
      "image": "redis:3.2.11",
      "labels": {
        "io.kubernetes.pod.namespace": "default"
        ...
      }
    }
  }
}

You can define a set of configuration templates to be applied when the condition matches an event. Templates define a condition to match on autodiscover events, together with the list of configurations to launch when this condition happens.

Conditions match events from the provider. Providers use the same format for Conditions that processors use.

Configuration templates can contain variables from the autodiscover event. They can be accessed under the data namespace. For example, with the example event, "${data.port}" resolves to 6379.

Filebeat supports templates for inputs and modules.

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains:
              docker.container.image: redis
          config:
            - type: docker
              containers.ids:
                - "${data.docker.container.id}"
              exclude_lines: ["^\\s+[\\-`('.|_]"]  # drop asciiart lines

This configuration launches a docker logs input for all containers running an image with redis in the name.

If you are using modules, you can override the default input and use the docker input instead.

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains:
              docker.container.image: redis
          config:
            - module: redis
              log:
                input:
                  type: docker
                  containers.ids:
                    - "${data.docker.container.id}"

When using autodiscover, you have to be careful when defining config templates, especially if they are reading from places holding information for several containers. For instance, under this file structure:

/mnt/logs/<container_id>/*.log

You can define a config template like this:

Wrong settings:

autodiscover.providers:
  - type: docker
    templates:
      - condition.contains:
          docker.container.image: nginx
        config:
          - type: log
            paths:
              - "/mnt/logs/*/*.log"

That would read all the files under the given path several times (one per nginx container). What you really want is to scope your template to the container that matched the autodiscover condition. Good settings:

autodiscover.providers:
  - type: docker
    templates:
      - condition.contains:
          docker.container.image: nginx
        config:
          - type: log
            paths:
              - "/mnt/logs/${data.docker.container.id}/*.log"
Kubernetesedit

The Kubernetes autodiscover provider watches for Kubernetes pods to start, update, and stop. These are the available fields on every event:

  • host
  • port
  • kubernetes.container.id
  • kubernetes.container.image
  • kubernetes.container.name
  • kubernetes.labels
  • kubernetes.namespace
  • kubernetes.node.name
  • kubernetes.pod.name

If the include_annotations config is added to the provider config, then the list of annotations present in the config are added to the event.

For example:

{
  "host": "172.17.0.21",
  "port": 9090,
  "kubernetes": {
    "container": {
      "id": "bb3a50625c01b16a88aa224779c39262a9ad14264c3034669a50cd9a90af1527",
      "image": "prom/prometheus",
      "name": "prometheus"
    },
    "labels": {
      "project": "prometheus",
      ...
    },
    "namespace": "default",
    "node": {
      "name": "minikube"
    },
    "pod": {
      "name": "prometheus-2657348378-k1pnh"
    }
  },
}

The configuration of templates and conditions is similar to that of the Docker provider. Configuration templates can contain variables from the autodiscover event. They can be accessed under data namespace.

The kubernetes autodiscover provider has the following configuration settings:

in_cluster
(Optional) Use in cluster settings for Kubernetes client, true by default.
host
(Optional) In case in_cluster is false, use this host to connect to Kubernetes API.
kube_config
(Optional) Use given config file as configuration for Kubernetes client.

Filebeat supports templates for inputs and modules.

filebeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - condition:
            equals:
              kubernetes.namespace: kube-system
          config:
            - type: docker
              containers.ids:
                - "${data.kubernetes.container.id}"
              exclude_lines: ["^\\s+[\\-`('.|_]"]  # drop asciiart lines

This configuration launches a docker logs input for all containers of pods running in the Kubernetes namespace kube-system.

If you are using modules, you can override the default input and use the docker input instead.

filebeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - condition:
            equals:
              kubernetes.container.image: "redis"
          config:
            - module: redis
              log:
                input:
                  type: docker
                  containers.ids:
                    - "${data.kubernetes.container.id}"