Load Balancing

Filebeat provides configuration options that you can use to fine tune load balancing when sending events to multiple hosts.

To enable load balancing, you specify loadbalance: true when you configure the output. For example:

output.logstash:
  hosts: ["localhost:5044", "localhost:5045"]
  loadbalance: true

The loadbalance option is available for Redis, Logstash, and Elasticsearch outputs. The Kafka output handles load balancing internally.

Filebeat can send events in a few different modes:

  • Send events to one host after another:

    By default, when you configure Filebeat to send events to multiple hosts (loadbalance: true), Filebeat will send the events to one host after another. This mode requires the least memory and CPU usage. This is not true load balancing in the sense that Filebeat doesn’t account for the load being processed by each host.

    The load balancer also supports multiple workers per host. The default is worker: 1. If you increase the number of workers, additional network connections will be used. The total number of workers participating in load balancing is number of hosts * workers.

    Example:

    filebeat.prospectors:
    - input_type: log
      paths:
        - /var/log/*.log
    output.logstash:
      hosts: ["localhost:5044", "localhost:5045"]
      loadbalance: true
      worker: 2

    In this example, there are 4 workers participating in load balancing.

  • Send events to N hosts in lock-step:

    You can configure Filebeat to send events to N hosts in lock-step by setting spool_size = N * bulk_max_size. In lock-step mode, the batch collected by the spooler is split up into smaller batches of size bulk_max_size. These smaller batches are load balanced between available connections. Filebeat waits for all sub-batches to be published before it retrieves another batch from the spooler.

    This mode requires more memory and CPU usage than the previous mode.

    Example:

    filebeat.prospectors:
    - input_type: log
      paths:
        - /var/log/*.log
    filebeat.spool_size: 4096
    output.logstash:
      hosts: ["localhost:5044", "localhost:5045"]
      loadbalance: true
      bulk_max_size: 2048