Filebeat provides configuration options that you can use to fine tune load balancing when sending events to multiple hosts.
To enable load balancing, you specify
loadbalance: true when you configure
the output. For example:
output.logstash: hosts: ["localhost:5044", "localhost:5045"] loadbalance: true
loadbalance option is available for Redis, Logstash, and Elasticsearch
outputs. The Kafka output handles load balancing internally.
Filebeat can send events in a few different modes:
Send events to one host after another:
By default, when you configure Filebeat to send events to multiple hosts (
loadbalance: true), Filebeat will send the events to one host after another. This mode requires the least memory and CPU usage. This is not true load balancing in the sense that Filebeat doesn’t account for the load being processed by each host.
The load balancer also supports multiple workers per host. The default is
worker: 1. If you increase the number of workers, additional network connections will be used. The total number of workers participating in load balancing is
number of hosts * workers.
filebeat.prospectors: - input_type: log paths: - /var/log/*.log output.logstash: hosts: ["localhost:5044", "localhost:5045"] loadbalance: true worker: 2
In this example, there are 4 workers participating in load balancing.
Send events to
Nhosts in lock-step:
You can configure Filebeat to send events to
Nhosts in lock-step by setting
spool_size = N * bulk_max_size. In lock-step mode, the batch collected by the spooler is split up into smaller batches of size
bulk_max_size. These smaller batches are load balanced between available connections. Filebeat waits for all sub-batches to be published before it retrieves another batch from the spooler.
This mode requires more memory and CPU usage than the previous mode.
filebeat.prospectors: - input_type: log paths: - /var/log/*.log filebeat.spool_size: 4096 output.logstash: hosts: ["localhost:5044", "localhost:5045"] loadbalance: true bulk_max_size: 2048