shipper section contains configuration options for the Beat and some
general settings that control its behaviour.
Here is an example configuration:
shipper: # The name of the shipper that publishes the network data. It can be used to group # all the transactions sent by a single shipper in the web interface. # If this options is not defined, the hostname is used. #name: # The tags of the shipper are included in their own field with each # transaction published. Tags make it easy to group servers by different # logical properties. tags: ["service-X", "web-tier"] # Uncomment the following if you want to ignore transactions created # by the server on which the shipper is installed. This option is useful # to remove duplicates if shippers are installed on multiple servers. ignore_outgoing: true # How often (in seconds) shippers are publishing their IPs to the topology map. # The default is 10 seconds. refresh_topology_freq: 10 # Expiration time (in seconds) of the IPs published by a shipper to the topology map. # All the IPs will be deleted afterwards. Note, that the value must be higher than # refresh_topology_freq. The default is 15 seconds. topology_expire: 15 # Configure local GeoIP database support. # If no paths are not configured geoip is disabled. #geoip: #paths: # - "/usr/share/GeoIP/GeoLiteCity.dat" # - "/usr/local/var/GeoIP/GeoLiteCity.dat"
You can specify the following options under the
The name of the Beat. If this option is empty, the
hostname of the server is
used. The name is included as the
shipper field in each published transaction. You can
use the name to group all transactions sent by a single Beat.
At startup, each Beat can publish its IP, port, and name to Elasticsearch. This information is stored in Elasticsearch as a network topology map that maps the IP and port of each Beat to the name that you specify here.
When a Beat receives a new request and response (called a transaction), the Beat can query
Elasticsearch to see if the network topology includes the IP and port of the source
and destination servers. If this information is available, the
client_server field in the
output is set to the name of the Beat running on the source server, and the
server field is set to the
name of the Beat running on the destination server.
To use the topology map in Elasticsearch, you must enable Elasticsearch as output and set the
save_topology option to true.
shipper: name: "my-shipper"
A list of tags that the Beat includes in the
tags field of each published
transaction. Tags make it easy to group servers by different logical properties.
For example, if you have a cluster of web servers, you can add the "webservers" tag
to the Beat on each server, and then use filters and queries in the
Kibana web interface to get visualisations for the whole group of servers.
shipper: tags: ["my-service", "hardware", "test"]
ignore_outgoing option is enabled, the Beat ignores all the
transactions initiated from the server running the Beat.
This is useful when two Beats publish the same transactions. Because one Beat
sees the transaction in its outgoing queue and the other sees it in its incoming
queue, you can end up with duplicate transactions. To remove the duplicates, you
can enable the
ignore_outgoing option on one of the servers.
For example, in the following scenario, you see a 3-server architecture where a Beat is installed on each server. t1 is the transaction exchanged between Server1 and Server2, and t2 is the transaction between Server2 and Server3.
By default, each transaction is indexed twice because Beat2
sees both transactions. So you would see the following published transactions
ignore_outgoing is false):
- Beat1: t1
- Beat2: t1 and t2
- Beat3: t2
To avoid duplicates, you can force your Beats to send only the incoming
transactions and ignore the transactions created by the local server. So you would
see the following published transactions (when
ignore_outgoing is true):
- Beat1: none
- Beat2: t1
- Beat3: t2
The refresh interval of the topology map in seconds. In other words, this setting specifies how often each Beat publishes its IP addresses to the topology map. The default is 10 seconds.
The expiration time for the topology in seconds. This is useful in case a Beat stops publishing its IP addresses. The IP addresses are removed automatically from the topology map after expiration. The default is 15 seconds.
The internal queue size for single events in the processing pipeline. The default value is 1000.
(DO NOT TOUCH) The internal queue size for bulk events in the processing pipeline. The default value is 0.
This configuration option is currently used by Packetbeat only.
The paths to search for GeoIP databases. The Beat loads the first installed GeoIP database that if finds. Then, for each transaction, the Beat exports the GeoIP location of the client.
The recommended values for geoip.paths are
Important: For GeoIP support to function correctly, the GeoLite City database is required.
Intro to Kibana
ELK for Logs & Metrics