WARNING: Version 5.5 of Packetbeat has passed its EOL date.
This documentation is no longer being maintained and may be removed. If you are running this version, we strongly advise you to upgrade. For the latest information, see the current release documentation.
Transaction Protocolsedit
The protocols
section of the packetbeat.yml
config file contains configuration options for each supported protocol,
including common options like enabled
, ports
, send_request
, send_response
, and options that are protocol-specific.
Currently, Packetbeat supports the following protocols:
- ICMP (v4 and v6)
- DNS
- HTTP
- AMQP 0.9.1
- Cassandra
- Mysql
- PostgreSQL
- Redis
- Thrift-RPC
- MongoDB
- Memcache
Example configuration:
packetbeat.protocols.icmp: enabled: true packetbeat.protocols.dns: ports: [53] packetbeat.protocols.http: ports: [80, 8080, 8000, 5000, 8002] packetbeat.protocols.amqp: ports: [5672] packetbeat.protocols.cassandra: ports: [9042] packetbeat.protocols.memcache: ports: [11211] packetbeat.protocols.mysql: ports: [3306] packetbeat.protocols.redis: ports: [6379] packetbeat.protocols.pgsql: ports: [5432] packetbeat.protocols.thrift: ports: [9090]
Common Protocol Optionsedit
The following options are available for all protocols:
enablededit
The enabled setting is a boolean setting to enable or disable protocols without having to comment out configuration sections. If set to false, the protocol is disabled.
The default value is true.
portsedit
Exception: For ICMP the option enabled
has to be used instead.
The ports where Packetbeat will look to capture traffic for specific protocols. Packetbeat installs a BPF filter based on the ports specified in this section. If a packet doesn’t match the filter, very little CPU is required to discard the packet. Packetbeat also uses the ports specified here to determine which parser to use for each packet.
send_requestedit
If this option is enabled, the raw message of the request (request
field) is
sent to Elasticsearch. The default is false. This option is useful when you want to
index the whole request. Note that for HTTP, the body is not included by
default, only the HTTP headers.
send_responseedit
If this option is enabled, the raw message of the response (response
field)
is sent to Elasticsearch. The default is false. This option is useful when you
want to index the whole response. Note that for HTTP, the body is not included
by default, only the HTTP headers.
transaction_timeoutedit
The per protocol transaction timeout. Expired transactions will no longer be correlated to incoming responses, but sent to Elasticsearch immediately.
ICMP Configuration Optionsedit
You can specify the following options in the icmp
section of the packetbeat.yml
config file:
enablededit
The ICMP protocol can be enabled/disabled via this option. The default is true.
If enabled Packetbeat will generate the following BPF filter: "icmp or icmp6"
.
DNS Configuration Optionsedit
The dns
section of the packetbeat.yml
config file specifies configuration options for the DNS protocol. The DNS protocol supports processing DNS messages on UDP. Here is a sample configuration section for DNS:
packetbeat.protocols.dns: ports: [53] include_authorities: true include_additionals: true
include_authoritiesedit
If this option is enabled, dns.authority fields (authority resource records) are added to DNS events. The default is false.
include_additionalsedit
If this option is enabled, dns.additionals fields (additional resource records) are added to DNS events. The default is false.
HTTP Configuration Optionsedit
The HTTP protocol has several specific configuration options. Here is a
sample configuration for the http
section of the packetbeat.yml
config file:
packetbeat.protocols.http: ports: [80, 8080, 8000, 5000, 8002] hide_keywords: ["pass", "password", "passwd"] send_headers: ["User-Agent", "Cookie", "Set-Cookie"] split_coookie: true real_ip_header: "X-Forwarded-For"
hide_keywordsedit
A list of query parameters that Packetbeat will automatically censor in
the transactions that it saves. The values associated with these parameters are replaced
by 'xxxxx'
. By default, no changes are made to the HTTP messages.
Packetbeat has this option because, unlike SQL traffic, which typically only contains the hashes of the passwords, HTTP traffic may contain sensitive data. To reduce security risks, you can configure this option to avoid sending the contents of certain HTTP POST parameters.
This option replaces query parameters from GET requests and top-level parameters from POST requests. If sensitive data is encoded inside a parameter that you don’t specify here, Packetbeat cannot censor it. Also, note that if you configure Packetbeat to save the raw request and response fields (see the send_request and the send_response options), sensitive data may be present in those fields.
redact_authorizationedit
When this option is enabled, Packetbeat obscures the value of
Authorization
and Proxy-Authorization
HTTP headers, and censors
those strings in the response.
You should set this option to true for transactions that use Basic Authentication because they may contain the base64 unencrypted username and password.
send_headersedit
A list of header names to capture and send to Elasticsearch. These
headers are placed under the headers
dictionary in the resulting JSON.
send_all_headersedit
Instead of sending a white list of headers to Elasticsearch, you can send all headers by setting this option to true. The default is false.
include_body_foredit
The list of content types for which Packetbeat exports the full HTTP payload. The HTTP body is available under
http.request.body
and http.response.body
for these Content-Types.
In addition, if send_response option is enabled, then the HTTP body is exported together with the HTTP
headers under response
and if
send_request enabled, then request
contains the entire HTTP message including the body.
In the following example, the HTML attachments of the HTTP responses are exported under the response
field and under
http.request.body
or http.response.body
:
packetbeat.protocols.http: ports: [80, 8080] send_response: true include_body_for: ["text/html"]
split_cookieedit
If the Cookie
or Set-Cookie
headers are sent, this option controls whether
they are split into individual values. For example, with this option set, an
HTTP response might result in the following JSON:
"response": { "code": 200, "headers": { "connection": "close", "content-language": "en", "content-type": "text/html; charset=utf-8", "date": "Fri, 21 Nov 2014 17:07:34 GMT", "server": "gunicorn/19.1.1", "set-cookie": { "csrftoken": "S9ZuJF8mvIMT5CL4T1Xqn32wkA6ZSeyf", "expires": "Fri, 20-Nov-2015 17:07:34 GMT", "max-age": "31449600", "path": "/" }, "vary": "Cookie, Accept-Language" }, "phrase": "OK" }
The default is false.
real_ip_headeredit
The header field to extract the real IP from. This setting is useful when
you want to capture traffic behind a reverse proxy, but you want to get the geo-location
information. If this header is present and contains a valid IP addresses, the
information is used for the real_ip
field.
max_message_sizeedit
If an individual HTTP message is larger than this setting (in bytes), it will be trimmed to this size. Unless this value is very small (<1.5K), Packetbeat is able to still correctly follow the transaction and create an event for it. The default is 10485760 (10 MB).
AMQP Configuration Optionsedit
The amqp
section of the packetbeat.yml
config file specifies configuration options for the AMQP 0.9.1
protocol. Here is a sample configuration:
packetbeat.protocols.amqp: ports: [5672] max_body_length: 1000 parse_headers: true parse_arguments: false hide_connection_information: true
max_body_lengthedit
The maximum size in bytes of the message displayed in the request or response fields. Messages that are bigger than the specified size are truncated. Use this option to avoid publishing huge messages when send_request or send_request is enabled. The default is 1000 bytes.
parse_headersedit
If set to true, Packetbeat parses the additional arguments specified in the headers field of a message. Those arguments are key-value pairs that specify information such as the content type of the message or the message priority. The default is true.
parse_argumentsedit
If set to true, Packetbeat parses the additional arguments specified in AMQP methods. Those arguments are key-value pairs specified by the user and can be of any length. The default is true.
hide_connection_informationedit
If set to false, the connection layer methods of the protocol are also displayed, such as the opening and closing of connections and channels by clients, or the quality of service negotiation. The default is true.
Cassandra Configuration Optionsedit
The following settings are specific to the Cassandra protocol. Here is a sample
configuration for the cassandra
section of the packetbeat.yml
config file:
packetbeat.protocols.cassandra: send_request_header: true send_response_header: true compressor: "snappy" ignored_ops: ["SUPPORTED","OPTIONS"]
send_request_headeredit
If this option is enabled, the raw message of the response (cassandra_request.request_headers
field)
is sent to Elasticsearch. The default is true. enable send_request
first before enable this option.
send_response_headeredit
If this option is enabled, the raw message of the response (cassandra_response.response_headers
field)
is included in published events. The default is true. enable send_response
first before enable this option.
ignored_opsedit
This option indicates which Operator/Operators captured will be ignored. currently support:
ERROR
,STARTUP
,READY
,AUTHENTICATE
,OPTIONS
,SUPPORTED
,
QUERY
,RESULT
,PREPARE
,EXECUTE
,REGISTER
,EVENT
,
BATCH
,AUTH_CHALLENGE
,AUTH_RESPONSE
,AUTH_SUCCESS
.
compressoredit
Configures the default compression algorithm being used to uncompress compressed frames by name. Currently only snappy
is can be configured.
By default no compressor is configured.
Memcache Configuration Optionsedit
The memcache
section of the packetbeat.yml
config file specifies configuration options for the memcache
protocol. Here is a sample configuration section for memcache:
packetbeat.protocols.memcache: ports: [11211] parseunknown: false maxvalues: 0 maxbytespervalue: 100 transaction_timeout: 200 udptransactiontimeout: 200
parseunknownedit
When this option is enabled, it forces the memcache text protocol parser to accept unknown commands.
The unknown commands MUST NOT contain a data part.
maxvaluesedit
The maximum number of values to store in the message (multi-get). All values will be base64 encoded.
The possible settings for this option are:
-
maxvalue: -1
, which stores all values (text based protocol multi-get) -
maxvalue: 0
, which stores no values (default) -
maxvalue: N
, which stores up to N values
maxbytespervalueedit
The maximum number of bytes to be copied for each value element.
Values will be base64 encoded, so the actual size in the JSON document will be 4 times the value that
you specify for maxbytespervalue
.
udptransactiontimeoutedit
The transaction timeout in milliseconds. The defaults is 10000 milliseconds.
Quiet messages in UDP binary protocol get responses only if there is an error.
The memcache protocol analyzer will wait for the number of milliseconds specified by
udptransactiontimeout
before publishing quiet messages. Non-quiet messages or
quiet requests with an error response are published immediately.
MySQL and PgSQL Configuration Optionsedit
You can specify the following options in the mysql
or pgsql
sections of the packetbeat.yml
config file:
max_rowsedit
The maximum number of rows from the SQL message to publish to Elasticsearch. The default is 10 rows.
max_row_lengthedit
The maximum length in bytes of a row from the SQL message to publish to Elasticsearch. The default is 1024 bytes.
Thrift Configuration Optionsedit
The Thrift protocol has several specific configuration options. Here is a
sample configuration for the thrift
section of the packetbeat.yml
config file:
packetbeat.protocols.thrift: transport_type: socket protocol_type: binary idl_files: ["tutorial.thrift", "shared.thrift"] string_max_size: 200 collection_max_size: 20 capture_reply: true obfuscate_strings: true drop_after_n_struct_fields: 100
transport_typeedit
The Thrift transport type. Currently this option accepts the values socket
for TSocket, which is the default Thrift transport, and framed
for the TFramed Thrift
transport. The default is socket
.
protocol_typeedit
The Thrift protocol type. Currently the only accepted value is binary
for
the TBinary protocol, which is the default Thrift protocol.
idl_filesedit
The Thrift interface description language (IDL) files for the service that Packetbeat is monitoring. Providing the IDL files is optional, because the Thrift messages contain enough information to decode them without having the IDL files. However, providing the IDL enables Packetbeat to include parameter and exception names.
string_max_sizeedit
The maximum length for strings in parameters or return values. If a string is longer than this value, the string is automatically truncated to this length. Packetbeat adds dots at the end of the string to mark that it was truncated. The default is 200.
collection_max_sizeedit
The maximum number of elements in a Thrift list, set, map, or structure. If a collection
has more elements than this value, Packetbeat captures only the
specified number of elements. Packetbeat adds a fictive last element ...
to the end
of the collection to mark that it was truncated. The default is 15.
capture_replyedit
If this option is set to false, Packetbeat decodes the method name from the reply and simply skips the rest of the response message. This setting can be useful for performance, disk usage, or data retention reasons. The default is true.
obfuscate_stringsedit
If this option is set to true, Packetbeat replaces all strings found in method parameters,
return codes, or exception structures with the "*"
string.
drop_after_n_struct_fieldsedit
The maximum number of fields that a structure can have before Packetbeat ignores the whole transaction. This is a memory protection mechanism (so that Packetbeat’s memory doesn’t grow indefinitely), so you would typically set this to a relatively high value. The default is 500.
MongoDB Configuration Optionsedit
The following settings are specific to the MongoDB protocol. Here is a sample
configuration for the mongodb
section of the packetbeat.yml
config file:
packetbeat.protocols.mongodb: send_request: true send_response: true max_docs: 0 max_doc_length: 0
The following two settings are useful for limiting the amount of data
Packetbeat indexes in the response
fields.
max_docsedit
The maximum number of documents from the response to index in the response
field. The
default is 10. You can set this to 0 to index an unlimited number of documents.
Packetbeat adds a [...]
line at the end to signify that there were additional documents
that weren’t saved because of this setting.
max_doc_lengthedit
The maximum number of characters in a single document indexed in the response
field. The default is 5000. You can set this to 0 to index an unlimited number
of characters per document.
If the document is trimmed because of this setting, Packetbeat adds the string ...
at the end of the document.
Note that limiting documents in this way means that they are no longer correctly formatted JSON objects.