Transaction Protocols Configurationedit

The protocols section of the packetbeat.yml config file contains configuration options for each supported protocol, including common options like enabled, ports, send_request, send_response, and options that are protocol-specific.

Currently, Packetbeat supports the following protocols:

  • ICMP (v4 and v6)
  • DNS
  • HTTP
  • AMQP 0.9.1
  • Cassandra
  • Mysql
  • PostgreSQL
  • Redis
  • Thrift-RPC
  • MongoDB
  • Memcache

Example configuration:

packetbeat.protocols.icmp:
  enabled: true

packetbeat.protocols.dns:
  ports: [53]

packetbeat.protocols.http:
  ports: [80, 8080, 8000, 5000, 8002]

packetbeat.protocols.amqp:
  ports: [5672]

packetbeat.protocols.cassandra:
  ports: [9042]

packetbeat.protocols.memcache:
  ports: [11211]

packetbeat.protocols.mysql:
  ports: [3306]

packetbeat.protocols.redis:
  ports: [6379]

packetbeat.protocols.pgsql:
  ports: [5432]

packetbeat.protocols.thrift:
  ports: [9090]

Common Protocol Optionsedit

The following options are available for all protocols:

enablededit

The enabled setting is a boolean setting to enable or disable protocols without having to comment out configuration sections. If set to false, the protocol is disabled.

The default value is true.

portsedit

Exception: For ICMP the option enabled has to be used instead.

The ports where Packetbeat will look to capture traffic for specific protocols. Packetbeat installs a BPF filter based on the ports specified in this section. If a packet doesn’t match the filter, very little CPU is required to discard the packet. Packetbeat also uses the ports specified here to determine which parser to use for each packet.

send_requestedit

If this option is enabled, the raw message of the request (request field) is sent to Elasticsearch. The default is false. This option is useful when you want to index the whole request. Note that for HTTP, the body is not included by default, only the HTTP headers.

send_responseedit

If this option is enabled, the raw message of the response (response field) is sent to Elasticsearch. The default is false. This option is useful when you want to index the whole response. Note that for HTTP, the body is not included by default, only the HTTP headers.

transaction_timeoutedit

The per protocol transaction timeout. Expired transactions will no longer be correlated to incoming responses, but sent to Elasticsearch immediately.

ICMP Configuration Optionsedit

You can specify the following options in the icmp section of the packetbeat.yml config file:

enablededit

The ICMP protocol can be enabled/disabled via this option. The default is true.

If enabled Packetbeat will generate the following BPF filter: "icmp or icmp6".

DNS Configuration Optionsedit

The dns section of the packetbeat.yml config file specifies configuration options for the DNS protocol. The DNS protocol supports processing DNS messages on UDP. Here is a sample configuration section for DNS:

packetbeat.protocols.dns:
  ports: [53]
  include_authorities: true
  include_additionals: true

include_authoritiesedit

If this option is enabled, dns.authority fields (authority resource records) are added to DNS events. The default is false.

include_additionalsedit

If this option is enabled, dns.additionals fields (additional resource records) are added to DNS events. The default is false.

HTTP Configuration Optionsedit

The HTTP protocol has several specific configuration options. Here is a sample configuration for the http section of the packetbeat.yml config file:

packetbeat.protocols.http:
  ports: [80, 8080, 8000, 5000, 8002]
  hide_keywords: ["pass", "password", "passwd"]
  send_headers: ["User-Agent", "Cookie", "Set-Cookie"]
  split_coookie: true
  real_ip_header: "X-Forwarded-For"

hide_keywordsedit

A list of query parameters that Packetbeat will automatically censor in the transactions that it saves. The values associated with these parameters are replaced by 'xxxxx'. By default, no changes are made to the HTTP messages.

Packetbeat has this option because, unlike SQL traffic, which typically only contains the hashes of the passwords, HTTP traffic may contain sensitive data. To reduce security risks, you can configure this option to avoid sending the contents of certain HTTP POST parameters.

This option replaces query parameters from GET requests and top-level parameters from POST requests. If sensitive data is encoded inside a parameter that you don’t specify here, Packetbeat cannot censor it. Also, note that if you configure Packetbeat to save the raw request and response fields (see the send_request and the send_response options), sensitive data may be present in those fields.

redact_authorizationedit

When this option is enabled, Packetbeat obscures the value of Authorization and Proxy-Authorization HTTP headers, and censors those strings in the response.

You should set this option to true for transactions that use Basic Authentication because they may contain the base64 unencrypted username and password.

send_headersedit

A list of header names to capture and send to Elasticsearch. These headers are placed under the headers dictionary in the resulting JSON.

send_all_headersedit

Instead of sending a white list of headers to Elasticsearch, you can send all headers by setting this option to true. The default is false.

include_body_foredit

The list of content types for which Packetbeat exports the full HTTP payload. The HTTP body is available under http.request.body and http.response.body for these Content-Types.

In addition, if send_response option is enabled, then the HTTP body is exported together with the HTTP headers under response and if send_request enabled, then request contains the entire HTTP message including the body.

In the following example, the HTML attachments of the HTTP responses are exported under the response field and under http.request.body or http.response.body:

packetbeat.protocols.http:
  ports: [80, 8080]
  send_response: true
  include_body_for: ["text/html"]

split_cookieedit

If the Cookie or Set-Cookie headers are sent, this option controls whether they are split into individual values. For example, with this option set, an HTTP response might result in the following JSON:

"response": {
  "code": 200,
  "headers": {
    "connection": "close",
    "content-language": "en",
    "content-type": "text/html; charset=utf-8",
    "date": "Fri, 21 Nov 2014 17:07:34 GMT",
    "server": "gunicorn/19.1.1",
    "set-cookie": { 
      "csrftoken": "S9ZuJF8mvIMT5CL4T1Xqn32wkA6ZSeyf",
      "expires": "Fri, 20-Nov-2015 17:07:34 GMT",
      "max-age": "31449600",
      "path": "/"
    },
    "vary": "Cookie, Accept-Language"
  },
  "phrase": "OK"
}

Note that set-cookie is a map containing the cookie names as keys.

The default is false.

real_ip_headeredit

The header field to extract the real IP from. This setting is useful when you want to capture traffic behind a reverse proxy, but you want to get the geo-location information. If this header is present and contains a valid IP addresses, the information is used for the real_ip field.

max_message_sizeedit

If an individual HTTP message is larger than this setting (in bytes), it will be trimmed to this size. Unless this value is very small (<1.5K), Packetbeat is able to still correctly follow the transaction and create an event for it. The default is 10485760 (10 MB).

AMQP Configuration Optionsedit

The amqp section of the packetbeat.yml config file specifies configuration options for the AMQP 0.9.1 protocol. Here is a sample configuration:

packetbeat.protocols.amqp:
  ports: [5672]
  max_body_length: 1000
  parse_headers: true
  parse_arguments: false
  hide_connection_information: true

max_body_lengthedit

The maximum size in bytes of the message displayed in the request or response fields. Messages that are bigger than the specified size are truncated. Use this option to avoid publishing huge messages when send_request or send_request is enabled. The default is 1000 bytes.

parse_headersedit

If set to true, Packetbeat parses the additional arguments specified in the headers field of a message. Those arguments are key-value pairs that specify information such as the content type of the message or the message priority. The default is true.

parse_argumentsedit

If set to true, Packetbeat parses the additional arguments specified in AMQP methods. Those arguments are key-value pairs specified by the user and can be of any length. The default is true.

hide_connection_informationedit

If set to false, the connection layer methods of the protocol are also displayed, such as the opening and closing of connections and channels by clients, or the quality of service negotiation. The default is true.

Cassandra Configuration Optionsedit

The following settings are specific to the Cassandra protocol. Here is a sample configuration for the cassandra section of the packetbeat.yml config file:

packetbeat.protocols.cassandra:
  send_request_header: true
  send_response_header: true
  compressor: "snappy"
  ignored_ops: ["SUPPORTED","OPTIONS"]

send_request_headeredit

If this option is enabled, the raw message of the response (cassandra_request.request_headers field) is sent to Elasticsearch. The default is true. enable send_request first before enable this option.

send_response_headeredit

If this option is enabled, the raw message of the response (cassandra_response.response_headers field) is included in published events. The default is true. enable send_response first before enable this option.

ignored_opsedit

This option indicates which Operator/Operators captured will be ignored. currently support: ERROR ,STARTUP ,READY ,AUTHENTICATE ,OPTIONS ,SUPPORTED , QUERY ,RESULT ,PREPARE ,EXECUTE ,REGISTER ,EVENT , BATCH ,AUTH_CHALLENGE,AUTH_RESPONSE ,AUTH_SUCCESS .

compressoredit

Configures the default compression algorithm being used to uncompress compressed frames by name. Currently only snappy is can be configured. By default no compressor is configured.

Memcache Configuration Optionsedit

The memcache section of the packetbeat.yml config file specifies configuration options for the memcache protocol. Here is a sample configuration section for memcache:

packetbeat.protocols.memcache:
  ports: [11211]
  parseunknown: false
  maxvalues: 0
  maxbytespervalue: 100
  transaction_timeout: 200
  udptransactiontimeout: 200

parseunknownedit

When this option is enabled, it forces the memcache text protocol parser to accept unknown commands.

The unknown commands MUST NOT contain a data part.

maxvaluesedit

The maximum number of values to store in the message (multi-get). All values will be base64 encoded.

The possible settings for this option are:

  • maxvalue: -1, which stores all values (text based protocol multi-get)
  • maxvalue: 0, which stores no values (default)
  • maxvalue: N, which stores up to N values

maxbytespervalueedit

The maximum number of bytes to be copied for each value element.

Values will be base64 encoded, so the actual size in the JSON document will be 4 times the value that you specify for maxbytespervalue.

udptransactiontimeoutedit

The transaction timeout in milliseconds. The defaults is 10000 milliseconds.

Quiet messages in UDP binary protocol get responses only if there is an error. The memcache protocol analyzer will wait for the number of milliseconds specified by udptransactiontimeout before publishing quiet messages. Non-quiet messages or quiet requests with an error response are published immediately.

MySQL and PgSQL Configuration Optionsedit

You can specify the following options in the mysql or pgsql sections of the packetbeat.yml config file:

max_rowsedit

The maximum number of rows from the SQL message to publish to Elasticsearch. The default is 10 rows.

max_row_lengthedit

The maximum length in bytes of a row from the SQL message to publish to Elasticsearch. The default is 1024 bytes.

Thrift Configuration Optionsedit

The Thrift protocol has several specific configuration options. Here is a sample configuration for the thrift section of the packetbeat.yml config file:

packetbeat.protocols.thrift:
  transport_type: socket
  protocol_type: binary
  idl_files: ["tutorial.thrift", "shared.thrift"]
  string_max_size: 200
  collection_max_size: 20
  capture_reply: true
  obfuscate_strings: true
  drop_after_n_struct_fields: 100

transport_typeedit

The Thrift transport type. Currently this option accepts the values socket for TSocket, which is the default Thrift transport, and framed for the TFramed Thrift transport. The default is socket.

protocol_typeedit

The Thrift protocol type. Currently the only accepted value is binary for the TBinary protocol, which is the default Thrift protocol.

idl_filesedit

The Thrift interface description language (IDL) files for the service that Packetbeat is monitoring. Providing the IDL files is optional, because the Thrift messages contain enough information to decode them without having the IDL files. However, providing the IDL enables Packetbeat to include parameter and exception names.

string_max_sizeedit

The maximum length for strings in parameters or return values. If a string is longer than this value, the string is automatically truncated to this length. Packetbeat adds dots at the end of the string to mark that it was truncated. The default is 200.

collection_max_sizeedit

The maximum number of elements in a Thrift list, set, map, or structure. If a collection has more elements than this value, Packetbeat captures only the specified number of elements. Packetbeat adds a fictive last element ... to the end of the collection to mark that it was truncated. The default is 15.

capture_replyedit

If this option is set to false, Packetbeat decodes the method name from the reply and simply skips the rest of the response message. This setting can be useful for performance, disk usage, or data retention reasons. The default is true.

obfuscate_stringsedit

If this option is set to true, Packetbeat replaces all strings found in method parameters, return codes, or exception structures with the "*" string.

drop_after_n_struct_fieldsedit

The maximum number of fields that a structure can have before Packetbeat ignores the whole transaction. This is a memory protection mechanism (so that Packetbeat’s memory doesn’t grow indefinitely), so you would typically set this to a relatively high value. The default is 500.

MongoDB Configuration Optionsedit

The following settings are specific to the MongoDB protocol. Here is a sample configuration for the mongodb section of the packetbeat.yml config file:

packetbeat.protocols.mongodb:
  send_request: true
  send_response: true
  max_docs: 0
  max_doc_length: 0

The following two settings are useful for limiting the amount of data Packetbeat indexes in the response fields.

max_docsedit

The maximum number of documents from the response to index in the response field. The default is 10. You can set this to 0 to index an unlimited number of documents.

Packetbeat adds a [...] line at the end to signify that there were additional documents that weren’t saved because of this setting.

max_doc_lengthedit

The maximum number of characters in a single document indexed in the response field. The default is 5000. You can set this to 0 to index an unlimited number of characters per document.

If the document is trimmed because of this setting, Packetbeat adds the string ... at the end of the document.

Note that limiting documents in this way means that they are no longer correctly formatted JSON objects.