Loading

Zeek Integration for Elastic

Version 4.1.0 (View all)
Subscription level
What's this?
Basic
Developed by
What's this?
Elastic
Ingestion method(s) File
Minimum Kibana version(s) 9.0.0
8.12.0
Note

This AI-assisted guide was validated by our engineers. You may need to adjust the steps to match your environment.

The Zeek integration for Elastic enables you to collect and analyze high-fidelity network security monitoring logs generated by Zeek's passive traffic analysis. This integration transforms raw network metadata into actionable insights within the Elastic Stack, allowing you to monitor network activity, detect threats, and investigate security incidents.

This integration has been tested against Zeek version 2.6.1, which was the primary version used during development. However, it's expected to work with all later versions, including the latest LTS releases.

For the integration to parse data correctly, you must configure Zeek with the json-logs policy.

This integration works by collecting logs that Zeek generates after performing passive network traffic analysis. Zeek produces various log files—such as conn.log, dns.log, and http.log—which contain detailed metadata about the network traffic it observes. You deploy an Elastic Agent on the host where Zeek is running or where its logs are stored. The agent is configured to read these JSON-formatted log files from the Zeek log directory and forward the data to your Elastic deployment, where it's normalized into the Elastic Common Schema (ECS) for analysis.

The Zeek integration collects a comprehensive range of network metadata by ingesting log files generated by the Zeek network security monitor. Each data stream corresponds to a specific Zeek log file and provides detailed insights into various network protocols and system activities.

The Zeek integration collects log messages of the following types:

  • capture_loss: Monitor packet loss at the sensor level to ensure data integrity.
  • connection: Metadata for TCP, UDP, and ICMP connections, including source and destination information, duration, and bytes transferred.
  • dce_rpc: Activity related to Distributed Computing Environment / Remote Procedure Calls (DCE/RPC).
  • dhcp: DHCP lease activity, including client identifiers and assigned IP addresses.
  • dnp3: Metadata for the DNP3 protocol used in SCADA and industrial control systems.
  • dns: Records of all DNS queries and responses observed on the network.
  • dpd: Logs for Dynamic Protocol Detection failures, helping you identify misconfigured or non-standard traffic.
  • files: Metadata for files transferred over the network, including hashes and file types.
  • ftp: Information about FTP sessions, including commands, responses, and file transfers.
  • http: Detailed HTTP request and response information, including URLs, methods, and headers.
  • intel: Matches from the Zeek Intelligence Framework when traffic aligns with your provided indicators.
  • irc: Activity from Internet Relay Chat (IRC) sessions.
  • kerberos: Metadata for Kerberos authentication requests and ticket assignments.
  • known_certs: Certificates observed on the network during TLS handshakes.
  • known_hosts: Addresses of hosts observed participating in network traffic.
  • known_services: Services identified as running on specific hosts and ports.
  • modbus: Modbus protocol activity for industrial control system monitoring.
  • mysql: Metadata for MySQL database queries and responses.
  • notice: Significant or unusual network events identified by Zeek's notice framework.
  • ntlm: Windows authentication activity using the NTLM protocol.
  • ntp: Network Time Protocol synchronization events and metadata.
  • ocsp: Online Certificate Status Protocol traffic for checking certificate revocation.
  • pe: Metadata for Portable Executable (PE) files transferred over the network.
  • radius: RADIUS authentication, authorization, and accounting activity.
  • rdp: Remote Desktop Protocol session metadata, including keyboard layout and cookie information.
  • rfb: Remote Frame Buffer (VNC) session activity.
  • signature: Traffic matching defined Zeek signatures for threat detection.
  • sip: Session Initiation Protocol (VoIP) traffic metadata.
  • smb_cmd: Information about SMB commands executed over the network.
  • smb_files: Metadata for files accessed or transferred using SMB.
  • smb_mapping: SMB share mapping activity and tree connects.
  • smtp: Email transaction metadata, including sender, receiver, and subject lines.
  • snmp: Simple Network Management Protocol traffic and command metadata.
  • socks: SOCKS proxy relay activity and destination information.
  • software: Versions and types of software identified running on network hosts.
  • ssh: Metadata for encrypted SSH sessions, including client and server versions.
  • ssl: TLS and SSL handshake information, including cipher suites and certificate details.
  • stats: Performance metrics for the Zeek process, such as memory usage and packet counts.
  • syslog: Syslog messages captured directly from network traffic.
  • traceroute: Detected traceroute attempts and network path information.
  • tunnel: Metadata for encapsulated or tunneled traffic, such as Teredo or GRE.
  • weird: Unexpected or malformed protocol behavior that might indicate network issues or attacks.
  • x509: Detailed metadata for X.509 certificates observed in encrypted traffic.

Integrating Zeek logs with Elastic provides you with a powerful solution for enhancing network visibility and security posture. You can use this integration to support the following use cases:

  • Real-time threat detection: You can use Elastic Security to alert on suspicious network patterns, such as data exfiltration using DNS or unauthorized SSH connections.
  • Incident response and forensics: You'll have access to a rich repository of protocol-specific metadata to investigate the scope and timeline of a security breach.
  • Network performance monitoring: You can analyze capture_loss and stats data to ensure your network monitoring infrastructure is operating efficiently.
  • Compliance and auditing: You can maintain long-term, searchable archives of network transactions, including file transfers and authentication events, to meet regulatory requirements.
  • Asset discovery: You can use known_hosts, known_services, and software data to automatically discover and inventory the devices and applications on your network.

Before you install the integration, you'll need to ensure your environment meets the following requirements.

You'll need to set up the following Elastic components:

  • You must install the Elastic Agent on the host where the Zeek logs are located and enroll it in Fleet.
  • You'll need a stable network connection between the Elastic Agent and the Elastic Stack (Elasticsearch and Kibana) to ship data and receive policy updates.
  • You must ensure the user account running the Elastic Agent has read permissions for the Zeek log directory, typically /opt/zeek/logs/current.

Your Zeek environment must meet these conditions:

  • The Elastic Agent must have read access to the directory where Zeek writes its logs, such as /opt/zeek/logs/current.

Elastic Agent must be installed on the host where Zeek is running or where its logs are stored. For detailed installation instructions, refer to the Elastic Agent installation guide. You can install only one Elastic Agent per host.

Elastic Agent is required to stream data from the log files and ship the data to Elastic, where the events will then be processed using the integration's ingest pipelines.

Before you can collect data, you must configure your Zeek instance to output logs in JSON format. This integration relies on JSON-formatted logs for accurate parsing and ingestion.

Follow these steps to enable JSON logging:

  1. Locate the site configuration file. Identify the local.zeek file for your site-specific configuration. Common paths include:
    • /opt/zeek/share/zeek/site/local.zeek (standard package install)
    • /usr/local/zeek/share/zeek/site/local.zeek (source install)
  2. Edit the configuration. Open the file with root privileges using a text editor.
  3. Enable JSON logging. Append the following line to the end of the file to force all log streams into JSON format:
    @load policy/tuning/json-logs.zeek
    		
  4. Save and exit. Save the changes to the file.
  5. Check the configuration. Use the Zeek control utility to verify that the scripts load correctly:
    sudo zeekctl check
    		
  6. Apply and restart. Deploy the new configuration and restart the Zeek processes:
    sudo zeekctl deploy
    		
  7. Verify JSON format. Check one of the current logs to ensure it's formatted as JSON. The output should start with a { character. For example:
    head -n 1 /opt/zeek/logs/current/conn.log
    		

For more information on setting up Zeek, refer to the following resources:

To set up the integration in Kibana:

  1. In Kibana, navigate to Management > Integrations.
  2. Search for Zeek and select the integration.
  3. Click Add Zeek.
  4. Assign the integration to an Elastic Agent policy.
  5. Configure the Base Path where Zeek logs are stored. Typical paths include:
    • /var/log/bro/current
    • /opt/zeek/logs/current
    • /usr/local/var/spool/zeek
  6. Configure the filename variables for the data streams you wish to collect. By default, the integration is configured to collect the following:
    • Filename of capture loss log file: ['capture_loss.log']
    • Filename of connection log: ['conn.log']
    • Filename of dce_rpc log file: ['dce_rpc.log']
    • Filename of dhcp log file: ['dhcp.log']
    • Filename of dnp3 log file: ['dnp3.log']
    • Filename of dns log file: ['dns.log']
    • Filename of the dpd log file: ['dpd.log']
    • Filename of the files log file: ['files.log']
    • Filename of ftp log file: ['ftp.log']
    • Filename of http log file: ['http.log']
    • Filename of intel log file: ['intel.log']
    • Filename of irc log file: ['irc.log']
    • Filename of kerberos log file: ['kerberos.log']
    • Filename of modbus log file: ['modbus.log']
    • Filename of mysql log file: ['mysql.log']
    • Filename of notice log file: ['notice.log']
    • Filename of ntlm log file: ['ntlm.log']
    • Filename of ntp log: ['ntp.log']
    • Filename of ocsp log file: ['ocsp.log']
    • Filename of pe log file: ['pe.log']
    • Filename of radius log file: ['radius.log']
    • Filename of rdp log file: ['rdp.log']
    • Filename of rfb log file: ['rfb.log']
    • Filename of signature log: ['signature.log']
    • Filename of sip log file: ['sip.log']
    • Filename of smb_cmd log file: ['smb_cmd.log']
    • Filename of smb_files log file: ['smb_files.log']
    • Filename of smb_mapping log file: ['smb_mapping.log']
    • Filename of smtp log file: ['smtp.log']
    • Filename of snmp log file: ['snmp.log']
    • Filename of socks log file: ['socks.log']
    • Filename of software log: ['software.log']
    • Filename of ssh log file: ['ssh.log']
    • Filename of ssl log file: ['ssl.log']
    • Filename of stats log file: ['stats.log']
    • Filename of syslog log file: ['syslog.log']
    • Filename of traceroute log file: ['traceroute.log']
    • Filename of tunnel log file: ['tunnel.log']
    • Filename of weird log file: ['weird.log']
    • Filename of x509 log file: ['x509.log']
    • Filename of Known Certs log: ['known_certs.log']
    • Filename of Known Services log: ['known_services.log']
    • Filename of Known Hosts log: ['known_hosts.log']
  7. Configure optional advanced settings:
    • Preserve original event: If enabled, this stores a raw copy of the original log in the event.original field. Use this for auditing or legal compliance, but be aware it increases storage requirements.
    • Tags: Add custom tags to your events for easier filtering (e.g., ['forwarded', 'zeek-sensor-1']).
    • Processors: Add Elastic Agent processors to filter or enhance data before ingestion.
  8. Click Save and continue or Add Integration.

Follow these steps to verify that the integration is working correctly and data is flowing into Elasticsearch:

  1. In Kibana, navigate to Fleet > Agents and verify that the Elastic Agent is online and has a healthy status.
  2. Generate network traffic or activity on the host monitored by Zeek to trigger log generation.
  3. Navigate to Discover.
  4. Select the logs-* data view.
  5. Filter the results using a KQL query such as data_stream.dataset : "zeek.connection".
  6. Verify that events are appearing with recent timestamps.
  7. Check that protocol-specific fields are populated correctly (for example, source.ip, destination.ip, zeek.connection.id).
  8. Navigate to Dashboards and search for "Zeek" to view the pre-built dashboards, such as the Network Traffic or Protocol Overview dashboards, and confirm they are populated with data.

For help with Elastic ingest tools, check Common problems.

You can resolve most issues by checking the following common configuration points:

  • Logs are not visible in Kibana: Verify that Zeek is actively monitoring traffic and writing logs to the expected location. Check that the Base Path in the integration settings correctly points to your Zeek log directory, such as /opt/zeek/logs/current/ or /var/log/bro/current/.
  • Zeek logs are not in JSON format: This integration requires logs to be in JSON format. If your logs are tab-separated, ensure you have added @load policy/tuning/json-logs.zeek to your local.zeek configuration file and applied the changes by running sudo zeekctl deploy.
  • Permission denied: The user account running the Elastic Agent must have read permissions for the Zeek log directory. Check directory permissions with ls -ld /opt/zeek/logs/current/ to ensure the agent can access the files.
  • Incomplete data collection: If only some logs are appearing, check the specific filename settings for each data stream in the integration configuration. If you have customized your Zeek configuration to use non-standard filenames, you must update the corresponding data stream settings to match the actual files, such as conn.log or http.log.
  • Data parsing errors: If you use custom Zeek scripts that modify standard log fields, the ingest pipeline might fail to parse the data correctly. Verify that your local.zeek configuration does not conflict with standard JSON output schemas.
  • Zeek service state: Ensure that Zeek is running and that the current directory is being updated. You can check the status of your Zeek nodes using the sudo zeekctl status command.

For more information and detailed configuration guides, refer to the official Zeek documentation:

For more information on architectures that can be used for scaling this integration, check the Ingest Architectures documentation.

In a high-volume environment, you may need to scale your Zeek deployment into a cluster. In this scenario, you can aggregate logs on a dedicated log collector host where the Elastic Agent is installed. This setup provides the following benefits:

  • Centralizes log collection to a single point for easier management.
  • Reduces the processing and resource load on your individual Zeek workers.

This integration supports the following inputs: These inputs can be used with this integration:

You can find more information about Zeek logs and configuration in the following resources:

The Zeek integration collects the following data streams:

The capture_loss data stream provides events from Zeek capture_loss logs, which track packet loss statistics during network traffic capture.

The connection data stream provides events from Zeek connection logs, which contain details about IP connections including source and destination information, protocol, and duration.

The dce_rpc data stream provides events from Zeek dce_rpc logs, documenting Distributed Computing Environment/Remote Procedure Call activity.

The dhcp data stream provides events from Zeek dhcp logs, containing details about DHCP lease activity and client information.

The dnp3 data stream provides events from Zeek dnp3 logs, documenting Distributed Network Protocol v3 traffic often used in SCADA systems.

The dns data stream provides events from Zeek dns logs, capturing DNS queries and responses including lookups and error codes.

The dpd data stream provides events from Zeek dpd logs, which track Dynamic Protocol Detection results and protocol mismatches.

The files data stream provides events from Zeek files logs, tracking files transferred over the network including hashes and file metadata.

The ftp data stream provides events from Zeek ftp logs, documenting FTP commands, responses, and file transfer metadata.

The http data stream provides events from Zeek http logs, capturing HTTP requests, responses, headers, and related metadata.

The intel data stream provides events from Zeek intel logs, recording occurrences where network activity matches entries in the Zeek Intelligence Framework.

The irc data stream provides events from Zeek irc logs, documenting Internet Relay Chat activity including commands and messages.

The kerberos data stream provides events from Zeek kerberos logs, capturing Kerberos authentication requests and ticket information.

The known_certs data stream provides events from Zeek Known Certs logs, which track SSL/TLS certificates observed on the network.

The known_hosts data stream provides events from Zeek Known Hosts logs, tracking IP addresses that have been observed acting as hosts.

The known_services data stream provides events from Zeek Known Services logs, documenting services (IP and port pairs) observed on the network.

The modbus data stream provides events from Zeek modbus logs, capturing Modbus protocol activity used in industrial control systems.

The mysql data stream provides events from Zeek mysql logs, documenting MySQL database connection attempts and command execution.

The notice data stream provides events from Zeek notice logs, which record interesting or unusual activity identified by Zeek scripts.

The ntlm data stream provides events from Zeek ntlm logs, documenting Windows NTLM authentication activity.

The ntp data stream provides events from Zeek ntp logs, capturing Network Time Protocol queries and responses.

The ocsp data stream provides events from Zeek ocsp logs, documenting Online Certificate Status Protocol requests and responses.

The pe data stream provides events from Zeek pe logs, documenting metadata for Portable Executable files observed on the network.

The radius data stream provides events from Zeek radius logs, capturing RADIUS authentication and accounting activity.

The rdp data stream provides events from Zeek rdp logs, documenting Remote Desktop Protocol connection details and capabilities.

The rfb data stream provides events from Zeek rfb logs, capturing Remote Frame Buffer activity often used by VNC.

The signature data stream provides events from Zeek signature logs, recording matches against Zeek's signature-based detection engine.

The sip data stream provides events from Zeek sip logs, capturing Session Initiation Protocol activity for VoIP and multimedia sessions.

The smb_cmd data stream provides events from Zeek smb_cmd logs, documenting individual commands within SMB sessions.

The smb_files data stream provides events from Zeek smb_files logs, tracking files accessed or transferred using SMB.

The smb_mapping data stream provides events from Zeek smb_mapping logs, documenting share mapping and tree connects in SMB.

The smtp data stream provides events from Zeek smtp logs, capturing email transaction details including sender, recipient, and path.

The snmp data stream provides events from Zeek snmp logs, documenting Simple Network Management Protocol activity.

The socks data stream provides events from Zeek socks logs, capturing SOCKS proxy connection details.

The software data stream provides events from Zeek software logs, documenting software versions and applications identified in network traffic.

The ssh data stream provides events from Zeek ssh logs, capturing SSH connection metadata including client/server strings and authentication status.

The ssl data stream provides events from Zeek ssl logs, documenting SSL/TLS handshake details, cipher suites, and certificate information.

The stats data stream provides events from Zeek stats logs, which record periodic memory and packet processing statistics for the Zeek process.

The syslog data stream provides events from Zeek syslog logs, capturing syslog messages observed over the network.

The traceroute data stream provides events from Zeek traceroute logs, documenting traceroute attempts identified in ICMP or UDP traffic.

The tunnel data stream provides events from Zeek tunnel logs, documenting encapsulated traffic such as Teredo, GTP, or GRE.

The weird data stream provides events from Zeek weird logs, recording protocol violations and other abnormal network behavior.

The x509 data stream provides events from Zeek x509 logs, containing detailed information about X.509 certificates seen in network traffic.

This integration includes one or more Kibana dashboards that visualizes the data collected by the integration. The screenshots below illustrate how the ingested data is displayed.