PingDirectory Integration for Elastic

Version 0.1.0 (View all)
Managed integration release status
Subscription level
What's this?
Basic
Developed by
What's this?
Elastic
Ingestion method(s) API, File, Network Protocol
Minimum Kibana version(s) 9.4.6
8.19.21
Important - Managed integrations on Elastic Cloud Hosted

You can use this integration as an Elastic Managed integration on Elastic Cloud Hosted deployments running this version or later.

The PingDirectory integration v0.1.0 is in technical preview

To use pre-release integrations, go to the Integrations page in Kibana, scroll down, and toggle on the Display beta integrations option.

Note

When using this integration as an Elastic Managed integration, only the following data streams are available: Group and User. These data streams use inputs compatible with the Elastic Managed deployment mode. Other data streams (Audit, HTTP Access, and LDAP Access) require Filestream or UDP inputs, which are not supported in the Elastic Managed deployment mode. To collect data from these data streams, use Elastic Agent.

PingDirectory is an enterprise-grade LDAP directory server and identity data store. It provides high-performance, scalable directory services for managing user identities, credentials, and access control across hybrid environments — combining authentication, authorization, and directory data management into a unified platform for critical identity infrastructure.

This integration collects data from PingDirectory log files using the Elastic Agent filestream input, or over the network using the Elastic Agent UDP input. It also collects user identity and group membership data from PingDirectory via the SCIM v2 API using the Elastic Agent CEL input. It provides visibility into HTTP access activity, directory change activity, identity lifecycle events, administrative operations, LDAP connection activity, and user accounts and groups managed within your PingDirectory environment.

The PingDirectory integration is compatible with PingDirectory version 11.0.0.2 and above.

This integration collects PingDirectory data from log files (or syslog over UDP) and from the SCIM v2 API for user identity and group membership records.

The integration uses the Elastic Agent filestream input to collect log files generated by PingDirectory on a configurable path, or the UDP input to receive logs forwarded by a PingDirectory Syslog Text Access Log Publisher. Each collection cycle:

  1. Reads log entries from the configured PingDirectory log files (HTTP detailed access log http-detailed-access.log, Audit log audit.log, and LDAP Access log access.log), or receives LDAP Access logs over UDP when syslog forwarding is used.
  2. Parses each log line to extract fields — for HTTP access, a grok pattern yields the timestamp, HTTP method, URL, status code, client IP, request headers, and response headers; for audit, a grok pattern yields the header timestamp and timezone, and the remaining semicolon-delimited header pairs (connection ID, operation ID, instance name, thread ID, client IP, requester DN) are extracted as key/value pairs; for LDAP access, a grok pattern extracts fields such as timestamp, connection ID, operation type, instance name, client and server IP/port, requester DN, and result code.
  3. For audit records, extracts LDIF body fields from the multiline audit record, including the target DN, the change type, changed ds-cfg-* configuration attributes, objectClass values, modifiers name, creators name, and entry UUID. Attribute-level detail for non-configuration entries, such as which user or group attributes a modify operation changed, is not extracted. For LDAP access, extracts operation-specific details including request type, search filters, response time, and result codes from the access log record.
  4. Emits each log entry as an individual event for ingestion and enrichment via the built-in ingest pipeline.

The integration also uses the Elastic Agent CEL input to collect user and group data from the PingDirectory SCIM v2 API. Each collection cycle:

  1. Authenticates against the PingDirectory REST API (/directory/v1/authenticate) using the configured bind_dn and password to obtain a short-lived access token.
  2. Paginates through all users via the SCIM v2 Users endpoint (/scim/v2/Users) and all groups via the SCIM v2 Groups endpoint (/scim/v2/Groups) using the configured batch size.
  3. Emits each user or group record as an individual event for ingestion and enrichment via the built-in ingest pipeline.
  4. Caches the access token across collection cycles and re-authenticates automatically when the token expires.

The PingDirectory integration collects the following types of data:

Data stream Description
http_access HTTP access logs generated by PingDirectory, including request methods, URLs, client information, response status codes, response sizes, request/response headers, redirect destinations, and request processing times for monitoring web traffic and troubleshooting HTTP-based services.
audit Audit logs generated by PingDirectory, including directory change operations (add, modify, delete, rename), target entry DNs, requester identities, client IP addresses, operation metadata, and timestamps for monitoring identity lifecycle events and administrative changes to directory entries.
ldap_access LDAP access logs generated by PingDirectory, including connection and disconnection events, LDAP requests and results, security negotiation details, client and server addresses, requester identities, and response times for monitoring LDAP client activity across your directory environment.
user User identity records retrieved from the PingDirectory SCIM v2 API (/scim/v2/Users), including usernames, email addresses, display names, names, titles, user types, and account metadata.
group Group membership records retrieved from the PingDirectory SCIM v2 API (/scim/v2/Groups), including group names, display names, members, and group metadata.
  • HTTP traffic monitoring: Track incoming HTTP requests, response status codes, request methods, and endpoint activity to gain visibility into web traffic patterns and service usage.

  • Identity change monitoring: Track add, modify, delete, and rename operations on directory entries to gain visibility into identity lifecycle events and administrative changes across your directory environment.

  • Operational troubleshooting and access analysis: Analyze failed requests, redirects, and response times for HTTP services, and investigate who made directory changes, from which client IP, using which application, and at what time — to troubleshoot issues, audit administrator activity, and support compliance reporting.

  • LDAP connection monitoring: Track connect, disconnect, and security negotiation events to gain visibility into client connection activity and TLS/cipher usage across your directory environment.

  • LDAP operational troubleshooting and access analysis: Analyze who performed which LDAP operations, from which client IP, with what result code, and at what response time to investigate failed binds, slow searches, and unauthorized access attempts.

  • User inventory and identity visibility: Track all user accounts managed in PingDirectory, including their attributes, email addresses, and account types.

  • Identity data enrichment: Correlate PingDirectory user records with other security and operational data in Elastic for unified identity context.

  • Group inventory and membership visibility: Track all groups managed in PingDirectory, including their members, display names, and group types.

This integration installs Elastic latest transforms. For more details, check the Transform setup and requirements.

  • PingDirectory deployment: An active PingDirectory server configured to generate HTTP Access, Audit, and LDAP Access logs, with the SCIM v2 API enabled and accessible when collecting user identity and group membership data.
  • HTTP Access logging enabled: The HTTP Access Logger must be enabled and configured to record HTTP request and response activity.
  • Audit logging enabled: The File-Based Audit Logger must be enabled and configured to record directory change operations.
  • LDAP Access logging enabled: The File-Based Access Logger must be enabled to record LDAP operations, or a Syslog Text Access Log Publisher must be configured to forward access logs over UDP. To configure the publisher: in the PingDirectory Administrative Console, go to Configuration → Log Publishers → New Log Publisher → Syslog Text Access Log Publisher, create a Syslog External Server with the Elastic Agent host and UDP port (must match the integration's Listen Port, default 9602), then enable and save the publisher.
  • Log access: Elastic Agent must have access to the HTTP Access, Audit, and LDAP Access log files on the PingDirectory host, or network access to receive the forwarded syslog UDP traffic.
  • Service account credentials: A bind DN (bind_dn) and password with sufficient permissions to authenticate via /directory/v1/authenticate and read users from /scim/v2/Users and groups from /scim/v2/Groups.
  • Network access: Elastic Agent must be able to reach the PingDirectory HTTPS endpoint when collecting user identity and group membership data.
  • SSL certificate: If PingDirectory uses a self-signed certificate, the certificate must be trusted by the Elastic Agent host.
  • Elastic Agent: Version 8.19+ or 9.1.0+ with Fleet enrollment.

This integration supports both Elastic Agentless-based and Agent-based installations.

Elastic Agent must be installed. For more details, check the Elastic Agent installation instructions. You can install only one Elastic Agent per host.

Elastic Agent is required to stream data from the syslog or log file receiver, poll the PingDirectory SCIM v2 API, and ship the data to Elastic, where the events will then be processed via the integration's ingest pipelines.

Agentless deployments are only supported in Elastic Serverless and Elastic Cloud environments. Agentless deployments provide a means to ingest data while avoiding the orchestration, management, and maintenance needs associated with standard ingest infrastructure. Using an agentless deployment makes manual agent deployment unnecessary, allowing you to focus on your data instead of the agent that collects it.

For more information, refer to Agentless integrations and Agentless integrations FAQ

Before configuring the integration for user identity and group membership collection, ensure that a PingDirectory account is available for API authentication and retrieval.

  1. Create or identify a service account in PingDirectory.

  2. Grant the account permission to:

    • Authenticate through the /directory/v1/authenticate endpoint.
    • Read user records through the /scim/v2/Users endpoint.
    • Read group records through the /scim/v2/Groups endpoint.
  3. Record the account's bind DN and password. These values are required when configuring the integration in Kibana.

  4. Verify that the SCIM v2 API is enabled and accessible from the Elastic Agent host.

  5. If HTTPS is configured with a self-signed certificate, export the certificate so it can be trusted by Elastic Agent.

Refer to the PingDirectory documentation for configuring SCIM 2.0 and authentication requirements.

  1. In Kibana, navigate to Fleet → Integrations and search for PingDirectory.

  2. Click Add PingDirectory.

  3. Configure the integration settings:

    • HTTP Access Logs Path: Specify the path to the PingDirectory HTTP Access log file(s), for example:

      /opt/pingdirectory/logs/http-access*
      		
    • Audit Logs Path: Specify the path to the PingDirectory Audit log file(s), for example:

      /opt/pingdirectory/logs/audit*
      		
    • To collect LDAP Access from a local log file, enable the filestream input and specify the Paths to the PingDirectory LDAP Access log file(s), for example: /opt/pingdirectory/logs/access*

    • To receive LDAP Access logs forwarded over the network, enable the UDP input and set the Listen Address and Listen Port to match the Syslog External Server configured on PingDirectory.

    • URL: The base URL of your PingDirectory instance, for example: https://pingdirectory.example.com:2443

    • Bind DN: The distinguished name used to authenticate, for example: cn=admin,dc=example,dc=com

    • Password: The password for the bind DN.

    • Batch Size: Number of user or group records to retrieve per API page (default: 500).

    • Interval: How frequently to poll for new data (default: 24h).

    • Configure any optional collection settings as required.

  4. If using a self-signed SSL certificate for SCIM collection, configure the SSL settings under Advanced options.

  5. Select Save and continue to save the integration.

  6. Add the integration to an existing Agent policy or create a new one.

  7. Verify that HTTP Access, Audit, LDAP Access, user identity, and group membership events are being ingested into Elasticsearch.

  1. In the top search bar in Kibana, search for Dashboards.
  2. In the search bar, type PingDirectory.
  3. Open the [Logs PingDirectory] HTTP Access dashboard and verify that the visualizations are populated with HTTP access data, including request methods, response status categories, and accessed endpoints.
  4. Open the [Logs PingDirectory] Audit dashboard and verify that the visualizations are populated with audit data, including change types, requester identities, target entries, and client IP addresses.
  5. Open the [Logs PingDirectory] LDAP Access dashboard and verify that the visualizations are populated with LDAP access data, including connection events, requester identities, result codes, and client IP addresses.
  6. Open the [Logs PingDirectory] User dashboard and verify that the visualizations are populated with user identity data, including usernames, email addresses, and account types.
  7. Open the [Logs PingDirectory] Group dashboard and verify that the visualizations are populated with group membership data, including group names, members, and membership counts.
  1. In the top search bar in Kibana, search for Transforms.
  2. Select the Data / Transforms from the search results.
  3. In the search bar, type PingDirectory.
  4. All transforms from the search results should indicate Healthy under the Health column.

An Elastic Transform is created for the user data stream to provide a view of the most recent user identity data. Use the destination alias from the table below to access the latest data for dashboards, rules, and other Elastic features.

Destination indices are aliased to logs-ping_directory_latest.<data_stream_name>.

Source Data Stream Destination Index Pattern Destination Alias
logs-ping_directory.user-* logs-ping_directory_latest.dest_user-1 logs-ping_directory_latest.user

For more information on architectures that can be used for scaling this integration, check the Ingest Architectures documentation.

  • No data collected: Verify that the PingDirectory HTTP Access Logger, File-Based Audit Logger, and File-Based Access Logger are enabled and generating log entries, or that the Syslog Text Access Log Publisher is enabled and pointed at the correct Elastic Agent host and port. Confirm that the configured log file paths or UDP listener match the locations of the HTTP access, audit, and LDAP access logs. For user identity and group membership data, verify that the PingDirectory SCIM v2 API is enabled and reachable from the Elastic Agent host, and that the configured URL, bind DN, and password are correct.
  • Authentication failures: Ensure the bind DN has permission to authenticate via /directory/v1/authenticate and read from /scim/v2/Users and /scim/v2/Groups.
  • SSL certificate errors: If PingDirectory uses a self-signed certificate, extract the certificate and configure it under the SSL settings of the integration, or add it to the Elastic Agent's trusted certificate store.
  • Token expiry: The integration automatically re-authenticates when the access token expires. If repeated auth failures occur, verify the bind DN password has not changed.

For help with Elastic ingest tools, check Common problems.

The http_access data stream provides HTTP access logs from PingDirectory.

The audit data stream provides audit logs from PingDirectory.

The ldap_access data stream provides LDAP access logs from ping_directory.

The user data stream provides user identity records collected from PingDirectory.

The group data stream provides group membership records collected from PingDirectory.

These inputs can be used with this integration:

To collect logs via API endpoint, configure the following parameters:

  • API Endpoint URL
  • API credentials (tokens, keys, or username/password)
  • Request interval (how often to fetch data)

These PingDirectory REST API endpoints are used by this integration:

Endpoint Method Data stream Description
/directory/v1/authenticate POST user, group Authenticate with bind DN and password to obtain an access token
/scim/v2/Users GET user Retrieve paginated user identity records
/scim/v2/Groups GET group Retrieve paginated group membership records

To facilitate user identity data, the source data stream-backed index .ds-logs-ping_directory.user-* is allowed to contain duplicates from each polling interval. The ILM policy logs-ping_directory.user-default_policy is added to this source index so it doesn't lead to unbounded growth. This means that in this source index data will be deleted after 30 days from ingested date.

This integration includes one or more Kibana dashboards that visualizes the data collected by the integration. The screenshots below illustrate how the ingested data is displayed.