Dataminr Pulse Integration User Guide

Embed Dataminr Pulse real-time, actionable intelligence directly into Elastic Security. Transform the earliest external threat signals from over 1.1 million public, deep, and dark web sources into Elastic-native detections, enriched indices, and automated workflows.

The Dataminr Pulse integration for Elastic Security seamlessly bridges the gap between external real-time data and your internal security operations. By leveraging the Elastic Agent with a CEL (Common Expression Language) input, the integration polls the Dataminr Pulse v4 API at configurable intervals—automatically managing OAuth token refresh and cursor-based pagination to ensure a continuous, resilient data flow.

Stay ahead of the threat curve and be the first to see rapidly emerging and evolving threats, vulnerabilities, exploits, ransomware activity, third-party incidents, and more—often hours or days before traditional sources.

Unmatched Coverage, Precision, and Granularity

With Dataminr Pulse for Cyber Risk, security teams gain a critical time advantage. Dataminr processes more than 45 terabytes of daily public data, leveraging over 55 proprietary LLMs and 15 years of historic alerting information. With multimodal fusion AI, GenAI, and Agentic AI deeply embedded into the platform, security teams can:

• Dynamically detect and defend digital assets beyond the perimeter.
• Unearth hidden threats and close blind spots with advanced processing of text, images, video, and machine signals.
• Leverage Agentic AI-powered Intel Agents to autonomously assemble adversary context, including TTPs, IOCs, CVEs, and MITRE ATT&CK® mappings.
• Proactively prioritize and patch fast-breaking vulnerabilities and exploits.

Accelerate Elastic Workflows with Actionable Context

• Native ECS Mapping: Ingested alerts are automatically mapped to the Elastic Common Schema (ECS) and stored in the logs-dataminr_pulse.alerts-* data stream for immediate correlation.
• Granular Entity Expansion: The integration automatically expands alerts containing multiple discovered entities, such as vulnerabilities, threat actors, malware, IP addresses, and URLs into individual documents for deep, granular analysis.
• Integrated Detection & Response: Populate Elastic Security signals and enrich indices to power advanced hunting, Kibana Dashboards, and Elastic Defend workflows.

Address Your Critical Use Cases with Dataminr

Before installing, ensure the following requirements are met.

The host running the Elastic Agent must have outbound HTTPS (port 443) access to the following endpoints:

1. Log in to elastic and navigate to the Integrations page under Data Management.
2. Search for Dataminr Pulse in the integrations catalog.
3. Click the Dataminr Pulse integration card, then click Add Dataminr Pulse.

To use this integration, you need a Dataminr Pulse API account with a valid Client ID and Client Secret. Contact your Dataminr account representative to obtain the API credentials.

After clicking Add Dataminr Pulse from the integrations page, fill in the configuration form.

1. Browse to the data management and select integrations.
2. Search ‘Dataminr Pulse’ to install the integration and follow the onscreen instructions.
3. Optionally, configure the Polling Interval, and Page Size. The defaults work for most deployments.
4. Select an existing Agent policy or create a new one.
5. Click Save and continue, then Save and deploy changes.

The table below describes all available configuration parameters.

• Polling Interval: Start with the default of 5m. Reduce to 1m only if you need near-real-time alert ingestion and your Dataminr API quota allows it.
• Page Size: The default of 40 works well for most deployments. Increase up to 100 if you expect high alert volumes to reduce the number of API calls.
• Preserve original event: Keep disabled in production. Enabling it roughly doubles the storage per document.

After deploying the integration, verify that data is flowing correctly.

1. Navigate to Assets > Fleet.
2. Under Agents locate your enrolled agent.
3. Verify the agent status is Healthy (green).
4. Click the agent name, then click the Logs tab.
5. Look for log entries showing successful execution of CEL script within agent (Ex: “Unit state changed cel-default (STARTING->HEALTHY): Healthy”)

1. Navigate to Discover and create a session
2. Set the Index pattern to logs-dataminr_pulse.alerts* under Data view
3. Select the time to be the last hour, and confirm documents are appearing.

1. Navigate to Data Management > Streams
2. Search for logs-dataminr_pulse.alerts*. Verify the index exists and the document count is increasing.

1. Navigate to Dashboards.
2. Search for Dataminr. The integration includes pre-built dashboards for alert monitoring.
3. Open a dashboard and verify it displays data.

To remove the Dataminr Pulse integration from an agent policy:

1. Navigate to Assets > Fleet > Policies
2. Click the policy that contains the Dataminr Pulse integration.
3. Locate the Dataminr Pulse integration entry and click the Actions menu (three dots), then select Delete integration.

If integration assets (dashboards, index templates, ingest pipelines) become corrupted or out of sync, you can reset them.

1. Navigate to Data Management > Integrations.
2. Click Dataminr Pulse.
3. Select the Settings tab.
4. Click Reinstall Dataminr Pulse. This reinstalls dashboards, index templates, and ingest pipelines to their default state.

The integration maps Dataminr Pulse alert fields to Elastic Common Schema (ECS) fields. Custom fields are stored under the dataminr_pulse namespace for reference.

Request tracing logs full HTTP request and response details, which is useful for diagnosing connectivity or authentication issues.

1. Navigate to Assets > Fleet > Policies
2. Click the policy that contains the Dataminr Pulse integration.
3. Locate the Dataminr Pulse integration entry and click the Actions menu (three dots), then select Edit integration.
4. Under advanced settings, set Enable request tracing to true.
5. Click Save and deploy changes.
6. To view traces, navigate to Assets > Fleet > Agents
7. Click the agent, then click Actions > Request diagnostics.
8. Download the diagnostics bundle and examine the HTTP trace logs in the agent log files.

1. Check Agent Status: Navigate to Fleet > Agents and verify the agent is Healthy.
2. Check Agent Logs: Click the agent and review the Logs tab for error messages.
3. Test Credentials: On the agent host, run:
   1. curl -X POST https://userauth.dataminr.com/auth/2/token -H "Content-Type: application/x-www-form-urlencoded" -d "grant_type=api_key&client_id=YOUR_ID&client_secret=YOUR_SECRET"
   2. A successful response returns a JSON object with dmaToken and expire fields.
4. Check Data Stream: In Dev Tools, run:
   1. GET logs-dataminr_pulse.alerts-*/_count
   2. If the count is 0, the integration is not receiving data from the API. Review the agent logs for details.

The integration uses document fingerprinting to prevent duplicates. If you observe duplicate documents:

1. Verify the ingest pipeline is installed by running in Dev Tools:
   1. GET _ingest/pipeline/logs-dataminr_pulse.alerts-*
   2. If missing, reset integration assets (see Reset Integration Assets).