PolySwarm Threat Intelligence

Version 0.1.0 (View all)
Subscription level
What's this?
Basic
Developed by
What's this?
Partner
Ingestion method(s) API
Minimum Kibana version(s) 9.1.0
8.19.0
The PolySwarm Threat Intelligence integration v0.1.0 is in technical preview

To use pre-release integrations, go to the Integrations page in Kibana, scroll down, and toggle on the Display beta integrations option.

The PolySwarm Threat Intelligence integration collects malicious artifact (file) indicators from the PolySwarm CTI API and normalizes them into Elastic Common Schema (ECS) threat.indicator.* fields. PolySwarm is a crowdsourced malware-detection marketplace: each artifact submitted to the network is scanned by many independent engines, and the resulting assertions are condensed into a single malicious-likelihood score called the PolyScore.

Because the integration writes to logs-ti_polyswarm.scans-*, which matches Kibana's default threat intelligence index pattern (logs-ti_*), ingested indicators appear automatically under Security → Explore → Intelligence and are usable by indicator-match detection rules with no additional configuration.

This integration has been tested against PolySwarm CTI API v3 (GET /v3/search/metadata/query). For supported Elastic Stack versions, see the version constraints shown on the integration's page in Kibana.

The Elastic Agent uses the CEL input to poll the PolySwarm metadata search API on a configurable interval. Each poll submits a server-side metadata query — by default artifact.created:>now-1d AND scan.detections.malicious:>0, a rolling one-day window restricted to artifacts at least one engine flagged as malicious — and pages through the results using the cursor PolySwarm returns in offset, continuing while has_more is true.

Every returned artifact becomes one indicator document. The ingest pipeline maps the artifact hashes, file metadata, and scan timestamps to ECS, derives threat.indicator.confidence from the PolyScore, and records the raw vendor payload under the polyswarm.* field group. A non-200 response from the API is indexed as a single event.kind: pipeline_error document carrying error.code and error.message, rather than as a false indicator.

This integration collects a single data stream:

  • scans: Malicious artifact indicators from PolySwarm metadata search. Each document is emitted with event.kind: enrichment, event.category: [threat], event.type: [indicator], and threat.indicator.type: file. Artifact hashes are mapped to threat.indicator.file.hash.{md5,sha1,sha256} and mirrored into related.hash so analysts can pivot on a hash without knowing which field holds it.

The integration also installs a latest_scans transform that deduplicates the source stream by SHA-256 into logs-ti_polyswarm_latest.scans, keeping one document per artifact. See Deduplication and IOC expiration below.

  • Driving indicator-match detection rules from the deduplicated latest-state index, so file hashes observed by your endpoints are matched against artifacts PolySwarm recently scored as malicious.
  • Enriching and triaging alerts in Elastic Security with PolyScore, malware family, and file metadata for a hash under investigation.
  • Building dashboards over the recent malicious-artifact landscape (volume over time, malware families, file types).
  • A self-managed or Elastic Cloud deployment. The supported Elastic Stack versions are listed on the integration's page in Kibana.
  • An Elastic Agent enrolled in Fleet. If you do not have one, see the Elastic Agent installation instructions. Only one Elastic Agent can be installed per host.
  • A PolySwarm account with CTI API access and an API key. Register at polyswarm.network. The API key is available from your account settings. Metadata search is a paid capability — confirm your plan includes it before configuring the integration.

This integration is deployed using Elastic Agent. For an end-to-end overview of installing the agent and adding an integration, see the Fleet and Elastic Agent guide.

  1. In Kibana, navigate to Management → Integrations and search for "PolySwarm".
  2. Select PolySwarm Threat Intelligence, then click Add PolySwarm Threat Intelligence.
  3. Configure the following settings:
    • URL: PolySwarm API base URL, without a trailing slash. Default https://api.polyswarm.network. Change only if PolySwarm provides a different host.
    • API Key: Your PolySwarm API key. Stored as a policy secret, masked in the UI, and redacted from the CEL input's logged state.
    • Interval: How often to poll the API. Default 5m. Supported units are h, m, and s.
    • Query: The server-side metadata query. Default artifact.created:>now-1d AND scan.detections.malicious:>0. Keep the scan.detections.malicious:>0 clause so only artifacts with at least one malicious detection are ingested as indicators — removing it ingests benign artifacts as threat indicators. See Tuning the query.
    • Community: The PolySwarm community to search. Default default.
    • Batch Size: Number of results requested per API request. Default 100. Available under Advanced options.
    • Preserve original event: Retains the raw PolySwarm JSON for each artifact in event.original. Default off. Enable it when you need an audit trail of exactly what the API returned.
    • Proxy URL (optional): Route the connection through an HTTP proxy, as http[s]://<user>:<password>@<host>:<port>. Available under Advanced options.
    • SSL Configuration (optional): Custom TLS settings, for example a corporate certificate authority. Available under Advanced options.
    • HTTP Client Timeout: How long to wait before declaring the HTTP connection timed out. Default 30s. Available under Advanced options.
  4. Select the agent policy to deploy to, then click Save and continue.

After deployment, verify that data is flowing:

  1. In Kibana, open Discover and select the logs-ti_polyswarm.scans-* data view. Documents should appear within one Interval.
  2. Check the latest_scans transform under Stack Management → Transforms: state should be started and documents_processed should grow over time. The transform runs every 30 seconds with a 120-second sync delay, so allow roughly three minutes after the first poll before the destination index populates.
  3. Open Security → Explore → Intelligence and confirm indicators with threat.feed.name: PolySwarm are listed.

The Query setting is passed straight through to PolySwarm's metadata search, so the indicator set is defined server-side. Two clauses matter:

  • artifact.created:>now-1d bounds the rolling window the integration re-reads on every poll. It must stay comfortably shorter than the transform's 48-hour retention window (see below) — widening it past 48 hours ingests artifacts that immediately fall outside the active view.
  • scan.detections.malicious:>0 is what makes each result an indicator. Keep it.

Useful refinements:

  • AND scan.latest_scan.polyscore:>0.7 — restrict ingestion to high-confidence artifacts only, cutting volume substantially.
  • AND scan.detections.malicious:>3 — require corroboration from multiple engines.
  • AND artifact.mimetype:"application/vnd.microsoft.portable-executable" — narrow to a single file type.

Indicators are held in two tiers, following the same pattern as Elastic's other threat intelligence integrations:

  • Source data stream logs-ti_polyswarm.scans-*: every poll re-ingests the artifacts still inside the rolling query window, duplicates included. This re-assertion acts as a heartbeat proving an indicator is still current. Retention bounds its growth — see Source data stream retention below.
  • Latest transform logs-ti_polyswarm_latest.scans: keeps exactly one document per threat.indicator.file.hash.sha256, the most recent by event.ingested. Its retention policy deletes indicators whose last re-assertion is older than 48 hours, so an artifact that ages out of the query window drops out of the deduplicated view. That expiration is applied on completed transform checkpoints rather than on a timer, so it stalls whenever indicator ingestion stops — see Expiration depends on continued ingestion.

For indicator-match rules and dashboards, always query the deduplicated index logs-ti_polyswarm_latest.scans rather than the raw source stream.

The package bounds the growth of the source data stream with a retention that depends on the deployment type:

Data stream Self-managed and Elastic Cloud Hosted (ILM policy) Serverless (data stream lifecycle)
logs-ti_polyswarm.scans-* logs-ti_polyswarm.scans-default_policy: roll over after 1d or 50 GB primary shard size, delete 5d after rollover delete 5d after rollover

On self-managed and Elastic Cloud Hosted deployments the ILM policy applies, and the data stream lifecycle shipped with the package is not used there. ILM counts the delete age from the rollover of a backing index, so a document can remain for up to the rollover age plus the delete age. On Serverless, ILM is not available and the data stream lifecycle applies instead. It also works per backing index: Elasticsearch rolls the write index over automatically on age, size, or document count, and deletes a backing index once the retention has passed since it rolled over. A document therefore stays for the retention plus up to one rollover interval. Neither mechanism affects the transform's destination index, whose expiration is governed by the transform retention policy described above.

Each document records the configured polling interval in labels.interval for expiration diagnostics.

Both tiers match the default logs-ti_* threat index pattern, so a query spanning that pattern reaches the raw stream as well as the deduplicated view. To tell them apart, every document carries labels.is_ioc_transform_source as a constant_keyword: "true" on the raw source stream, "false" on the transform's latest index. Filter on labels.is_ioc_transform_source: "false" to get only the deduplicated, currently-asserted set.

Elastic's prebuilt threat intelligence rules already use this label — their indicator query excludes labels.is_ioc_transform_source: "true" — so they read only the deduplicated tier with no configuration on your part.

Because expiration is driven by recency of re-assertion, a polling interval that approaches 48 hours can let still-current indicators drop out of the latest index between polls.

The transform's retention policy is applied as part of a completed checkpoint, not on an independent timer. A checkpoint only completes when new documents matching the transform's source query arrive — and that query admits only documents that carry threat.indicator.file.hash.sha256 and no error.message. If indicator ingestion stops — agent down, network outage, revoked or expired API key — no further checkpoints complete and the indicators already in logs-ti_polyswarm_latest.scans are not expired. They stay in the index, and in any indicator-match rule reading it, until ingestion resumes.

This is the opposite of a quiet drain: the active view looks healthy while the intelligence behind it goes stale. Plan for it:

  • Alert on ingest freshness, excluding error documents. A rejected API key does not stop documents from arriving. Every poll still indexes an event.kind: pipeline_error document into logs-ti_polyswarm.scans-* with a fresh event.ingested, while no indicator document does. An alert that only watches that data stream for recent event.ingested values therefore stays green straight through an authentication outage — the very case most likely to cause one. Scope the alert to the documents that can actually advance a checkpoint, mirroring the transform's own source query. Set the rule's index to logs-ti_polyswarm.scans-* and its KQL query to:

    threat.indicator.file.hash.sha256: * and not error.message: *
    		

    Alert when that count stays at zero for a few hours. Monitoring the agent and the logs-ti_polyswarm.latest_scans-default-<version> transform directly works too. One of these is your only reliable outage signal — the deduplicated index will not show one.

  • Add a freshness bound at query time where staleness matters. Rules and dashboards reading the latest index can constrain event.ingested (for example, event.ingested >= now-48h) so an outage removes indicators from their scope regardless of whether the transform has run.

Once ingestion resumes, the next completed checkpoint applies the retention policy and the backlog of expired indicators is removed in one pass.

Enable the prebuilt detection rule "Threat Intel Hash Indicator Match" (Security → Rules → Detection rules → Add Elastic rules), or create a custom indicator-match rule with:

  • Index patterns: your endpoint and log indices (for example, logs-*).
  • Indicator index patterns: logs-ti_polyswarm_latest.scans.
  • Indicator mapping: file.hash.sha256 MATCHES threat.indicator.file.hash.sha256. Add OR clauses for sha1 and md5 to widen coverage.

The integration ships one saved search, [PolySwarm] Current indicators (re-asserted in the last 48h), available under Discover. It scopes to the deduplicated tier (labels.is_ioc_transform_source: "false") and bounds event.ingested to the last 48 hours, so it reflects only indicators PolySwarm is still re-asserting.

That event.ingested bound is what makes the view correct during an outage. As described in Expiration depends on continued ingestion, the transform cannot expire indicators while collection is stopped, so a query without a freshness bound keeps returning them. Use this saved search wherever acting on stale intelligence would be a problem.

Note that the prebuilt rules do not apply such a bound: their indicator query filters on labels.is_ioc_transform_source and on @timestamp, which for this integration carries the artifact's creation time rather than when the indicator was last re-asserted. If that distinction matters to you, build a custom rule on the saved search's criteria instead.

The first poll happens after one Interval (default 5m), and the transform adds up to ~2.5 minutes on top of that. If logs-ti_polyswarm.scans-* is still empty after 10 minutes:

  1. Verify the agent is healthy and on the current policy revision. Fleet → Agents → your agent should report Healthy with the latest revision.
  2. Enable agent monitoring to see CEL polling logs. Fleet → Agent policies → your policy → Settings → Agent monitoring → enable "Collect agent logs". Then Fleet → Agents → your agent → Logs and search for cel or polyswarm.
  3. Check for error documents. A non-200 response is indexed rather than dropped: search logs-ti_polyswarm.scans-* for event.kind: pipeline_error and read error.code. 401/403 means the API key is invalid, expired, or lacks metadata-search entitlement; 429 means you are exceeding your plan's rate limit — increase Interval.
  4. Confirm the query returns results. A narrow Query or a community with no recent malicious artifacts legitimately yields zero documents. Widen the artifact.created window temporarily to confirm connectivity.

Documents that fail in the ingest pipeline are indexed with event.kind: pipeline_error and the processor error in error.message. Search logs-ti_polyswarm.scans-* for event.kind: pipeline_error and read error.message to identify the offending field. This works on every supported stack version.

On deployments where the data stream failure store is enabled you can additionally check GET .fs-logs-ti_polyswarm.scans-*/_count, which should be 0.

PolySwarm's per-file analysis metadata (polyswarm.exiftool, polyswarm.lief, polyswarm.pefile, polyswarm.scan) is mapped as flattened precisely because its schema varies by file type.

The logs-ti_polyswarm.latest_scans-default-<version> transform only processes documents that have threat.indicator.file.hash.sha256 set. If source documents exist but documents_processed stays at 0, confirm the source documents are indicators rather than event.kind: pipeline_error records, then restart the transform from Stack Management → Transforms.

For PolySwarm-side issues (API entitlements, rate limits, query syntax), consult the PolySwarm API documentation.

Each poll pages through the entire result set for the configured query, so ingest volume is driven by the query window and the community's artifact rate, not by Interval. Shortening Interval refreshes indicators sooner but re-reads the same window more often, multiplying source-stream volume. The transform collapses the duplicates, so cluster sizing for the latest index should be based on the count of unique SHA-256 values in the window rather than raw event volume.

If you need broader coverage, prefer a wider Query on a moderate interval over a very short interval. Running several policies with disjoint queries (for example, split by mimetype) spreads load across agents without re-reading the same artifacts.

  • CEL input: Polls the PolySwarm metadata search API, paging with the cursor returned in offset while has_more is true, and redacts the API key from the CEL input's logged state. Note that redaction applies to the input's state, not to HTTP request traces: if request tracing is enabled for debugging, the trace files contain the API key in the Authorization header and must be handled as credentials.
  • GET /v3/search/metadata/query?query={query}&community={community}&limit={batch_size}&offset={cursor} — Returns artifacts matching a server-side metadata query, along with has_more and the next-page offset. Authenticated with the API key as a raw Authorization header token.