Recorded Future

Ingest threat intelligence indicators from Recorded Future risk lists with Elastic Agent.

Version
1.26.0 (View all)
Compatible Kibana version(s)
8.13.0 or higher
Supported Serverless project types

Security
Observability
Subscription level
Basic
Level of support
Elastic

The Recorded Future integration fetches risklists from the Recorded Future API. It supports domain, hash, ip and url entities.

In order to use it you need to define the entity and list to fetch. Check with Recorded Future for the available lists for each entity. To fetch indicators from multiple entities, it's necessary to define one integration for each.

Alternatively, it's also possible to use the integration to fetch custom Fusion files by supplying the URL to the CSV file as the Custom URL configuration option.

Expiration of Indicators of Compromise (IOCs)

The ingested IOCs expire after certain duration. An Elastic Transform is created to faciliate only active IOCs be available to the end users. This transform creates a destination index named logs-ti_recordedfuture_latest.threat-1 which only contains active and unexpired IOCs. The destination index also has an alias logs-ti_recordedfuture_latest.threat. When setting up indicator match rules, use this latest destination index to avoid false positives from expired IOCs. Please read ILM Policy below which is added to avoid unbounded growth on source .ds-logs-ti_recordedfuture.threat-* indices.

ILM Policy

To facilitate IOC expiration, source datastream-backed indices .ds-logs-ti_recordedfuture.threat-* are allowed to contain duplicates from each polling interval. ILM policy is added to these source indices so it doesn't lead to unbounded growth. This means data in these source indices will be deleted after 5 days from ingested date.

NOTE: For large risklist downloads, adjust the timeout setting so that the Agent has enough time to download and process the risklist.

An example event for threat looks as following:

{
    "@timestamp": "2024-05-09T12:24:05.286Z",
    "agent": {
        "ephemeral_id": "b0d47395-89bd-40e7-8018-57fdcc0cf1b8",
        "id": "013c7177-2e5d-40da-9e17-9ee5d2249880",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.12.2"
    },
    "data_stream": {
        "dataset": "ti_recordedfuture.threat",
        "namespace": "ep",
        "type": "logs"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "013c7177-2e5d-40da-9e17-9ee5d2249880",
        "snapshot": false,
        "version": "8.12.2"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "threat"
        ],
        "dataset": "ti_recordedfuture.threat",
        "ingested": "2024-05-09T12:24:15Z",
        "kind": "enrichment",
        "risk_score": 75,
        "timezone": "+00:00",
        "type": [
            "indicator"
        ]
    },
    "input": {
        "type": "log"
    },
    "log": {
        "file": {
            "path": "/tmp/service_logs/rf_file_default.csv"
        },
        "offset": 57
    },
    "recordedfuture": {
        "evidence_details": [
            {
                "criticality": 2,
                "criticality_label": "Suspicious",
                "evidence_string": "2 sightings on 1 source: PolySwarm. Most recent link (Mar 23, 2024): https://polyswarm.network/scan/results/file/63212aa8c94098a844945ed1611389b2e1c9dc3906a5ba9d7d0d320344213f4f",
                "mitigation_string": "",
                "name": "linkedToMalware",
                "rule": "Linked to Malware",
                "sightings_count": 2,
                "sources": [
                    "source:doLlw5"
                ],
                "sources_count": 1,
                "timestamp": "2024-03-23T17:10:20.642Z"
            },
            {
                "criticality": 3,
                "criticality_label": "Malicious",
                "evidence_string": "3 sightings on 3 sources: Polyswarm Sandbox Analysis, Recorded Future Triage Malware Analysis, PolySwarm. Most recent link (Mar 23, 2024): https://polyswarm.network/scan/results/file/63212aa8c94098a844945ed1611389b2e1c9dc3906a5ba9d7d0d320344213f4f",
                "mitigation_string": "",
                "name": "positiveMalwareVerdict",
                "rule": "Positive Malware Verdict",
                "sightings_count": 3,
                "sources": [
                    "source:hzRhwZ",
                    "source:ndy5_2",
                    "source:doLlw5"
                ],
                "sources_count": 3,
                "timestamp": "2024-03-23T16:36:02.000Z"
            }
        ],
        "name": "63212aa8c94098a844945ed1611389b2e1c9dc3906a5ba9d7d0d320344213f4f",
        "risk_string": "2/17"
    },
    "tags": [
        "forwarded",
        "recordedfuture"
    ],
    "threat": {
        "feed": {
            "name": "Recorded Future"
        },
        "indicator": {
            "file": {
                "hash": {
                    "sha256": "63212aa8c94098a844945ed1611389b2e1c9dc3906a5ba9d7d0d320344213f4f"
                }
            },
            "provider": [
                "PolySwarm",
                "Polyswarm Sandbox Analysis",
                "Recorded Future Triage Malware Analysis"
            ],
            "scanner_stats": 4,
            "sightings": 5,
            "type": "file"
        }
    }
}

Exported fields

FieldDescriptionType
@timestamp
Event timestamp.
date
cloud.image.id
Image ID for the cloud instance.
keyword
data_stream.dataset
Data stream dataset name.
constant_keyword
data_stream.namespace
Data stream namespace.
constant_keyword
data_stream.type
Data stream type.
constant_keyword
event.dataset
Event dataset
constant_keyword
event.module
Event module
constant_keyword
host.containerized
If the host is a container.
boolean
host.os.build
OS build information.
keyword
host.os.codename
OS codename, if any.
keyword
input.type
Type of Filebeat input.
keyword
labels.is_ioc_transform_source
Field indicating if its the transform source for supporting IOC expiration. This field is dropped from destination indices to facilitate easier filtering of indicators.
constant_keyword
log.flags
Flags for the log file.
keyword
log.offset
Offset of the entry in the log file.
long
recordedfuture.evidence_details.criticality
double
recordedfuture.evidence_details.criticality_label
keyword
recordedfuture.evidence_details.evidence_string
keyword
recordedfuture.evidence_details.mitigation_string
keyword
recordedfuture.evidence_details.name
keyword
recordedfuture.evidence_details.rule
keyword
recordedfuture.evidence_details.sightings_count
integer
recordedfuture.evidence_details.sources
keyword
recordedfuture.evidence_details.sources_count
integer
recordedfuture.evidence_details.timestamp
date
recordedfuture.list
User-configured risklist.
keyword
recordedfuture.name
Indicator value.
keyword
recordedfuture.risk_string
Details of risk rules observed.
keyword
threat.feed.name
Display friendly feed name
constant_keyword

Changelog

VersionDetailsKibana version(s)

1.26.0

Enhancement View pull request
Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.

8.13.0 or higher

1.25.1

Bug fix View pull request
Adjust field mappings for transform destination index.

8.12.0 or higher

1.25.0

Enhancement View pull request
Decode Evidence_Details field. This is a breaking change since the mapping is changed.

8.12.0 or higher

1.24.0

Enhancement View pull request
Add destination index alias and fix docs.

8.12.0 or higher

1.23.0

Enhancement View pull request
Add dashboards and list field

8.12.0 or higher

1.22.0

Enhancement View pull request
Set sensitive values as secret.

8.12.0 or higher

1.21.0

Enhancement View pull request
Make threat.indicator.url.full available for rule detections.

8.8.0 or higher

1.20.2

Enhancement View pull request
Changed owners

8.8.0 or higher

1.20.1

Bug fix View pull request
Fix exclude_files pattern.

8.8.0 or higher

1.20.0

Enhancement View pull request
Limit request tracer log count to five.

8.8.0 or higher

1.19.0

Enhancement View pull request
ECS version updated to 8.11.0.

8.8.0 or higher

1.18.1

Bug fix View pull request
Fix the parse of providers information.

8.8.0 or higher

1.18.0

Enhancement View pull request
Improve 'event.original' check to avoid errors if set.

8.8.0 or higher

1.17.0

Enhancement View pull request
ECS version updated to 8.10.0.

8.8.0 or higher

1.16.0

Enhancement View pull request
Add DLM policy. Add owner.type to package manifest. Update format_version to 3.0.0

Enhancement View pull request
Add tags.yml file so that integration's dashboards and saved searches are tagged with "Security Solution" and displayed in the Security Solution UI.

8.8.0 or higher

1.15.1

Bug fix View pull request
Replace dotted YAML keys.

1.15.0

Enhancement View pull request
Update package-spec to 2.10.0.

8.8.0 or higher

1.14.0

Enhancement View pull request
Update package to ECS 8.9.0.

8.8.0 or higher

1.13.0

Enhancement View pull request
Document duration units.

8.8.0 or higher

1.12.0

Enhancement View pull request
Ensure event.kind is correctly set for pipeline errors.

8.8.0 or higher

1.11.0

Enhancement View pull request
Update package to ECS 8.8.0.

8.8.0 or higher

1.10.0

Enhancement View pull request
Add IOC field to transform source to easily filter destination indices with unexpired indicators

8.8.0 or higher

1.9.0

Enhancement View pull request
Support for IoC Expiration

8.8.0 or higher

1.8.0

Enhancement View pull request
Add a new flag to enable request tracing

8.7.1 or higher

1.7.0

Enhancement View pull request
Scrape provider details from evidence field.

8.0.0 or higher

1.6.0

Enhancement View pull request
Update package to ECS 8.7.0.

8.0.0 or higher

1.5.0

Enhancement View pull request
Update package to ECS 8.6.0.

8.0.0 or higher

1.4.1

Bug fix View pull request
Use ECS definition for threat.indicator.geo.location.

8.0.0 or higher

1.4.0

Enhancement View pull request
Update package to ECS 8.5.0.

1.3.0

Enhancement View pull request
Update package to ECS 8.4.0

8.0.0 or higher

1.2.1

Bug fix View pull request
Expose request timeout setting and increase it to 5m.

Bug fix View pull request
Do not fail on invalid URLs.

8.0.0 or higher

1.2.0

Enhancement View pull request
Update categories to include threat_intel.

8.0.0 or higher

1.1.0

Enhancement View pull request
Update package to ECS 8.3.0.

8.0.0 or higher

1.0.1

Enhancement View pull request
update readme added link to recorded future API documentation

8.0.0 or higher

1.0.0

Enhancement View pull request
Make GA

8.0.0 or higher

0.1.3

Enhancement View pull request
Update package descriptions

0.1.2

Enhancement View pull request
Add field mapping for event.created

0.1.1

Enhancement View pull request
Add documentation for multi-fields

0.1.0

Enhancement View pull request
Initial release

On this page