Alienvault OTX Integration

edit

Alienvault OTX Integration

edit

Version

1.26.0 (View all)

Compatible Kibana version(s)

8.13.0 or higher

Supported Serverless project types
What’s this?

Security
Observability

Subscription level
What’s this?

Basic

Level of support
What’s this?

Elastic

This integration is for Alienvault OTX. It retrieves indicators for all pulses subscribed to a specific user account on OTX

Configuration

edit

To use this package, it is required to have an account on Alienvault OTX. Once an account has been created, and at least 1 pulse has been subscribed to, the API key can be retrieved from your user profile dashboard. In the top right corner there should be an OTX KEY.

Logs

edit
Threat
edit

Retrieves all the related indicators over time, related to your pulse subscriptions on OTX.

Exported fields
Field Description Type

@timestamp

Event timestamp.

date

cloud.image.id

Image ID for the cloud instance.

keyword

data_stream.dataset

Data stream dataset name.

constant_keyword

data_stream.namespace

Data stream namespace.

constant_keyword

data_stream.type

Data stream type.

constant_keyword

event.dataset

Event dataset

constant_keyword

event.module

Event module

constant_keyword

host.containerized

If the host is a container.

boolean

host.os.build

OS build information.

keyword

host.os.codename

OS codename, if any.

keyword

input.type

Type of Filebeat input.

keyword

log.flags

Flags for the log file.

keyword

log.offset

Offset of the entry in the log file.

long

otx.content

Extra text or descriptive content related to the indicator.

keyword

otx.description

A description of the indicator.

keyword

otx.id

The ID of the indicator.

keyword

otx.indicator

The value of the indicator, for example if the type is domain, this would be the value.

keyword

otx.title

Title describing the indicator.

keyword

otx.type

The indicator type, can for example be "domain, email, FileHash-SHA256".

keyword

threat.feed.dashboard_id

Dashboard ID used for Kibana CTI UI

constant_keyword

threat.feed.name

Display friendly feed name

constant_keyword

threat.indicator.file.hash.pehash

The file’s pehash, if available.

keyword

threat.indicator.first_seen

The date and time when intelligence source first reported sighting this indicator.

date

threat.indicator.last_seen

The date and time when intelligence source last reported sighting this indicator.

date

threat.indicator.modified_at

The date and time when intelligence source last modified information for this indicator.

date

Example

An example event for threat looks as following:

{
    "@timestamp": "2024-03-08T02:55:33.690Z",
    "agent": {
        "ephemeral_id": "8edc1f21-05cd-4fa5-aadc-66e64f44856a",
        "id": "f29e7d89-991e-4f0a-838f-9c2eb93d876e",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.12.1"
    },
    "data_stream": {
        "dataset": "ti_otx.threat",
        "namespace": "ep",
        "type": "logs"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "f29e7d89-991e-4f0a-838f-9c2eb93d876e",
        "snapshot": false,
        "version": "8.12.1"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "threat"
        ],
        "created": "2024-03-08T02:55:33.690Z",
        "dataset": "ti_otx.threat",
        "ingested": "2024-03-08T02:55:45Z",
        "kind": "enrichment",
        "original": "{\"count\":40359,\"next\":\"https://otx.alienvault.com/api/v1/indicators/export?types=domain%2CIPv4%2Chostname%2Curl%2CFileHash-SHA256\\u0026modified_since=2020-11-29T01%3A10%3A00+00%3A00\\u0026page=2\",\"previous\":null,\"results\":{\"content\":\"\",\"description\":null,\"id\":1251,\"indicator\":\"info.3000uc.com\",\"title\":null,\"type\":\"hostname\"}}",
        "type": [
            "indicator"
        ]
    },
    "input": {
        "type": "httpjson"
    },
    "otx": {},
    "tags": [
        "preserve_original_event",
        "forwarded",
        "otx-threat"
    ],
    "threat": {
        "indicator": {
            "type": "domain-name",
            "url": {
                "domain": "info.3000uc.com"
            }
        }
    }
}
Pulses Subscribed (Recommended)
edit

Retrieves all indicators from subscribed pulses on OTX from API /api/v1/pulses/subscribed using Filebeat’s CEL input. The following subscriptions are included by this API:

  • All pulses by users you are subscribed to
  • All pulses you are directly subscribed to
  • All pulses you have created yourself
  • All pulses from groups you are a member of
Indicators of Comprosie (IoC) Expiration
edit

Pulses Subscribed datastream also supports IoC expiration by using latest transform. Below are the steps on how it is handled:

  1. All the indicators are retrieved into source indices named logs-ti_otx.pulses_subscribed-* using CEL input and processed via ingest pipelines. These indicators have a property named expiration which is either a null value or a timestamp such as "2023-09-07T00:00:00". When the value is null or if the timestamp value is less than current timestamp now(), the indicator is not expired, and hence is still active.
  2. A latest transform is continuosly run on source indices. The purpose of this transform is to:

    • Move only the active indicators from source indices into destination indices named logs-ti_otx_latest.pulses_subscribed-<NUMBER> where NUMBER indicates index version.
    • Delete expired indicators based on the expiration timestamp value.
  3. All the active indicators can be retrieved using destination index alias logs-ti_otx_latest.pulses_subscribed which points to the latest destination index version.

    • Note: Do not use the source indices logs-ti_otx.pulses_subscribed-*, because when the indicators expire, the source indices will contain duplicates. Always use the destination index alias: logs-ti_otx_latest.pulses_subscribed to query all active indicators.
Exported fields
Field Description Type

@timestamp

Event timestamp.

date

data_stream.dataset

Data stream dataset name.

constant_keyword

data_stream.namespace

Data stream namespace.

constant_keyword

data_stream.type

Data stream type.

constant_keyword

event.dataset

Event dataset

constant_keyword

event.module

Event module

constant_keyword

input.type

Type of Filebeat input.

keyword

labels.is_ioc_transform_source

Indicates whether an IOC is in the raw source data stream, or the in latest destination index.

constant_keyword

log.flags

Flags for the log file.

keyword

log.offset

Offset of the entry in the log file.

long

otx.content

keyword

otx.count

integer

otx.created

date

otx.description

keyword

otx.expiration

date

otx.id

The ID of the indicator.

keyword

otx.indicator

keyword

otx.is_active

integer

otx.prefetch_pulse_ids

boolean

otx.pulse.adversary

keyword

otx.pulse.attack_ids

keyword

otx.pulse.author_name

keyword

otx.pulse.created

date

otx.pulse.description

keyword

otx.pulse.extract_source

keyword

otx.pulse.id

keyword

otx.pulse.industries

keyword

otx.pulse.malware_families

keyword

otx.pulse.modified

date

otx.pulse.more_indicators

boolean

otx.pulse.name

keyword

otx.pulse.public

integer

otx.pulse.references

keyword

otx.pulse.revision

integer

otx.pulse.targeted_countries

keyword

otx.pulse.tlp

keyword

otx.role

keyword

otx.t

double

otx.t2

double

otx.t3

double

otx.title

keyword

threat.feed.dashboard_id

Dashboard ID used for Kibana CTI UI

constant_keyword

threat.feed.name

Display friendly feed name

constant_keyword

threat.indicator.file.hash.pehash

The file’s pehash, if available.

keyword

threat.indicator.first_seen

The date and time when intelligence source first reported sighting this indicator.

date

threat.indicator.last_seen

The date and time when intelligence source last reported sighting this indicator.

date

threat.indicator.modified_at

The date and time when intelligence source last modified information for this indicator.

date

Example

An example event for pulses_subscribed looks as following:

{
    "@timestamp": "2023-08-08T05:05:15.000Z",
    "agent": {
        "ephemeral_id": "c12b4f72-265e-41f0-96e0-103c81de7908",
        "id": "8299ae35-ee0e-4107-9acb-1b6acfdda1fb",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.13.0"
    },
    "data_stream": {
        "dataset": "ti_otx.pulses_subscribed",
        "namespace": "32586",
        "type": "logs"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "8299ae35-ee0e-4107-9acb-1b6acfdda1fb",
        "snapshot": false,
        "version": "8.13.0"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "threat"
        ],
        "dataset": "ti_otx.pulses_subscribed",
        "ingested": "2024-08-02T06:03:28Z",
        "kind": "enrichment",
        "original": "{\"content\":\"\",\"count\":2,\"created\":\"2023-08-08T05:05:15\",\"description\":\"\",\"expiration\":null,\"id\":3454375108,\"indicator\":\"pinup-casino-tr.site\",\"is_active\":1,\"prefetch_pulse_ids\":false,\"pulse_raw\":\"{\\\"adversary\\\":\\\"\\\",\\\"attack_ids\\\":[\\\"T1531\\\",\\\"T1059\\\",\\\"T1566\\\"],\\\"author_name\\\":\\\"SampleUser\\\",\\\"created\\\":\\\"2023-08-22T09:43:18.855000\\\",\\\"description\\\":\\\"\\\",\\\"extract_source\\\":[],\\\"id\\\":\\\"64e38336d783f91d6948a7b1\\\",\\\"industries\\\":[],\\\"malware_families\\\":[\\\"WHIRLPOOL\\\"],\\\"modified\\\":\\\"2023-08-22T09:43:18.855000\\\",\\\"more_indicators\\\":false,\\\"name\\\":\\\"Sample Pulse\\\",\\\"public\\\":1,\\\"references\\\":[\\\"https://www.cisa.gov/news-events/analysis-reports/ar23-230a\\\"],\\\"revision\\\":1,\\\"tags\\\":[\\\"cisa\\\",\\\"backdoor\\\",\\\"whirlpool\\\",\\\"malware\\\"],\\\"targeted_countries\\\":[],\\\"tlp\\\":\\\"white\\\"}\",\"role\":null,\"t\":0,\"t2\":0.0050694942474365234,\"t3\":2.7960586547851562,\"title\":\"\",\"type\":\"domain\"}",
        "type": [
            "indicator"
        ]
    },
    "input": {
        "type": "cel"
    },
    "otx": {
        "count": 2,
        "created": "2023-08-08T05:05:15.000Z",
        "expiration": "2023-08-13T05:05:15.000Z",
        "id": "3454375108",
        "is_active": 1,
        "prefetch_pulse_ids": false,
        "pulse": {
            "attack_ids": [
                "T1531",
                "T1059",
                "T1566"
            ],
            "author_name": "SampleUser",
            "created": "2023-08-22T09:43:18.855Z",
            "description": "",
            "extract_source": [],
            "id": "64e38336d783f91d6948a7b1",
            "industries": [],
            "malware_families": [
                "WHIRLPOOL"
            ],
            "modified": "2023-08-22T09:43:18.855Z",
            "more_indicators": false,
            "name": "Sample Pulse",
            "public": 1,
            "references": [
                "https://www.cisa.gov/news-events/analysis-reports/ar23-230a"
            ],
            "revision": 1,
            "targeted_countries": [],
            "tlp": "white"
        },
        "t": 0,
        "t2": 0.0050694942474365234,
        "t3": 2.7960586547851562
    },
    "tags": [
        "preserve_original_event",
        "forwarded",
        "otx-pulses_subscribed",
        "cisa",
        "backdoor",
        "whirlpool",
        "malware"
    ],
    "threat": {
        "indicator": {
            "provider": "OTX",
            "type": "domain-name",
            "url": {
                "domain": "pinup-casino-tr.site"
            }
        }
    }
}

Changelog

edit
Changelog
Version Details Kibana version(s)

1.26.0

Enhancement (View pull request)
Add "preserve_original_event" tag to documents with event.kind set to "pipeline_error".

8.13.0 or higher

1.25.3

Bug fix (View pull request)
Fix labels.is_ioc_transform_source values

8.13.0 or higher

1.25.2

Bug fix (View pull request)
Add missing fields in transform

8.13.0 or higher

1.25.1

Bug fix (View pull request)
Fix ECS date mapping on threat fields.

8.13.0 or higher

1.25.0

Enhancement (View pull request)
Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.

8.13.0 or higher

1.24.1

Bug fix (View pull request)
Fix type-mapping inconsistency for otx.id field.

8.12.0 or higher

1.24.0

Enhancement (View pull request)
Set sensitive values as secret.

8.12.0 or higher

1.23.2

Enhancement (View pull request)
Changed owners

8.10.3 or higher

1.23.1

Bug fix (View pull request)
Fix IOC expiration duration character casting.

8.10.3 or higher

1.23.0

Enhancement (View pull request)
Append hash and IP values to related.* fields

8.10.3 or higher

1.22.0

Enhancement (View pull request)
Limit request tracer log count to five.

8.10.3 or higher

1.21.0

Enhancement (View pull request)
ECS version updated to 8.11.0.

8.10.3 or higher

1.20.0

Enhancement (View pull request)
Improve event.original check to avoid errors if set.

8.10.3 or higher

1.19.0

Enhancement (View pull request)
Add Pulses Subscribed datastream to support IoC expiration

Enhancement (View pull request)
Add DLM policy. Add owner.type to package manifest.

8.10.3 or higher

1.18.0

Enhancement (View pull request)
ECS version updated to 8.10.0.

8.7.1 or higher

1.17.0

Enhancement (View pull request)
The format_version in the package manifest changed from 2.11.0 to 3.0.0. Removed dotted YAML keys from package manifest. Added owner.type: elastic to package manifest.

8.7.1 or higher

1.16.0

Enhancement (View pull request)
Add tags.yml file so that integration’s dashboards and saved searches are tagged with "Security Solution" and displayed in the Security Solution UI.

8.7.1 or higher

1.15.0

Enhancement (View pull request)
Update package-spec to 2.10.0.

8.7.1 or higher

1.14.0

Enhancement (View pull request)
Update package to ECS 8.9.0.

8.7.1 or higher

1.13.0

Enhancement (View pull request)
Document duration units.

8.7.1 or higher

1.12.0

Enhancement (View pull request)
Document valid duration units.

8.7.1 or higher

1.11.0

Enhancement (View pull request)
Ensure event.kind is correctly set for pipeline errors.

8.7.1 or higher

1.10.0

Enhancement (View pull request)
Update package to ECS 8.8.0.

8.7.1 or higher

1.9.0

Enhancement (View pull request)
Add a new flag to enable request tracing

8.7.1 or higher

1.8.0

Enhancement (View pull request)
Update package to ECS 8.7.0.

8.0.0 or higher

1.7.1

Enhancement (View pull request)
Honor preserve_original_event setting.

8.0.0 or higher

1.7.0

Enhancement (View pull request)
Update package to ECS 8.6.0.

8.0.0 or higher

1.6.1

Enhancement (View pull request)
Add support to drop empty documents

8.0.0 or higher

1.6.0

Enhancement (View pull request)
Update package to ECS 8.5.0.

8.0.0 or higher

1.5.0

Enhancement (View pull request)
Update package to ECS 8.4.0

8.0.0 or higher

1.4.2

Bug fix (View pull request)
Fix proxy URL documentation rendering.

8.0.0 or higher

1.4.1

Enhancement (View pull request)
Update categories to include threat_intel.

8.0.0 or higher

1.4.0

Enhancement (View pull request)
Update package to ECS 8.3.0.

8.0.0 or higher

1.3.2

Enhancement (View pull request)
Update readme file to add documentation link

8.0.0 or higher

1.3.1

Enhancement (View pull request)
Update package descriptions

8.0.0 or higher

1.3.0

Enhancement (View pull request)
Update to ECS 8.2

8.0.0 or higher

1.2.2

Enhancement (View pull request)
Add field mapping for event.created

8.0.0 or higher

1.2.1

Enhancement (View pull request)
Add documentation for multi-fields

8.0.0 or higher

1.2.0

Enhancement (View pull request)
Update to ECS 8.0

8.0.0 or higher

1.1.0

Enhancement (View pull request)
Adding threat.feed fields and dashboards

8.0.0 or higher

1.0.3

Bug fix (View pull request)
Change test public IPs to the supported subset

8.0.0 or higher

1.0.2

Enhancement (View pull request)
Bump minimum version

8.0.0 or higher

1.0.1

Enhancement (View pull request)
Update title and description.

1.0.0

Enhancement (View pull request)
Initial release