Collective Intelligence Framework v3 Integration

edit

Collective Intelligence Framework v3 Integration

edit

Version

1.15.0 (View all)

Compatible Kibana version(s)

8.13.0 or higher

Supported Serverless project types
What’s this?

Security
Observability

Subscription level
What’s this?

Basic

Level of support
What’s this?

Community

This integration connects with the REST API from the running CIFv3 instance to retrieve indicators.

Expiration of Indicators of Compromise (IOCs)

edit

Indicators are expired after a certain duration. An Elastic Transform is created for a source index to allow only active indicators to be available to the end users. The transform creates a destination index named logs-ti_cif3_latest.dest_feed* which only contains active and unexpired indicators. Destination indices are aliased to logs-ti_cif3_latest.feed. The indicator match rules and dashboards are updated to show only active indicators.

Indicator Type Indicator Expiration Duration

ipv4-addr

45d

ipv6-addr

45d

domain-name

90d

url

365d

file

365d

All Other Types

Derived from IOC Expiration Duration setting

ILM Policy
edit

To facilitate IOC expiration, source datastream-backed indices .ds-logs-ti_cif3.feed-* are allowed to contain duplicates. ILM policy logs-ti_cif3.feed-default_policy is added to these source indices so it doesn’t lead to unbounded growth. This means data in these source indices will be deleted after 5 days from ingested date.

Data Streams

edit
Feed
edit

The CIFv3 integration collects threat indicators based on user-defined configuration including a polling interval, how far back in time it should look, and other filters like indicator type and tags.

CIFv3 confidence field values (0..10) are converted to ECS confidence (None, Low, Medium, High) in the following way:

CIFv3 Confidence ECS Conversion

Beyond Range

None

0 - <3

Low

3 - <7

Medium

7 - 10

High

Exported fields
Field Description Type

@timestamp

Event timestamp.

date

cif3.application

The application used by the indicator, such as telnet or ssh.

keyword

cif3.asn

AS Number of IP.

integer

cif3.asn_desc

AS Number org name.

keyword

cif3.cc

Country code of GeoIP.

keyword

cif3.city

GeoIP city information.

keyword

cif3.confidence

The confidence on a scale of 0-10 that the tags appropriately contextualize the indicator.

float

cif3.count

The number of times the same indicator has been reported with the same metadata by the same provider.

integer

cif3.deleted_at

The indicator expiration timestamp.

date

cif3.description

A description of the indicator.

keyword

cif3.expiration_duration

The configured expiration duration.

keyword

cif3.indicator

The value of the indicator, for example if the type is fqdn, this would be the value.

keyword

cif3.indicator_iprange

IPv4 or IPv6 IP Range.

ip_range

cif3.indicator_ipv4

IPv4 address.

ip

cif3.indicator_ipv4_mask

subnet mask of IPv4 CIDR.

integer

cif3.indicator_ipv6

singleton IPv6 address.

keyword

cif3.indicator_ipv6_mask

subnet mask of IPv6 CIDR.

integer

cif3.indicator_ssdeep_chunk

SSDEEP hash chunk.

text

cif3.indicator_ssdeep_chunksize

SSDEEP hash chunk size.

integer

cif3.indicator_ssdeep_double_chunk

SSDEEP hash double chunk.

text

cif3.itype

The indicator type, can for example be "ipv4, fqdn, email, url, sha256".

keyword

cif3.latitude

Latitude of GeoIP.

keyword

cif3.location

Lat/Long of GeoIP.

geo_point

cif3.longitude

Longitude of GeoIP.

keyword

cif3.portlist

The port or range of ports used by the indicator.

text

cif3.protocol

The protocol used by the indicator.

text

cif3.provider

The source of the indicator information.

keyword

cif3.rdata

Extra text or descriptive content related to the indicator such as OS, reverse lookup, etc.

keyword

cif3.reference

A reference URL with further info related to the indicator.

keyword

cif3.region

GeoIP region information.

keyword

cif3.tags

Comma-separated list of words describing the indicator such as "malware,exploit".

keyword

cif3.timezone

Timezone of GeoIP.

text

cif3.uuid

The ID of the indicator.

keyword

data_stream.dataset

Data stream dataset.

constant_keyword

data_stream.namespace

Data stream namespace.

constant_keyword

data_stream.type

Data stream type.

constant_keyword

event.dataset

Event dataset

constant_keyword

event.module

Name of the module this data is coming from.

constant_keyword

input.type

Type of Filebeat input.

keyword

labels.is_ioc_transform_source

Indicates whether an IOC is in the raw source data stream, or the in latest destination index.

constant_keyword

log.flags

Flags for the log file.

keyword

log.offset

Offset of the entry in the log file.

long

threat.feed.name

Display friendly feed name

constant_keyword

threat.indicator.first_seen

The date and time when intelligence source first reported sighting this indicator.

date

threat.indicator.last_seen

The date and time when intelligence source last reported sighting this indicator.

date

threat.indicator.modified_at

The date and time when intelligence source last modified information for this indicator.

date

threat.indicator.tls.client.ja3

An md5 hash that identifies clients based on their TLS handshake.

keyword

Example

An example event for feed looks as following:

{
    "@timestamp": "2024-08-01T08:05:14.040Z",
    "agent": {
        "ephemeral_id": "b351d699-2fd0-49f7-99e1-a7a471a29a62",
        "id": "36b03887-7783-4bc4-b8c5-6f8997e4cd1a",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.13.0"
    },
    "cif3": {
        "deleted_at": "2022-09-03T20:25:53.000Z",
        "expiration_duration": "45d",
        "indicator": "20.206.75.106",
        "itype": "ipv4",
        "portlist": "443",
        "uuid": "ac240898-1443-4d7e-a98a-1daed220c162"
    },
    "data_stream": {
        "dataset": "ti_cif3.feed",
        "namespace": "26457",
        "type": "logs"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "36b03887-7783-4bc4-b8c5-6f8997e4cd1a",
        "snapshot": false,
        "version": "8.13.0"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "threat"
        ],
        "created": "2024-08-01T08:05:14.040Z",
        "dataset": "ti_cif3.feed",
        "ingested": "2024-08-01T08:05:26Z",
        "kind": "enrichment",
        "original": "{\"application\":\"https\",\"asn\":8075,\"asn_desc\":\"microsoft-corp-msn-as-block\",\"cc\":\"br\",\"city\":\"campinas\",\"confidence\":10,\"count\":1,\"firsttime\":\"2022-07-20T20:25:53.000000Z\",\"group\":[\"everyone\"],\"indicator\":\"20.206.75.106\",\"indicator_ipv4\":\"20.206.75.106\",\"itype\":\"ipv4\",\"lasttime\":\"2022-07-20T20:25:53.000000Z\",\"latitude\":-22.9035,\"location\":[-47.0565,-22.9035],\"longitude\":-47.0565,\"portlist\":\"443\",\"protocol\":\"tcp\",\"provider\":\"sslbl.abuse.ch\",\"reference\":\"https://sslbl.abuse.ch/blacklist/sslipblacklist.csv\",\"region\":\"sao paulo\",\"reporttime\":\"2022-07-21T20:33:26.585967Z\",\"tags\":[\"botnet\"],\"timezone\":\"america/sao_paulo\",\"tlp\":\"white\",\"uuid\":\"ac240898-1443-4d7e-a98a-1daed220c162\"}",
        "type": [
            "indicator"
        ]
    },
    "input": {
        "type": "httpjson"
    },
    "network": {
        "protocol": "https",
        "transport": "tcp"
    },
    "related": {
        "ip": [
            "20.206.75.106"
        ]
    },
    "tags": [
        "preserve_original_event",
        "forwarded",
        "cif3-indicator",
        "botnet"
    ],
    "threat": {
        "indicator": {
            "as": {
                "number": 8075,
                "organization": {
                    "name": "microsoft-corp-msn-as-block"
                }
            },
            "confidence": "High",
            "first_seen": "2022-07-20T20:25:53.000Z",
            "geo": {
                "country_iso_code": "br",
                "location": {
                    "lat": -22.9035,
                    "lon": -47.0565
                },
                "region_name": "sao paulo",
                "timezone": "america/sao_paulo"
            },
            "ip": "20.206.75.106",
            "last_seen": "2022-07-20T20:25:53.000Z",
            "marking": {
                "tlp": "WHITE"
            },
            "modified_at": "2022-07-21T20:33:26.585967Z",
            "name": "20.206.75.106",
            "provider": "sslbl.abuse.ch",
            "reference": "https://sslbl.abuse.ch/blacklist/sslipblacklist.csv",
            "sightings": 1,
            "type": "ipv4-addr"
        }
    }
}

Changelog

edit
Changelog
Version Details Kibana version(s)

1.15.0

Enhancement (View pull request)
Add "preserve_original_event" tag to documents with event.kind set to "pipeline_error".

8.13.0 or higher

1.14.4

Bug fix (View pull request)
Use triple-brace Mustache templating when referencing variables in ingest pipelines.

8.13.0 or higher

1.14.3

Bug fix (View pull request)
Fix labels.is_ioc_transform_source values

8.13.0 or higher

1.14.2

Bug fix (View pull request)
Add missing fields in transform

8.13.0 or higher

1.14.1

Bug fix (View pull request)
Fix ECS date mapping on threat fields.

8.13.0 or higher

1.14.0

Enhancement (View pull request)
Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.

8.13.0 or higher

1.13.1

Bug fix (View pull request)
Adjust field mappings for transform destination index.

8.12.0 or higher

1.13.0

Enhancement (View pull request)
Improve handling of empty responses.

8.12.0 or higher

1.12.0

Enhancement (View pull request)
Support for IOC expiration

8.12.0 or higher

1.11.0

Enhancement (View pull request)
Set sensitive values as secret.

8.12.0 or higher

1.10.2

Bug fix (View pull request)
Clean up null handling

8.7.1 or higher

1.10.1

Enhancement (View pull request)
Changed owners

8.7.1 or higher

1.10.0

Enhancement (View pull request)
Limit request tracer log count to five.

8.7.1 or higher

1.9.0

Enhancement (View pull request)
ECS version updated to 8.11.0.

8.7.1 or higher

1.8.0

Enhancement (View pull request)
Set community owner type.

8.7.1 or higher

1.7.0

Enhancement (View pull request)
ECS version updated to 8.10.0.

8.7.1 or higher

1.6.0

Enhancement (View pull request)
The format_version in the package manifest changed from 2.11.0 to 3.0.0. Removed dotted YAML keys from package manifest. Added owner.type: elastic to package manifest.

8.7.1 or higher

1.5.0

Enhancement (View pull request)
Add tags.yml file so that integration’s dashboards and saved searches are tagged with "Security Solution" and displayed in the Security Solution UI.

8.7.1 or higher

1.4.0

Enhancement (View pull request)
Update package-spec to 2.9.0.

8.7.1 or higher

1.3.0

Enhancement (View pull request)
Update package to ECS 8.9.0.

8.7.1 or higher

1.2.0

Enhancement (View pull request)
Document duration units.

8.7.1 or higher

1.1.0

Enhancement (View pull request)
Document valid duration units.

8.7.1 or higher

1.0.0

Enhancement (View pull request)
Release Collective Intelligence Framework as GA.

8.7.1 or higher

0.8.0

Enhancement (View pull request)
Ensure event.kind is correctly set for pipeline errors.

0.7.0

Enhancement (View pull request)
Update package to ECS 8.8.0.

0.6.0

Enhancement (View pull request)
Add a new flag to enable request tracing

0.5.0

Enhancement (View pull request)
Update package to ECS 8.7.0.

0.4.1

Enhancement (View pull request)
Honor preserve_original_event setting.

0.4.0

Enhancement (View pull request)
Update package to ECS 8.6.0.

0.3.1

Bug fix (View pull request)
Use ECS definition for threat.indicator.geo.location.

0.3.0

Enhancement (View pull request)
Update package to ECS 8.5.0.

0.2.2

Bug fix (View pull request)
Remove duplicate field.

0.2.1

Enhancement (View pull request)
Fix documentation build error

0.2.0

Enhancement (View pull request)
Labelling with Threat Intelligence category

0.1.0

Enhancement (View pull request)
Initial draft of the package