The plugin references the message field (displayed as event.original in the image below) as the source. When the document is ingested, a SHA-256 hash is generated and populates the document _id . If the same message is subsequently pulled via the RSS feed, instead of creating a new document it updates the matching document.

One drawback we found is that there are outlier scenarios where a duplicate message is not identified as a duplicate because of slight variations. The appearance of a definite article like “the” can cause a new message to be interpreted as different, resulting in a new document being created. We are currently working on testing the title field to replace the message field in hopes of remediating the outliers.

Our project also contains a section dedicated to issues we encounter as we enhance and update this project.

key : The fingerprint method to use. Value can be any of the following: SHA1, SHA256, SHA384, SHA512, MD5, MURMUR3, MURMUR3_128, IPV4_NETWORK, UUID, PUNCTUATION. We simply opted to use Alexander Marquardt’s example key, since there isn’t a default value and it is a bit arbitrary.
method: We chose to use SHA-256 because it is often associated with file, message, and data integrity verification. If the message changes, the hash changes; if the message is unaltered, then the hash is unaltered.
source: The name(s) of the source field(s) whose contents will be used to create the fingerprint. We opted to use the message field; the data populated in the message field is essentially a summary of the article. We figured the message field was a safe bet, but we are testing additional fields due to the aforementioned outliers.
target: The name of the field where the generated fingerprint will be stored. We will chat about this more in the output section, but this is where the SHA-256 hash is sent and it acts as our document _id to alleviate the mass duplication we encountered during our initial testing phase.
concatenate_sources: When set to true and the method isn’t UUID or PUNCTUATION, the plugin concatenates the names and values of all fields given in the source option into one string (like the old checksum filter) before doing the fingerprint computation. We needed this. We tried it without and we got duplicates.
mutate (copy): Copy an existing field to another field. We noticed a couple of RSS feeds were publishing dates in a field called updated versus published. We also recognized the majority were sending publishing date values to the published field. The majority won and we decided to copy the values in the updated to published. We wanted to keep our designated time field consistent for visualizations and dashboards.
mutate (remove field): If this filter is successful, remove arbitrary fields from this event. We opted to use this and it was purely a creative decision (ha!). We aren’t huge fans of keeping excess fields hanging around. We are Elasticians and a lot of our job is data validation and ensuring users optimize whenever and wherever possible. The Feed fields redirected us to unreadable content and the message field was inherently copied to the event.original (per ECS) so it was essentially duplicated values (aka unnecessary).
conditionals (if and else): We noticed Cisco Talos and The Register were sending their publication dates to the field update as opposed to published; they are outliers because the majority sends publication dates to published so we created a conditional leading to a consistent time field.

Enterprise Search

Observability

Security

Elastic Cloud

Elastic (ELK) Stack

Elastic 8.5 released

Upgrade the Elastic Stack

Documentation

ElasticON Comes To You 2022

We're hiring

Power of Elastic

Improving digital customer experiences

Evolving the DevOps lifecycle

Security without limits

Public Sector

Financial Services

Telecommunications

Healthcare

Technology

Retail and Ecommerce

Media and Entertainment

Manufacturing and Automotive

Cybersecurity solutions for a riskier world

Enterprise Search

Observability

Security

Getting started

Support

Contact us

Jaguar Land Rover

Emirates NBD

Zurich Insurance

Documentation

Blogs

Training

Events

Community

Consulting

Driving quantified success with Elastic Enterprise Search

Get started with Elasticsearch

Observability Engineer training

About

Careers

Press

Partners

Investor Relations

Elastic Excellence Awards

Why now is the time to move critical databases to the cloud

Enterprise Search

Observability

Security

Elastic Cloud

Elastic (ELK) Stack

Elastic 8.5 released

Upgrade the Elastic Stack

Documentation

ElasticON Comes To You 2022

We're hiring

Power of Elastic

Improving digital customer experiences

Evolving the DevOps lifecycle

Security without limits

Public Sector

Financial Services

Telecommunications

Healthcare

Technology

Retail and Ecommerce

Media and Entertainment

Manufacturing and Automotive

Enterprise Search

Observability

Security

Getting started

Support

Contact us

Jaguar Land Rover

Emirates NBD

Zurich Insurance

Documentation