Find out how Siren uses the real-time capabilities of the Elastic Stack to power their investigative intelligence platform. Learn how they use Elastic to tackle big challenges: tracking relationships between diverse data and diverse analyst needs.

At Siren, we build a platform used for “investigative intelligence” in Law Enforcement, Intelligence, and Financial Fraud. Investigative intelligence is a specialisation of data analytics that serves the needs of those that are typically hunting for bad actors. Such investigations are the primary focus of law enforcement and intelligence, but are also critical to uncovering financial crime activities and for threat hunting in cybersecurity.
At the heart of Siren is the Elastic Stack. With its real-time responses and ability to deal with ever-increasing amounts of structured and unstructured data, <a href="/elasticsearch">Elasticsearch</a> provides the ideal fabric for investigative analysis. Furthermore, thanks to the Siren and Elastic partnership, all the advanced features of an Elastic Platinum subscription can be used in Siren as part of US Federal Siren deployments, giving investigators unprecedented flexibility, capabilities, and operational resilience. At Siren, we set out to use Elasticsearch to tackle two of the biggest problems in investigative intelligence: disjointed data and disjointed tools. With that in mind, in this blog post, we would like to share part of our approach with the broader community of technologists and architects who are trying to get the most out of their data.
<h2>Two challenges for modern investigative analytics</h2>At the data level, the main challenge in investigative intelligence is disjointed data: investigators need to be able to follow non-obvious relationships across a myriad of diverse data sets and data sources.&nbsp;
A more subtle challenge, however, is the diversity of the analysis needs, which traditionally forced organizations and analysts to either use many disconnected tools and backends (APIs) or embark on building expensive and rigid ad-hoc integrations.&nbsp;
For example, investigators certainly need link analysis — to see and explore the connections between records — but they also need fast drilldowns, business intelligence (BI)-style visuals, and text search and analysis for unstructured data.&nbsp;
Our challenge from the onset has been: How can we deliver a unified investigative analytics platform that is also architecturally modern and easy to deploy?
<h2>Stepping up to the challenge</h2>With its real-time responses, its array of <a href="/guide/en/elasticsearch/reference/7.9/search-aggregations.html">real-time analytics functions</a>, and its powerful search engine — capable of dealing with fuzzy searches and noisy data — Elasticsearch immediately stood out as the backend providing the ideal starting point for investigative analysis.
That is why we built the Siren platform — a unified tool that enables big, disconnected data analytics — on top of Elasticsearch and Kibana.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt2d030b406ac6417a/5f761aeddac9270efe35f0c7/blog-siren-platform.png" data-sys-asset-uid="blt2d030b406ac6417a" alt="The Siren platform" style="display: block; margin: auto; width:500px;">
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt189f1fb680ad09c5/5f6a6a252b93e84e4c32a297/blog-siren-graph-analysis-example.jpg" data-sys-asset-uid="blt189f1fb680ad09c5" alt="" style="display: block; margin: auto;">
<h2>Analysing disjointed data in Elasticsearch: Data model, joins, and link analysis</h2>The investigative world is made up of disjointed data that needs to be connected. People (for example) are connected to vehicles they own, which are connected to locations where they’ve been, which may be connected to events, and so on.
In Elasticsearch these are typically recorded in separate indices, possibly coming from all sorts of sources. Siren leverages the real-time speed of Elasticsearch to tie this data together for investigators, regardless of source or index.
<h3>Tying data together with an associative data model</h3>In Siren Investigate — Siren’s frontend built on <a href="/kibana">Kibana</a> — administrators or advanced analysts define an Associative Data Model on top of their existing data, and this data model then drives all the analytic operations.&nbsp;
The Data Model editor is visual, allowing you to define how tables are interconnected, typically by shared keys, which are then used to join the records. For example, an Associative Data Model in law enforcement can be defined to connect tables which contain persons with vehicles, cases, automatic camera licence plate readings, and more.
One uses the visual editor to specify the primary and foreign keys to be used as associations. For example, here are the connections for the Crimes index:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc03f34ff4e2c9684/5f6a6a50b038186a244ba518/blog-siren-visual-editor.jpg" data-sys-asset-uid="bltc03f34ff4e2c9684" alt="" style="display: block; margin: auto;">
And the overall model can also be seen as a single picture, such as this graph visualization of the connections between persons and other entities.&nbsp;
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltdc27c3420fabf654/5f6a6aab128a9c4e4d50c138/blog-siren-analyzed-emails.jpg" data-sys-asset-uid="bltdc27c3420fabf654" alt="" style="display: block; margin: auto;">
<figcaption>A view of the connections between persons and other entities
</figcaption>In another example — a cybersecurity scenario — it is common to use concepts such as IPs, MD5 hash values, emails or user IDs to tie together security logs. The following screenshot shows the relationships between different IPs.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltcdbf230d9b2e27d3/5f6a6a80128a9c4e4d50c134/blog-siren-ip-relations.jpg" data-sys-asset-uid="bltcdbf230d9b2e27d3" alt="" style="display: block; margin: auto;">
<figcaption>A view of the relations between different IPs
</figcaption>All the examples above share specific identifiers, but, as we’ll see later, “fuzzy” relations can be similarly accounted for (e.g., relations coming from Natural Language Processing (NLP) or Entity Resolution).
<h3>Data model-powered associative drilldowns (and link analysis)</h3>The data model enables a special kind of investigative capabilities called associative drilldowns and link analysis&nbsp;investigations.
Let’s see this in action with a financial investigation example, where we have companies that have received investments by investors, as well as articles that mention companies (and often their investments). This is represented by the following data model:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blta49497ea09bfc836/5f6a6a66b038186a244ba51c/blog-siren-articles.jpg" data-sys-asset-uid="blta49497ea09bfc836" alt="" style="display: block; margin: auto;">
<figcaption>Here, the articles-to-companies relation comes from the NLP engine.
</figcaption>Thanks to the data model in Siren, we can drill down based on what’s connected to a set of records.&nbsp;
For example, in the screenshot below, the relational navigator, the user sees in real time how many records are connected (351,243 articles and 41,298 investments) to the current set and is navigable with the click of a button.&nbsp;
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltfb61ae1cd1578159/5f6a6af394533468847809f2/blog-siren-drilldown-1.jpg" data-sys-asset-uid="bltfb61ae1cd1578159" alt="" style="display: block; margin: auto;">
Drilling into investments in the relational navigator button brings us to a tailored dashboard, where we can drill down further.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt3a50ef7f9d89073b/5f6a6b0a8350724defa21ab0/blog-siren-drilldown-2.jpg" data-sys-asset-uid="blt3a50ef7f9d89073b" alt="" style="display: block; margin: auto;">
The process can be then repeated — for example, to move from here to the 2,535 investors who made these investments in 2012.
Under the hood, these real-time interactive associative capabilities and the relational button make use of the <a href="https://siren.io/elasticsearch-investigative/">Siren Federate Plugin</a>, which extends the Elasticsearch query DSL to include cluster scalable join/correlation capabilities.&nbsp;
Siren Federate also enables working across different backends: it has a series of drivers that can see data in remote backends as if they were in Elasticsearch (<a href="https://siren.io/the-alternatives-to-etl-into-elasticsearch-virtualize-or-reflect/">virtual indices</a>).
<h3>Pivoting to graph mode: Siren Link analysis</h3>Being able to do associative drilldowns is great, but there are questions that no dashboard can answer.&nbsp;
For example:
<ul>
	<li>Which investors invested in which company?</li>
	<li>Are they investing together or in groups?</li>
	<li>Are there groups that appear to be investing in competing companies?</li>
</ul>For these questions, Siren's ability to move from dashboards to link analysis is key.
I simply dragged and dropped the filtered Investments dashboards and I can quickly see how they connect
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltdf679f6942e75875/5f6a6b75f0d2b04fae76ca69/blog-siren-link-analysis.jpg" data-sys-asset-uid="bltdf679f6942e75875" alt="" style="display: block; margin: auto;">
<h4>Elasticsearch aggregations for big (graph) link analysis&nbsp;</h4>Elasticsearch aggregations can be used on demand to summarize the graph. The sidebar of the link analysis visualization allows you to choose the aggregation criteria to display edges which summarize (e.g., count/rollup) all the nodes between two entities.&nbsp;
In this example we’re counting the number of articles which co-mention the two companies but also the significance of the co-mention — as outputted by the incredibly useful <a href="/guide/en/elasticsearch/reference/7.9/search-aggregations-bucket-significantterms-aggregation.html">significant terms</a> aggregation in Elasticsearch.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltaefb9e9523ec374e/5f6a6b20a4d7b56a23867264/blog-siren-agg-link-analysis.jpg" data-sys-asset-uid="bltaefb9e9523ec374e" alt="blog-siren-agg-link-analysis.jpg"> 
<figcaption>Link analysis: Articles displayed as aggregate links between companies
</figcaption><h3>Efficient shortest path queries in Elasticsearch</h3>Finding the shortest and most significant path across connected records (phone calls, messages, social links) is a typical example of a widely used investigative graph algorithm.
Efficient shortest path in Elasticsearch is another operation made possible by the Siren Federate plugin technology. <a href="https://www.youtube.com/watch?v=UYlubDhpHGA&feature=youtu.be">Here it is in action finding connections between two users, six phone hops away</a>.
Another very important investigative graph algorithm is the ability to find a “common communicator” among nodes. In the following screenshots we find that a common communicator exists between these three companies.&nbsp;
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltdd267d30223bad85/5f6a6bdf5e85ad4dee190e17/blog-siren-shortest-path.jpg" data-sys-asset-uid="bltdd267d30223bad85" alt="" style="display: block; margin: auto;">
<figcaption>Identifying Microsoft as the common communicator as mentioned in articles with other companies
</figcaption><h3>Web services and advanced geo/time/spatial analysis</h3>Sometimes it’s not possible to get an answer with a simple operation, such as with a shared key join.&nbsp; Siren supports calling remote web services and fitting their results back in the data model.&nbsp;
This capability can be used in many ways — for example, to pull in data on demand (e.g., to access remote knowledge or reference data) and ask for advanced computations on demand.&nbsp;
Let’s take for example a COVID-19 simulation scenario: have two phones been in physical proximity for more than 15 minutes? In the next screenshot, Siren is configured to use a web service (which implements the complex logic required to deal with noisy and spotty data) and makes the results available for analysis.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt0c091d1aabd3f757/5f6a6bf7a4d7b56a2386726a/blog-siren-geo-dashboard.jpg" data-sys-asset-uid="blt0c091d1aabd3f757" alt="" style="display: block; margin: auto;">
As expected, geo time analysis is critical in investigative intelligence, and Siren builds on the extraordinary <a href="/maps">geo capabilities of Elasticsearch</a> to provide this in an analyst-interactive way. The following screenshots illustrate some of the capabilities, which include graph over time evolution and analyst-activated Elasticsearch stored layers.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt4aab0edb1e52660e/5f6a6c0c6f40ab4648b5e50b/blog-siren-geo-world.jpg" data-sys-asset-uid="blt4aab0edb1e52660e" alt="" style="display: block; margin: auto;">
<figcaption>Using the timeline mode to view spatial and temporal data
</figcaption><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt51aa4a14c3844db9/5f6a6c254a19ef4fa853eee3/blog-siren-geo-gb.jpg" data-sys-asset-uid="blt51aa4a14c3844db9" alt="" style="display: block; margin: auto;">
<h2>Conclusion&nbsp;</h2>Elasticsearch is the ideal centerpiece backend for large-scale, interactive structured and unstructured data analysis. It was a natural foundational choice for Siren in its mission to provide a unified intelligence analytics experience — and connect disjointed data.Interested? Try Siren now with our freely available <a href="https://siren.io/downloads/">Siren Community Edition</a> and our nice <a href="https://siren.io/getting-started/">getting started tutorial</a>.&nbsp;
<h2>Learn more</h2><a href="/webinars/elastic-and-siren-protecting-people-assets-and-networks">Elastic and Siren: Protecting people, assets, and networks (video)</a>
<h2>About Siren</h2><a href="https://www.siren.io">Siren</a> provides investigative intelligence based on Elasticsearch to some of the world’s largest and most complex organizations.&nbsp;
<h2>About Dr. Giovanni Tummarello</h2>Giovanni Tummarello, Ph.D is a Computer Scientist and entrepreneur, co-founder and Chief Product Officer at <a href="https://www.siren.io/">Siren.io</a>. He led the team at the National University of Ireland Galway <a href="https://scholar.google.com/citations?user=x-8b-ZkAAAAJ&hl=en">researching</a> on Knowledge Graphs, Search Engines and related UI/UX which then spun off into Siren. Previously, while at the FBK Institute in Trento Italy he led a Semantic Web team and co-founded business information company Spaziodati.eu.&nbsp;

blog-generic-black-banner.png

blog-thumb-siren.png

Investigative analysis of disjointed data in Elasticsearch with the Siren Platform

92754888-4a281080-f359-11ea-9b83-8ba7b421e968.jpg

Co-founder & Chief Product Officer

Dr. Giovanni Tummarello

Articles By

Register to Attend

Sign In to Attend

Register to Watch

You'll also receive an email with related content

Watch Now

Learn More

Contact Info

See All Posts

What to explore next...

Load more news

Load more press releases

Load More

View more learning opportunities

Translate Content

Articles by Dr. Giovanni Tummarello

Investigative analysis of disjointed data in Elasticsearch with the Siren Platform

Products & Solutions

Company

Resources

Featured topics

News

Articles by Dr. Giovanni Tummarello

Investigative analysis of disjointed data in Elasticsearch with the Siren Platform