Learn how to make advanced data science visualizations by using Vega to combine the strengths of Jupyter Notebooks and Kibana dashboards.

In this blog post you’ll learn how to create visualizations for Kibana dashboards from <a href="https://jupyter.org/">Jupyter Notebooks</a>. The good news for you as a Python developer: You won’t have to spend time writing cumbersome plain JSON documents or even JavaScript code. Under the hood, we’re going to make use of Kibana’s Vega plugin which became <a href="https://www.elastic.co/guide/en/kibana/current/whats-new.html#vega-generally-available">generally available</a> with Elastic Stack 7.10. Jupyter Notebooks are one of the natural habitats of data scientists, offering great ways to explore and visualize data to gain new insights. An important part of a data scientist’s workflow is the final step of sharing that knowledge or deploying a notebook’s content to a production environment. While Jupyter Notebooks themselves are web-based and a static HTML rendered version can be shared easily, a notebook — with its focus on including the full process and code —&nbsp; might not be the right fit for the target audience. This is where Kibana can play a role in a data scientist’s arsenal of tools. Data visualization offers great opportunities to share analysis results with experts and non-experts alike. There’s already a great ecosystem of scientific visualization libraries for Python and Jupyter Notebooks, but they have their pain points. For static data or batch-processed analysis results, it might be sufficient to publish visualizations as non-interactive charts or as part of a workflow using a static website generator. However, visualizations that update in real time when new data becomes available can be difficult with Jupyter Notebooks. On the other hand, Kibana offers a runtime environment to easily create <a href="https://www.elastic.co/guide/en/kibana/7.9/tutorial-build-dashboard.html">real-time charts and dashboards</a> backed by Elasticsearch’s powerful aggregations. However, out of the box, Kibana lacks some of the features data scientists can do much more easily using scientific visualization frameworks. Things like charts using small multiples, binned histograms, scatterplot matrices and boxplot or violin charts. So how can we get the best of both worlds?&nbsp; Luckily, we can make use of a visualization technology that’s usable in both Jupyter Notebooks and Kibana: <a href="https://vega.github.io/vega/">Vega</a>.&nbsp; <h2>Tie Jupyter Notebooks and Kibana together with Vega</h2> Vega allows us to describe how data should be visualized, independent of any programming language. Vega specifications are plain JSON files though, so writing more complex visualizations can be cumbersome. To help with this, we can use the <a href="https://altair-viz.github.io/">Altair library</a> for Python as a wrapper to let us write Vega specifications more easily. The full workflow looks like this: <img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt9a22e745a219c238/5fcc20eca84f233eaf60eefa/blog-jupyter-kibana-1.jpg" data-sys-asset-uid="blt9a22e745a219c238" alt="" style="display: block; margin: auto;"> With this set of libraries the main benefit is that a data scientist can continue to work with the environment/stack they are most familiar with (Python/Jupyter Notebooks), without touching raw Vega JSON specifications or coding custom JavaScript visualizations. So the whole workflow is a good example of how the language-independent Vega spec allows you to move and deploy between stacks (e.g., Python/Jupyter vs. JavaScript/Kibana). <h2>Save time with example workflows</h2> Who here wants to write more JSON? No one? I didn't think so... To get you started, we've created a <a href="https://github.com/walterra/jupyter2kibana">set of Jupyter Notebooks</a> to demonstrate the workflow and some useful approaches which allow you to create scientific visualizations which are otherwise tricky to produce using out-of-the-box Kibana visualizations. Please note that you’ll need to use at least Elastic Stack 7.9 and Altair 4.1 to make sure the underlying shared Vega version supports the created visualizations. The examples begin by demonstrating the general workflow and then guide you how to move on to more complex setups. Let's briefly explore some of the contents and goals of the notebooks. <h2>Data prep and workflow explanation using binned histograms&nbsp; </h2><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt03ac72ed973c0afb/5fcc212d50460c30dd53ed51/blog-jupyter-kibana-2.png" data-sys-asset-uid="blt03ac72ed973c0afb" alt="" style="display: block; margin: auto;">The first set of notebooks cover how to get data from Elasticsearch via <a href="https://eland.readthedocs.io/">eland</a>, and discuss some different approaches for visualizing small multiples of <a href="https://walterra.github.io/jupyter2kibana/viz-1a-flights-histogram.html">binned histograms</a> and <a href="https://walterra.github.io/jupyter2kibana/viz-1b-flights-splom.html">scatterplots</a> with eland and Altair. They cover the basic requirements to successfully create and publish a custom visualization from Jupyter to Kibana.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt8abed062d6e44cf7/5fcc213dd5cc79476633f653/blog-jupyter-kibana-3.jpg" data-sys-asset-uid="blt8abed062d6e44cf7" alt="" style="display: block; margin: auto;"> <h2>Working with Elastic machine learning</h2> We prepared some examples that allow you to visualize results created using <a href="https://www.elastic.co/what-is/elasticsearch-machine-learning">Elastic machine learning</a>. There’s a set of notebooks that covers how to create a dashboard with results from <a href="https://walterra.github.io/jupyter2kibana/viz-2b-cars-outlier.html">outlier detection</a> including a Vega based slider to adjust the threshold for highlighting. The overall workflow can also be used to create custom charts for evaluating results from <a href="https://walterra.github.io/jupyter2kibana/viz-3a-iris-classification.html">classification</a> tasks. Finally, we conclude the examples with some more charts for visualizing <a href="https://walterra.github.io/jupyter2kibana/viz-4a-anomaly-detection.html">anomaly detection results</a>. <img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt2e1edd6c75ebb2b9/5fcc215159bd313bf831142c/blog-jupyter-kibana-4.jpg" data-sys-asset-uid="blt2e1edd6c75ebb2b9" alt="" style="display: block; margin: auto;"><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt1cb4956dec9d35d9/5fcc216de53f3355047db1cd/blog-jupyter-kibana-5.jpg" data-sys-asset-uid="blt1cb4956dec9d35d9" alt="" style="display: block; margin: auto;"> All notebooks can be found on Github here: <a href="https://github.com/walterra/jupyter2kibana">https://github.com/walterra/jupyter2kibana</a> <h2>Summary</h2> In this blog post we outlined a workflow for data scientists to combine the power and flexibility of Jupyter Notebooks with an option to deploy visualizations to Kibana dashboards. Going through the above examples should give you a good understanding of the workflow. If you have any questions about the workflows, I encourage you to reach out on our <a href="https://discuss.elastic.co">Discuss forums</a>. Would you like to try it out for yourself? The quickest way to get started is to get a free <a href="https://www.elastic.co/elasticsearch/service">14-day trial of the Elasticsearch Service</a>, or <a href="https://www.elastic.co/downloads/">download the Elastic Stack</a>.

blog-banner-jupyter-kibana.jpg

blog-thumb-jupyter-kibana.jpg

How to bring Jupyter Notebook visualizations to Kibana dashboards with Vega

Learn how to apply user annotations, a new feature in Elastic machine learning, to augment your ML results with the information you have about your data.

User annotations are a new machine learning feature in Elasticsearch available from 6.6 onwards. They provide a way to augment your machine learning jobs with descriptive domain knowledge. When you run a machine learning job, its algorithm is trying to find anomalies —&nbsp;but it doesn’t know what the data itself is about.&nbsp;The job wouldn't know, for&nbsp;example, whether it was dealing with&nbsp;CPU usage or network throughput. User annotations offer a way to augment the results with the knowledge you as a user have about the data.&nbsp;
In this blog post, we’ll&nbsp;show you how user annotations work and how to apply them to different use cases. We’ll be&nbsp;analysing&nbsp;data&nbsp;from <a href="https://apps.tirol.gv.at/hydro/#/Wasserstand/?station=201525" target="_BLANK">Hydro Online</a>&nbsp;—&nbsp;an open data portal run by Austria’s Tyrolean local government. Hydro Online&nbsp;offers an interface to investigate weather sensor data&nbsp;such as&nbsp;rainfall accumulation, river height, or snowpack totals. As described in one of our <a href="/blog/importing-csv-and-log-data-into-elasticsearch-with-file-data-visualizer">previous blog posts</a>, the File Data Visualizer offers a robust way to ingest data from CSV data, as is found in this case.
<h2>Usage</h2>Let’s start with a single metric job that analyses river height measurements of the River Grossache going through the village of Kössen. Once the job is created, the Single Metric Viewer can be used to add annotations to the results of the analysis. Simply drag over a time range in the chart to create an annotation. A flyout element will pop up to the right, which allows you to add a custom description. In the example below, we annotate an anomalous river height (a&nbsp;major flooding occurred on that date). By creating the annotation, you can make that knowledge available to other users.
<img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/bltb1c486237bfd181d/5cb5a5433aebf35f29365dab/download" data-sys-asset-uid="bltb1c486237bfd181d" alt="" "="">
The annotation is visible&nbsp;in both the chart itself&nbsp;as well as in the Annotations table below it. The label visible in the first column of the table can be used to identify the annotation in the chart. These labels are dynamically&nbsp;created&nbsp;for the annotations on display. When hovering over a row in the Annotations table, the corresponding annotation will also be highlighted in the chart above it.
The annotations created for each job can also be accessed from the Job Management page, where they are displayed in their own tab by expanding a row in the list of jobs. Each annotation in the table includes a link in the right hand column, which takes you back to the Single Metric Viewer with a focus on the time range covered by the annotation. These permalinks can also be shared with others. This means you can use annotations to create bookmarks on particular anomalies to revisit them later on.
<img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/bltccdf9900f2d7720d/5cb5a564b790c01122391df1/download" data-sys-asset-uid="bltccdf9900f2d7720d" alt="" "="">
If there are multiple annotations covering the same time range, annotations will be vertically distributed in the chart to avoid overlap. To edit or delete an annotation, simply click on it in the chart. The flyout element will open again to the right where you’ll be able to edit the text or delete the annotation. From 6.7 onwards, this can also be done by using the edit button in the Annotations table, making&nbsp;this functionality available from the Job Management page too.
<img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/bltb137bc3438e7860b/5cb5a583791abe5a29a1d4f5/download" data-sys-asset-uid="bltb137bc3438e7860b" alt="" "="">
Now that we've covered the basic functionality of how to create and work with user annotations, let's&nbsp;move on to some more use cases.
<h2>Using annotations to verify expected anomalies</h2>Annotations can be used to supply a ground truth to verify if a machine learning job comes up with expected results. In the following example, we are&nbsp;again looking at the river level data from <a href="https://apps.tirol.gv.at/hydro/#/Wasserstand/?station=201525" target="_BLANK">Hydro Online</a> and are now aiming to&nbsp;automatically&nbsp;overlay historic events as annotations on the anomaly results. As a data scientist, for example, your work might include obtaining and preparing both the source data you want to analyse as well as the data set to verify the results.
For our own analysis, we need the raw dataset.
<img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/blte10b1840268283bc/5cb5a5a382d7ee1622f86b1a/download" data-sys-asset-uid="blte10b1840268283bc" alt="" "=""> 
Luckily, in this case, in addition to investigating data via the web interface, we can also download historic data&nbsp;for further analysis. For this example we’ll use the <a href="https://ehyd.gv.at/eHYD/MessstellenExtraData/owf?id=202382&file=8">river height data</a> of the River Grossache measured at the “Huette” measurement point. The annotations covering the desired ground truth will be created from a <a href="https://www.tirol.gv.at/fileadmin/themen/umwelt/wasserkreislauf/wasserstand/downloads/Tafeln_Koessen_Huette.pdf" target="_BLANK">document describing severe river heights and floods</a>.
<table style="width: 100%">
<tbody>
<tr>
	<td style="border: 0px; width: 50%">
		<img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/bltf6aa0ed4408c2c99/5cb5a6dc3e0de5c323b87308/download" data-sys-asset-uid="bltf6aa0ed4408c2c99" alt="" "="">
	</td>
	<td style="border: 0px; width: 50%">
		<img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/blt7c57cde13dd3e014/5cb5a5d6791abe5a29a1d4fb/download" data-sys-asset-uid="blt7c57cde13dd3e014" alt="" "="">
	</td>
</tr>
</tbody>
</table>In addition to&nbsp;using the UI previously&nbsp;described,&nbsp;machine learning annotations are&nbsp;stored as documents in a separate standard Elasticsearch index.&nbsp;Annotations can also be created programmatically or manually using standard Elasticsearch APIs. Annotations are stored in a version-specific index, and should be accessed via the aliases <code>.ml-annotations-read</code> and <code>.ml-annotations-write</code>. For this example, we'll add annotations to reflect the historic river events before creating our machine learning job.
<pre class="prettyprint">{
 "_index":".ml-annotations-6",
 "_type":"_doc",
 "_id":"DGNcAmoBqX9tiPPqzJAQ",
 "_score":1.0,
 "_source":{
 "timestamp":1368870463669,
 "end_timestamp":1371015709121,
 "annotation":"2013 June; 770 m3/s; 500 houses flooded.",
 "job_id":"annotations-leukental-4d-1533",
 "type":"annotation",
 "create_time":1554817797135,
 "create_username":"elastic",
 "modified_time":1554817797135,
 "modified_username":"elastic"
 }
}
</pre>We’ll now create a machine learning job to find anomalies in maximum river height&nbsp;using a&nbsp;name that matches the <code>job_id</code> field from the annotation above so that it picks up the manually created annotations. This is how this job looks in the Single Metric wizard once we ingest&nbsp;the historic river data into an Elasticsearch index:
<img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/blt1787a021b6855637/5cb5a71fb790c01122391e1f/download" data-sys-asset-uid="blt1787a021b6855637" alt="" "="">
The important bit here is that the job name we chose matches the one used for the annotations. Once we run the job and move to the Single Metric Viewer, we&nbsp;should see annotations corresponding to the anomalies in river height that the machine learning job detected:
<img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/blt28e9133b42e9986b/5cb5a73830313775298d66b2/download" data-sys-asset-uid="blt28e9133b42e9986b" alt="image9.png">
This technique offers a great way to verify if the analysis you’re running is valid when compared to pre-existing validation data stored as annotations.
<h2>Annotations for system events</h2>In addition to the user-generated annotations&nbsp;above, the machine learning backend automatically creates annotations in some circumstances for system events.
<img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/blt26871e9e546a8b1c/5cb5a74c3e0de5c323b87310/download" data-sys-asset-uid="blt26871e9e546a8b1c" alt="" "="">
The screenshot above shows an example of an automatically created annotation. In this case, a real-time machine learning job was run, but data ingestion wasn’t able to keep up with the ingestion rate required for the job. This meant&nbsp;documents were added to the index after the job had run its analysis on the bucket. The automatically created annotation highlights this issue that was previously hard to spot and debug. The annotation text features detail&nbsp;the identified problem and provide a suggestion on how to solve it —&nbsp;in this case increasing the query_delay setting.
<h2>Alerting integration</h2>Even before the availability of user annotations for machine learning, you could&nbsp;use <a href="https://www.elastic.co/guide/en/x-pack/current/watcher-getting-started.html">Watcher</a> to create alerts based on anomalies identified by machine learning jobs. While that is a great improvement when compared to alerting on basic thresholds, the alerts may&nbsp;be&nbsp;too granular for&nbsp;the target group that receives the alerts. As a user of machine learning jobs, annotations can give you a way to curate what gets triggered as Watcher alerts and what gets passed on to other stakeholders. Since annotations are stored in their own Elasticsearch index, you can use Watcher to simply react to newly created documents in that index and trigger notifications. Watcher can also be configured to <a href="https://www.elastic.co/guide/en/watcher/current/configuring-slack.html">send alerts to a Slack channel</a>. The following configuration gives you an example on how to create a watch to trigger Slack messages when a new annotation gets created:
<pre class="prettyprint">{
 "trigger": {
 "schedule": {
 "interval": "5s"
 }
 },
 "input": {
 "search": {
 "request": {
 "search_type": "query_then_fetch",
 "indices": [
 ".ml-annotations-read"
 ],
 "rest_total_hits_as_int": true,
 "body": {
 "size": 1,
 "query": {
 "range": {
 "create_time": {
 "gte": "now-9s"
 }
 }
 },
 "sort": [
 {
 "create_time": {
 "order": "desc"
 }
 }
 ]
 }
 }
 }
 },
 "condition": {
 "compare": {
 "ctx.payload.hits.total": {
 "gte": 1
 }
 }
 },
 "actions": {
 "notify-slack": {
 "transform": {
 "script": {
 "source": "def payload = ctx.payload; DateFormat df = new SimpleDateFormat(\"yyyy-MM-dd'T'HH:mm:ss.SSS'Z'\"); payload.timestamp_formatted = df.format(Date.from(Instant.ofEpochMilli(payload.hits.hits.0._source.timestamp))); payload.end_timestamp_formatted = df.format(Date.from(Instant.ofEpochMilli(payload.hits.hits.0._source.end_timestamp))); return payload",
 "lang": "painless"
 }
 },
 "throttle_period_in_millis": 10000,
 "slack": {
 "message": {
 "to": [
 "#&lt;slack-channel&gt;"
 ],
 "text": "New Annotation for job *{{ctx.payload.hits.hits.0._source.job_id}}*: {{ctx.payload.hits.hits.0._source.annotation}}",
 "attachments": [
 {
 "fallback": "View in Single Metric Viewer http://&lt;kibana-host&gt;:5601/app/ml#/timeseriesexplorer?_g=(ml:(jobIds:!({{ctx.payload.hits.hits.0._source.job_id}})),refreshInterval:(pause:!t,value:0),time:(from:'{{ctx.payload.timestamp_formatted}}',mode:absolute,to:'{{ctx.payload.end_timestamp_formatted}}'))&_a=(filters:!(),mlSelectInterval:(interval:(display:Auto,val:auto)),mlSelectSeverity:(threshold:(color:%23d2e9f7,display:warning,val:0)),mlTimeSeriesExplorer:(zoom:(from:'{{ctx.payload.timestamp_formatted}}',to:'{{ctx.payload.end_timestamp_formatted}}')),query:(query_string:(analyze_wildcard:!t,query:'*')))",
 "actions": [
 {
 "name": "action_name",
 "style": "primary",
 "type": "button",
 "text": "View in Single Metric Viewer",
 "url": "http://&lt;kibana-host&gt;:5601/app/ml#/timeseriesexplorer?_g=(ml:(jobIds:!({{ctx.payload.hits.hits.0._source.job_id}})),refreshInterval:(pause:!t,value:0),time:(from:'{{ctx.payload.timestamp_formatted}}',mode:absolute,to:'{{ctx.payload.end_timestamp_formatted}}'))&_a=(filters:!(),mlSelectInterval:(interval:(display:Auto,val:auto)),mlSelectSeverity:(threshold:(color:%23d2e9f7,display:warning,val:0)),mlTimeSeriesExplorer:(zoom:(from:'{{ctx.payload.timestamp_formatted}}',to:'{{ctx.payload.end_timestamp_formatted}}')),query:(query_string:(analyze_wildcard:!t,query:'*')))"
 }
 ]
 }
 ]
 }
 }
 }
 }
}
</pre>In the configuration above, just replace <code>&lt;slack-channel&gt;</code> and <code>&lt;kibana-host&gt;</code> with your settings and use it to create an <a href="https://www.elastic.co/guide/en/kibana/current/watcher-create-advanced-watch.html">advanced watch</a>. Once everything is set up, you should receive a Slack notification every time you create a new annotation —&nbsp;including the annotation text and a link back to Single Metric Viewer.
<img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/blt8fc884c15ecd2fcf/5cb5a765fe8386be234acb7e/download" data-sys-asset-uid="blt8fc884c15ecd2fcf" width="650">
<h2>Summary</h2>In this article we introduced the new annotations feature for Elasticsearch machine learning. It can be used for adding annotations via the UI&nbsp;and for system annotations triggered via backend tasks. These annotations are available as bookmarks via the Job Management page and are sharable as links with others. Annotations can be created programmatically from external data to be used as a ground truth overlay for detected anomalies. Finally, in combination with Watcher and the slack action in Elasticsearch, we’ve seen how annotations can be used for curated alerting. Have fun with annotations, and find us on the <a href="https://discuss.elastic.co/">Discuss forums</a>&nbsp;if you have any questions.

ml-annotations-header-optimized.jpg

ml-annotation-edit-thumbnail.png

User annotations for Elastic machine learning

Introduced in Elastic Stack 6.3, job validation for machine learning catches misconfigurations and helps you tweak bucket spans, partition fields and more.

Articles by Walter Rafelsberger

How to bring Jupyter Notebook visualizations to Kibana dashboards with Vega

User annotations for Elastic machine learning

How to Optimize Elasticsearch Machine Learning Job Configurations using Job Validation

Products & Solutions

Company

Resources

Featured topics

News

Articles by Walter Rafelsberger

How to bring Jupyter Notebook visualizations to Kibana dashboards with Vega

User annotations for Elastic machine learning

How to Optimize Elasticsearch Machine Learning Job Configurations using Job Validation