May 3, 2017 Engineering

Operational Analytics with Elasticsearch at Elastic{ON} 2017 - Part 2

By Dale McDiarmidJesse Lovelace

This blog post continues the description of the Elastic{ON} Operational Analytics demo, describing how the georeferenced WiFi data collected in the previous post was visualised using a custom tile map of Pier 48, the Elastic{ON} venue. Whilst using this demo as an example, this same approach could potentially be reapplied to visualise any geo-enriched metrics on a custom map. For example, are you using Beats to capture and ingest geocoded metrics for each server in your 100,000 sq-ft data centre? Read on and use the following approach to plot server metrics over a floor plan of the data centre.

Background

In the previous blog post, we described the approach used to collect wireless network usage metrics at Elastic{ON} 2017. Using a combination of Packetbeat, SNMP and Ingest Node all DNS and HTTP traffic was encoded with a coordinates describing the approximate location of the source before being indexed into Elasticsearch. More specifically, the locations referred to Ruckus WiFi access points that the device generating a packet was likely connected to. Similarly, sentiment data collected from happy or not consoles was also geo-enriched with the location of each terminal.

Now that we had geo-enriched data, we needed to address the challenge of plotting attendee activity and sentiment over a floor plan of the Elastic{ON} venue. From a Kibana visualization perspective, this represented the largest technical challenge. By default, the Kibana tile map visualization utilises the Elastic Tile Service to render a world map over which Elasticsearch data points can be plotted. However, since version 4.2 this visualization also allows the user to specify any WMS compliant map server as the data source. This would obviously require us to host a WMS service to serve the tiles of our floor plan.

A WMS tile service effectively serves images of a predetermined size (256x256) for the coordinates and zoom level requested. These “tiles” can either be pre-generated and served from a datasource or generated dynamically (typically a combination of both with heavy caching involved). We therefore had two fundamental choices:

  1. The WMS compliant Tile Service to use
  2. The datasource for the tiles

Multiple approaches were considered beyond the scope of this blog. Our final choices were largely motivated by the desire for a simple and repeatable process, limited time and the relative small volume of data that would be involved - the comparative size of the data would be tiny in comparison to the challenge of a full global tile service with multiple zoom levels!

GeoServer provides an open source, feature rich and accessible means of providing a WMS service. For newbies to the domain, the simple setup and UI were decisive.

Considering the above criteria and choice of WMS service our data sources were limited to:

  • A TIFF image of the floor map which could be georeferenced thus producing a GeoTIFF as described in this earlier post plotting NHL match data over a hockey rink.
  • A vectorised representation of the data in the form of ESRI(tm) Shapefiles (*.shp) as supported by GeoServer.

Whilst the former represented the simplest process, early experiments resulted in issues with more complex granular images and zoom levels. The hockey rink represented a simplified set of black lines - floor plans with text and labels are more detailed. As the user zoomed, the image would be sampled to generate the tiles resulting in poor images. We considered using different source images for different zoom levels - this would complicate the process as well as making it image dependent and harder to replicate for future readers.

The vectorised approach we describe below fulfilled all our requirements - simple, repeatable and more importantly decent results at all zoom levels! By providing GeoServer with a vector representation, it is able to more effectively sample the data at different zoom levels. More importantly, the vector could consist of “layers”, each representing a different feature e.g. chairs, which could in turn be configured to only be drawn at specific zoom levels e.g. don’t show text at higher zoom levels.

On settling on the vectorised source we describe the process assuming the user has either an svg or vector based pdf of the desired image. Courtesy of our design team we started with the following.

Pre-requisites

In devising the following approach, we aimed to only use open source tooling to maximise accessibility. Although we do appreciate commercial design tools (such a Adobe Illustrator) and think that its plugins for georeferencing would likely simplify number of steps or provide more flexibility. Furthermore, it is very likely that the process we describe below could be further refined and optimised. Contributions and suggestions are welcome. The following tools and minimum versions are required:

The Process

Preparing the Data

The following section creates the necessary shapefiles from a raw pdf or svg for publishing in GeoServer.

For the purposes of brevity, we will assume our Elastic{ON} floor map is greatly simplified to the following features for the following example:

  • Outer walls
  • Inner walls
  • Demo booths
  • Demo monitors (the little grey boxes inside the teal demo booths)

Removing all other features from our image and retaining the above simplifies the image greatly:

In practice we simply repeated the same process adding and styling more layers as required to deliver the earlier image. The above simplified image as a svg is available here for you to use when following the process below. The complete Elastic{ON} floor map is also available here as a pdf for testing. For those wishing to skip this section, the shapefiles for the simplified svg can also be downloaded here. The following process can be applied to either a vector pdf or svg.

Create Layers

Within the context of serving tiles through a WMS service, a layer can be thought of as those image features you wish to render at the same zoom level and with the same style. For the above image we have 4 obvious layers: the outer walls, inner walls, demo booths and demo monitors. Whilst svg has no concept of layers, their use in vector pdfs is optional. Any existing layers may also not represent those features with the same desired styling, thus requiring modifications.  We use Inkscape's vector manipulation capabilities to open our pdf/svg and group image features together to form the desired layers.

Whilst the following describes the steps for creating the “outer_walls” layer using our simplified floor plan, the same steps are required for any pdf/svg:

  1. Open the pdf/svg in Inkscape. If opening a pdf you will be prompted to select the precision. For simpler images the value selected here shouldn’t matter. However, if you have meshes consider how accurately you want them to be rendered. High values will result in potentially larger shape files and slower tile rendering.
  2. By default your image will have a single layer consistent with the filename. Select Layer | Add Layer from the main menu. When prompted provide an appropriate name e.g. “outer_walls”. For “position” the value “Above current” can be selected.
  3. Select all outer walls in the image. Right click selecting the menu option “Move to Layer”. Select “outer_walls” from the subsequent menu.
  4. The layers menu, accessible in the footer, is useful for hiding and showing specific layers at a time.

Disabling all layers other than “outer_walls” should produce the following:

Repeat the above for the remaining layers to create “inner walls”,”demo booths” and “demo monitors”.

Scaling the Image

Once all layers have been created the size of the respective image must be considered. The layers created previously will each be exported to a dxf intermediary format. By default the dimensions of this file will be later directly interpreted as longitude and latitudes by QGIS, with the units ignored. Values higher than 90 and 180 will therefore be invalid positions for latitude and longitude respectively, resulting in incomplete/incorrect rendering. Whilst more complex solutions exist, involving transformations in QGIS, we can simply ensure the values fall within the required scaled - ideally covering sufficient area for Kibana’s default zoom whilst avoiding polar extremes. We have 2 potential solutions to this problem:

  1. Select a different set of units which cause the dimensions to fall within the desired range. The units mentioned above are irrelevant as they are later ignored.
  2. Re-scale the image to the sizes required using an appropriate set of units.

Our example svg has dimensions of 1530 x 990 px. Converted to cm, this conveniently equates to 43.18 x 27.94.  The latitude and longitude range of the image will therefore be 0°N, 43.18°E (top right). To scale the image or simply change the units, use File | Document Properties.

Export to DXF

Each of layer’s created above must be imported into QGIS. Currently the most only common format supported by both tools is Desktop Cutting Plotter format (dxf). This therefore acts as an intermediate representation only. To ensure each layer can be treated independently in QGIS, we save each layer as a separate .dxf file in InkScape. Repeat the following process for each layer. We use “outer_walls” as an example:

  1. Select File | Save As
  2. Specify dxf as the save format, using the name of layer you intend to save as the filename, before clicking “Save”.

  1. On clicking “Save”, you will be presented with a dialog requesting which layers to include in the dxf file. Ensure the following options are specified correctly before selecting “OK”:
    • The “Base unit” determined earlier, which delivers the correct range of values for latitude and longitude, should be specified. For our example svg - “cm”.
    • “Layer Export Selection” should be set to “By name match”.
    • “Layer match name” should be set to the name of the layer being saved e.g. “outer_walls”.

All other values can use respective defaults.

  1. Repeat the above process for the remaining 3 layers.

If using the example svg, you should have 4 dxf files on completion of the above process - one for each of the layers.

Geo Reference

Our dxf files created above currently lack geo information i.e. the latitude and longitude of each layer. For this purpose we utilise QGIS.

The following provides the steps for geo referencing a single layer, using the “outer_walls” dxf file as an example. This process should be repeated for each layer generated previously.

  1. Open GIS and start a new project.
  2. Drag the dxf file, representing the layer for referencing, onto the canvas.
  3. QGIS will prompt for the Coordinate Reference System to use. A plethora of options and standards are available well beyond the scope of this blog or writer’s knowledge. The default WGS 84 represents the system used by GPS and can be used here.

  1. Rename the layer using the layers panel as shown below:

Repeat the above process for all layers. For the example, svg the result should look like the following: Note how the styles have been lost. These will need to be recreated later in GeoServer.

Note the coordinates of the image features in the footer. These should be same as the dimensions earlier identified in Inkscape. Our previous units have been interpreted as lat/longs.

Export to Shapefiles Files

GeoServer accepts a variety of data source formats. For vectorised files, ESRI(tm) Shapefiles (*.shp) represent the closest to a standard can be produced by both open source and commercial tools. These include the geo information added above and required to render tiles through the WMS service to Kibana.

Repeat the following process for each layer, exporting each to a separate shp file. Note that each layer export will generate a number of additional files (dbf,prj,qpj,shx), in addition to the shp file, that are later required. Again we use our “outer_walls” layer for purposes of example.

  1. Via the “Layers Panel”, right click on the layer selecting “Save As” as illustrated below.

  1. On the subsequent dialog simple save the file to a unique folder with the name of the layer. All defaults can be utilised.

Repeat the above process for each layer, saving the files to the same folder. For the simplified example svg the result should be 20 files i.e.

Loading the Data

The previous process should have yielded a folder of files, including a shp file for each layer. This represents our vector data, complete with geo referencing, for use in GeoServer. The installation of GeoServer is beyond the scope of this blog, with excellent installation instructions available, we have provided a docker container convenience. The reader is additionally encouraged to read the basic GeoServer concepts of workspaces, data sources, layers and styles.

Running Geoserver in a Container

A simple GeoServer Docker container is available here. To build and run the container, follow the steps in the README provided.  Or to just pull and run it:

   docker run -p 8080:8080 m3ntat/geoserver

The default credentials are admin/geoserver.

Importing

The following describes the process for importing the data created in the previous section. We assume the user is hosting GeoServer on localhost at port 8080 and importing the files generated from the simplified Elastic{ON} floorplan.

  1. Navigate to the admin interface of GeoServer at http://localhost:8080/geoserver/web/
  2. Select the menu option Data | Workspaces
  3. From the subsequent page select “Add new workspace”. Create a workspace with an appropriate name and uri. For the purposes for example, we use “Elastic_ON_Simple” for each.

  1. From the workspace listings, select the newly created workspace. Enable the WMS service on the subsequent page and save.

  1. Copy the folder of shape files produced in the previous section to the GeoServer data directory (default location).
  2. Select the menu option Data | Stores, followed by the option “Add new Store”. From the subsequent page select “Directory of spatial files (shapefiles)”.
  3. From the subsequent form, select your created Workspace and provide a name for the datasource. Browse to select your directory of shapefiles. The following illustrates the process for the shapefiles created earlier for the simplified svg.

  1. On clicking “Save” the user will be asked to publish each of the layers detected in the shapefiles. For the example, this should require 4 layers to be published.

Create Styles

In order to determine how they are rendered to tiles, each layer must have styles applied.

GeoServer supports several mechanisms for providing styles. The default SLD, which we use below, is well documented with plenty of cookbook examples. It is, however, a little clunky - relying on an XML definitions. Others may wish to explore the CSS extension beyond the scope of this blog.

Styles can be potentially reused across layers if the desired appearance is the same (you may just have different layers for different zoom levels). For our simplified Elastic{ON} svg, however, each of our 4 layers require a different appearance - we therefore create a style per layer of the same name.

Repeat the following process for each of your layers. If using the example svg, download the XML styles for each layer from here.

  1. Select the menu option Data | Styles
  2. On the following page click “Add a new style”
  3. On the following form, provide a name for the style and select the workspace created earlier. For the example svg, the style name can be consistent with its target layer. Other images may require a naming scheme for generic styles applicable to many layers. In the Style Editor panel copy the contents of the style provided for the layer e.g.outer_walls. An example is shown below.

  1. Click “Submit” to save the style.

Add and Publish Layers

With your datasource and styles created you are ready to add and publish each layer. Repeat the following for each layer. We use the layer “outer_walls” from the example svg for purposes of example:

  1. From the left menu select Data | Layers. On the subsequent page click “Add a new layer”.
  2. From the “Add layer from” dropdown select the workspace:datasource combined option for those created earlier. In our example, this will be Elastic_ON_Simple:Elastic_ON_Simple”.
  3. A list of layers will be presented indicating their current state. Those yet to be published will include a “Publish” link. Select for the next layer to be published - “outer_walls” in our example.
  4. The following form must be completed with the following options completed in sequence:
    1. The fields name and title should be populated with the filename. Adjust if required.
    2. Under “Coordinate Reference Systems” click “Find”. The the popup dialog search for “4326” and select the option presented.

  1. Under “Bounding Boxes” click the option “Computer from data” followed by “Compute from native bounds”. This should populate the two sets of lat long bounding box fields.

  1. On the same form, select the top tab “Publishing”. From the drop down “Default Style” select the style you wish to apply to the layer e.g. “Elastic_ON_Simple:outer_walls”.

  1. Scroll to the bottom of the page and click “Save”.

Group the Layers

Once the above process has been completed for each layer, one final step is required to make the layers available together as a single group of tiles forming the complete image. To achieve this we create a Layer Group.

For our simple floor plan example this process is simple. For more complex images the drawing order of the layers maybe important - especially if the layers overlap.

The following process creates a layer group we can in turn reference in Kibana.  The sequence of steps is again important:

  1. Select the left menu option Data | Layer Groups. From the subsequent menu option click “Add new layer group”.
  2. Provide a name and title to reference the group - these will be publically exposed. For our example, use “Elastic_ON_Simple”.
  3. Select the workspace created earlier.
  4. Click “Add Layer…” and select each “outer_walls”. Repeat this process for each layer, adding them to the group in the desired rendering order.  For the example svg use the order order “outer_walls”, “inner_walls”, “demo_booths” and “demo_stands”.

  1. Click the button “Generate Bounds”
  2. Finally, click “Save”.

Identify Geo Points

You have noticed but the positions of our the features on our floor plan have fairly arbitrary latitude and longitudes - based on the earlier sizing and unit specification within InkScape.  In order to geo enrich our documents with specific points on the floor plan, we must first identify the location of each feature!  This is unfortunately currently a fairly manual exercise. For Elastic{ON} we simply recorded the latitude and longitude of each Ruckus AP.  These we in turn mapped to their mac address, allowing us to enrich packetbeat at index time with a simple lookup using the translate plugin in Logstash.

Several options exist for identifying the latitude and longitude of a feature:

  1. Use QGIS to hover over specific points on the map and record the location displayed in the footer bar.
  2. In GeoServer navigate to Data | Layer Preview on the left menu.  For the layer group created above, select the link “OpenLayers”.  This will provide a preview of the map in separate window.  Hover over specific points to identify the lat lon.  For example, below we identify the location of the corner of the demo stand where this demo was initially presented.

For custom images e.g. a data centre, this tool can be used to capture the effective location of each feature e.g. a server.

Add some sample Data

Unfortunately due to data sensitivities we are unable to provide the data collected during Elastic{ON}. To complete the example, however, we will need to index some data into Elasticsearch for rendering on our custom tile map in Kibana.  We have provided a sample random data generator here with instructions for running.  

Configure Kibana

Finally, we need to configure Kibana.  Record the name of both your workspace and layer group as these will be required. For generic use cases:

  1. Navigate to Kibana and create a Tile Map visualisation for your dataset.  Set the geo field per normal. The test generator provided above indexes into the “elastic_on_simple” index.
  2. In the options tab, complete the following settings:

Setting

Value

Description

WMS url*

http://<host>:<port>/geoserver/<workspace>/wms

Set <host> and <port> to the location of your geoserver instance. For the example svg th <workspace> value is Elastic_ON_Simple.

WMS layers*

<layer_group>

Name of your layer group. For sample svg “Elastic_ON_Simple”.

WMS version*

1.3.0

WMS Version - 1.3.0 for Geoserver.

WMS format*

image/png

Either image/png or image/jpeg. Former supports transparent layers.

  1. Apply the settings and wollah!

For our simplified floor plan example simply open the visual “Elastic_ON_Simple” in kibana.  The data generated is random. Feel free to adjust the styling of the map.

Thanks to above process and amazing design team at Elastic, we were able to produce the following with a complete floor plan:

Final Thoughts

The above represents the minimum steps required to render a custom complex image in Kibana effective at multiple zoom levels.  For very simple images, the steps described in the earlier post here maybe sufficient.

The above does not address ensuring your GeoServer is production ready. In small environments with a low number of users the above may be sufficient with respect to performance.  However, the user is encouraged to read the basic recommended steps for productizing a GeoServer deployment here. More advanced topics to deliver a more scalable environment e.g. caching and clustering, are beyond the scope of this post.

Finally we have assumed all of our layers can to be rendered at every zoom level - part of the motivation for our stylized design.  In extremely complex images this is unlikely to be the case with a need for the same features to be hidden or even rendered using different layers and styles at different zoom levels. GeoServer provides Style Rules to achieve this with documentation the reader can follow if required.