28 March 2018 Engineering

Here to Help... (An Elastic{ON} Canvas Story)

By Jamie Smith

This is the second in a series about Canvas at Elastic{ON} 2018. Part 1 covers visualizing attendee caffeine intake at the conference.

Along with all of the great talks, demos, and of course, coffee, at Elastic{ON} there was an Ask Me Anything (or “AMA”) Booth, where users, customers, and partners could go for some quality time with the team. This booth was always packed - I mean that it was a madhouse (in a good way). This was the place to go with specific questions– questions about products, processes, use cases, or enhancements. There seemed to be at least ten Elasticians staffing the booth throughout the conference, and at times there seemed to be too many to count. Support engineers, consultants, product managers, and developers engaged in deep conversation, sketching and designing on the tabletop with dry-erase markers or in notebooks. Even I managed to answer a few questions!

This is the second installment in the Canvas Story series, where we dig in to some of the very cool Canvas workpads that were on display at Elastic{ON} 2018 (you can check out the Canvas demo in the keynote). We’ll go over the final layout and configuration for each area, and delve into some changes that I would make, in hindsight.

AMA Metrics

What is Canvas?

Now available as a technology preview, Canvas is a creative space for live data. To learn more about why Canvas came to be, check out the introductory blog post; to keep up to date with the status of Canvas, jump over to the dedicated Canvas mini-site.

AMA Metrics Canvas: Break it Down

Gathering the Data

You may have seen the Elasticians with their iPads, phones, and computers filling out the internal “AMA Survey” app. This is how the data made its way into the cluster. The application is a simple React app, which records the data in Elasticsearch:

AMA App

We captured the group that the Elasticians report through, the types of products discussed, and one or more use cases the conversations centered on. The time spent on the conversations was bucketed, enabling us to gauge the complexity of the issues.

The Layout

Much like the Café Canvas from our previous installment, this workpad isn’t as complicated as it looks. Breaking it down there are only six different types of data elements on this workpad, with multiple instances containing slightly different parameters (We obviously leveraged folks more artistically creative than I to come up with the graphics):

Marked-up AMA Workpad

Around the Horn

Group Bubbles

Starting at the top left, we see the breakdown of the groups which were staffing the booth throughout the conference. Each of these actually consist of four different elements - a graphic with a “face” shape, a little circle, and a couple markdown elements:

Dev Group Bubble

After the markdown element is added, you can tweak the settings in the control panel, or edit in code. When I switch to code I like to format it nicely, rather than have it in one long line, so it is a bit easier to read. The code for the above markdown pieces looks like:

filters | 
    markdown {escount index="amaresponses" q="business_group:Engineering"} 
         font={font family="Open Sans, Helvetica, Arial, sans-serif" 
              size=24 
              align="center" 
              color="#FFFFFF" 
              weight="bold" 
              underline=false 
              italic=false} 
    | render 

The filters command simply applies any global filters from the page to this element, it is pretty standard to start with this. This is a markdown element, and all that we need is the count of the documents in the amaresponses index which have a business_group of Engineering. The rest of the config is setting the appearance, ending in the render command.

If you look at the whole workpad you’ll notice that there are some groups with more than one word in the name- the config for this has to escape the quotes. For example, the escount element for the Developer Evangelist group is:

{escount index="amaresponses" q="business_group:\"Dev Rel\""}

Which allows us to get those that match exactly.

Jug(gling) the Data

As we move around clockwise, we get to the products breakdown. This was a place that I underestimated how many questions we would handle - The Elasticsearch category quickly made it to 100%! We had to change the denominator in the “image fill” elements. We ended up with the denominator for the Elasticsearch category to be 800, while the others were set to 400.

The image-fill element is straightforward - One “empty” image, one “full” image, and a number [0-1]. The empty and full jars looked like this:

half empty or half full?

The configuration behind the jars, recalling that the denominator for the Elasticsearch term is higher than the others, was:

filters 
    | escount index="amaresponses" q="topics:Elasticsearch" 
    | math "value / 800" 
    | if {compare "gt" to=1} then=1 else={context} 
    | revealImage origin="bottom" 
        image={asset "asset-d8aad426-206b-4ae9-9340-9f4b1f740386"}  
        emptyImage={asset "asset-7aeb2f8e-27af-4cf8-98b1-b4b6c3d80a74"} 
    | render

Again, for those elements which have two words in the topic, we have to escape the quotes in the query:

| escount index="amaresponses" q="topics:\"Elastic Cloud Enterprise\"" 

The Beats bucket was also a little more complicated, with multiple values:

| escount index="amaresponses" q="topic:Filebeat OR Packetbeat OR Auditbeat OR Filebeat OR Winlogbeat OR Heartbeat".

This is similar to the markdown queries from the group section. A slightly different query, and some math to measure against our goal. One important part is the if statement - if {compare "gt" to=1} then=1 else={context}, which makes sure that we don’t exceed 1.0. The numbers above the jars

Elasticsearch Jar

Is just another markdown element, tweaked with italics:

filters 
    | markdown {escount index="amaresponses" q="Kibana"} 
         font={font family="Open Sans, Helvetica, Arial, sans-serif"  
             size=35  
             align="center"  
             color="#444444"  
             weight="bold"  
             underline=false 
             italic=true} 
    | render

Dynamic Bar Chart

The next element in our tour is the use case summary. In a preliminary version of this workpad this was also done with an image reveal, but some last-minute tweaks to the Canvas plugin allowed us to use a standard plot with some axis definitions to come up with the final look.

We start out configuring it using the sidebar, until we can’t:

Config Panel for Bar Chart

The first pass is decent:

Rough draft

But I would like to see a little separation between the bars, so we duck down into the code editor and set the bars manually (bars=.8)

filters 
    | timelion query=".es(index=amaresponses, split=use_case:10).label('$1', '.*>.*:(.*)>.*')" interval="9999d" 
    | pointseries x="value" y="label" 
    | plot font={font family="'Open Sans', Helvetica, Arial, sans-serif" 
                    size=15 
                    align="left" 
                    color="#000000" 
                    weight="bold" 
                    underline=false 
                    italic=false}
           legend=false 
           xaxis=false 
           yaxis={axisConfig position="right"} 
           defaultStyle={seriesStyle points="0" bars=".8" color="#f5cc5d" lines="0" horizontalBars=true}
    | render css="                                                                                                                                  
    .flot-x-axis {                                                                                                                                  
        z-index: 1;                                                                                                                                 
    }                                                                                                                                               
    .flot-y-axis .flot-tick-label {                                                                                                                 
    transform: translate(-1025px,-3px);                                                                                                              
        max-width: fit-content !important;
        white-space: nowrap;                                                                                                                                                                                                                  
    } 
"

Yielding

Final Bar Chart

Which looks more along the lines of what I was thinking of.

This one, admittedly, is a bit more complicated than others. Even though most of it is the formatting, we should will go into a bit more detail. We again start with the filters, but rather than esdocs or escount we are using Timelion, adding in a tricky interval which returns just one total per use case rather than an entire histogram. By specifying the parameterized label we can extract the use case name. Compare the Timelion results without the label (left) and with the label (right). If we hadn’t done this the long q:*> use_case:Blah part would have been in the names.

Timelion Comparison

After the Timelion query we pipe it through a pointseries, with the y-axis coming from the terms, and the x-axis the value. The remainder of the code deals with layout.

Consultation Time Series

The bottom of the workpad has the count of visits broken down by time. This also uses a Timelion query, but this time actually caring about the time. The code for this:

filters 
    | timelion q=".es(index='amaresponses')" 
         from="2018-02-27T00:00:00" 
         to="2018-03-02T10:59:59" 
    | pointseries x="@timestamp" y="value"
    | plot defaultStyle={seriesStyle points="0" bars="10" lines="0" color="#263746"}  
         legend=false  
         yaxis=false 
    | render css="
    .canvas_element {
        overflow: hidden;
    }
    .flot-x-axis .flot-tick-label {transform: translateY(-20px); margin:0px;}
    " containerStyle={containerStyle border="-2px undefined undefined"}

This gets the bucketed count from Timelion, but forces it to only use the time window of the event. In this case the x-axis of the pointseries is the timestamp, while the height of the bars indicates the count. The bars get a width of 10, while there are neither points nor lines configured.

Time on Our Side

The last element on our tour is the pocket watch. It might look the most complex but that’s just Canvas trickery - the pocket watch is really just a pie chart on top of a graphic. If I rearrange you can see the individual pieces.

Image Stacking

Transparent backgrounds and creative layering let you make fantastic visuals. The configuration of the pie chart is pretty simple:

filters 
    | esdocs index="amaresponses" 
    | ply by="time_spent" expression=${rowCount | as "count"} 
    | pointseries color="time_spent" size="count" 
    | pie font={font family="'Open Sans', Helvetica, Arial, sans-serif" 
                    size=14 
                    align="left" 
                    color="#000000" 
                    weight="bold" 
                    underline=false 
                    italic=false} 
         palette={palette "#00A8E1" "#00556F" gradient=true} 
         labelRadius=90

We start with the filters, then query the esdocs for all documents in the amaresponses index. The pipeline hits something we haven’t seen yet - the ply operation, which subdivides a datatable and passes the resulting tables into an expression, merging the output. So for each of the values of time_spent it counts the rows and returns the count for bucket. If I modify the code to show the debug output:

filters 
    | esdocs index="amaresponses" 
    | ply by="time_spent" expression=${rowCount | as "count"} 
    | render as=debug

The debug output shows explains what is going on, and what the resulting data looks like:

{
  "datatable": {
    "columns": [
      {
        "name": "time_spent",
        "type": "string"
      },
      {
        "name": "count",
        "type": "number"
      }
    ],
    "rows": [
      {"count": 22, "time_spent": ["15+"]},
      {"count": 35, "time_spent": ["5-10"]},
      {"count": 8,  "time_spent": ["10-15"]},
      {"count": 35, "time_spent": ["0-5"]}
    ],
    "type": "datatable"
  },
  "paginate": true,
  "perPage": 10
}

The pointseries color="time_spent" size="count" is doing what it sounds like – breaking them down colored by the time_spent, and the size based on the number of items in that window. This data is sent through the pie operation, with some style information and a labelRadius attribute, which controls where the labels go relative to the width of the element.

Ch-ch-ch-ch-changes

There are a couple of things that I would do differently if I were to modify this to be a little better at handling the different number of questions that we got for each of the product groups. I guess they aren’t things that I would do differently - I actually did make the changes in a copy of the canvas.

Alternate AMA Metrics with sub-buckets

This one uses a mix of image reveal and image repeat to convey the difference in sizes. Rather than several big jars of varying sizes, I instead made the big jar just hold fifty answers - and when that jar gets full it fills in one of the “overflow” jars, along with the overall tally markdown still at the top. Here is a closeup of the Elasticsearch and Kibana jars:

Close up of Jars

I did have to tweak the code (and create the small jar empty and full images, which are the same as the big jars, with a little border added to leave a little room around them). The small jars are an image repeat:

filters 
    | escount index="amaresponses" q="Elasticsearch" 
    | math "value/50" 
    | repeatImage 
        emptyImage={asset "asset-2ad47327-52ca-43ff-b287-dc688a51942a"} 
        image={asset "asset-5c698545-3156-40ec-96da-939da0f41955"} 
        size=20 
        max=15 
    | render

The math "value / 50" gives us one jar per 50 items (rounded down to a whole number), and specified how big each jar should be (size=20) and the maximum number of jars to show is (max=15).

Want to try the data?

Head on over to our examples repo, and under canvas you can find AMA Responses collateral. This includes the data from the conference, the actual canvas workpad which you can import, and even the code for the home assistant config which includes the script we used.