Kurrently in Kibana - May 18, 2015

Welcome to Kurrently in Kibana, your source for keeping up with the Kibana project. With information on project releases, work in progress, and learning resources, you can Kount on this weekly update to let you know about the latest Kibana goodness coming your way!

Kibana 4.1: Getting Closer...

The team has been plugging away at the last few 4.1 issues, mostly trying to get some tile map stuff cleaned up so we can merge it.

Lately, we:

  • Merged pinned filters! Dance party!
  • Fixed a bug in which X-axis values would rarely appear out of order.
  • Removed some stale code.
  • Turned off a history recording feature we weren’t using to save memory.
  • Fixed a bug in which advanced settings for aggregations were shown by default.
  • Jim Unger joined the Kibana team. Welcome, Jim! :)
  • Fixed the Windows build system. 
  • Fixed the tile map memory leak, correctly.

Kibana Forum

Just like the other projects here at Elastic, Kibana now has a forum for discussions and questions that was recently launched. This awesome new service lets us keep Github for getting work done, while fostering a community of Kibana experts to kick around questions, ideas and solutions in a totally async and persistent fashion. Check it out, and enjoy!

Elasticsearch Stuff We’re Giddy About: _path.

We’ve bandied this idea around for over a year and it's almost ready to use.

Pull: https://github.com/elastic/elasticsearch/pull/10980

Purpose: `path` allows us to specify what data we want from a request in a super granular way. For example, let’s say Kibana is retrieving unique web requests over time; a date_histogram agg, with a cardinality agg inside of it.

```
curl -XGET  'localhost:9200/logstash-*/_search’ -d ‘
{
  “aggs”: {
    “2”: {
      “date_histogram”: {
        “field”: “@timestamp”,
        “interval”: “30s”,
      },
      “aggs”: {
        “1”: {
          “cardinality”: {
            “field”: “request”
          }
        }
      }
    }
  }
}
```

We’re going to get back something like this for every bucket. So, you know, like 1000 of these.

```
{
  "key_as_string" : "2015-04-25T07:06:30.000Z”,  // We don’t need this
  "key" : 1429945590000, // Oooo important
  "doc_count" : 6716, // Nope
  "1" : {
    "value" : 441 // Yep
  }
},
```

So this thing was 4 keys, only 2 of which we actually need, and one of them is a big long string. Move enough of those over the wire and it's a whole lot of data. Not to mention the memory considerations of parsing all those and keeping around a big honking object on which we’ll only ever access half the properties.

Ok, now let's bring https://github.com/elastic/elasticsearch/pull/10980 into the fray:

```
curl -XGET  'localhost:9200/logstash-*/_search?_path=aggregations.2.buckets.1,aggregations.2.buckets.key -d [All that other stuff from above]
```

We get back a nice clean result with only the stuff we need.

```
{
  "key" : 1429945740000,
  "1" : {
    "value" : 1
   }
}
```

It's something like a 68% reduction in data moved over the wire. And it's not just the search API -- it applies to every API. If you need to do something like enumerate the hostnames of every node in your cluster, it's more like a 99% reduction. Sweet.


See You Next Week!

Did you know? Elasticsearch and Apache Lucene, Logstash and Kibana all have weekly updates published to the Elastic blog. We'll bring you the newest, stuff in next week's Kurrently in Kibana.