The Story of Sense - Announcing Sense 2.0.0-beta1
Want to download Sense? Take it for a spin.
The Beginning
As with any good project, Sense was conceived while drinking beer, after a
rowing session on the
Amstel. My friend,
Jasper, and I were talking about the up and
coming yearly
hackathon at Jasper’s
company. The hackathon follows a strict procedure dictating
how ideas are selected and teams are formed. At the time, both Jasper and I
were committed to work with
Anne Veling on an
alternative history screen in Chrome browser.
Jasper and I decided that although both of us were happy users of Elasticsearch, there was no good tool around to allow you to submit requests to its rich REST API. As shameful as it was, the best tool was still the terminal based cURL command. All you have to do is to spend 5 seconds in the terminal submitting requests with a body to figure out just how inconvenient it is. Simple things, like fixing a typo, required re-typing the entire command or fighting the terminal multi line support. It had to be possible to find some web-based JSON editor and put it to good work.
I did some research and phoned Anne. I told him I was still committed to the History Page project but since we were all avid Elasticsearch users, maybe we could better spend our time building a cool Console for it. We could simply use the Ace online editor and build a knowledge base that would automatically recognize the API and give proper contextual suggestions. Anne immediately got enthusiastic about it and agreed that it would be cool. However, he had to turn the idea down as it violated one of the hackathon basic rules: it must be done in a weekend. Clearly what I was suggesting couldn’t. In the end, we went ahead and built the most awesome Chrome History Page replacement ever.
Nevertheless, I had been challenged and I just had to prove that it could be done. The next weekend (and a few extra evenings :) ) I sat down and built it. Sense was born. It was buggy as hell but it worked. I showed it to people and they got excited.
From that point on I spent an evening a week extending the Knowledge Base and fixing bugs. Sense had started gaining traction and I was afraid people would be stuck on an old buggy version for ever. To counter that I decided to release Sense as a Chrome Extension so it would be updated automatically when ever I released. One by one features like the History Panel was added as I needed them.
Around that time I joined Elastic. I was happy to hear many people in the company were using Sense and were excited about it. I explicitly remember talking to Clint, who said "You know what Sense should do? It should use this format and allow you to have multiple requests in the editor". Another challenge had been made and, of course, I had to build it. This one was not as trivial and I had to dive deeper into the bowels of Ace. The result was a new Ace mode (the highlighting logic used by Ace) which was tightly integrated into Sense’s suggestion engine.
To appreciate what this means, look at this old screenshot of Sense:
On the top you would type an API url and on the left hand side the JSON body. Everything was nicely separated and I could use Ace’s standard JSON mode. Now however, I had to figure out how to support the following format:
GET _cluster/health
POST index/_settings
{
"index": { "number_of_replicas": 3 }
}
That meant that Ace had to know how to parse 3 different things: an HTTP method, a URL and optionally a JSON body. Even worse, these 3 different modes were not as clearly separated as before. The only way to know that a JSON body is complete is by counting braces. It took some work and customization to Ace but I managed to pull it off and the Sense syntax you now know was born (Thanks Clint!).
During the day job, my primary task became the development of Marvel, our Management and Monitoring solution for Elasticsearch (side note: Marvel has just been reborn (see " Shield, Watcher, and Marvel 2.0.0 GA Released"). Marvel was to be free for development use and it made sense to incorporate Sense into Marvel. This allowed me to actually work on Sense during the day and also put it in front of many more users than before. It also meant that Sense was worked on by actual bona fideJavaScript developers, who helped to clean it up and teach me all the latest coolness in Browser Land.
During this period Sense went through a couple of rewrites. Most notably, I rewrote the two separate suggestion engines for URLs and JSON and built a single suggestion engine that could work in both these contexts, and allow a third context: URL parameters.
The new engine is also smart enough to maintain multiple suggestion contexts.
For example, consider the
_search API sort parameter:
GET _search
{
"sort": [
"timestamp": "desc",
"price": {
"order": "desc".
"missing": "last"
},
"nested_filter": { "term": { ... }},
"_score"
]
}
Depending on where you are, Sense need to suggest simple values (like
_score), more complex structures (like order and missing), or even an
arbitrary filter (like
nested_filter). All of those suggestion paths are
processed at once and are excluded as they become irrelevant.
Marvel 1.x was built on top of Kibana 3.0, which was a great data exploration and dashboarding tool. However, the Kibana team had a bigger vision. Kibana 4.x is designed to be a generic application platform on which people can easily build UI apps which work with Elasticsearch as a backend. In fact, Marvel 2.0 is the very first app to use Kibana as a platform.
Taking a step back and looking at Sense - a generic console to interact with the Elasticsearch API, I realized that it totally fits the Kibana vision of an app. Today, I’m very happy to announce Sense 2.0 as an open-source Kibana app, free to use in development and production.
Sense 2.0 also comes with great new improvements. Here are the highlights (a complete list is available here):
- Elasticsearch 2.0
- The Sense knowledge base has been updated to support Elasticsearch 2.0, and includes autocomplete suggestions for the new pipeline aggregations.
- Multiple Request Execution
When testing or trying things out you frequently need to repeat the same sequence of commands. Just write them all in Sense, select and submit multiple requests to Elasticsearch.
Sense will send the requests to Elasticsearch one by one, and show the output in the right pane as Elasticsearch responds. This is very handy when debugging an issue or trying query combinations in multiple scenarios.
- Multiple Request Copy-Paste
Selecting multiple requests also allows you to auto format and copy them as cURL in one go. For example:
will add the following to your clipboard:
# Delete all data in the `website` index curl -XDELETE "http://localhost:9200/website" # Create a document with ID 123 curl -XPUT "http://localhost:9200/website/blog/123" -d' { "title": "My first blog entry", "text": "Just trying this out...", "date": "2014/01/01" }'Of course, you can also paste multiple cURL commands and Sense will parse them correctly as well.
We are initially releasing Sense 2.0.0 as a beta release, in recognition that there is more work to be done. We expect the GA to follow soon.
To know more and try Sense 2.0 out, please check out the new documentation. Let us know what you think on the forums and open an issue if you have suggestions or run into bugs.