17 Januar 2014 Engineering

Introducing: The cat API

Von Drew Raines


Perhaps the biggest success of Elasticsearch is its APIs. Interacting with the system is so simple it still catches me off guard, four years after I first tried it out. The engine powering this simplicity is JSON, a straightforward form of structured text birthed out of the rise of JavaScript. JSON is easy to understand and parse, and because of that, supported by almost every programming language in existence.

Humans, however, are not programming languages. JSON’s strength is that it’s plaintext, which makes it possible for our eyes to parse, but merely looking at human-readable characters isn’t the same as actually understanding the information. With any more than the most trivial structure in a JSON doc, we typically reach for the nearest pretty-printer. Unfortunately, pretty-printing often still does not translate into actionable knowledge. In fact, the addition of whitespace often makes life even more difficult as it eats up precious space in a terminal window.

“Not a problem,” you say. “JSON is so simple all I need is $LANG and five minutes.” Unfortunately, JSON and $LANG, along with speaking, walking, producing coherent sentences, and almost any other task in life, is a bit harder when you’re woken up from deep sleep by your phone alerting you to a system outage.

The 3 AM Page

Imagine this has just happened to Teresa. The monitoring system noticed that her cluster is red. A common first step in this moment of life as an Elasticsearch cluster administrator is to take a look at the logs on the master node. Which node is master? Armed with the comprehensive output of the cluster state API, she’s off to the races.

% curl 'es1:9200/_cluster/state?pretty'
  "master_node" : "Wjf_YVvySoK8TE41yORt3A",

OK, not quite. That’s just the node ID. Which node is that?

"nodes" : {
  "56RhV2ecT3OIZFzUYVYwNQ" : {
    "name" : "Midnight Sun",
    "transport_address" : "inet[/]",
    "attributes" : { }
  "pyqzjh_nRx6rapL-CBvsyA" : {
    "name" : "Urthona",
    "transport_address" : "inet[/]",
    "attributes" : { }
  "Wjf_YVvySoK8TE41yORt3A" : {
    "name" : "Lasher",
    "transport_address" : "inet[/]",
    "attributes" : { }

Her tired eyes move back and forth a few times and figure out that in order to get to Wjf_YVvySoK8TE41yORt3A she must connect to

She grumbles and pores over the logs. She notices that a node has failed some pings. She’s no fool. She’s seen this before and it wasn’t the network. The JVM is likely in trouble on that node.

First up, she checks on the cluster’s health.

% curl es1:9200/_cluster/health?pretty
  "cluster_name" : "foo",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 4,
  "number_of_data_nodes" : 4,
  "active_primary_shards" : 283,
  "active_shards" : 566,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 1

Uh oh! Red! Teresa is a bit paranoid, so she checks health across the whole cluster.

% for i in 1 2 3 4 5; do curl es${i}:9200/_cluster/health; echo; done


A bit verbose, but it got the job done. All the nodes agree; at least, the ones that are responding. A node definitely is missing, which confirms the ping failures, but that doesn’t explain the red cluster. She sits back nods off for ten minutes. When she wakes up, by chance she notices in the JSON soup splattered all over her screen that there’s an unassigned shard.

“Hm, which one is that?” she asks herself. Experienced with the APIs, she cleverly attaches the level parameter to /_cluster/health to dig deeper.

% curl es1:9200/_cluster/health?level=shards&pretty
    "foo-20140116" : {
      "status" : "red",
      "number_of_shards" : 2,
      "number_of_replicas" : 0,
      "active_primary_shards" : 1,
      "active_shards" : 1,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 1,
      "shards" : {
        "0" : {
          "status" : "red",
          "primary_active" : false,
          "active_shards" : 0,
          "relocating_shards" : 0,
          "initializing_shards" : 0,
          "unassigned_shards" : 1

Now she’s getting somewhere. The foo-20140116 index was created today by Logstash. For some reason shard 0 doesn’t have an active primary, which must have been on the node that isn’t up at the moment. “What happened to the replicas?” she thinks.

Teresa starts the missing node, flips on a replica, and heads back to bed. She can figure that out in the morning.

A new kind of API

If it was as difficult to read that short tale as it was to write it, I apologize. Fortunately there is light at the end of the curly brace.

Let’s see what Teresa’s night would have looked like if she was able to work with some slightly different APIs.

The first thing she needed to do was find the master. It would have been nice if in one, single glorious line she got the node and host information.

% curl es1:9200/_cat/master
Wjf_YVvySoK8TE41yORt3A es1 Lasher 

Puurfect! Instead of messing with the logs, however, she really just needs to get a bird’s-eye view of the current node situation.

% curl es1:9200/_cat/nodes
es1 35 79 0.00 d * Lasher       
es2 40 88 0.00 d m Midnight Sun 
es4 49 89 0.00 d m Urthona      
es5 40 75 0.00 d m Chimera      

She immediately can tell that she’s missing a node, and which one! Next up is to figure out why the cluster is red. Is it thirty indices or only one?

% curl es1:9200/_cat/indices | grep ^red
red   foo-20140116 2 0 30620 1 78.6mb 78.6mb 

Looks like it’s only one. How many shards are missing?

% curl es1:9200/_cat/shards/foo-20140116
foo-20140116 0 p UNASSIGNED                                    
foo-20140116 1 p STARTED    30620 78.6mb Chimera 

It’s easy to see now that half of the primaries are gone and there aren’t replicas configured for foo 20140116. After starting up es3, a cluster-wide health check:

% for i in 1 2 3 4 5; do ssh es${i} curl -s localhost:9200/_cat/health; done
1389940476 18:05:40 foo green 5 5 10 10 0 0 0 
1389940477 18:05:40 foo green 5 5 10 10 0 0 0 
1389940479 18:05:40 foo green 5 5 10 10 0 0 0 
1389940480 18:05:40 foo green 5 5 10 10 0 0 0 
1389940480 18:05:40 foo green 5 5 10 10 0 0 0 

Nice and succinct, where anomalies can easily be caught before precious minutes are wasted in data that doesn’t lead you to informed decisions.

Numbers Everywhere

This may not seem like an improvement to you. We’ve gone from the explicit, labeled JSON to columns of random numbers. To alleviate the transition headache, every cat endpoint takes a v parameter to turn on verbose mode. It will output a header row labeling each column.

% curl 'es1:9200/_cat/health?v'
epoch      timestamp cluster status nodeTotal nodeData shards pri relo init unassign 
1389963537 18:06:03  foo     green          5        5     10  10    0    0        0 


Now we can see that the numbers correspond directly to the key/value pairs that appear in the cluster health API. We can also use these headers to selectively output columns relative to our context. Suppose we’re tracking a long cluster recovery and we want to see our unassigned shards number precipitously drop. We could just output all the numbers. A cleaner approach would be to filter every thing except the number we care about.

% while true; do curl 'es1:9200/_cat/health?h=epoch,timestamp,cluster,status,unassign'; sleep 30; done
1389969492 06:38:12 foo yellow 262
1389969495 06:38:15 foo green 250
1389969498 06:38:18 foo green 237

Column management

One of the major motivations behind cat is to speak Unix fluently. In this case, a simple … | awk '{print $1, $2, $3, $4, $11}' would have sufficed, but some APIs have many more non-default columns that you can only get to with h.

Let’s say Teresa experienced some high heap usage on her nodes and she does a lot of sorting and faceting, common users of fielddata cache. She would like to compare fielddata cache usage and heap across nodes, a task that’s technically possible with the node stats API, but becomes impractical-to-impossible after two nodes. With the cat nodes API, it’s simply a matter of knowing a few column names.

% curl 'es1:9200/_cat/nodes?h=host,heapPercent,heapMax,fielddataMemory' | sort -rnk2
es4 61 29.9gb 14.4gb
es3 58 29.9gb 16.5gb
es5 40 29.9gb    5gb
es2 33 29.9gb  8.2gb
es1 20 29.9gb  3.4gb

Sorting by percentage of heap used makes it very clear there is some rough correlation between heap and fielddata use.

Byte and time resolution

What if she wanted to sort by fielddataMemory? ES provides human-readable conversions from bytes but this actually makes it harder for sort. She can supply the bytes flag to specify the unit of precision.

% curl 'es1:9200/_cat/nodes?h=host,heapPercent,heapMax,fielddataMemory&bytes=b' | sort -rnk4
es3 58 29.9gb 17805705171
es4 61 29.9gb 15550755044
es2 33 29.9gb  8880273008
es5 40 29.9gb  5449302354
es1 20 29.9gb  3687354160

The same kind of resolution calculation for time works as well. If she has a column like merges.total_time that she wants in seconds, she can supply a time parameter with ms.

% curl 'es1:9200/_cat/nodes?h=host,mtt&time=s' | sort -rnk2
es4 910
es3 902
es2 278
es1 190
es5  99


How did she know that mtt would give her merges.total_time? Each API supports a flag help with all the possible column headers.

% curl 'es1:9200/_cat/nodes?help' | fgrep merge
merges.current           | mc,mergesCurrent          | number of current merges                
merges.current_docs      | mcd,mergesCurrentDocs     | number of current merging docs          
merges.current_size      | mcs,mergesCurrentSize     | size of current merges                  
merges.total             | mt,mergesTotal            | number of completed merge ops           
merges.total_docs        | mtd,mergesTotalDocs       | docs merged                             
merges.total_size        | mts,mergesTotalSize       | size merged                             
merges.total_time        | mtt,mergesTotalTime       | time spent in merges                    

Any of merges.total_time, mtt, or mergesTotalTime would have worked. She picked mtt since it’s short and esoteric, like a good Unix admin prefers.

Much more!

health, nodes, master, and shards are just a few. There are APIs for indices, recovery progress, and more!


cat is an evolution of ad hoc tools produced in the field of large clusters running on early versions of Elasticsearch. It was clear that the APIs were a generation ahead. They were excellent for machines while they were merely usable for humans. cat aims to fit in those places where JSON doesn’t – the Unix pipe, the chat window, the tweet. In an era of unprecedented bandwidth of communication, it’s the low-bandwidth, the lightweight, that we reach for most often. It’s fitting that they also happen to be the places cats seem to appear most.

We would love to hear your feedback on the cat API. Let us know what you think!