Playing HTTP Tricks with Nginx

One of the defining features of Elasticsearch is that it’s exposed as a (loosely) RESTful service over HTTP.

The benefits are easy to spell out, of course: the API is familiar and predictable to all web developers. It’s easy to use with “bare hands” via the curl command, or in the browser. It’s easy to write API wrappers in various programming languages.

Nevertheless, the importance of the HTTP-based nature of Elasticsearch is rooted deeper: in the way it fits into the existing paradigm of software development and architecture.

HTTP As an Architectural Paradigm

The typical modern software or information system is quite frequently a collection of loosely coupled services, communicating over network: typically, via HTTP. Design-wise, the important aspect of this approach is that you can always “rip apart” the chain of services, and insert another component, which adds or changes functionality, into the “stack.” In the old days, this has been traditionally called “middleware,” but it resurfaced in context of RESTful web services, for example as Rack middleware,
used notably in the Ruby on Rails framework.

HTTP is particularly well suited for such architectures, because its perceived shortcomings (lack of state, text-based representation, URI-centric semantics, …) turn into an advantage: neither “middleware” has to accommodate for something specific in the “chain”, and just passes along status codes, headers, and bodies. In this sense, HTTP is functionally transparent – it doesn’t matter, for example, if you fetch an image from the original web server or a cache on a different continent. It’s still the same “resource.”

Caching is a prime example of this aspect of HTTP, presented already in Roy Fielding’s seminal work on RESTful architectures. (For a thorough information on the subject, see Ryan Tomayko’s Things Caches Do and Mark Nottingham’s Caching Tutorial.)

Technically, the cache operates as a proxy here – it “stands for” some other component in the stack.

But proxies can do so much more. A good example is authentication and authorization: a proxy can intercept requests to a service, perform authentication and/or authorization routines, and either allow or deny access to the client.

This type of proxy is usually called a reverse proxy. The name makes sense when you consider that a traditional proxy “forwards” traffic from a local network to the remote network (the internet), which is reversed here, because the “reverse” proxy forwards requests from the internet to a “local” backend. Such a proxy could be implemented in a programming language like Java or Go, or with a framework like Node.js. Alternatively, we could use a configurable webserver like Nginx.

Nginx

Nginx is an open source web server, originally writen by Igor Sysoev, focused on high performance, concurrency and low memory footprint. (For a detailed technical overview, see the relevant chapter of the The Architecture of Open Source Applications book.)

Nginx has been designed with a proxy role in mind from the start, and supports many related configuration directives an options. It is fairly common to run Nginx as a load balancer in front of Ruby on Rails of Django applications. Many large PHP applications even put Nginx in front of Apache running mod_php to accelerate serving static content and scale the application. Most parts of this article assume a standard Nginx installation, but the advanced parts rely on the Lua module for Nginx.

To run Nginx as a “100% transparent” proxy for Elasticsearch, we need a very minimal configuration:

http {
  server {
    listen 8080;
    location / {
      proxy_pass http://localhost:9200;
    }
  }
}

When we execute a request to http://localhost:8080, we’ll get a response from Elasticsearch running on port 9200.

This proxy is of course quite useless — it just hands over data between the client and Elasticsearch; though astute readers might have guessed that it already adds something to the “stack,” namely the logging of every request.

Use Cases

In this article, we’ll go through some of the more interesting use cases for Nginx as a reverse proxy for Elasticsearch.

Persistent HTTP Connections

Let’s start with a very simple example: using Nginx as a proxy which keeps persistent (“keep-alive”) connections to Elasticsearch. Why we would like to do it? The primary reason would be to relieve Elasticsearch of the stress from opening and closing a connection for each request when using a client without support for persistent connections. Elasticsearch has many more responsibilities than just handling the networking, and opening/closing connections wastes valuable time and resources (such as open files limit).

The full configuration is available, like all examples in this article, in this gist.

events {
    worker_connections  1024;
}
http {
  upstream elasticsearch {
    server 127.0.0.1:9200;
    keepalive 15;
  }
  server {
    listen 8080;
    location / {
      proxy_pass http://elasticsearch;
      proxy_http_version 1.1;
      proxy_set_header Connection "Keep-Alive";
      proxy_set_header Proxy-Connection "Keep-Alive";
    }
  }
}

Let’s launch Nginx with this configuration:

$ nginx -p $PWD/nginx/ -c $PWD/nginx_keep_alive.conf

When you execute a request directly to Elasticsearch, you’ll notice that the number of opened connections is increasing all the time:

$ curl 'localhost:9200/_nodes/stats/http?pretty' | grep total_opened
# "total_opened" : 13
$ curl 'localhost:9200/_nodes/stats/http?pretty' | grep total_opened
# "total_opened" : 14
# ...

But it’s a completely different story when using Nginx — the number of opened connections stays the same:

$ curl 'localhost:8080/_nodes/stats/http?pretty' | grep total_opened
# "total_opened" : 15
$ curl 'localhost:9200/_nodes/stats/http?pretty' | grep total_opened
# "total_opened" : 15
# ...

Simple Load Balancer

With a very small change to the configuration, we can use it to pass requests to multiple Elasticsearch nodes, and use Nginx as a light-weight load balancer:

events {
    worker_connections  1024;
}
http {
  upstream elasticsearch {
    server 127.0.0.1:9200;
    server 127.0.0.1:9201;
    server 127.0.0.1:9202;
    keepalive 15;
  }
  server {
    listen 8080;
    location / {
      proxy_pass http://elasticsearch;
      proxy_http_version 1.1;
      proxy_set_header Connection "Keep-Alive";
      proxy_set_header Proxy-Connection "Keep-Alive";
    }
  }
}

As you can see, we’ve added two additional nodes in the upstream directive. Nginx will now automatically distribute requests, in a round-robin fashion, across these servers, spreading the load on the Elasticsearch cluster evenly across the nodes:

$ curl localhost:8080 | grep name
# "name" : "Silver Fox",
$ curl localhost:8080 | grep name
# "name" : "G-Force",
$ curl localhost:8080 | grep name
# "name" : "Helleyes",
$ curl localhost:8080 | grep name
# "name" : "Silver Fox",
# ...

This is a desirable behaviour, because it prevents hitting a single “hot” node, which has to perform the regular node duties and also route all the traffic and perform all the other associated actions. To change the configuration, we just need to update the list of servers in the upstream directive, and send Nginx a reload signal.

For more information about Nginx’s load balancing features, including different balancing strategies, setting “weights” for different nodes, health checking and live monitoring, please see the Load Balancing with NGINX and NGINX Plus article.

(Note, that the official Elasticsearch clients can perform such load balancing by themselves, with the ability to automatically reload list of nodes in the cluster, retrying a request on another node, etc.)

Conclusion

In this article, we’ve made great use of the fact that Elasticsearch is a service exposed via HTTP. We’ve seen how HTTP fits very well conceptually into the current paradigm of designing software architectures as independent, decoupled services, and how Nginx can be used as a high-performing, customizable proxy. Happy proxying!