2013年09月24日

Release the clients! Ruby, Python, PHP, Perl

By Clinton Gormley

Today, the Elasticsearch developer team is excited to announce the release of official Elasticsearch clients for Ruby, Python, PHP and Perl, with other languages to follow in the future. All of the clients are licensed under the Apache 2 open source license.

Why are we providing official clients?

The Elasticsearch community has done a great job over the years of providing clients for many languages. The feature set of these clients varies greatly: some are low level thin wrappers, some are high level with many abstractions. Some focus on the Elasticsearch API, while others focus on the networking layer. Some have extensive coverage of the Elasticsearch APIs and some just implement the few APIs that the developer needs.

And they are all implemented in the preferred style of the developer.

A number of our users have multiple languages in their codebase and want to use Elasticsearch from all of them. Differences between clients can make this more complicated than it need be.

What do these clients provide?

We have written these clients with the following goals in mind:

No opinions!

It should be as simple as possible to go from the Elasticsearch reference docs directly to the client of your choice. All of the Elasticsearch APIs provided by these clients are direct translations of the native Elasticsearch REST interface. There should be no guessing required.

Play nicely with the cluster

While it is easy to make HTTP calls to Elasticsearch, handling dynamic node detection or failover when nodes disappear is tricky. These clients provide a solid networking base for playing nicely with the cluster.

Full, consistent coverage of the APIs

We have implemented the full REST interface. Nothing is missing. And the method calls and parameters are consistent between languages: you can make the same calls from any of the supported clients. That said, each client still feels like it belongs to the language it is written in. It won’t feel like you are programming Java in Ruby, or C++ in PHP.

Transport abstraction

These clients are not tied to a particular networking module. Different HTTP backends can be plugged in for different environments, or to match the HTTP client that you are already using. But this abstraction also allows us to plugin different transport protocols in the future, which are more efficient than HTTP.

Extend and conquer

The API that these clients provide is a thin wrapper around the REST interface. Perhaps their style is different from your own, and you’d prefer more abstractions. We hear you, so we have built these clients to be extensible. We have tried to do the hard stuff for you, to provide you with a solid foundation for building your own interface.This is the beauty of open source: our users will have great ideas that haven’t even occurred to us. Now you also have the tools to implement them, without having to start from scratch.

Supported by Elasticsearch

Because we wrote these clients and they are tested by us, we can provide official support for them. When bugs are found, we can fix them quickly – it’s our job.

Downloading the clients

Each client is available from GitHub and also from the standard distribution network for each language, where one exists.

Note: This is the first release of these clients. Please download them, try them out and give us feedback. What doesn’t work? What is missing? What can we do better?

Elasticsearch for Perl

GitHub repo: https://github.com/elasticsearch/elasticsearch-perl
Issues list: https://github.com/elasticsearch/elasticsearch-perl/issues

Installing

cpanm Elasticsearch

Example usage

use Elasticsearch;

# Connect to localhost:9200
my $es = Elasticsearch->new();

# Round-robin between two nodes
my $es = Elasticsearch->new(
    nodes => [
        'search1:9200',
        'search2:9200'
    ]
);

# Connect to cluster at search1:9200, sniff all nodes 
# and round-robin between them
my $es = Elasticsearch->new(
    nodes    => 'search1:9200',
    cxn_pool => 'Sniff'
);

# Index a document
$es->index(
    index   => 'my_app',
    type    => 'blog_post',
    id      => 1,
    body    => {
        title   => 'Elasticsearch clients',
        content => 'Interesting content...',
        date    => '2013-09-24'
    }
);

# Get the document
my $doc = $es->get(
    index   => 'my_app',
    type    => 'blog_post',
    id      => 1
);

# Search
my $results = $es->search(
    index => 'my_app',
    body  => {
        query => {
            match => { title => 'elasticsearch' }
        }
    }
);

From the developer

My name is Clinton Gormley and I’ve been using Perl since 1999 and Elasticsearch from the first release back in early 2010. At the time I was working for the largest obituary and family announcement website in Europe and we desperately needed a solution for full text search.

I kept putting it off because the options that existed back then made me want to cry. But eventually, I couldn’t put it off any longer. Before starting, I searched Google again, just in case a different solution had become available, and discovered an interview with Shay Banon discussing version 0.4 of Elasticsearch. I clicked through to the website and it looked too good to be true: simple, scalable, easy to start using. I tried it out and found that it lived up to its promises!

The first thing that I needed was a Perl API so I wrote ElasticSearch.pm and have been maintaining it ever since. ElasticSearch.pm is used by metacpan.org amongst others.

The new Elasticsearch client is a complete rewrite of the original codebase with a number of improvements, especially in extensibility, logging and handling node failover, that would have been difficult to add in the old version. I apologise in advance for the change in naming: ElasticSearch to Elasticsearch…

Currently there is only support for using HTTP::Tiny as a backend, but I will soon be releasing backends for HTTP::Lite, LWP, libcurl and AnyEvent too. Also, I will release an ElasticSearch::Compat module which will make it easy to migrate your existing code to the new Elasticsearch client.

In the meantime, please try out the new client and let me know of anything that is missing via the GitHub issues list.

Elasticsearch for PHP

GitHub repo: https://github.com/elasticsearch/elasticsearch-php
Issues list: https://github.com/elasticsearch/elasticsearch-php/issues

Installing

Add `elasticsearch/elasticsearch` to your `composer.json`:

{
    "require": {
        "elasticsearch/elasticsearch": "~0.4"
    }
}

Then install with composer

curl -s http://getcomposer.org/installer | php
php composer.phar install

Example usage

require 'vendor/autoload.php';
use Elasticsearch;

// Connect to localhost:9200:
$es = new Elasticsearch\Client();

// Round-robin between two nodes:
$es = new Elasticsearch\Client(
    array(
        'hosts' => array(
            'search1:9200',
            'search2:9200'
        )
    )
);

//Connect to cluster at search1:9200, sniff all nodes and round-robin between them:
$es = new Elasticsearch\Client(
    array(
        'hosts' => array('search1:9200'),
        'connectionPoolClass' => '\Elasticsearch\ConnectionPool\SniffingConnectionPool'
    )
);


// Index a document:
$es->index(
    array(
        'index' => 'my_app',
        'type'  => 'blog_post',
        'id'    => 1,
        'body'  => array(
            'title'   => 'Elasticsearch clients',
            'content' => 'Interesting content...',
            'date'    => '2013-09-24'
        )
    )
);

//Get the document:
$doc = $es->get(
    array(
        'index' => 'my_app',
        'type'  => 'blog_post',
        'id'    => 1
    )
);

// Search:
$params = array(
    'index' => 'my_app',
    'type'  => 'blog_post'
);
$params['body']['query']['match']['title'] = 'elasticsearch';
$results = $es->search($params);

From the developer

My name is Zachary Tong and I’ve been working in PHP off and on for about 12 years. I had the pleasure of meeting Elasticsearch about two years ago around release 0.18.6.1. We’ve been in a steady relationship ever since.

At the time I was working as a molecular biologist and was desperately dissatisfied with available biomedical search engines. Always being an evening-hacker at heart, I obtained a lease to the NLM Medline database of biomedical abstracts and began trying to develop a better search solution. Wouldn’t you know it…search is a Hard Problem!

I stumbled onto Elasticsearch at that point and was excited by the combination of features + performance (even on my modest laptop). Although that particular search project never left my laptop, I’ve continued using Elasticsearch for a variety of personal and professional projects since.

Coming from a non-information-retrieval background (and largely non-computer-science background), many of the concepts in Elasticsearch were foreign to me. Shards, indexes, TF-IDF, analyzers, ngrams, what? It can be overwhelming for newcomers to the search field. I find that I learn best by teaching, so over the last two years I’ve written a number of articles on my personal blog.

Although I’ve been coding and writing about Elasticsearch for a long time, I have never really participated in the open source community until recently. My first big forray into OSS was with a PHP client for Elasticsearch. This caught the attention of developers at Elasticsearch and the rest is history.

I’m very excited about the clients being released today. My hope is to free developers from the drudgery of “boring” low-level interaction with an Elasticsearch cluster. Using this client as a base we can start to build more interesting components, like bundles for popular frameworks, or advanced ORMs that abstract away the cluster entirely.

Elasticsearch for Python

GitHub repo: https://github.com/elasticsearch/elasticsearch-py
Issues list: https://github.com/elasticsearch/elasticsearch-py/issues

Installing

pip install elasticsearch

Example usage

import elasticsearch

# Connect to localhost:9200 by default:
es = elasticsearch.Elasticsearch()

# Round-robin between two nodes:
es = elasticsearch.Elasticsearch(["search1:9200", "search2:9200"])

# Connect to cluster at search1:9200, sniff all nodes and round-robin between them
es = elasticsearch.Elasticsearch(["search1:9200"], sniff_on_start=True)

# Index a document:
es.index(
    index="my_app",
    doc_type="blog_post",
    id=1,
    body={
        "title": "Elasticsearch clients",
        "content": "Interesting content...",
        "date": date(2013, 9, 24),
    }
)

# Get the document:
es.get(index="my_app", doc_type="blog_post", id=1)

# Search:
es.search(index="my_app", body={"query": {"match": {"title": "elasticsearch"}}})

From the developer

My name is Honza Král, I chose Python as my primary language in 2006 and haven’t looked back. Before I joined Elasticsearch in April of 2013 my job was to develop and maintain backends for a few bigger content websites based on the Django web framework. Before that I spent my days as an ETL guy working in data warehousing

I learned about Elasticsearch as part of my job when I was looking for a search engine to power our full-text search and was immediately captured by it’s balance of ease-of-use and powerful feature set. It reminded me of my days as a data guy and dove right in.

Ever since the beginning of my programming career I was drawn towards open-source and, whenever possible, would contribute back to the project I am using. It was no surprise then that once I learned that there is an opening for a Python guy on the team to work on the new client I couldn’t let it pass by.

Since the new clients we’re releasing today are a very close mapping to the REST API it should be very easy for any elasticsearch user to learn and use, as well as flexible enough for any possible use-case. This should enable us to work well with existing clients and provide a clean migration path for people who would wish to migrate – we will try and work with the maintainers of existing clients to port their code to be a layer on top of this new client, thus enabling developers to keep their code while getting the benefit of using our new client underneath.

Elasticsearch for Ruby

GitHub repo: https://github.com/elasticsearch/elasticsearch-ruby
Issues list: https://github.com/elasticsearch/elasticsearch-ruby/issues

Installing

gem install elasticsearch

Example usage

require 'elasticsearch'

# Connect to localhost:9200 by default:
es = Elasticsearch::Client.new log: true

# Round-robin between two nodes:
es = Elasticsearch::Client.new hosts: ['search1:9200', 'search2:9200']

# Connect to cluster at search1:9200, sniff all nodes and round-robin between them
es = Elasticsearch::Client.new hosts: ['search1:9200'], reload_connections: true

# Index a document:
es.index index: 'my_app',
         type:  'blog_post',
         id: 1,
         body: {
          title:   "Elasticsearch clients",
          content: "Interesting content...",
          date:    "2013-09-24"
         }

# Get the document:
es.get index: 'my_app', type: 'blog_post', id: 1

# Search:
es.search index: 'my_app',
          body: { query: { match: { title: 'elasticsearch' } } }

From the developer

My name is Karel Minařík, and I discovered Elasticsearch in late 2010, when I was researching alternatives for CouchDB-Lucene, which I was using on a project at the time. As I’m quite often quoted, I initially considered the project a hoax; there just can’t exist such a magical project with so many buzzwords (HTTP, JSON, schema-free, distributed, …), I was saying to myself. Nevertheless, I installed the project, glanced on the source, and it not only did what it said on the tin — it surpassed all expectations, up to the level where my colleagues didn’t believed me when I was demoing it.

Our application was written in Ruby, and we had a pretty good idea how we would like to “talk” to Elasticsearch: we need a DSL for its DSL! One late evening, I have sketched such Ruby DSL, and after some time, we had a brand new client as a Rubygem, called Slingshot and later renamed to Tire. Little did I know that this project will get nearly 1,500 watchers and 500 forks at Github, that I’ll solve more then 700 issues, and the library will grow to be loved and hated by many users.

Tire managed to solve many issues for a nice Ruby and Rails integration, and it also managed to make some seriously bad decisions. Today marks a fresh start for me, and for Ruby users of Elasticsearch. Today, we’re releasing a suite of clients with common design and semantics, which aims to provide a solid foundation for further extensions, integrations and experiments.

If you liked Tire, you can be sure that all the convenience for painless Rails integration will come soon. If you hated Tire, I’d like to invite you to kick the tires on the new Ruby client. I took great care to avoid all the bad decisions of Tire, and make the library — or, in fact a collection of three libraries — as modular, extensible and robust as much as I could. Your feedback, opinions and critique — either via Github issues, via IRC or via e-mail — is most welcome!