Getting Started with LIRE and Elasticsearch

Installing

The LIRE plugin for Elasticsearch is called elasticsearch-image and is available on Github. It bundles LIRE and all we have to do is to install a version that matches our Elasticsearch version. Unfortunately, it currently lags a little behind the Elasticsearch releases, but as Elasticsearch is quick to adopt new Lucene versions, this is probably to be expected for plugins that tie in all the way down to the Lucene layer.

The latest release of the plugin is 1.2.0 and it was built against Elasticsearch 1.0.1. In this article we will be using Elasticsearch 1.0.3, because the only changes between those versions are bug fixes. The master branch is currently targeting Elasticsearch 1.1.0, but it is still a work in progress as not all are passing.

bin/plugin -install com.github.kzwang/elasticsearch-image/1.2.0

Mappings

In order to get started, we need to create an index with appropriate mappings. For this test, the cluster only has one node, and we don’t intend to grow it any further so we only use one shard in the index. What is more interesting is the mappings:

    "image": {
        "properties": {
            "name": {
                "type": "string"
            },
            "image": {
                "type": "image",
                "feature": {
                    "CEDD": {
                        "hash": "BIT_SAMPLING"
                    },
                    "JCD": {
                        "hash": ["BIT_SAMPLING", "LSH"]
                    },
                    "FCTH": {}
                }
            }
        }
    }

We define a type, called image, and it has two properties: name, the file name of the image and image which will contain the actual image data. The first is a simple string type, like normal, but the latter has type image. The image datatype is defined by the plugin and is how Elasticsearch knows to utilize the LIRE plugin. The data type supports different types of fetures and these in turn provide different ways of matching the query image with the indexed images when searching.

curl -XPOST 'https://cluster_id-eu-west-1.foundcluster.com:9243/images' -d '{
    "settings" : {
        "number_of_shards" : 1
    },
    "mappings" :{
    "image": {
        "properties": {
            "name": {
                "type": "string"
            },
            "image": {
                "type": "image",
                "feature": {
                    "CEDD": {
                        "hash": "BIT_SAMPLING"
                    },
                    "JCD": {
                        "hash": ["BIT_SAMPLING", "LSH"]
                    },
                    "FCTH": {}
                }
            }
        }
    }
}}'

Indexing Images

Indexing a document containing image data is not much different from indexing any other document. The API is the same, but the exception is that we want to put binary data in the image field. Since JSON does not support binary data as a native value, the plugin expects the data to be base 64 encoded as a JSON string. To distinguish these strings from normal strings, we also have to define the mappings explicitly for the image field.

Base64 encoding is a transformation that encodes 3 bytes of data into four printable charachters. Most programming languages have utilities to do this and if not there is usually not difficult to adapt an example found online.

For this article, I have written a small example in Scala that indexes the 25k images data set from Flickr. Using the sun.misc.BASE64Encoder it is pretty straight forward to create JSON documents for each image file. The only gotcha is removing the newlines, like this:

import java.nio.file.Files
import java.nio.file.Path
import sun.misc.BASE64Encoder
case class ImageDocument(name: String, data: Array[Byte]) {
  def this(path: Path) = this(path.toFile().getName(), Files.readAllBytes(path))
  def toJson = {
    s"""
    {
        "name" : "$name",
        "image" :"${new BASE64Encoder().encode(data).replaceAll("[\n\r]", "")}"
    }"""
  }
}

To retrieve all the images in a certain path, we define the following function:

  def getData(): Stream[ImageDocument] = {
    for {
      childPath <- Files.newDirectoryStream(path).asScala.toStream
      if childPath.toFile().getName().endsWith(".jpg")
    } yield new ImageDocument(childPath)
  }

We then push the stream of images in batches to Elasticsearch. First by creating a transport client to my Found cluster and wait for it to connect:

val settings = ImmutableSettings.settingsBuilder()
  .put("transport.type", "no.found.elasticsearch.transport.netty.FoundNettyTransportModule")
  .put("transport.found.api-key", "api-key")
  .put("cluster.name", "cluster_id")
  .put("client.transport.ignore_cluster_name", false)
  .build();
val client = {
  val address = new InetSocketTransportAddress("clusterid-region.foundcluster.com", 9343);
  ElasticClient.fromClient(new TransportClient(settings).addTransportAddress(address))
}
println("Waiting for clusterstatus yellow.")
val clusterStatus = client.client.admin().cluster().prepareHealth().setWaitForYellowStatus().execute().actionGet().getStatus().name()
println(s"Got clusterstatus $clusterStatus")

With the client ready, the actual indexing is done like this:

import ExecutionContext.Implicits.global
val bulkLoads = images.grouped(bulkSize).map {
  batch =>
    client.bulk(
      batch.map(image => index into "images/image" source new StringSource(image.toJson)): _*)
}.toStream
Future.sequence(bulkLoads).onComplete(result => {
  println("Closing client")
  client.close()
})
for (bulk <- bulkLoads) {
  bulk.onSuccess {
    case bulkResult => {
      println(s"Bulkresult ${bulkResult.getItems()}")
      for (result <- bulkResult.getItems()) {
        if (result.isFailed()) {
          println(s"Failed response: [${result.getFailure()}][${result.getFailureMessage()}]")
        } else {
          println("Bulk success")
        }
      }
    }
  }
}
val failureCount = (Future sequence bulkLoads.map(_.map(_.asScala.count(_.isFailed())))).map(_.sum)
failureCount.onSuccess {
  case count: Int => {
    println(s"Found ${images.size} images, $count failed to index")
  }
}

The complete example can be found in the gist.

Searching

The Elasticsearch-image plugin also provides a new query type, aptly named image. Using it with the rest API looks like this:

curl -XPOST 'localhost:9200/images/_search' -d '{
    "fields": [
        "name"
    ],
    "query": {
        "image": {
            "image": {
                "feature": "CEDD",
                "image": "<Base64 encoded image binary>",
                "hash": "BIT_SAMPLING",
                "limit": 10
            }
        }
    }
}

Don’t load the images from Elasticsearch unless you have to. In this example query we use the fields parameter to load only the names of the images. This is because the images are typically large, and it really slows down searches if you want Elasticsearch to deliver this data. You can read more about this and similar tuning options in the article Managing Elasticsearch fields when searching.

Strictly speaking, the fields and limit parameters are optional, but they are higly recommended for performance reasons.

def toQuery: String = {
  s"""
  {   
      "image": {
          "image": {
              "feature": "CEDD",
              "image": "${new BASE64Encoder().encode(data).replaceAll("[\n\r]", "")}",
              "hash": "BIT_SAMPLING",
              "limit": 10
          }
      }
  }"""
}

client.prepareSearch("images").setQuery(image.toQuery).addField("name").get()

Searching for images similar to: im10314.jpg
Found: [im10314.jpg] with score: [1.0]
Found: [im21075.jpg] with score: [0.0362315]
Found: [im4494.jpg] with score: [0.03478625]
Found: [im23103.jpg] with score: [0.0347418]
Found: [im13182.jpg] with score: [0.031561606]
Found: [im2384.jpg] with score: [0.030222585]
Found: [im18142.jpg] with score: [0.03011181]
Found: [im2031.jpg] with score: [0.028815333]
Found: [im7330.jpg] with score: [0.024395658]
Found: [im22767.jpg] with score: [0.02146201]

New leader: im10.jpg, score: 0.5457066
New leader: im10064.jpg, score: 0.55895704
New leader: im10069.jpg, score: 0.61380416
New leader: im10090.jpg, score: 0.7782447
New leader: im10622.jpg, score: 0.9452252
New leader: im10817.jpg, score: 0.9933184

Images similar to: im10817.jpg
Found: [im10817.jpg] with score: [1.0]
Found: [im3443.jpg] with score: [1.0]
Found: [im8545.jpg] with score: [0.97995514]
Found: [im1360.jpg] with score: [0.5906452]
Found: [im19056.jpg] with score: [0.3436884]
Found: [im9820.jpg] with score: [0.31441176]
Found: [im12341.jpg] with score: [0.30534726]
Found: [im6995.jpg] with score: [0.3012863]
Found: [im6038.jpg] with score: [0.2511092]
Found: [im5160.jpg] with score: [0.2422185]

Plane Images

We have seen some similarity, but honestly I’m not yet impressed. In this final attempt, I handpick five images of the same motif: an airplane against a blue sky. I add them to the same index and execute the same query to see if any of them are retrieved within the top ten.

Image credits:

• Bertknot
• Hammerin Man
• Leo Hidalgo
• Steven Conry
• Joe Hunt

Images similar to: 8401586753_91a218f95e_o.jpg
Found: [8401586753_91a218f95e_o.jpg] with score: [1.0]
Found: [im14728.jpg] with score: [0.07146341]
Found: [im2633.jpg] with score: [0.059009902]
Found: [im13879.jpg] with score: [0.05875]
Found: [im2643.jpg] with score: [0.04318436]
Found: [im12893.jpg] with score: [0.039066523]
Found: [im23867.jpg] with score: [0.037845306]
Found: [im14107.jpg] with score: [0.033956632]
Found: [im15524.jpg] with score: [0.032459892]
Found: [im21666.jpg] with score: [0.03232]

Images similar to: 14668291623_2cbb04aa1c_o.jpg
Found: [14668291623_2cbb04aa1c_o.jpg] with score: [1.0]
Found: [im12011.jpg] with score: [0.058475036]
Found: [im18417.jpg] with score: [0.058243487]
Found: [im23867.jpg] with score: [0.049384493]
Found: [im12893.jpg] with score: [0.046724197]
Found: [im4108.jpg] with score: [0.045620646]
Found: [im15983.jpg] with score: [0.04494405]
Found: [im14728.jpg] with score: [0.0403279]
Found: [im17209.jpg] with score: [0.03635637]
Found: [im4504.jpg] with score: [0.03228649]

Images similar to: 14264287401_e68c94ee85_o.jpg
Found: [14264287401_e68c94ee85_o.jpg] with score: [1.0]
Found: [im3471.jpg] with score: [0.058208954]
Found: [im15179.jpg] with score: [0.0427226]
Found: [im3716.jpg] with score: [0.04201087]
Found: [im22055.jpg] with score: [0.04192308]
Found: [im12893.jpg] with score: [0.040504586]
Found: [im14161.jpg] with score: [0.038947366]
Found: [im12820.jpg] with score: [0.038699917]
Found: [im4108.jpg] with score: [0.036421873]
Found: [im9682.jpg] with score: [0.031241378]

Images similar to: 11731068223_2bb5258d7a_o.jpg
Found: [11731068223_2bb5258d7a_o.jpg] with score: [1.0]
Found: [im16938.jpg] with score: [0.07487332]
Found: [im13637.jpg] with score: [0.06531328]
Found: [im17338.jpg] with score: [0.065182954]
Found: [im22588.jpg] with score: [0.057351522]
Found: [im11164.jpg] with score: [0.05494677]
Found: [im794.jpg] with score: [0.048871726]
Found: [im13775.jpg] with score: [0.045117185]
Found: [im5373.jpg] with score: [0.043741353]
Found: [im24607.jpg] with score: [0.042783484]

Images similar to: 3893372643_a10546bacb_o.jpg
Found: [3893372643_a10546bacb_o.jpg] with score: [1.0]
Found: [im11164.jpg] with score: [0.058427628]
Found: [im14188.jpg] with score: [0.051731724]
Found: [im16314.jpg] with score: [0.041284915]
Found: [im11603.jpg] with score: [0.03983957]
Found: [im22150.jpg] with score: [0.039609164]
Found: [im20188.jpg] with score: [0.039445836]
Found: [im10609.jpg] with score: [0.038841564]
Found: [im22588.jpg] with score: [0.034103587]
Found: [im22229.jpg] with score: [0.033843976]

The surprising result in this test is that none of the handpicked images matched one another (among the top ten at least), but there seems to be more similar results along with a few odd ones. Many of the images found contain blue skies and a few of them even show an airplane. One explanation for this improvement could be that the handpicked images are larger, but one cannot really conclude on this as it is not a statistically significant sample.

Conclusion

The elasticsearch-image plugin is the way to go if Elasticsearch is what you got and content based image search is what you need. That said, it is still a little rough around the edges at times, but hopefully future releases will address that as well as targeting newer Elasticsearch versions.

Although we only had time to test one algorithm this time, LIRE has a lot more to offer and just as there are different kinds of images like photos, drawings or t-shirt patterns to name a few, it is not immediately given which one is the best fit for a given use case.