<html><head></head><body><p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.
</p>
<p>Elasticsearch has some very good support for geo locations. You can store locations, use distances to filter and even use distances to group using aggregations. Elasticsearch also has something called geo shapes - no worries, we&apos;ll explain later on what they are - to select locations within a certain area. If you combine these geo shapes with filters and something called the percolator, you can create a basic classification system. Curious? Read on to learn about the Elasticsearch percolator and geo support in Elasticsearch. To explain the concepts we use a sample application based on spring boot and angularjs. Before we jump into the technical stuff, we&apos;ll talk about the data lifecycle that you need for all data projects.
</p><section id="data-lifecycle-5-stages-of-handling-data" class="level2">
<h2><a href="#data-lifecycle-5-stages-of-handling-data">Data Lifecycle, 5 Stages of Handling Data</a></h2>
<p>The first step is to obtain the data, the second step to transform the data into the form we want to store it. This is the third step, storing the data. The fourth step is to use the data in reports, analysis or in our case a dashboard. The final step is to learn from the data, improve the import or transport process or track how people are using your data and use the usage data as new data.
</p>
<p>In this sample we use two data sources. A csv file with all the Dutch zip codes and a JavaScript file containing the geo shapes for the Dutch provinces.
</p>
<section id="obtaining-the-data" class="level3">
<h3><a href="#obtaining-the-data">Obtaining the Data</a></h3>
<p><em>csv file with Dutch zip codes</em>
</p>
<p>You can download the csv file from the following website: <a href="http://www.postcodedata.nl/download/">http://www.postcodedata.nl/download/</a>
</p>
<p>The following code block show one line of the csv file:
</p>
<pre class="prettyprint">395614;&quot;7940XX&quot;;79408888;7940;&quot;XX&quot;;4;12;&quot;mixed&quot;;&quot;Troelstraplein&quot;;&quot;Meppel&quot;;1082;&quot;Meppel&quot;;119;&quot;Drenthe&quot;;&quot;DR&quot;;&quot;52.7047653217626&quot;;&quot;6.1977201775604&quot;;&quot;209781.52077777777777777778&quot;;&quot;524458.25733333333333333333&quot;;&quot;postcode&quot;;&quot;2014-04-10 13:20:28&quot;
</pre>
<p><em>JS file with the Dutch provinces</em>
</p>
<p>The province shape data is coming from a small GitHub project containing the geo shapes for all the Dutch provinces: <a href="https://gist.github.com/lekkerduidelijk/4387055">https://gist.github.com/lekkerduidelijk/4387055</a>
</p>
</section>
<section id="transforming-the-data" class="level3">
<h3><a href="#transforming-the-data">Transforming the Data</a></h3>
<p>Transformation of the zip codes was not too hard. We had a csv file that was imported into our application. It was mapped onto a much easier object containing less fields. Only the required fields were kept.
</p>
<p>The province data was a two dimensional array, the first dimension representing the different provinces. The second dimension is a set of strings with lat,lon doubles in there. We manually extracted the different provinces into a file for each province containing the one dimensional array containing strings. So the files look like this: [&#x201C;52.829113,6.151108&#x201D;,&#x201C;52.814176,6.177200&#x201D;, &#x2026;]
</p>
<p>The string containing two doubles might seem unuseful and a first citizen to be transformed, however as we will discuss in the Elasticsearch geo support section, we can create a <em>GeoPoint</em> from such a string.
</p>
</section>
<section id="storing-the-data" class="level3">
<h3><a href="#storing-the-data">Storing the Data</a></h3>
<p>The zip codes were imported using a Java application that created a number of index requests. With Java and the Jackson library creating documents to index becomes easy, just create the object, serialize it using Jackson and pass it on to the Elasticsearch API. In a later section we have a short look at the structure of the object.
</p>
<p>Often when using Elasticsearch, you will create a lot of documents. In the percolator case we create just 12 documents, they are special documents though. They are percolator queries, this is something special from Elasticsearch where you store the query instead of the document and then match documents against queries. The queries we store will be filtered queries using the <a href="http://www.elastic.co/guide/en/elasticsearch/reference/1.4/query-dsl-geo-polygon-filter.html">Geo Polygon Filter</a>.
</p>
</section>
<section id="using-the-data" class="level3">
<h3><a href="#using-the-data">Using the Data</a></h3>
<p>To show what we can do with geo shapes and the percolator we want an application with maps. Without maps, what is the use for geo locations. We are going to show you how to find geo locations for cities, then using these locations we use the percolator to find the province for the location. Finally, we use the percolator geo shape as stored in the query to highlight the province on the map. The next image shows the application we are creating.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c9a3dc542cb38c084c1/download?uid=blt17954fe5bd44733d?uid=blt17954fe5bd44733d" data-sys-asset-uid="bltcf3ae450b04329d4" alt="screenshot created application">
</p><figcaption>screenshot created application</figcaption>
<p>By now you should have an idea what we did with the data, in the next section we are explaining the basic support for geo in Elasticsearch.
</p>
</section>
</section><section id="elasticsearch-geo-support" class="level2">
<h2><a href="#elasticsearch-geo-support">Elasticsearch Geo Support</a></h2>
<section id="indexing-geo-points" class="level3">
<h3><a href="#indexing-geo-points">Indexing Geo Points</a></h3>
<p>Elasticsearch can index geo_points, a geo_point is represented by two doubles. One double being the latitude and the other being the longitude. Using this point Elasticsearch can perform calculations like distances and do grouping using distances as well. If you want to store a location as a geo_point, you have to explicitly mention this in the mapping. If you do not do this, you will end up with just to fields of type double. The following code block shows the mapping for a field called <em>location</em> of type geo_point.
</p>
<pre class="sourceCode JavaScript prettyprint">{
  &quot;properties&quot;: {
    &quot;location&quot;: {
      &quot;type&quot;: &quot;geo_point&quot;
    }
  }
}
</pre>
<p>With the mapping in place we can add geo_points. There are three different formats for inserting geo points:
</p>
<ol type="1">
	<li>string -&gt; location: &#x201C;lat,lon&#x201D;</li>
	<li>object -&gt; location: {&#x201C;lat&#x201D;: .., &#x201C;lon&#x201D;: ..}</li>
	<li>array -&gt; location: [lon,lat]</li>
</ol>
<p>Notice the difference in order of lat and lon in options 1 and 3. The reason for this difference can historically be tracked to how cartographers disagree with computer scientists on what is the correct order. We prefer option 2, as we can be explicit about what the lat is and what the lon is.
</p>
<p>Read more about the geo_point type in the reference manual of Elasticsearch: <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-geo-point-type.html">https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-geo-point-type.html</a>
</p>
<p>In the sample project we model our entities as Plain Old Java Objects (POJO) and serialize them using Jackson. Our PostalCode object contains an object Location which has two properties called lat and lon. When serializing this object to a string based JSON notation, this will result in option 2 <em>object</em> based notation of the geo_type.
</p>
<pre class="sourceCode JavaScript prettyprint">{
  &quot;location&quot;: {
    &quot;lat&quot;: ...,
    &quot;lon&quot;:...
  }
}
</pre>
</section>
<section id="filtering-on-geo-points" class="level3">
<h3><a href="#filtering-on-geo-points">Filtering on Geo Points</a></h3>
<p>If you have an index available with some geo points, you can create filters to select documents based on these geo points. There are a number of filters available:
</p>
<ul>
	<li>geo_bounding_box -&gt; A rectangle where all points need to be in</li>
	<li>geo_distance -&gt; A central point with a radius</li>
	<li>geo_distance_range -&gt; More like a donut shape</li>
	<li>geo_polygon -&gt; Create your own polygon using locations</li>
</ul>
<p>To get a feeling about the structure of such a request, let us have a look at a query that makes use of the geo_distance filter. In my database I have all zip codes of the Netherlands and I would like to filter on all postal codes that are within a 1 km radius of my home. Before we can do this query, we need to find the coordinates for my home. Of course I can query the postal code database, but it is more fun to use a specific website for this. Using this website I get the coordinates of my home: <a href="http://www.latlong.net/convert-address-to-lat-long.html">http://www.latlong.net/convert-address-to-lat-long.html</a>
</p>
<p>The coordinates are: lat-52.060669,lon-4.494025
</p>
<p>The query that would find all zip codes within a 1km radius of my home would be:
</p>
<pre class="sourceCode JavaScript prettyprint">GET /geostuff/_search
{
  &quot;query&quot;: {
    &quot;filtered&quot;: {
      &quot;filter&quot;: {
        &quot;geo_distance&quot;: {
          &quot;distance&quot;: &quot;1km&quot;,
          &quot;location&quot;: {
            &quot;lat&quot;: 52.060669,
            &quot;lon&quot;: 4.494025
          }
        }
      }
    }
  }
}
</pre>
<p>If you want more information on the geo filters, there is this very good online resource: <a href="http://www.elastic.co/guide/en/elasticsearch/guide/current/geoloc.html">http://www.elastic.co/guide/en/elasticsearch/guide/current/geoloc.html</a>
</p>
<p>In the application that we are creating, we want to find the province where a city is in. The way we want to do this is by using the <em>GeoPolygonFilter</em>. This is an expensive filter, but for the amount of points we have it is good to use. The input for this filter is a set of points that together create a polygon. It would look for all the points that fall within that polygon. So to find all zip codes for the province <em>zuid-holland</em> we could use the following query:
</p>
<pre class="sourceCode JavaScript prettyprint">GET /geostuff/_search
{
  &quot;query&quot;: {
    &quot;filtered&quot;: {
      &quot;filter&quot;: {
        &quot;geo_polygon&quot;: {
          &quot;location&quot;: {
            &quot;points&quot;: [
              &quot;52.275300,4.453033&quot;,
              &quot;52.224435,4.410805&quot;,
              ... left out around 250 other points
              &quot;52.309633,4.564373&quot;,
              &quot;52.328625,4.492962&quot;
            ]
          }
        }
      }
    }
  }
}
</pre>
<p>How does this helps us to find the province for a city? That is where the percolator comes in, this percolator is the main topic for the next section.
</p>
</section>
</section><section id="using-percolator-to-find-provinces" class="level2">
<h2><a href="#using-percolator-to-find-provinces">Using Percolator to Find Provinces</a></h2>
<p>The percolator is what is called an inverse query. What is meant with that is that we store the query and use a document to find a query that matches the document. Usually this would be the other way around, store documents and use a query to find documents. So why is this interesting? Take a look at the previous code sample, this is a query for one province. We have 12 in the Netherlands. What we do is store all 12 queries in the percolator and then use the coordinates of a city to go over all queries and see if one matches. The matching query should be for the province that the city is.
</p>
<section id="storing-the-percolated-queries" class="level3">
<h3><a href="#storing-the-percolated-queries">Storing the Percolated Queries</a></h3>
<p>Storing a percolator query is the same as any other document, you just store them in a separate type of your index. You can store them in their own index, that does not really change the idea. So for now, store them as a separate type. In our sample we store the queries using the following Java code.
</p>
<pre class="sourceCode java prettyprint">  &amp;public void doCreatePercolatorQuery(String province) {
    client.prepareIndex(GEO_INDEX, &quot;.percolator&quot;, &quot;province_&quot; + province)
        .setSource(jsonBuilder()
            .startObject()
                .field(&quot;query&quot;, createQuery(province + &quot;.txt&quot;))
                .field(&quot;province&quot;, province)
            .endObject()
        ).setRefresh(true).get();
  }

  private FilteredQueryBuilder createQuery(String provincePoints) {
    final List<geopoint> polygon = new ArrayList&lt;&gt;();
    List<string> geo = null;
    try {
      geo = new ObjectMapper().readValue(new ClassPathResource(provincePoints).getInputStream(), List.class);
    } catch (IOException e) {
      throw new RuntimeException(&quot;Cannot read province file with points &quot; + provincePoints);
    }

    geo.stream().forEach(s -&gt; {
      polygon.add(new GeoPoint(s));
    });

    GeoPolygonFilterBuilder geoPolygonFilterBuilder = FilterBuilders.geoPolygonFilter(&quot;location&quot;);
    polygon.stream().forEach(geoPolygonFilterBuilder::addPoint);
    return filteredQuery(matchAllQuery(), geoPolygonFilterBuilder);
  }
</string></geopoint></pre>
<p>Notice how we use the <em>jsonBuilder</em> to create the percolator document for the index GEO_INDEX and type .percolator. Each document gets the id equal to the name of the province prepended with the word province. Then in the field query we store the create query obtained from the method <em>createQuery</em>. In this method we read a file containing the array with all the geo_points, just like in the example query in one of the previous code blocks. Executing this code for all provinces gives us 12 percolator queries in the index GEO_INDEX, which is a constant containing geostuff by the way.
</p>
</section>
<section id="using-the-percolator-end-point" class="level3">
<h3><a href="#using-the-percolator-end-point">Using the Percolator End Point</a></h3>
<p>Now we want to use the coordinates of my home town to find the province using the percolator API. The following code block shows the request in JSON format and well as the response.
</p>
<pre class="sourceCode JavaScript prettyprint">GET /geostuff/locations/_percolate
{
  &quot;doc&quot;: {
    &quot;location&quot;: {
      &quot;lat&quot;: 52.060893,
      &quot;lon&quot;: 4.534213
    }
  }
}

The response:
{
   &quot;took&quot;: 3,
   &quot;_shards&quot;: {
      &quot;total&quot;: 1,
      &quot;successful&quot;: 1,
      &quot;failed&quot;: 0
   },
   &quot;total&quot;: 1,
   &quot;matches&quot;: [
      {
         &quot;_index&quot;: &quot;geostuff-20150306114746&quot;,
         &quot;_id&quot;: &quot;province_zuidholland&quot;
      }
   ]
}
</pre>
<p>Notice that we only get back the _id of the document, we can store more fields in the percolator document, however to obtain them we have to explicitly request the document using the standard get request. For the sample application we want to print the found province on a map and therefore we need all the points of the shape, and so we have to obtain the percolated document. To be complete the next code block shows the Java code to do the same.
</p>
<pre class="sourceCode java prettyprint">    public String checkLocationForProvince(double longitude, double latitude) {
      XContentBuilder docToCheck = jsonBuilder()
            .startObject()
            .startObject(&quot;location&quot;)
            .field(&quot;lat&quot;, latitude)
            .field(&quot;lon&quot;, longitude)
            .endObject()
            .endObject();
      PercolateSourceBuilder.DocBuilder builder = new PercolateSourceBuilder.DocBuilder();
      builder.setDoc(docToCheck);

      PercolateResponse matches = client.preparePercolate()
            .setPercolateDoc(builder)
            .setIndices(&quot;geostuff&quot;)
            .setDocumentType(&quot;locations&quot;)
            .get();
    if (matches.getMatches().length &gt; 0) {
      return matches.getMatches()[0].getId().toString();
    }
    return &quot;Not in a province&quot;;
  }
</pre>
</section>
</section><section id="the-sample-application" class="level2">
<h2><a href="#the-sample-application">The Sample Application</a></h2>
<p>You can find the sample application on my GitHub account: <a href="https://github.com/jettro/geo-elastic">https://github.com/jettro/geo-elastic</a>
</p>
<p>The sample is created using spring boot and maven. If you have that installed you can run it using: mvn package &amp; Java -jar target/geo-elastic-0.1-SNAPSHOT.jar
</p>
<p>It tries to connect to a cluster named performance, if you want a different cluster name change the name of the cluster in src/main/resources/application.properties
</p>
<p>If you want to import the postal codes as well, you have to download the following file: <a href="http://download.postcodedata.nl/data/postcode_NL_head.csv.zip">http://download.postcodedata.nl/data/postcode_NL_head.csv.zip</a>
</p>
<p>Unzip the file into the project directory: src/main/resources and start the application as mentioned before. Now go to the <em>maintain</em> tab (see the following image) and push the <em>Restore percolators</em> button and after that the <em>Restore postal codes</em> button. If everything goes well you should see <em>471993</em> postal codes right below the restore button.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c9a3dc542cb38c084c2/download?uid=blt657a3ee5d24d0963?uid=blt657a3ee5d24d0963" data-sys-asset-uid="bltd7d041c1a94f1bfa" alt="screenshot maintain page">
</p><figcaption>screenshot maintain page</figcaption>
<p>Before leaving you experimenting with this code I want to mention the angular-leaflet-directive, which made it very easy to integrate the map powered by leaflet: <a href="https://github.com/tombatossals/angular-leaflet-directive">https://github.com/tombatossals/angular-leaflet-directive</a>
</p>
</section><section id="summarizing" class="level2">
<h2><a href="#summarizing">Summarizing</a></h2>
<p>That&#x2019;s it, we went from discussing the Data Lifecycle to showing it in practice with a sample based on zip codes and provinces in The Netherlands. You have seen how to import a csv file, create the mapping for Elasticsearch for geo_points. With the indexed data we went on to introduce geo filters and especially the geo polygon filter. In the end all this was used with the percolator to find the province for a city based on its coordinates.
</p>
<p>The final step in the data lifecycle is learning from how your application is used. Since this is a sample application this might feel artificial. A good scenario could be to record what cities people search for, if a city is searched for more often we can add it to the presets. Also we can count the number of found provinces and change the colors of the provinces that are found most often.
</p>
<p>Now imagine what cool map based products you can create, because we all want cool map based products.
</p>
</section></body></html>

Combining Geo Points With the Elasticsearch Percolator

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Blog Footer

Jettro Coenradie

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content.

You'll also receive an email with related content.

You'll also receive an email with related content

Load More

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Newsletter GDPR Text

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

Elasticsearch Platform

ELK Stack

Elastic Cloud

Observability

Security

Search

By industry

By solution

Customer spotlight

Developers

Connect

Learn

Help

See what's happening at Elastic

Author

Articles by Jettro Coenradie

Combining Geo Points With the Elasticsearch Percolator

Follow us

About us

Join us

Press

Partners

Trust & Security

Investor relations

EXCELLENCE AWARDS