<html><head></head><body><h2>Content Repository &amp; Search</h2>At <a href="http://www.nuxeo.com/" target="_blank">Nuxeo</a>, our job is to provide a content repository, a stack of associated services and the needed assembly tools so that people can build their own content-centric business application in a clean, maintainable and scalable way. When it comes to scalability, the first challenge is usually not so much the storage volume but the ability to quickly execute search queries. Indeed, most of the screens of applications heavily rely on queries: searching for documents under a given hierarchy, in a given state or associated to a task... 
<img src="https://api.contentstack.io/v2/assets/575e4cf69985d58976377be6/download?uid=blt75ed3068a42613ef?uid=blt75ed3068a42613ef" data-sys-asset-uid="blt707b124f4b1639a9" alt="Nuxeo1.png" width="558" height="315" style="width: 558px; height: 315px;">
 As a result, pretty much all screens of the application are issuing one or several queries. In addition, platform users can configure both the data structures and the screens so the used queries can vary a lot and can become very complex.
<h2>Our History with Lucene</h2>We&apos;ve been building a content repository for more than 10 years, so we know about this search challenge. Since everything we do is open source, we have a long history with Apache Lucene. Strangely enough, we started using Lucene at a time where our repository was running on Python/Zope. We used <a href="http://lucene.apache.org/pylucene/" target="_blank">PyLucene</a> to build an XML-RPC search service. This hybrid solution was a pain to setup (compile Lucene to native code via GCJ), but when it was finally up and running the performance was just amazing. When we moved the whole platform to Java, Lucene was logically one of the building blocs we re-used. At this time, our repository backend was <a href="https://jackrabbit.apache.org/" target="_blank">Apache JackRabbit</a> and since the built-in Lucene index was not enough to fulfill our requirements for complex queries, we integrated <a href="http://www.compass-project.org/" target="_blank">Compass Core</a> that was providing a transactional layer on top of Lucene (and had an interesting <a href="https://github.com/kimchy/compass" target="_blank">lead developer</a>). This Lucene integration was not so successful for us since we ended up with a lot of missing sync and transaction deadlocks issues at the index level. Along with Jackrabbit limitations, we decide to re-write completely the repository implementation, 100% SQL based and then 100% ACID. The result is a very reliable storage where everything relies on the SQL database. 
<img src="https://api.contentstack.io/v2/assets/575e4cf658208ba076e29231/download?uid=blt4a9b60c9ec8fcbd2?uid=blt4a9b60c9ec8fcbd2" data-sys-asset-uid="bltd3b173fb7231fe22" alt="Nuxeo2.png" width="684" height="641" style="width: 684px; height: 641px;">
However, there are some limitations in terms of queries: <ul>
	<li>full-text capabilities are poor in most databases</li>
	<li>some queries can not be fast</li>
	<li>scale out is complex</li>
</ul>So in 2013, we started implementing Elasticsearch connector for the Nuxeo Platform with the goal to get rid of these problems: spend less time on complex SQL tuning and focus on Elasticsearch index mapping.
<h1>Elasticsearch Integration Overview</h1><h2>Hybrid Storage</h2>The idea was to build an additional index on top of the repository:
<ul>
	<li>Nuxeo Repository keeps its primary index based on SQL 
	<ul>
		<li>ACID but limited in power</li>
	</ul></li>
	<li>Elasticsearch will provide an additional index 
	<ul>
		<li>full featured index, but asynchronous</li>
	</ul></li>
</ul><img src="https://api.contentstack.io/v2/assets/575e4cf69985d58976377beb/download?uid=blt743065bdd4df9a61?uid=blt743065bdd4df9a61" data-sys-asset-uid="blt97e03d6bae19ee2c" alt="Nuxeo3.png" width="688" height="694" style="width: 688px; height: 694px;">
One of the main advantages of this approach is that queries are written once using NXQL and then, depending on the configuration, it will be executed by the repository or the Elasticsearch index. On a per query basis, transactional behavior or search speed can be favored. Later, when we introduced the MongoDB backend for the repository, this demarcation became synchronous vs asynchronous, but still, Elasticsearch remains the solution for blazing fast search. How do we know that? Well, we did actual benchmarks!
<h2>Performances</h2>As soon as we had a first version of the <code>nuxeo-elasticsearch</code>&#xA0;connector, we started doing performances benchmarks. One of the first tests we did was comparing the performances the same NXQL query between a Nuxeo Repository internal SQL index and the Elasticsearch index. The results are very significants: 
<img src="https://api.contentstack.io/v2/assets/575e4cf73dc542cb38c0948c/download?uid=blt154be0f546687635?uid=blt154be0f546687635" data-sys-asset-uid="blta80b909aee9491b3" alt="Nuxeo4.png" width="485" height="409" style="width: 485px; height: 409px;">
Response time is far smaller and remains stable under load.
<img src="https://api.contentstack.io/v2/assets/575e4cf743e9adc538714c45/download?uid=blta6f5f83ecc2f55b3?uid=blta6f5f83ecc2f55b3" data-sys-asset-uid="blte6a99906f3f38720" alt="Nuxeo5.png" width="536" height="209" style="width: 536px; height: 209px;">
Throughput increases linearly under load. Happy with these first results, we finished the integration and did some more benchmarks. One of the key performance aspects is re-indexing speed:
<ul>
	<li>because it is useful for initial migration</li>
	<li>because we have several use cases for recurse re-indexing</li>
</ul>We tested re-indexing speed and measured a throughput of about 3,500 documents/s. This is fast enough so that a full re-indexing would not be a problem. But, we actually saw that the bottleneck for the re-indexing was not Elasticsearch, but the repository SQL backend. So, we ran again the same test with the MongoDB backend and measured a re-indexing throughput of 10,000 documents/s. We also tested the Elasticsearch capacity to handle scale out. For that, we injected a lot of queries on a Nuxeo Repository via REST API. The limit reached was about 3,000 queries/s. Then adding a second Elasticsearch node gave us about 6,000 queries/s, showing a linear scale-out capability. During these tests, we leveraged the fact that Elasticsearch keeps a copy of the source JSON document stored inside the index: this allows us to off-load the query and the retrieval from the database to Elasticsearch. 
<img src="https://api.contentstack.io/v2/assets/575e4cf70342dfd738264384/download?uid=blta0a0723a452ed26d?uid=blta0a0723a452ed26d" data-sys-asset-uid="blt17fe8cf7d3870534" alt="Nuxeo6.png" width="722" height="408" style="width: 722px; height: 408px;">
<h2>This is magic!</h2>Within the technical team, we were very impressed by the Elasticsearch technology and how well everything was working. Then, the Solutions Architects team started playing with it and they discovered that in addition of high-end performances, Elasticsearch also brings a lot of new features (more on this later). Finally, it was our client&apos;s turn to discovered Elasticsearch. For several of them, solving their performances or scalability concerns was as simple as to say&#xA0;&quot;please activate the <code>nuxeo-elasticsearch</code> plugin!&quot;. This is so true, that we integrated the <code style="line-height: 1.6em;" rel="line-height: 1.6em;">nuxeo-elasticsearch</code> plugin as the default deployment option in Nuxeo 6.0 and also &quot;back-ported&quot; to make it available for people using the previous LTS.
<h1>Technical Integration of Elasticsearch</h1>We knew from Elasticsearch previous work on integrating Lucene that the main challenges we had to address were: 
<ul>
	<li>Handle security filtering</li>
	<li>Keep the index synchronized with the repository</li>
	<li>Mitigate eventually consistency side effects</li>
</ul><h2>Security Filtering</h2>Inside the Nuxeo Repository, the documents are associated with security descriptors (ACLs) that are used to determine &quot;who can do what&quot;&#xA0;on the document. This is also true when it comes to viewing a document, so this also applies to search: search results must be filtered according to user credentials. Doing post-filtering is not an option as we would loose all the added-value of Elasticsearch (speed, aggregates...). In the past, we also tested solutions based on joining Lucene Indexes together: this ended up being a pain and since anyway it was not compliant by Elasticsearch distributed architecture, this option was ruled out. So, we decided to index the ACLs inside Elasticsearch. For that, we build a synthesis of the users and groups who have access to the document and we make it part of the JSON representation of the document.
<pre>ecm:acl: [&quot;Management&quot;, &quot;Supervisors&quot;, &quot;bob&quot;]
</pre>Then, when translating NXQL queries to Elasticsearch JSON DSL, Nuxeo will always add a filter clause of <code>ecm:acl</code>. Because the ACLs can be inherited, a change of security will usually trigger a recursive re-index, putting more pressure on the infrastructure that must keep the index in sync with the repository.
<h2>Keeping the Index in Sync</h2>Our past experience with Lucene has led us to believe that trying to make Lucene transactional is a mistake. So, we decided to accept this limitation and build a system that will make the indexing safe even if asynchronous and non-transactional. The idea is simple:
<ul>
	<li>collect events occurring during the transaction</li>
	<li>de-duplicate them if needed</li>
	<li>if the transaction on the repository commits: create an asynchronous job to update Elasticsearch</li>
</ul>To make this safe, the asynchronous jobs must be persistent and re-triable, but since the Nuxeo Platform uses Redis to back the processing queues, this is not an issue. 
<img src="https://api.contentstack.io/v2/assets/575e4cf79e7a83165490d2a0/download?uid=bltd0c635916fd63de2?uid=bltd0c635916fd63de2" data-sys-asset-uid="blt847bda1de6c012ba" alt="Nuxeo7.png" width="683" height="453" style="width: 683px; height: 453px;">
<h2>Mitigating Consistency Issues</h2>Search is executed against data structures that require some processing to build, which means it is best to batch up some documents before building one. So while Elasticsearch guarantees that documents can be retrieved by ID immediately, it may take a second before Lucene makes the same document visible to search. 
<img src="https://api.contentstack.io/v2/assets/575e4cf743e9adc538714c43/download?uid=blt9d537fd83d905f3c?uid=blt9d537fd83d905f3c" data-sys-asset-uid="bltc893a915cdee1fa3" alt="Nuxeo8.png" width="461" height="201" style="width: 461px; height: 201px;">
This is actually a very good trade-off considering the speed it provides and the very short time interval during which problems may be visible. Still, there are some cases where this can be an issue. If a user lists documents to process and validate one of them, he logically expects this document to disappear from the list. But when the list is built using Elasticsearch, there might be a small &quot;refresh issue&quot; because the index is not updated in real time.
To mitigate this issue, we added a &quot;pseudo real time mode&quot;: when the indexing events are caused by a &quot;UI Thread&quot;, the indexing jobs is run on the <code>afterCompletion</code> of the repository transaction and will issue a <code>refresh()</code>.
<h2>Results in Production</h2>We have some customers with past bad experience with Compass/Lucene and logically they were worried about index consistency when their fist Nuxeo + Elasticsearch clusters went live. So, we built <a href="https://github.com/nuxeo/esync" target="_blank">some tools</a> to check the Elasticsearch index and verify that it is really &quot;in sync&quot; with the repository. Having these tools was useful to make everyone more relaxed, and we did not find any real issue. The feedback of our customers regarding Elasticsearch integration is very good.
<h1>Additional Benefits</h1>Integrating Elasticsearch into the Nuxeo Platform gave us a way to really boost the performance of the platform. However, we also quickly saw that Elasticsearch can provide us more than raw performances.
<h2>Aggregates</h2>Once we had integrated Elasticsearch as a backend for NXQL Queries, it became possible to leverage the aggregate feature. To do so, we extended the PageProvider model (our model for named queries) to allow to define aggregates and the result is a powerful Faceted Search feature. 
<img src="https://api.contentstack.io/v2/assets/575e4cf79e7a83165490d2ab/download?uid=blt3c95ea1d88192d82?uid=blt3c95ea1d88192d82" data-sys-asset-uid="bltfe954fdcd71fdb2f" alt="Nuxeo10.png" width="637" height="360" style="width: 637px; height: 360px;">
<h2>Advanced Indexing </h2>Elasticsearch index semantic is much richer than what we had inside a SQL Database. This allows for index fine tuning like:
<ul>
	<li><a href="http://www.nuxeo.com/blog/elasticsearch-for-nuxeo-optimize-multilingual-search-results/" target="_blank">multi language support using multiple analyzers and copy_to</a></li>
	<li>compound fields created using groovy scripts</li>
	<li>geolocation search</li>
</ul>To be able to leverage that, we had to extend our NXQL query language to support some <a href="https://doc.nuxeo.com/display/NXDOC/NXQL#NXQL-ElasticsearchNXQLHints" target="_blank">Elasticsearch specific hints</a>.
<h2>Audit Trail</h2>Audit Log is a typical nightmare for an SQL database: write an intensive and huge amount of free form entries. 
<img src="https://api.contentstack.io/v2/assets/575e4cf758208ba076e2924f/download?uid=blt70af40f34b237744?uid=blt70af40f34b237744" data-sys-asset-uid="blte02d98150debbb80" alt="Nuxeo11.png" width="728" height="434" style="width: 728px; height: 434px;">
Here we can use Elasticsearch as a primary storage for what he does the best a write once, search many. In addition, it allows for very cool queries mixing documents and history.
<h2>Statistics</h2>This open the way for doing real-time data analytics on the documents and Audit Log that are available in Elasticsearch. For this, we added a <a href="http://www.nuxeo.com/blog/maximize-potential-elasticsearch-new-api-passthrough/" target="_blank">Nuxeo Elasticsearch pass-through</a> that basically re-exposes the Elasticsearch HTTP on top of <code>nuxeo-elasticsearch</code>, the main goal being to integrate the security filtering. We use this feature to provide configurable dashboards for workflows and searches: 
<img src="https://api.contentstack.io/v2/assets/575e4cf743e9adc538714c50/download?uid=blt2bd67e7be53f3cb5?uid=blt2bd67e7be53f3cb5" data-sys-asset-uid="blt3e3c7fa341f5704b" alt="Nuxeo12.png" width="634" height="446" style="width: 634px; height: 446px;">
<h1>About the Next Steps</h1>Because we are very happy with the features we gained from integrating Elasticsearch, we want to go further. The roadmap includes:
<ul>
	<li>upgrade to Elasticsearch 2.0!</li>
	<li>add more automatic de-normalization
	<ul><li>use Elasticsearch to search through relations for example</li></ul></li>
	
	<li>leverage Elasticsearch percolator feature
	<ul>
		<li>be able to notify users when a change in the repository impacts them</li>
	</ul></li>
	<li>integrate Shield
		<ul><li>provide index level security for people using Nuxeo in a multi-tenant environment</li></ul></li>

<hr>
<img src="https://api.contentstack.io/v2/assets/575e4cf89e7a83165490d2cf/download?uid=bltf5d70811388d4adc?uid=bltf5d70811388d4adc" data-sys-asset-uid="blt136e5407d844f7ba" alt="Tiry-2014.png" width="146" height="146" style="line-height: 1.6em; width: 146px; height: 146px; float: right; margin: 15x;">Thierry Delprat joined Nuxeo in 2005 as Chief Technology Officer. As CTO, he guides the architectural development of the Nuxeo Platform including the adoption of Java as the platform for innovation. 
Prior to joining Nuxeo, Thierry worked for over 7 years at Unilog, with progressively senior experience across different branches of the consulting company. He was also a technical architect at Cryo-Networks (infrastructure for online games), and has participated in start-up companies. 
Thierry graduated from the Ecole Centrale de Nantes.
	
</ul></body></html>

nuxeo-search-lucene.jpg

nuxeo-search-lucene-thumb.jpg

Nuxeo, Search and Lucene, Oh My!

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Blog Footer

Thierry Delprat

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content.

You'll also receive an email with related content.

You'll also receive an email with related content

Load More

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

Elasticsearch Platform

ELK Stack

Elastic Cloud

Observability

Security

Search

By industry

By solution

Customer spotlight

Developers

Connect

Learn

Help

See what's happening at Elastic

Author

Articles by Thierry Delprat

Nuxeo, Search and Lucene, Oh My!

Follow us

About us

Join us

Press

Partners

Trust & Security

Investor relations

EXCELLENCE AWARDS