<a href="https://www.bol.com/nl/index.html" target="_blank">Bol.com</a> started in 1999 and has grown from an online bookstore to an online superstore with a wide variety of products, including books, segways, shoes, saunas, swimming pools, and much more. Since 2010, Bol.com has also opted to become a platform for other sellers to (re)sell their products and/or sell the same products bol.com offers but with different conditions.Right now we have over 11 million products available, 6.2 million active customers, and about 230,000 active sellers a month.<h2>What's the problem?</h2>Like any e-commerce site, we want to help our customers find what they are looking for and part of that comes in the form of a search engine.Currently we use Endeca for site search and we have a 'flat' document model for products, which means that we join data relevant for search for each product in one flat document and for each product we select the offer (which contains information like price, availability, seller) from a set of offers we think is most relevant based on some rules. This means price and availability are part of the product document and we can only use one offer:<table><tbody><tr><td>productid </td><td>title </td><td>author </td><td>price </td><td>availability </td><td>review </td></tr><tr><td>1 </td><td>Under the naked sun </td><td>Isaac Asimov </td><td>10 </td><td>within 24h </td><td>4 stars</td></tr></tbody></table>Functionally this limits the customers. Let's say I have a product with an offer from bol.com for 20 Euros and an offer from Seller A for 18 Euros and our offer selection says bol.com has the best offer. The price-facet is based on the 'best buy' price so if a customer selects a price between 15-19 Euros the product will disappear because it does not apply to the filter. This is bad for our customers because there actually is a relevant offer, and it’s also bad for sellers because their offer isn't shown.We want the customer to be in more control. One of the major things we need to solve is that we need a scalable way to model all offers for each product.<h2>Other challenges</h2>We've been working with <a href="/products/elasticsearch" target="_blank">Elasticsearch</a> and the Elastic Stack for a while and recently also for an application for our professional sellers. Elasticsearch seems to be very scalable and flexible so we wanted to see if Elasticsearch could solve our modelling challenge and whether it comes close to the performance we get out of our current search engine. To give an indication, we had about 2,400 requests a second on our search engine during the holiday season, and expect a lot more this year.Another crucial challenge is the number of updates due to the amount of products, offers, and sellers — about 500K offer updates (price or availability changes) and 200K content updates a day (and mostly during office hours). Sometimes we have peaks of 1.5 million product updates and 20 million offer updates a day. On average this will increase in the coming years because we are growing, with more products, more sellers, more offers, and more updates and inserts.So choosing the model is not just about IF we can model product offers correctly, it's also thinking about how we can set up Elasticsearch in such a way that it can handle all those updates in near real-time while processing the number of requests. For this article we'll stick mostly to modelling and performance from a request perspective. <h2>Challenge recap</h2>So to recap our challenges:<ul><li>Hierarchical data model, multiple levels</li><li>High volume searches and data changes</li><li>Complex query requirements<ul><li>Both product and offer fields in query</li><li>Aggregations on both levels</li></ul></li></ul>I'll get back to 'complex query requirements' in a moment but in general we have come up with a principle which states 'although we value performance at index time, we value performance at query time more'. We used this principle in the decision process.<h2>Elasticsearch and data modelling</h2>Elasticsearch has a few modelling options for our case:<ol><li>A single document for each offer with product information inline</li><li>A single document for each product, where the best offer is added inline</li><li>A single document for each production with all offers added inline</li><li>A single document for each product with all offers added inline, but also marked as nested objects in the Elasticsearch mapping</li><li>A separate document for each product and separate documents for each offer, using the parent-child functionality of Elasticsearch</li></ol><img src="https://api.contentstack.io/v2/assets/576b1541d3526d584e7e42f1/download?uid=blt8cbdb0c7817d72f3" data-sys-asset-uid="blt8cbdb0c7817d72f3" alt="bolcom-data-modelling.jpeg"/>ApproachOption 3&nbsp;is the situation we have now so we skipped that one, option 2&nbsp;was also quickly discarded because you lose the relation between the data of an offer if you have more than one offer. That left us with three models (1, 4&nbsp;and 5). But how can you test which one is best? We first started by actually creating representative indices for all these options to test against.We used a Node.js script for creating random data:Product<ul><li>Title: two random nouns from noun list</li><li>Category: pick one out of 26 nouns</li><li>Half have no offer, half between 1-4</li></ul>Offer<ul><li>Random price between 1-20</li><li>Seller: pick one out of 10k unique sellers</li></ul>We also want to test with different index sizes so we flushed the test data out to disk in three flavors - each flavor keeping its own bulk size of 100K - 1M, 10M and 100M products. We ended up with nine indices to test with, for each model we had three flavors. Now that we've got the indices, let's talk about the queries.<h2>Complex query requirements</h2>For our test we came up with four use cases that we want to test, each one with increasing complexity but still a representative query that customers perform each day. Below is an overview of the first three use cases.Use case A is a regular query where we need to display the cheapest offer and aggregate of some offer attributes. For use case B we ‘selected’ the deliverycode with value ‘0’ which means that we need to filter use case A and select the correct cheapest offer and aggregate on the correct offers. Use case C is ordering the selection on cheapest offer first. Here we need to display the cheapest offer and again aggregate on the correct offers.Use Cases:<img src="https://api.contentstack.io/v2/assets/576b156b1f82392051463d82/download?uid=blt9ca1ba1192ae887d" data-sys-asset-uid="blt9ca1ba1192ae887d" alt="use-cases-1.png"/>Use case D is special case and can best be explained with the following example<img src="https://api.contentstack.io/v2/assets/576b156685453e6f4e3b51d2/download?uid=bltaf077ed6b9e7ea01" data-sys-asset-uid="bltaf077ed6b9e7ea01" alt="rollup.PNG"/>What you see here is the ‘rollup’ functionality, which means that we want to display closely related products (we call them product families, products are the same except on 1 or 2 attributes) as 1 result. Here you can see the Apple iPad, all black but with different GB sizes. This results in the following requirements for the fourth use case:<img src="https://api.contentstack.io/v2/assets/576b15701f82392051463d87/download?uid=blt732c3b464a0da261" data-sys-asset-uid="blt732c3b464a0da261" alt="use-cases-4.png" width="626" height="340" style="width: 626;height: 340;"/>The complexity here is doing the aggregation on the set of products within the family/rollup and selecting the best offer to display. Now we’ve defined our use cases, let’s start testing.<h2>Testing</h2>Again we used a Node.js script to query and good old Excel to generate some graphs. The Node.js script used an array of terms, they have an overlap with the test data but a percentage results in 0 results, just like on our website.You could also specify how many queries to execute (we used 1,000 and 10,000) and it wrote out the response times to file. During the first run the response times were terrible, what was wrong? Well, rookie mistake, we only provided Elasticsearch with 1 GB of HEAP_SPACE, so after we set that to 32 GB (yes I know, more than 30 is an overkill but for testing it was fine) thinks were looking much better.So after setting that straight the result of our tests can be seen in the graphs below, the left side is average time in ms for all requests during the run, on the bottom the 4 different use cases.<img src="https://api.contentstack.io/v2/assets/576b155485453e6f4e3b51c2/download?uid=blt275b9e552cda6017" data-sys-asset-uid="blt275b9e552cda6017" alt="result-graph-1.png"/><img src="https://api.contentstack.io/v2/assets/576b155985453e6f4e3b51c7/download?uid=blte4471a1ad98f02c4" data-sys-asset-uid="blte4471a1ad98f02c4" alt="result-graph-2.png"/><img src="https://api.contentstack.io/v2/assets/576b15611f82392051463d75/download?uid=blt764993e2874c9500" data-sys-asset-uid="blt764993e2874c9500" alt="result-graph-3.png"/>Some remarks about the graphs and outcome:<ul><li>Regarding the 100m run, we did several tests but forgot to run the 'doc' queries&nbsp;for use case C and D but we expect the same outcome</li><li>As you can also see for query C there are no parent-child results, that's because we couldn't get the query to work for that use case. It doesn't mean that it's not possible but not in the timeframe we had</li><li>Doc-type offers is slow because we needed to aggregate on productid and the counts needed to be on product level. This was especially clear with the 100 million document index</li><li>The 'clear' winner is the nested document model except for the use case D&nbsp;so we need to figure out a way of doing that with a faster query or in a completely different way</li></ul>The downside of nested documents is the update behaviour, if only one offer changed for a doc with five offers, the whole document needs to be reindexed. But remember our principle? It's not a big issue if the data is behind for, let's say, 15-20 minutes. Not to say it's not a challenge...<h2>Conclusions</h2>Based on the outcome of the test we draw the following conclusions:<ol><li>Parent/Child has limitations when combining cross-level queries with aggregations in one go</li><li>Doc (with offer being the main entity) not as fast as we’d expected but that's because we needed top_hits aggregation</li><li>Nested was the 'winner' although 'rollup' was still slow</li><li>Elasticsearch scales predictably</li></ol>Based on the outcome, we decided to move forward and we are now in the process of building part of the search and browse of our webshop on Elasticsearch.The modelling and testing was done in cooperation with Anne Veling (beyondtrees.com). <hr/>Maarten Roosendaal is an IT Architect at bol.com and his main focus is on scalable search and SEO solutions. He has a background in Java/J2EE and previously worked for Sogyo and Capgemini. When not spending time with family or working, you can find him either in the gym, watching NBA/NCAA basketball, or at a movie theater.

banner-bol-com-birds-paradise.jpg

bolcom-logo.jpg

Finding a Scalable Data Model for Search @ bol.com

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Maarten Roosendaal

Надсилаючи цю форму ви погоджуєтеся з <a href="/legal/terms-of-use">Умовами надання послуг Elastic</a>. Ваші особисті дані будуть опрацьовані відповідно з <a href="/legal/privacy-statement">Положеннями про конфіденційність Elastic</a>.

Newsletter GDPR Text - Ukrainian

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Intentionally Blank

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

提交即表示您確認已詳閱並同意我們的<a href="/legal/elastic-cloud-account-terms" target="_blank">服務條款</a>，且同意 Elastic 使用您在上方提供的個人資料<a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">與您聯絡</a>，為您提供相關產品和服務的資訊。如需詳細說明或隨時想停止收到產品和服務資訊，請參閱 <a href="/legal/privacy-statement/" target="_blank">Elastic 的隱私聲明</a>。

Newsletter GDPR Text - TW

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.By submitting this form, I agree to Elastic sharing my contact information with Google Cloud. I also agree to Elastic giving my information to the vendor for the purpose of shipping my food package.

GDPR Text + Google Cloud

Inviando la registrazione (1) esprimi il consenso ai <a href="/legal/terms-of-use">termini e condizioni di Elastic</a> e <a href="/legal/privacy-statement">all’informativa sulla privacy</a>, nonché (2) a ricevere saltuarie e-mail.

Newsletter GDPR Text - Italian

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Load more

Overview

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content.

You'll also receive an email with related content.

You'll also receive an email with related content

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

Author

Articles by Maarten Roosendaal

Finding a Scalable Data Model for Search @ bol.com