<p>Outbrain is the world's leading content discovery platform, bringing personalized, relevant online, mobile and video content to audiences. Outbrain helps publishers engage, monetize and understand their audiences through data, and helps marketers to reach their most relevant audiences.</p><p>Outbrain serves over 200 billion personalized content recommendations every month and reaches over 500 million unique monthly visitors from across the globe.</p><figure><div data-img-caption-container="true" style="display: inline-block; text-align: center;"><img src="https://api.contentstack.io/v2/uploads/59271cb9606bd5216cf7c118/download?uid=blt3460c0a2b65ad745" data-sys-asset-uid="blt3460c0a2b65ad745" alt="img1 outbrain.png" /><figcaption  style="text-align: center;" style="text-align: center;">Outbrain widget in a CNN article</figcaption></div></figure><p>In this article we would like to share how we reduced the problem of recommending content to a user, to a search problem for users' implied interests, and how this enabled us to solve some of our major scale bottlenecks, and to increase the complexity of our online prediction models.</p><h3>Jungle of Market Rules, Personalization, and Scale</h3><p>Our content discovery engine finds content that is most likely to be relevant to each user's interests, as manifested in their browsing patterns. In addition to being personalized, Outbrain's recommendations must comply with a long list of market rules and restrictions. For articles that pass all market rules, a ranking function is applied to select the most relevant content for a given user.</p><p>One simple example of a market rule is geographic targeting. This rule enables a marketer to target an audience from a specific location, namely to ensure its content is recommended to users from that location only.</p><p>Outbrain publishers and marketers have hundreds of possible market rule configurations, and these must be enforced in real time at a 35K/sec throughput of requests on millions of potential content recommendations and at a latency of under 100 milliseconds per request.</p><p>While increasing throughput is horizontally scalable and can be achieved by adding more serving machines, inventory size is much more difficult to scale, especially with a growing set of market rules. After several architecture design attempts, we realized that Elasticsearch, being a distributed scalable search platform, can provide us with the solution.</p><p>Elasticsearch has three key properties for our serving-stack needs. First, it is able to score documents by their relevance to a query. Second, it is able to filter documents by certain attributes. Finally, it does this efficiently and at scale both in terms of throughput and inventory size.</p><p>Therefore, by translating the problem of recommending content to a user into a search problem for users' implied interests, we can base our recommender system on Elasticsearch and handle the aforementioned scale challenges.</p><p>Let's now describe how this translation is being done.</p><h3>Every Potential Recommendation is an Elasticsearch Document</h3><p>Every article in the inventory of our potential content recommendations is indexed as a separate Elasticsearch document. We index its title, other meta-data, and most importantly, a set of semantic features that will be used to determine its relevance to each potential user.</p><p>Those features are generated offline using an NLP-based semantic engine, and include high-level categories, more specific topics, explicit entities appearing in the article, and so forth.</p><figure><div data-img-caption-container="true" style="display: inline-block; text-align: center;"><img src="https://api.contentstack.io/v2/uploads/5936d8e1d3cc76eb3be152d0/download?uid=bltc282264d04d544e6" data-sys-asset-uid="bltc282264d04d544e6" alt="image1.jpg" /><figcaption  style="text-align: center;" style="text-align: center;">A "Westworld season one explosive finale" article and its semantic features</figcaption></div></figure><figure>&nbsp;<img src="https://api.contentstack.io/v2/uploads/59271d0788b3d11379392987/download?uid=bltb04f046f20a6ae0b" data-sys-asset-uid="bltb04f046f20a6ae0b" alt="img3 outbrain.png" width="600" /><figcaption  style="text-align: center;" style="text-align: center;">Representing the article as an Elasticsearch document. Every feature is defined in a separate field</figcaption></figure><h3>Recommendations and Search: Your user IS the query</h3><p>The next step after indexing our inventory of potential content recommendations, was to translate users' implied interests to an Elasticsearch query.</p><p>Users at Outbrain have 'profiles' that represent their interests. These are generated based on the semantics of the content each user reads.</p><figure><div data-img-caption-container="true" style="display: inline-block; text-align: center;"><img src="https://api.contentstack.io/v2/uploads/5936d92b5f89c17842a1d539/download?uid=blt3c3d1f7505523681" data-sys-asset-uid="blt3c3d1f7505523681" alt="image2.jpg" /><figcaption  style="text-align: center;" style="text-align: center;">A profile of a user having five main interests, confidence scores indicate affinity strength</figcaption></div></figure><p>We can then incorporate user interests into the Elasticsearch query such that all relevant content recommendations are retrieved. Specifically, we generated a 'should' query, so that any document matching at least one clause of the query is eligible to be returned to the user. Documents with more than one match will be scored higher. The 'boost' parameter can be used to specify that not all matches contribute equally to the final score.</p><figure><center><img src="https://api.contentstack.io/v2/uploads/59271d205e123fe06b32f171/download?uid=blt6cd6d7e405ed7f92" data-sys-asset-uid="blt6cd6d7e405ed7f92" alt="img5 outbrain.png" width="600"/></center><figcaption><em>This user reads largely about television, investments, and specifically the entity 'dolores'</em></figcaption></figure><h3>Translating Market Rules to Elasticsearch Filters</h3><p>Translating market rules to Elasticsearch filters seems rather natural, and it is. Every rule needs to be represented both on the document side and the query side.</p><p>One simple example is geographic targeting of users. Imagine a New York music scene site that wishes to increase its traffic and opens an Outbrain campaign. On the business side, as it sells tickets to New York shows only, it is not interested in receiving traffic from users outside the US. Implementing this market rule has two main stages. First, a new 'geo' field is indexed for the relevant Elasticsearch documents. Secondly, a new 'terms' filter is added to the user query. This filter is defined on the 'geo' field and its value depends on the origin country of the user.</p><figure><center><img src="https://api.contentstack.io/v2/uploads/59271d2916939f147dd9ff89/download?uid=blt9a0e99a6b812449b" data-sys-asset-uid="blt9a0e99a6b812449b" alt="img6 outbrain.png" width="600"/></center><figcaption><em>The new 'geo' field in the index</em> <center><img src="https://api.contentstack.io/v2/uploads/59271d4b26a967c762dc3b20/download?uid=blt43519c1670badff3" data-sys-asset-uid="blt43519c1670badff3" alt="img7 outbrain.png" width="600"/></center></figcaption></figure><img/><p>Interestingly, even this rather simple market rule has a twist. What happens if a marketer is indifferent to user geographic location? In the naive approach, all the articles (Elasticsearch documents) of this marketer would have had an empty value in their 'geo' field. However, in that setting, this document will actually be filtered out as there will be no match between the query (user) 'geo' term filter and the values of the 'geo' field in the document. As a result, this marketer will essentially reach no users at all.</p><p>To solve this, we added a wildcard term 'all' both to the index and to the query. This way, a user will receive content recommendations from campaigns that explicitly target the user's location, as well as content that has no geographic targeting specifications.</p><center><img src="https://api.contentstack.io/v2/uploads/59271d5d6b26a41763042e1e/download?uid=bltbc21bda1fb40a5e7" data-sys-asset-uid="bltbc21bda1fb40a5e7" alt="img8 outbrain.png" width="600"/></center><h3>Incorporating Machine Learned Models</h3><p>To incorporate more complicated machine learned prediction models into our recommender system, Elasticsearch provides us with its flexible plugin capabilities. To efficiently implement a custom scoring function on our index, we wrap native Java code into a stand-alone jar and deploy it on our Elasticsearch cluster as a registered plugin.</p><p>The script_score query clause instructs Elasticsearch to use the custom plugin for scoring potential content recommendations.</p><p>We therefore gain high flexibility in terms of our scoring mechanism, while not compromising on efficiency and scalability.</p><h3>Optimizing the Elasticsearch System for Throughput and Latency</h3><p>To enable Outbrain's recommendation system to serve millions of requests per minute at under 100 millisecond latency and reasonable HW cost required substantial work. This section shares the more interesting parts of our optimization effort.</p><h3>System Overview</h3><p>The recommendation system is made up of two independent services which interact with the Elasticsearch backend. An Indexing Service (writing the inventory) and a Recommendations Service (querying for the best candidates)</p><p><img src="https://api.contentstack.io/v2/uploads/592df49db9de65fe44bf890d/download?uid=blte5d41d15b6569626" data-sys-asset-uid="blte5d41d15b6569626" alt="Outbrain Guest Blog.png" style="background-color: initial;"/>In order to achieve optimal query latency and throughput, the system was designed to separate indexing from search and hence enable the search phase to interact with an optimised, read-only index. The indexing process takes place once every half hour, after which the index is made read-only and can be queried.</p><h3>Hardware and Size</h3><p>The machines allocated as data nodes are as follows:</p><ul><li>2 x Processors of 12 hyperthreaded cores(Intel E5-2630 v2)</li><li>128 GB RAM.</li><li>2 x 500G SATA drives for OS(mdadm raid1)</li><li>2 x 1Gbe NIC in bond</li></ul><p>The Elasticsearch system was given 26 physical servers for data nodes and 6 servers for master nodes to be divided into two clusters (indexing/searching).</p><h3>Our Custom Elasticsearch Plugin</h3><p>Along with the indexed document data (which is updated every half-hour), our system requires additional, frequently updated (every 2 minutes) document-related data which is used for the purpose of scoring the candidates for recommendation.</p><p>In order to keep our read-only index unaffected during the search phase, we implemented an Elasticsearch plugin which supports updates of additional document-related data. Keeping in mind our latency/throughput goal, we chose the off-heap Chronicle Map technology as the backing storage in order to avoid possible GC costs.</p><p>Our scoring script uses that off-heap data when evaluating the rank of queried documents.</p><h3>Optimizing Our System</h3><p>Our optimization work can be divided into two main effort areas: (1) application level effort, optimizing our index and queries; and (2) system level effort, optimizing the Elasticsearch setup and structure.</p><h3>Optimizing Index and Queries</h3><p>Taking advantage of the fact that our index is read-only at search phase, all queries are made to use cachable filters only. This approach proved to be very effective in achieving improved latency.</p><p>Working with slow-log queries, we identified the most time consuming query filters and moved all possible filter term parameters to be calculated at indexing time, making them part of our document fields.</p><p>To make the index as small as possible, we optimized our field definitions using an elaborate field mappings template <a href="https://drive.google.com/file/d/0B_VHhLbrgvgnblFnQmYzbkRQT0k/view">(link)</a> allowing for stored-only/indexed-only fields, disabling norms (we do our own scoring) etc.</p><p>To gain latency and throughput by having our search requests run on a single segment, we perform a force merge operation before making the index searchable. Note that merging an index into a single segment has to be specified explicitly (we advise to read the documentation here).</p><p>Working with a single shard replicated across all nodes (having all our data on every node of the cluster) enabled our queries to run with the "_local" query preference. In this way, the search operation can be completed locally, saving the possible extra network hops to neighboring nodes to complete the search request.</p><h3>Optimising Elasticsearch Nodes and Clusters</h3><p>Having a small index (~2GB) made it possible for us to define the data storage on tmpfs, gaining all the advantages of RAM vs. Disk.</p><p>Our initial system design was a two-cluster Hot-Cold setup. Indexing all nodes in cluster A while searching on cluster B and flipping once indexing is done.<img src="https://api.contentstack.io/v2/uploads/592df2a5d618228557490154/download?uid=blt4aee99bab17b4eb2" data-sys-asset-uid="blt4aee99bab17b4eb2" alt="Outbrain Guest Blog (1).png"/></p><p>With increased traffic to the recommendation system, we found this approach to be wasteful, since actually only half of the HW capacity was utilized at any given time. We then switched to a different cluster setup with small indexing and large searching clusters.</p><p><img src="https://api.contentstack.io/v2/uploads/592df2f97f64a9805742cc94/download?uid=blt5dd6aa9b5e6513a5" data-sys-asset-uid="blt5dd6aa9b5e6513a5" alt="Outbrain Guest Blog (2).png"/></p><p>For speed, the indexing is performed on a single, non-replicated shard. When indexing is finished a <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html">forceMerge</a> operation is invoked merging all segments into a single one and the newly created index is backed-up to a shared network drive. The backed-up index is then restored on the search cluster and is replicated to all its data nodes and is made searchable.</p><p>This design proved to be very effective and practically doubled our system's capacity. Our concern that the restoring process will interfere with the ongoing search proved to be unjustified.</p><h3>Search Threads and Queue</h3><p>Starting with the default number of search threads and queue size proved right away inappropriate for our needs and had to be adjusted. Very quickly we experienced a cluster with saturated queues, terrible latency, and total breakdown. Thread limits and queue size dynamic configuration proved to be very useful while looking for the best numbers to fit our needs. From our tests and for our needs (with a search-only cluster and no indexing) we found that the number of search threads is close to the number of available virtual cores on the data nodes.</p><h3>Two Data Node Instances per Single Physical Server</h3><p>Following a few optimization phases, we were ready to stress-load our system with traffic to test its throughput limits. To our surprise and puzzlement, no matter how much traffic we loaded, the data nodes did not go much beyond 50% CPU utilization, although the number of search threads was defined as the number of virtual cores (24). Searching for clues, we disabled one of the CPU sockets. As we anticipated, the throughput remained pretty much the same.</p><p>Assuming that a single Elasticsearch instance could not gain the full capacity of a two socket processor setup, we decided to install a second Elasticsearch instance on each physical machine. Each node instance limited to its own cpu socket and memory, which was achieved by cpuset(cgroups) command, and by the tmpfs mount option(mpol=local). This change in effect doubled our throughput and made it possible to utilize 80-90% of the servers' processing power.</p><h3>Summary</h3><p>Starting with the approach that things would work for us out-of-the-box, we very quickly realized that customization is required to adjust the Elasticsearch system to fit our specific recommendation serving use-case. Following trial and error, we are now able to serve 800,000 req/min at under 100MS on a cluster of 48 data nodes.</p><p>This capacity to quickly validate business rules and subsequently match user interests to available content enables us to utilize more complex personalization models, improving the quality and relevance of Outbrain's recommendations. This improves the very core of our business.</p><p>We truly hope that this report and the details above will prove to be interesting and useful.</p><p>We would also like to thank Andrei Stefan and Honza Kral from Elastic for their help in building our system.</p><p><a href="mailto:sliberman@outbrain.com"><em>Sonya Liberman</em></a><em> - Personalisation Team Lead (Recommendation Group Outbrain)</em></p><p dir="ltr"><a href="mailto:bbrutman@outbrain.com"><em>Baruch Brutman</em></a><em> - Infra Team Lead (Recommendation Group Outbrain)</em></p><p dir="ltr"><a href="mailto:shaharf@outbrain.com"><em>Shahar Fleischman</em></a><em> - Production Engineer (Recommendation Group Outbrain)</em></p>

blog-content-lens-fb.jpg

blog-content-lens-thumb.jpg

Looking at Content Recommendation Through a Search Lens

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Shahar Fleischman

<p>Надсилаючи цю форму ви погоджуєтеся з <a href="/legal/terms-of-use">Умовами надання послуг Elastic</a>. Ваші особисті дані будуть опрацьовані відповідно з <a href="/legal/privacy-statement">Положеннями про конфіденційність Elastic</a>.</p>

Newsletter GDPR Text - Ukrainian

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Communication Preferences - ONLY for /communication-preferences

<p>By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.</p>

Cloud Trial GDPR Text

Intentionally Blank

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Serverless GDPR Text

<p>提交即表示您確認已詳閱並同意我們的<a href="/legal/elastic-cloud-account-terms" target="_blank">服務條款</a>，且同意 Elastic 使用您在上方提供的個人資料<a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">與您聯絡</a>，為您提供相關產品和服務的資訊。如需詳細說明或隨時想停止收到產品和服務資訊，請參閱 <a href="/legal/privacy-statement/" target="_blank">Elastic 的隱私聲明</a>。</p>

Newsletter GDPR Text - TW

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p><p>By submitting this form, I agree to Elastic sharing my contact information with Google Cloud. I also agree to Elastic giving my information to the vendor for the purpose of shipping my food package.</p>

GDPR Text + Google Cloud

<p>Inviando la registrazione (1) esprimi il consenso ai <a href="/legal/terms-of-use">termini e condizioni di Elastic</a> e <a href="/legal/privacy-statement">all’informativa sulla privacy</a>, nonché (2) a ricevere saltuarie e-mail.</p>

Newsletter GDPR Text - Italian

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Newsletter GDPR Text

Load more

Overview

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content.

You'll also receive an email with related content.

You'll also receive an email with related content

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

Author

Articles by Shahar Fleischman

Looking at Content Recommendation Through a Search Lens