In <a href="/blog/how-kenna-security-speeds-up-elasticsearch-indexing-at-scale-part-1">part one</a> of this blog series I laid out all the techniques my company, <a href="https://www.kennasecurity.com/">Kenna Security</a>, used to speed up indexing while scaling its cluster. In part two, I want to share some of the techniques we used to speed up search while increasing our document count to over four billion documents.
<h2>Group Your Data</h2>
To start, I want to talk about data organization. Many people use <a href="/elasticsearch">Elasticsearch</a> for storing logs. Most logging clusters are set up so indexes are based on dates. One of the biggest reasons logging clusters are set up that way is because that it is a naturally easy way to filter down data when you are searching. If you want to search two days worth of data, Elasticsearch only has to query two indexes.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt408f38fa7012b51c/5c57d4f3713ebdec0ba06f33/Kenna_Search.png" data-sys-asset-uid="blt408f38fa7012b51c" style="display: block;margin: auto;" />
This means less shards to search, which leads to faster searches. This concept of grouping data to help speed up search can also be applied to a non-logging cluster.
When Kenna first started using Elasticsearch, all our data was in a single, small index. As the amount of data increased, we had to increase the number of shards in that index. As our shard count grew, our search speed slowed. To speed things up, we decided to split up our data by client. Each client now lives on its own index.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt1d1ed4484c761435/5c57d4f18d25f6030ca042ed/Kenna_Client_Index.png" data-sys-asset-uid="blt1d1ed4484c761435" style="display: block;margin: auto;" />
This made the most sense for us because when we look up data, 99% of the time it is by client. Now, when we run a search for a client ,we only have to look at a smaller subset of shards instead of them all. This has allowed us to maintain fast search speeds while adding more and more data to the cluster.
<h2>Filters are Friends</h2>
In addition to data organization, you can also speed up searches by optimizing their structure. Elasticsearch has two types of search modes: queries and filters.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltab172bfbde2f9bde/5c57d4e7893c9ee20b66f4f2/Kenna_Queries_FIlters.png" data-sys-asset-uid="bltab172bfbde2f9bde" style="display: block;margin: auto;" />
Queries, which have to score documents, are a lot more work for Elasticsearch. Filters, which don’t score documents, are less work and therefore faster. At Kenna, we exclusively use filters because we know their benefits. However, we didn’t fully appreciate those benefits until March, 2016.
In March, 2016, we upgraded to Elasticsearch 5.0. During the upgrade we ran into an <a href="https://github.com/elastic/elasticsearch/pull/23797">Elasticsearch bug</a> that caused some of our filters to be scored. Just how much more work was this for our cluster? Scoring a couple of heavily used filters caused so much extra work that our production cluster was rendered unusable for an entire week until the bug was patched. This experience taught us how important filters truly are and that you should use them whenever possible.
<h2>Filters</h2>
If any part of your search doesn’t need to be scored, move it to a filter block. If you do need parts of your search scored, consider using a `bool` query. This allows you to combine scoring blocks, like `must` and non scoring blocks, like `filter`.<pre class="prettyprint">GET /_search { "query": { "bool" : { "must" : { "term" : { "user" : "kimchy" } }, "filter": { "term" : { "tag" : "tech" } } } } } </pre>
With our data organized with filters, we were in a pretty good place with our Elasticsearch cluster by the end of 2017. This was when we finally started making some optimizations we had originally backlogged. One of those optimizations was to store IDs as keywords.
<h2>Storing IDs as Keywords</h2>
This suggestion is one I heard over and over again at <a href="/resources">Elasticsearch training</a>. (If you have never been to Elasticsearch training, I highly recommend it! I have been to training three times now and each time, I have come away with actionable items that we were able to use to improve our cluster.) Basically, whenever you are storing IDs that are never going to be used for range searches, you want to store them as keywords. The reason for this is that keywords are optimized for terms searches. Integers, or numeric mapping types, are optimized for range searches. Upon learning this, I didn’t think it would have much of an impact for us at Kenna, so I made a ticket and forgot about it.
It was over a year until we finally pulled that ticket out of the backlog and acted on it. The results we got were more than we bargained for. When we finally made the switch from integers to keywords we saw a 30% increase in search speed across the board. We immediately wished we had made the change sooner —but better late than never, right?
The last optimization I want to share is yet another from Elasticsearch training, but one also backlogged because we didn’t think it was that important. Yet again, we were wrong, and had to learn this lesson the hard way.
<h2>Don’t Let Your Users Slow You Down</h2>
One day we were monitoring Elasticsearch, when out of nowhere, all the nodes maxed out on CPU and load.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt830e542e89137990/5c57d4e102ac90e80ba0a878/Kenna_Production.png" data-sys-asset-uid="blt830e542e89137990" style="display: block;margin: auto;" />
My team scrambled, trying to figure out what was causing the load. We started sifting through the slow logs trying to figure out what queries were running, and came across this gem. 
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt56dc1a497d59cfb5/5c57d4e0c0eac4f70b57acdd/Kenna_Or_Statements.png" data-sys-asset-uid="blt56dc1a497d59cfb5" style="display: block;margin: auto;" />
A ton of “OR” statements paired with a ton of leading wildcards all in one query string from hell. This brings me to my last, and probably one of the most important pieces of advice I have: don't let your users slow you down!
It is crazy easy to slap a search box on your site and then send whatever is put into it to Elasticsearch. DON’T DO IT! Limit what users can and can’t search. We solved this problem at Kenna by defining keywords for users which they can use to search. We also spent time writing up more documentation to educate users about the fields available for searching.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc948eecca0a2fdf3/5c57d4d365ace9e30b316279/Kenna_Asset_Search_Examples.png" data-sys-asset-uid="bltc948eecca0a2fdf3" />
With these changes users’ searches are now more targeted, accurate, and much easier for Elasticsearch to handle. At the end of the day, everyone wins!
<h2>Recap: Speed up Searching at Scale</h2>
<ol>
 <li>Group your data</li>
 <li>Use filters whenever possible</li>
 <li>Store IDs as keywords</li>
 <li>Don’t let your users slow you down</li>
</ol>
<h2>Planning Ahead</h2>
When we started out using Elasticsearch to handle all our client’s search needs it seemed like we could do no wrong. Once our data size started growing though, we quickly realized we were going to have to be smarter about how we were using Elasticsearch. Try to apply these search techniques when your cluster is small and it will make scaling a whole lot easier. Thanks to all these indexing and searching optimizations, Kenna’s Elasticsearch cluster is now one of the most stable pieces of its infrastructure, and we plan to keep it that way for a long time.
<hr />
Molly Struve (<a href="https://twitter.com/molly_struve">@molly_struve</a>) is a Sr. Site Reliability Engineer at <a href="https://www.kennasecurity.com/">Kenna Security</a>. She has been working at Kenna and with Elasticsearch for over three years. During that time, she helped lead the team charged with scaling Kenna’s Elasticsearch cluster which now holds four billion documents and updates over 200 million of them a day. When she isn’t wrangling Elasticsearch, she can be found fulfilling her need for speed by riding and jumping her show horses.

Kenna_Production.png

How Kenna Security Speeds Up Search at Scale using Elasticsearch - Part 2

Kenna Security utilizes Elasticsearch to allow clients to customize how they organize their data, making search speed a top priority.

Everyone wants their <a href="/elasticsearch">Elasticsearch</a> cluster to index and search faster, but optimizing both at scale can take planning. In 2015, Kenna's cluster held 500 million documents, a million of which were processed every day. At the time, our poorly configured Elasticsearch cluster was the least stable piece of our infrastructure and could barely keep up as our data size grew. Today, our cluster holds 4 billion documents and we process over 200 million of them a day, with ease. Building a cluster to meet all of our indexing and searching demands was not easy. But with a lot of persistence and a few "OH CRAP!" moments, we got it done and learned a hell of a lot along the way. In this two part blog, I will share the techniques we used while building our Elasticsearch cluster at Kenna to get us to where we are today. However, before I dive into all the Elasticsearch fun, I first want to tell you a little bit about Kenna.
<h3>What is Kenna Security?</h3>
	<a href="https://www.kennasecurity.com/">Kenna</a> helps Fortune 500 companies manage their cybersecurity risk. The average company has 60 thousand assets. An asset is basically anything with an IP address. The average company also has 24 million vulnerabilities. A vulnerability is how you can hack an asset. With all this data it can be extremely difficult for companies to know what they need to focus on and fix first. That's where Kenna comes in. At Kenna, we take all of that data and we run it through our proprietary algorithms. Those algorithms then tell our clients which vulnerabilities pose the biggest risk to their infrastructure so they know what they need to fix first.
We initially store all of this data in MySQL, which is our source of truth. From there, we index the asset and vulnerability data into Elasticsearch:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt5430cc38d24a8623/5c57d6c5bc5d07ed0b48c8e6/MySQL_to_Elasticsearch.png" data-sys-asset-uid="blt5430cc38d24a8623" alt="MySQL to Elasticsearch.png" width="400" style="display: block; margin: auto;">
Elasticsearch is what allows our clients to really organize their data in any way they need, making search speed a top priority. At the same time, data is constantly changing in MySQL, so indexing is also extremely important in order to keep Elasticsearch insync with it.
<h2>Part 1: Speeding up Indexing at Scale</h2>Let's start with how we were able to scale our indexing capacity. For our use case, we set a great refresh interval from the beginning, one of the best things we did early on in running Elasticsearch.
<h3>Refresh Interval</h3>By default, Elasticsearch uses a <a href="/guide/en/elasticsearch/reference/current/tune-for-indexing-speed.html#_increase_the_refresh_interval">one-second refresh interval</a>. Refreshing an index takes up considerable resources, which takes away from other resources you could use for indexing. One of the easiest ways to speed up indexing is to increase your refresh interval. By increasing it you will decrease the number of refreshes an index does and thus free up resources for indexing. We have ours set at 30 seconds.
<pre class="prettyprint">PUT /my_index/_settings 
{
 "index" : {
 "refresh_interval": "30s"
 }
}
</pre>Client data updates are handled by background jobs, so waiting an extra 30 seconds for data to be searchable is not a big deal. If 30 seconds is too long for your use case, try 10 or 20. Anything greater than one will get you indexing gains.
Once we had our refresh interval set it was smooth sailing — until we got our first big client. This client had 200,000 assets and over 100 million vulnerabilities. After getting their data into MySQL, we started to index it into Elasticsearch but it was slow going. After a couple days of indexing we had to tell this client that it was going to take two weeks to index all of their data. Obviously, this wasn't going to work long term. The solution was bulk processing.
<h3>Bulk Processing</h3>We found that indexing 1000 documents in each bulk request gave us the biggest performance boost. The size you want to make your batches will depend on your document size. For tips on finding your optimal batch size, check out <a href="/guide/en/elasticsearch/reference/6.6/tune-for-indexing-speed.html#_use_bulk_requests">Elastic's suggestions for bulk processing.</a>
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltae90001115ad8e3d/5c99ef34228f2c4259417ec6/single-vs-bulk.png" data-sys-asset-uid="bltae90001115ad8e3d" alt="" width="600" style="display: block; margin: auto;">
<figcaption>Indexing documents individually vs. Indexing in a single batch using a bulk request</figcaption>Bulk processing alone got us through a solid year of growth. Then, this past year, MySQL got an overhaul and suddenly Elasticsearch couldn't keep up. During peak indexing periods, node CPU would max out and we started getting a bunch of 429 (TOO_MANY_REQUESTS) errors. This is when we became aware of threads and the role they play in indexing.<h3>Route Your Documents</h3>When you are indexing a set of documents the number of threads needed to complete the request depends on how many shards on which those documents belong. Let's look at two batches of documents:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltf0a14e7d7663bf69/5c98d78398d5092a05465ef7/8-threads.png" data-sys-asset-uid="bltf0a14e7d7663bf69" width="400" alt="" style="display: block; margin: auto;">
It's going to take four threads to index each batch because each one needs to talk to four shards. An easy way to decrease the number of threads you need for each request is to group your documents by shard. Going back to our example, if you group the documents by shard you can cut the number of threads needed to execute the request in half.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt6a944c89ca3a153c/5c57d69c8c741b010cbf6ac1/4_threads_to_ES_alt_img.png" data-sys-asset-uid="blt6a944c89ca3a153c" width="400" display:="" block;="" margin:="" auto;"="" style="display: block; margin: auto;" alt="">
Seems easy, right? Except, how do you know what shard a document belongs on? The answer is <a href="/guide/en/elasticsearch/reference/current/mapping-routing-field.html">routing</a>. When Elasticsearch is determining what shard to put a document on it uses this formula:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blte3258656b5bb7239/5c57d460713ebdec0ba06f2d/code.jpg" data-sys-asset-uid="blte3258656b5bb7239" alt="" width="600" style="display: block; margin: auto;">
The routing value will default to the document _id or you can set it yourself. If you set it yourself, then you can use it when you are indexing documents. Let's look at that example again. Since our route value corresponds with the shard a document belongs on, we can use that to group our documents and reduce the number of threads needed to execute each request.
At Kenna, we have a parent/child relationship between assets and vulnerabilities. This means the parent, or asset_id, is used for routing. When we are indexing vulnerabilities, we group them by asset_id to reduce the number of threads needed to fulfill each request.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt1b0dbb9f30407276/5c57d45ba4f817ef0b0e809d/vulnerabilities-by-asset.jpg" data-sys-asset-uid="blt1b0dbb9f30407276" width="600" alt="Vulnerabilities by Asset" style="display: block; margin: auto;">
Grouping documents by their routes has allowed us to ramp up our indexing considerably while keeping the cluster happy.
<h3>Recap: Speeding up Indexing at Scale</h3><ol>
	<li>Toggle your refresh interval</li>
	<li>Bulk process documents</li>
	<li>Route your documents</li>
</ol><h3>Planning Ahead</h3>These techniques have gotten Kenna to the point where we can process over 200 million documents a day and still have room to grow. When your cluster is small, and you are not processing a lot of data, these small adjustments are easy to overlook. However, as you scale, I guarantee these techniques will be invaluable. Plan ahead and start implementing these indexing strategies now so you can avoid slow downs in the future.
Indexing speed is only half of the picture. At Kenna, search is also a top priority for our clients. Check out part two of this blog, where I will dive into all the ways Kenna was able to speed up its search while scaling its cluster.
<hr>Molly Struve (<a href="https://twitter.com/molly_struve">@molly_struve</a>) is a Sr. Site Reliability Engineer at <a href="https://www.kennasecurity.com/">Kenna Security</a>. She has been working at Kenna and with Elasticsearch for over 3 years. During that time, she helped lead the team charged with scaling Kenna's Elasticsearch cluster which now holds 4 billion documents and updates over 200 million of them a day. When she isn't wrangling Elasticsearch, she can be found fulfilling her need for speed by riding and jumping her show horses.

abstract-ai-art-373543 (1).jpg

abstract-ai-art-373543.jpg

How Kenna Security Speeds Up Elasticsearch Indexing at Scale  - Part 1

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Molly Struve

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

提交即表示您确认您已阅读并同意我们的<a href="/cn/legal/elastic-cloud-account-terms" target="_self">服务条款</a>，并且您同意通过您提供的上述联系方式接收<a href="/cn/legal/privacy-statement#how-we-use-the-information" target="_self">所选的通信</a>。如需了解更多信息，或想随时取消订阅，请参阅 <a href="/cn/legal/privacy-statement/" target="_self">Elastic 的隐私声明</a>。

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

提交即表示您确认您已阅读并同意我们的<a href="/cn/legal/serverless-preview-terms" target="_self">服务条款</a>，并且 Elastic 可以使用您提供的上述详细信息与您联系，以向您介绍我们相关的<a href="/cn/legal/privacy-statement#how-we-use-the-information" target="_self">产品和服务</a>。如需了解更多信息，或想随时取消订阅，请参阅 <a href="/cn/legal/privacy-statement/" target="_self">Elastic 的隐私声明</a>。

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

提交即表示您同意 <a href="/cn/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud 服务条款</a>。我们将会按照 <a href="/cn/legal/privacy-statement">Elastic 的隐私声明</a>处理您的个人数据。

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

提交即表示您确认您已阅读并同意我们的<a href="/cn/legal/terms-of-use" target="_self">服务条款</a>，并且 Elastic 可以使用您提供的上述详细信息与您联系，以向您介绍我们相关的<a href="/cn/legal/privacy-statement#how-we-use-the-information" target="_self">产品和服务</a>。如需了解更多信息，或想随时取消订阅，请参阅 <a href="/cn/legal/privacy-statement/" target="_self">Elastic 的隐私声明</a>。

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

本页内容尚不支持所选语言。Elastic 正在不断努力，以实现对多种语言内容的支持。感谢您在此期间给予的耐心与陪伴！

Author

Articles by

Learn more

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

查看本次网络研讨会在我的时区的开始时间

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

无法实时参加？注册后我们会给您发送录像。您还将收到相关内容的电子邮件。

Author

Articles by Molly Struve

How Kenna Security Speeds Up Search at Scale using Elasticsearch - Part 2

How Kenna Security Speeds Up Elasticsearch Indexing at Scale - Part 1

Sign up for Elastic Cloud free trial