Elasticsearch can help you analyze images for facial recognition. Learn more about how to detect and encode facial information — and find matches in search — using Python.

Have you ever tried to search for objects in images? Elasticsearch can help you store, analyze, and search for objects in images or videos.
In this quick tutorial, we’ll show you how to build a system for facial recognition with Python. Learn more about how to detect and encode facial information — and find matches in search. 
 <h2>A quick walkthrough of the basics</h2>Need a refresher? Let's briefly review some fundamentals about a few basic concepts.
 <h3>Facial recognition</h3>Facial recognition is the process of identifying a person using their face — for example, for the purpose of implementing an authentication mechanism (like unlocking your smartphone). It captures, analyzes, and compares patterns based on the person's facial details. This process can be divided into three phases:&nbsp;
<ul>
	<li aria-level="1">Face detection: Identify human faces in digital images</li>
	<li aria-level="1">Face data encoding: Convert facial features into digital representation</li>
	<li aria-level="1">Face match: Search and compare facial features</li>
</ul>We’ll walk you through each phase in our example.
 <h3>128-dimension vector</h3>Facial features can be transformed into a set of digital information in order to be stored and analysed.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt4f879e5217b5bb57/606cabbc2c290626b4382280/facial-recognition-blog-vector.png" data-sys-asset-uid="blt4f879e5217b5bb57" alt="facial-recognition-blog-vector.png" "="" width="317" height="469" style="width: 317px; height: 469px; display: block; margin: auto;">
<h3>Vector data type</h3>Elasticsearch offers the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/dense-vector.html"><code>dense_vector</code></a> data type to store dense vectors of float values. The maximum number of dimensions in a vector shouldn’t exceed 2048, which is enough to store the face features representation.
Now, let's implement all these concepts.
 <h2>Everything you need</h2>To detect faces and encode the information, you’ll need the following:
<ul>
	<li aria-level="1"><a href="https://www.python.org/downloads/">Python</a>: We’ll be using Python 3 in this example, which <a href="https://www.python.org/downloads/">you can find here</a>. Installation and download steps will depend on your operating system.</li>
	<li aria-level="1">An Elasticsearch cluster: You can spin up a cluster in a <a href="https://cloud.elastic.co/registration">free trial of Elastic Cloud</a>.&nbsp;</li>
	<li aria-level="1"><a href="https://pypi.org/project/face-recognition/">Face-recognition library</a>: A simple face recognition Python library.</li>
	<li aria-level="1"><a href="https://elasticsearch-py.readthedocs.io/en/v7.10.1/">Python Elasticsearch client</a>: The official Python client for Elasticsearch.</li>
</ul>Please note we’ve tested the following instructions on Ubuntu 20.04 LTS and Ubuntu 18.04 LTS. Some changes may need to be made depending on your operating system.
 <h2>Install Python and Python libraries</h2>Ubuntu 20 and other versions of Debian Linux are delivered with the installation of Python 3. If this isn’t the case with your system, you can download and install Python <a href="https://www.python.org/downloads/">using this link</a>.
To confirm that your version is the latest, you can run the following command:&nbsp;
<pre class="prettyprint">sudo apt update 
sudo apt upgrade
</pre>Confirm the Python version is 3.x:
<pre class="prettyprint">python3 -V
</pre>Install pip3 to manage the Python libraries:
<pre class="prettyprint">sudo apt install -y python3-pip
</pre>Install cmake needed for the <code>face_recognition</code> library:
<pre class="prettyprint">pip3 install CMake
</pre>Add the cmake bin folder to your $PATH directory:
<pre class="prettyprint">export PATH=$CMake_bin_folder:$PATH
</pre>At the end, install the following libraries before starting scripting our main program:
<pre class="prettyprint">pip3 install dlib 
pip3 install numpy 
pip3 install face_recognition 
pip3 install elasticsearch
</pre> <h2>Detect and encode facial information from an image</h2>Using the <code>face_recognition</code> library, we can detect the faces from an image and transform the facial features into a 128-dimension vector.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltba1ba92ea79adfda/606cabdbf46b594bd59f6b02/facial-recognition-blog-vector-facial-features.png" data-sys-asset-uid="bltba1ba92ea79adfda" alt="facial-recognition-blog-vector-facial-features.png" "="" width="381" height="239" style="width: 381px; height: 239px; display: block; margin: auto;">
Create a getVectorFromPicture.py file:
<pre class="prettyprint">touch getVectorFromPicture.py
</pre>Append the file with following script:
<pre class="prettyprint">import face_recognition 
import numpy as np 
import sys 
image = face_recognition.load_image_file("$PATH_TO_IMAGE") 
# detect the faces from the images 
face_locations = face_recognition.face_locations(image) 
# encode the 128-dimension face encoding for each face in the image 
face_encodings = face_recognition.face_encodings(image, face_locations) 
# Display the 128-dimension for each face detected 
for face_encoding in face_encodings: 
 print("Face found ==&gt; ", face_encoding.tolist())
</pre>Let’s execute <code>getVectorFromPicture.py</code> to get the facial features representation for images of Elastic’s founders. The <code>$PATH_TO_IMAGE</code> variable needs to be changed in the script to set the image file name.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt2fcb4d8cc1d9008a/606cac60a9cf2022b93564bf/facial-recognition-blog-elastic-founders.png" data-sys-asset-uid="blt2fcb4d8cc1d9008a" alt="facial-recognition-blog-elastic-founders.png" width="565" height="352" style="width: 565px; height: 352px; display: block; margin: auto;">
We can now store the facial features representation into Elasticsearch.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt09790e6298edb432/606cac86d8743426b393db5b/facial-recognition-blog-features-representation.png" data-sys-asset-uid="blt09790e6298edb432" alt="facial-recognition-blog-features-representation.png" width="341" height="230" style="width: 341px; height: 230px; display: block; margin: auto;">
Let’s first create the index with mapping that contains a field with <code>dense_vector</code> type:
<pre class="prettyprint"># Store the face 128-dimension in Elasticsearch 
## Create the mapping 
curl -XPUT "http://localhost:9200/faces" -H 'Content-Type: application/json' -d' 
{ 
 "mappings" : { 
 "properties" : { 
 "face_name" : { 
 "type" : "keyword" 
 }, 
 "face_encoding" : { 
 "type" : "dense_vector", 
 "dims" : 128 
 } 
 } 
 } 
}'
</pre>We need to create one document for each face representation, which can be done using the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html">Index API</a>:
<pre class="prettyprint">## Index the face feature representation
curl -XPOST "http://localhost:9200/faces/_doc" -H 'Content-Type: application/json' -d'
{
 "face_name": "name",
 "face_encoding": [
 -0.14664565,
 0.07806452,
 0.03944433,
 ...
 ...
 ...
 -0.03167224,
 -0.13942884
 ]
}'
</pre>
<h2>Match the faces</h2>Let's say we have indexed four documents in Elasticsearch that contain each facial representation of Elastic’s founders. We can now use another image of our founders to match the individual images.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt654b494db42a931c/606cad11efddb74bd62b8576/facial-recognition-blog-elastic-founders-match.png" data-sys-asset-uid="blt654b494db42a931c" alt="facial-recognition-blog-elastic-founders-match.png" width="591" height="394" style="width: 591px; height: 394px; display: block; margin: auto;">
Create a recognizeFaces.py file:&nbsp;
<pre class="prettyprint">touch recognizeFaces.py
</pre>In this script, we’ll extract the vectors for each face detected from the input image, and we use the vectors for each face to build the query to send to Elasticsearch:
Import the libraries:
<pre class="prettyprint">import face_recognition 
import numpy as np 
from elasticsearch import Elasticsearch 
import sys
</pre>Add the following section to connect to Elasticsearch:
<pre class="prettyprint"># Connect to Elasticsearch cluster 
from elasticsearch import Elasticsearch 
es = Elasticsearch( 
cloud_id="cluster-1:dXMa5Fx...", 
 http_auth=("elastic", "&lt;password&gt;"), 
)
</pre>We will use the <code>cosineSimilarity</code> function to calculate the measure of cosine similarity between a given query vectors and document vectors stored into Elasticsearch.
<pre class="prettyprint">i=0 
for face_encoding in face_encodings: 
 i += 1 
 print("Face",i) 
 response = es.search( 
 index="faces", 
 body={ 
 "size": 1, 
 "_source": "face_name", 
 "query": { 
 "script_score": { 
 "query" : { 
 "match_all": {} 
 }, 
 "script": { 
 "source": "cosineSimilarity(params.query_vector, 'face_encoding')", 
 "params": { 
 "query_vector":face_encoding.tolist() 
 } 
 } 
 } 
 } 
 } 
 )
</pre>Let’s assume the score under 0.93 is considered an unknown face:
<pre class="prettyprint"> for hit in response['hits']['hits']: 
 #double score=float(hit['_score']) 
 if (float(hit['_score']) &gt; 0.93): 
 print("==&gt; This face match with ", hit['_source']['face_name'], ",the score is" ,hit['_score']) 
 else: 
 print("==&gt; Unknown face")
</pre>Let's execute our script:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc9d46c6ceaba0074/606cad301609c821964d9038/facial-recognition-blog-execute-script.gif" data-sys-asset-uid="bltc9d46c6ceaba0074" alt="facial-recognition-blog-execute-script.gif" style="display: block; margin: auto;">
The script was able to detect all faces with score match higher than 0.93.
 <h2>Go a step further with advanced search</h2>Facial recognition and search can be combined for advanced use cases. You can use Elasticsearch to build more sophisticated queries like <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-queries.html"><code>geo-queries</code></a>, <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-bool-query.html"><code>query-dsl-bool-query</code></a>, and <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html"><code>search-aggregations</code></a>.
As an example, the following query applies <code>cosineSimilarity</code> search on a specific location within a 200-km radius:
<pre class="prettyprint">GET /_search 
{ 
 "query": { 
 "script_score": { 
 "query": { 
 "bool": { 
 "must": { 
 "match_all": {} 
 }, 
 "filter": { 
 "geo_distance": { 
 "distance": "200km", 
 "pin.location": { 
 "lat": 40, 
 "lon": -70 
 } 
 } 
 } 
 } 
 }, 
 "script": { 
 "source": "cosineSimilarity(params.query_vector, 'face_encoding')", 
 "params": { 
 "query_vector":[ 
 -0.14664565,
 0.07806452,
 0.03944433,
 ...
 ...
 ...
 -0.03167224,
 -0.13942884
 ] 
 } 
 } 
 } 
 } 
}
</pre>Combining <code>cosineSimilarity</code> with other Elasticsearch queries give you unlimited possibilities to implement more complex use cases.
 <h2>Conclusion</h2>Facial recognition can be relevant in many use cases, and you may already use it in your daily life. The concepts described above can be generalized to any object detection in images or videos, so you may extend your use case to a very large utilization.&nbsp;
Elasticsearch can help simplify your advanced requirements. Try it out with a <a href="https://cloud.elastic.co/registration">free 14-day trial of Elastic Cloud</a>, our official managed Elasticsearch offering, and let us know what you think in our <a href="https://discuss.elastic.co/c/elastic-stack/elasticsearch/6">Discuss forums</a>.

blog-banner-elasticsearch-service.png

blog-thumb-release-elasticsearch.png

How to build a facial recognition system using Elasticsearch and Python

Elasticsearch can be optimized for any use case. In this post, learn how to size and benchmark your Elastic Stack deployment for logs and metrics observability.

With Elasticsearch, it's easy to hit the ground running. When I built my first Elasticsearch cluster, it was ready for indexing and search within a matter of minutes. And while I was pleasantly surprised at how quickly I was able to deploy it, my mind was already racing towards next steps. But then I remembered I needed to slow down (we all need that reminder sometimes!) and answer a few questions before I got ahead of myself. Questions like:<ul><li>How confident am I that this will work in production?</li><li>What is the throughput capacity of my cluster?</li><li>How is it performing?</li><li>Do I have enough available resources in my cluster?</li><li>Is it the right size?</li></ul>If these sound familiar, great! These are questions all of us should be thinking about as we deploy new products into our ecosystems. In this post, we'll tackle performance Elasticsearch benchmarking and sizing questions like the above. We’ll go beyond “it depends” to equip you with a set of methods and recommendations to help you size your Elasticsearch cluster and benchmark your environment. As sizing exercises are specific to each use case, we will focus on <a href="https://www.elastic.co/log-monitoring" target="_self">logging</a> and <a href="https://www.elastic.co/infrastructure-monitoring" target="_self">metrics</a> in this post.<h2>Think like a service provider</h2>When we define the architecture of any system, we need to have a clear vision about the use case and the features that we offer, which is why it’s important to think as a service provider — where the quality of our service is the main concern. In addition, the architecture can be influenced by the constraints that we may have, like the hardware available, the global strategy of our company and many other constraints that are important to consider in our sizing exercise.Note that with the <a href="https://www.elastic.co/elasticsearch/service" target="_self">Elasticsearch Service on Elastic Cloud</a>, we take care of a lot of the maintenance and data tiering that we describe below. We also provide a predefined observability template along with a checkbox to enable the shipment of logs and metrics to a dedicated monitoring cluster. Feel free to spin up a <a href="https://www.elastic.co/cloud/" target="_self">free trial cluster</a> as you follow along.<h2>Computing resource&nbsp;basics</h2>Performance is contingent on how you're using Elasticsearch, as well as what you're running it on. Let's review some fundamentals around computing resources. For each search or indexing operation the following resources are involved:<h3>Storage: Where data persists</h3><ul><li>SSDs are recommended whenever possible, in particular for nodes running search and index operations. Due to the higher cost of SSD storage, a <a href="https://www.elastic.co/blog/implementing-hot-warm-cold-in-elasticsearch-with-index-lifecycle-management" target="_self">hot-warm architecture</a> is recommended to reduce expenses.</li><li>When operating on bare metal, local disk is king!</li><li>Elasticsearch does not need redundant storage (RAID 1/5/10 is not necessary), logging and metrics use cases typically have at least one replica shard, which is the minimum to ensure fault tolerance while minimizing the number of writes.</li></ul><h3>Memory: Where data is buffered</h3><ul><li>JVM Heap:Stores metadata about the cluster, indices, shards, segments, and fielddata. This is ideally set to 50% of available RAM.</li><li>OS Cache:Elasticsearch will use the remainder of available memory to cache data, improving performance dramatically by avoiding disk reads during full-text search, aggregations on doc values, and sorts.</li></ul><h3>Compute: Where data is processed</h3>Elasticsearch nodes have thread pools and thread queues that use the available compute resources. The quantity and performance of CPU cores governs the average speed and peak throughput of data operations in Elasticsearch.<h3>Network: Where data is transferred</h3>The network performance — both bandwidth and latency — can have an impact on the inter-node communication and inter-cluster features like <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.9/modules-cross-cluster-search.html" target="_self">cross-cluster search</a> and <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.9/xpack-ccr.html" target="_self">cross-cluster replication</a>.<h2>Sizing by data volume</h2>For metrics and logging use cases, we typically manage a huge amount of data, so it makes sense to use the data volume to initially size our Elasticsearch cluster. At the beginning of this exercise we need to ask some questions to better understand the data that needs to be managed in our cluster.<ul><li>How much raw data (GB) we will index per day?</li><li>How many days we will retain the data?</li><li>How many days in the hot zone?</li><li>How many days in the warm zone?</li><li>How many replica shards will you enforce?</li></ul>In general we add 5% or 10% for margin of error and 15% to stay under the disk watermarks. We also recommend adding a node for hardware failure.<h3>Let’s do the math</h3><ul><li>Total Data (GB) = Raw data (GB) per day * Number of days retained * (Number of replicas + 1) * Indexing/Compression Factor</li><li>Total Storage (GB) = Total data (GB) * (1 + 0.15 disk Watermark threshold + 0.1 Margin of error)</li><li>Total Data Nodes = ROUNDUP(Total storage (GB) / Memory per data node / Memory:Data ratio) In case of large deployment it's safer to add a node for failover capacity.</li></ul>1.2 is an average ratio we have observed with throughout deployments.To get the value that relates to your data, index logs and metrics and divide the index size (without replica) by the raw volume size.<h2>Example: Sizing a small cluster</h2>You might be pulling logs and metrics from some applications, databases, web servers, the network, and other supporting services . Let's assume this pulls in 1GB per day and you need to keep the data 9 months.You can use 8GB memory per node for this small deployment. Let’s do the math:<ul><li>Total Data (GB)&nbsp;= 1GB x (9 x 30 days) x 2 = 540GB</li><li>Total Storage (GB)= 540GB x (1+0.15+0.1) = 675GB</li><li>Total Data Nodes = 675GB disk / 8GB RAM /30 ratio = 3 nodes</li></ul>Let’s see how simple it is to build this deployment on <a href="https://www.elastic.co/cloud" target="_self">Elastic Cloud</a>:<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt839edb187d5c4ecb/5f9ad378878c0d548cdc2608/blog-logs-metrics-sizing-small-cloud.png" data-sys-asset-uid="blt839edb187d5c4ecb" alt="Sizing your small cluster in Elastic Cloud"/><h3>Example: Sizing a large deployment</h3>Your small deployment got successful and more partners want to use your Elasticsearch service, you may need resizing your cluster to much the new requirements.Let’s do the math with the following inputs:<ul><li>You receive 100GB per day and we need to keep this data for 30 days in the hot zone and 12 months in the warm zone.</li><li>We have 64GB of memory per node with 30GB allocated for heap and the remaining for OS cache.</li><li>The typical memory:data ratio for the hot zone used is 1:30 and for the warm zone is 1:160.</li></ul>If we receive 100GB per day and we have to keep this data for 30 days, this gives us:<ul><li>Total Data (GB) in the hot zone = (100GB x 30 days * 2) = 6000GB</li><li>Total Storage (GB) in the hot zone = 6000GB x (1+0.15+0.1) = 7500GB</li><li>Total Data Nodes in the hot zone = ROUNDUP(7500 / 64 / 30) + 1 = 5 nodes</li><li>Total Data (GB) in the warm zone = (100GB x 365 days * 2) = 73000GB</li><li>Total Storage (GB) in the warm zone = 73000GB x (1+0.15+0.1) = 91250GB</li><li>Total Data Nodes in the warm zone = ROUNDUP(91250 / 64 / 160) + 1 = 10 nodes</li></ul>Let’s see how simple it is to build this deployment on <a href="https://www.elastic.co/cloud" target="_self">Elastic Cloud</a>:<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt47b6081e39428a2a/5f9ad3946c73a0751fc7a376/blog-logs-metrics-sizing-large-cloud.jpg" data-sys-asset-uid="blt47b6081e39428a2a" alt="Sizing your large cluster in Elastic Cloud"/><h2>Benchmarking</h2>Now that we have our cluster(s) sized appropriately, we need to confirm that our math holds up in real world conditions. To be more confident before moving to production, we will want to do benchmark testing to confirm the expected performance, and the targeted SLA.For this benchmark, we will use the same tool our Elasticsearch engineers use <a href="https://github.com/elastic/rally" target="_self">Rally</a>. This tool is simple to deploy and execute, and completely configurable so you can test multiple scenarios. Learn more about <a href="https://www.elastic.co/blog/rally-1-0-0-released-benchmark-elasticsearch-like-we-do" target="_self">how we use Rally at Elastic</a>.To simplify the result analysis, we will split the benchmark into two sections, indexing and search requests.<h2>Indexing benchmark</h2>For the indexing benchmarks we are trying to answers the following questions:<ul><li>What is the maximum indexing throughput for my clusters?</li><li>What is the data volume that I can index per day?</li><li>Is my cluster oversized or undersized ?</li></ul>For the purpose of this benchmark we will use a 3 nodes cluster with the following configuration for each nodes:<ul><li>8 vCPU</li><li>Standard persistent disk (HDD)</li><li>32GB/16 heap</li></ul><h3>Indexing benchmark #1:</h3>The data set used for this benchmark is <a href="https://github.com/elastic/rally-tracks/tree/master/metricbeat" target="_self">Metricbeat</a> data with the following specifications:<ul><li>1,079,600 documents</li><li>Data volume: 1.2GB</li><li>AVG document size: 1.17 KB</li></ul>The indexing performance will depend also on the performance of the indexing layer, in our case Rally. We'll execute multiple benchmark runs to figure out the optimal bulk size and the optimal thread count in our case.&nbsp;We’ll start with 1 Rally client to find the optimal batch size. Starting from 100, then increasing by double in subsequent runs, which shows that we have an optimal batch size of 12k (around 13.7 MB) and we can reach 13K events indexed per second.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltf7fbc948de75b747/5f9ad3cf1f9166620ed86cf8/blog-logs-metrics-sizing-batch-1.png" data-sys-asset-uid="bltf7fbc948de75b747"/>Next, using a similar method, we find that the optimal number of clients is 16 allowing us to reach 62K events indexed per second.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt8e20acb021757a45/5f9ad3e36c73a0751fc7a380/blog-logs-metrics-sizing-clients-1.png" data-sys-asset-uid="blt8e20acb021757a45"/>Our cluster can handle a maximum indexing throughput of 62K events per second. To go further we would need to add a new node.I was curious to see how many indexing requests one node can handle, So I’ve executed the same track on a one node cluster and a two node cluster to see the difference.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc1f831d98ba4a354/5f9ad4001a81c1644e996ea7/blog-logs-metrics-sizing-clusters-1.jpg" data-sys-asset-uid="bltc1f831d98ba4a354"/><h4>Conclusion: For my test environment, the maximum indexing throughput is...</h4><ul><li>With 1 node and 1 shard we got 22K events per second.</li><li>With 2 nodes and 2 shards we got 43k events per second.</li><li>With 3 nodes and 3 shards we got 62k events per second.</li></ul>Any additional indexing request will be put in the queue, and when the queue is full the node will send rejection responses.Our dataset impacts the cluster performance, which is why it’s important to execute the tracks with your own data.<h3>Indexing benchmark #2:</h3>For the next step, I have executed the same tracks with the <a href="https://github.com/elastic/rally-tracks/tree/master/http_logs" target="_self">HTTP server log data</a> using the following configuration:<ul><li>Volumes: 31.1GB</li><li>Document: 247,249,096</li><li>Avg document size: 0.8 KB</li></ul>Optimal bulk&nbsp;size is 16K documents.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt28f6345ba1491485/5f9ad457fe49b57a83a15327/blog-logs-metrics-sizing-batch-2.png" data-sys-asset-uid="blt28f6345ba1491485" alt="blog-logs-metrics-sizing-batch-2.png"/>Yielding an optimal number of 32 clients.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltd3fcf8723b3f4c14/5f9ad48b83a0a3620dd3b218/blog-logs-metrics-sizing-clients-2.png" data-sys-asset-uid="bltd3fcf8723b3f4c14"/>And the maximum indexing throughput for the http server log data is 220K events per second.<h2>Search benchmark</h2>To benchmark our search performance, we will consider using 20 clients with a targeted throughput of 1000 OPS.For the search, we will execute three benchmarks:<h3>1. Service time for queries</h3>We will compare the service time (90 percentiles) for a set of queries.<h4>Metricbeat data set</h4><ul><li>auto-date-historgram</li><li>auto-data-histogram-with-tz</li><li>date-histogram</li><li>Date-histogram-with-tz</li></ul>We can observe that auto-data-histogram-with-tz query have the higher service time.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt7d612d0591299c2f/5f9ad4a3fe49b57a83a15331/blog-logs-metrics-sizing-metricbeat.png" data-sys-asset-uid="blt7d612d0591299c2f"/><h4>HTTP server log data set</h4><ul><li>Default</li><li>Term</li><li>Range</li><li>Hourly_agg</li><li>Desc_sort_timestamp</li><li>Asc_sort_timestamp</li></ul>We can observe that desc_sort_timestamp and desc_sort_timestamp queries have the higher service time.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt44d39fb4b1ba5c50/5f9ad4c267a7c354926033c5/blog-logs-metrics-sizing-http.png" data-sys-asset-uid="blt44d39fb4b1ba5c50"/><h3>2. Service time for parallel queries</h3>Let's see the 90 percentile service time increased if I execute the queries in <a href="https://esrally.readthedocs.io/en/0.5.1/track.html" target="_self">parallel</a>.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt5358122b351cdd0d/5f9ad50a2425cd7a8af696cc/blog-logs-metrics-sizing-parallel.png" data-sys-asset-uid="blt5358122b351cdd0d"/><h3>3. Index rate and service time with parallel indexing</h3>We'll execute a <a href="https://esrally.readthedocs.io/en/0.5.1/track.html" target="_self">parallel</a> indexing task and the searches to see the indexing rate and the service time for our queries.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltdebe9d92ea2aa969/5f9ad52883a0a3620dd3b222/blog-logs-metrics-sizing-indexing.png" data-sys-asset-uid="bltdebe9d92ea2aa969"/>Let's see the 90 percentile service time for the queries increased when executing in parallel with indexing operations.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt949a10ab19056b1f/5f9ad53883a0a3620dd3b226/blog-logs-metrics-sizing-index-rate.png" data-sys-asset-uid="blt949a10ab19056b1f"/><h3>Reading our results</h3>With 32 clients for indexing and 20 users for search we have:<ul><li>Indexing throughput 173K, less than 220K noted previously.</li><li>Search throughput 1000 events per second.</li></ul><h2>Conclusion</h2>The sizing exercise equips you with a set of methods to calculate the number of nodes that you need based on data volume. In order to best plan for the future performance of your cluster, you will also need to benchmark your infrastructure — something we've already taken care of in <a href="https://www.elastic.co/cloud" target="_self">Elastic Cloud</a>. Since the performance will depend also on your use case, we recommend using your data and your queries to be closest to the reality. More details about how to define a custom workload in the <a href="https://esrally.readthedocs.io/en/stable/adding_tracks.html" target="_self">Rally documentation</a>.After the benchmark exercise, you should have a better understanding of your infrastructure performance and you can <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.9/tune-for-indexing-speed.html" target="_self">continue to fine tune Elasticsearch to improve your indexing speed</a>. Additionally, <a href="https://www.elastic.co/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right" target="_self">learn how to use Rally to get your Elasticsearch right.</a>

blog-banner-charts-monitor.png

blog-thumb-charts-monitor.png

Benchmarking and sizing your Elasticsearch cluster for logs and metrics

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Yacine Younes

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

제출하면 귀하가 <a href="/kr/legal/elastic-cloud-account-terms" target="_self">Elastic 서비스 약관</a>을 읽고 이에 동의하며, 위에 제공된 연락처로 <a href="/kr/legal/privacy-statement#how-we-use-the-information" target="_self">선택한 커뮤니케이션</a>을 수신하는 데 동의한 것으로 간주됩니다. 언제든지 수신을 거부할 수 있으며 자세한 내용은 <a href="/kr/legal/privacy-statement/" target="_self">Elastic 개인정보 취급방침</a>을 참조하시기 바랍니다.

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

제출하면 귀하가 Elastic <a href="/kr/legal/serverless-preview-terms" target="_self">서비스 약관</a>을 읽고 이에 동의하며, 위에 제공된 세부 정보를 사용하여 Elastic이 관련 제품 및 서비스에 대해 귀하에게 <a href="/kr/legal/privacy-statement#how-we-use-the-information" target="_self">연락할 수 있음</a>을 이해한 것으로 간주됩니다. 언제든지 수신을 거부할 수 있으며 자세한 내용은 <a href="/kr/legal/privacy-statement/" target="_self">Elastic 개인정보 취급방침</a>을 참조하시기 바랍니다.

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

제출하면 <a href="/kr/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud 서비스 약관</a>에 동의하시게 됩니다. 개인 데이터는 <a href="/kr/legal/privacy-statement">Elastic의 개인정보 취급방침</a>에 따라 처리됩니다.

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

제출하면 귀하가 Elastic <a href="/kr/legal/terms-of-use" target="_self">서비스 약관</a>을 읽고 이에 동의하며, 위에 제공된 세부 정보를 사용하여 Elastic이 관련 제품 및 서비스에 대해 귀하에게 <a href="/kr/legal/privacy-statement#how-we-use-the-information" target="_self">연락할 수 있음</a>을 이해한 것으로 간주됩니다. 언제든지 수신을 거부할 수 있으며 자세한 내용은 <a href="/kr/legal/privacy-statement/" target="_self">Elastic 개인정보 취급방침</a>을 참조하시기 바랍니다.

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

이 페이지의 콘텐츠는 선택하신 언어로 제공되지 않습니다. Elastic은 다양한 언어로 콘텐츠를 제공하기 위해 최선을 다하고 있습니다.조금만 더 기다려주세요!

Author

Articles by

Learn more

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

내 표준 시간대로 웨비나 시작 시간 확인하기

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

참여가 어려우신가요? 등록하시면 녹화 자료를 보내드리겠습니다. 이메일로 관련 콘텐츠도 받아보실 수 있습니다.

Small image for

On-demand webinar

Featured webinar

Register to Attend

Share on Facebook

Reset all

Upcoming webinar

View more posts

Print

Hosted by

Sign In to Attend

View next

Follow us on Facebook

Share

Highlights

Read less

You'll also receive an email with related content

이메일을 통해 관련 내용을 보내드리겠습니다.

Thank you for registering. We will send you a confirmation email soon.

등록해 주셔서 감사합니다. 곧 확인 이메일을 보내드리겠습니다.

Author

Articles by Yacine Younes

How to build a facial recognition system using Elasticsearch and Python

Benchmarking and sizing your Elasticsearch cluster for logs and metrics

Sign up for Elastic Cloud free trial