Tech Topics

Performance Impact of Meltdown on Elasticsearch

著者
Elastic Engineering

The old battle between security and performance strikes again: Linux has recently added a new security feature called kernel page table isolation (KPTI) in order to mitigate the so-called Meltdown attack. It was included in kernel 4.15-rc4 and has also been backported by Linux distributors.

The goal of KPTI is to separate the page tables from user space and kernel space completely to avoid attackers reading data from kernel space. With this change, an intermediate step is necessary when the system switches context from user mode to kernel mode, e.g. when handling an interrupt or a system call. If you are interested in more details, please see a walkthrough of the patches in Linux Weekly News.

These kernel patches rely on the presence of a processor feature called process context identifier (PCID) to reduce the impact on performance. On Linux, you can check whether your processor supports this features with grep "pcid" /proc/cpuinfo. While x86 CPUs have this feature since around 2010, if you are using virtual machines it may be the case that your hypervisor does not expose this feature to the guest OS.

Impact on Elasticsearch on Bare Metal

We were interested in the performance implications of this kernel patch on Elasticsearch. Therefore, we ran several macrobenchmarks with Rally. At the time we started our tests, there were no prebuilt kernels with the feature turned on, so we had to build our own kernel based on Ubuntu mainline v4.15-rc6, configured with CONFIG_PAGE_TABLE_ISOLATION=y. Meanwhile kernel patches have been backported and it is not necessary to build a custom kernel any more.

In the baseline configuration we turned the feature off with the kernel command line parameter nopti ( it shows up in the kernel log as "Kernel/User page tables isolation: disabled on command line."). We contrasted this baseline configuration with the feature turned on (it shows up as "Kernel/User page tables isolation: enabled" in the kernel log).

We tested on several bare-metal systems with the following specifications:

  • CPU: CPU: Intel(R) Core(TM) i7-6700 (Skylake), Intel(R) Xeon (R) Processor E3-1270 v5 (Skylake), or Intel(R) Core(TM) i7-7700 (Kaby Lake)
  • RAM: 32GB - 64GB
  • OS: Ubuntu 16.04.2 LTS on kernel version 4.15.0-041500rc6-generic (custom built kernel with CONFIG_PAGE_TABLE_ISOLATION=y)
  • Elasticsearch 6.1.1 (stock configuration, 4GB heap space)

We conducted the majority of our tests on systems with SSDs but also tested on spinning disks and NVMe drives to see whether the kind of storage had any effect. For our tests, the load test driver was on a dedicated machine.

We tested several data sets from our standard benchmarking suite and checked the impact on indexing throughput. While we have also tested the impact on search latency in several scenarios, we did not see any significant difference in our tests.

For the majority of workloads we also did not see any statistically significant impact on indexing throughput. For very I/O intensive workloads we did see reductions in indexing throughput of roughly 4%. We could not observe any significantly different behavior for different types of storage. We also deliberately turned off PCID by providing the kernel boot parameter nopcid and have observed reductions in indexing throughput of up to 20%.

Impact on Elasticsearch on Elastic Cloud

We have also tested the performance impact of the Meltdown patches on Elastic Cloud. Our benchmarking configuration:

  • Underlying VM: EC2 i3.8xlarge allocator (vCPU Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz)
  • OS: Ubuntu Xenial 16.04.03 LTS
  • AWS Xen hypervisor exposing the pcid feature to our guest.
  • An Elastic Cloud 8G cluster running Elasticsearch 6.1.1 with default configuration (4G heap)
  • An Elastic Cloud 64G cluster running Elasticsearch 6.1.1 with default configuration (31G heap)

All tests were conducted against the same VM and cluster endpoint.

  • Test #1 (Baseline). Kernel 4.4.0-1044-aws. The AWS-optimised kernel in the 4.4 tree provided by Canonical’s kernel team before the KPTI/KAISER patches landed.
  • Test #2 (PCID 0|128 kernel|userland). Kernel 4.4.0-1048-aws patched for Meltdown.

For very I/O intensive workloads we have seen reductions of up to 16% in the worst case on smaller instance types (4G heap) between baseline and patched version. This worst case scenario becomes less pronounced on larger instance types (31G heap). However, we recommend that you test the implications for your use case to get a more accurate picture. We are also currently investigating several options for reducing the performance hit.

Summary

Although we saw a performance impact due to kernel page table isolation, the worst drop in indexing throughput our tests was around 4%. While we have checked a variety of workloads in our benchmarking environment we recommend that you measure the impact with your workload in your environment to get an accurate picture of the impact in your situation. You should also check that your system has PCID support as this feature is important to reduce the performance impact of split page tables for user space and kernel space. Please also check your distribution’s documentation for further notes.

The title image Melting has been created by Tatiana Vdb and is licensed as CC BY 2.0.