Learn how the Elastic Operations team integrates Vault and Kubernetes in an engineering deep dive.

<html><head></head><body><p>On Elastic&apos;s Infrastructure team, we are always looking for opportunities to introduce automation and reduce operational burdens. Function-as-a-service solutions such as AWS Lambda have allowed us to simplify operations for some services, but applications that require persistent runtimes have different challenges. There are many potential solutions to this problem: fleets of autoscaled AWS ECS instances, platforms such as Google App Engine or AWS Elastic Beanstalk, and more. Our work on this has led to investigating various container schedulers - and with such a large community and broad feature set, <a href="https://kubernetes.io/">Kubernetes</a> has proven a useful platform for our use case.</p><p>Adopting a solution as powerful and flexible as Kubernetes entails some non-trivial work to bring it into an existing operations workflow, not the least of which is providing secrets or otherwise sensitive information to running applications within a <code>Pod</code> or <code>Deployment</code>. This post will explain our approach to integrating <a href="https://www.vaultproject.io/">Hashicorp Vault</a> (our chosen secret management solution) with Kubernetes.</p><h3 id="introduction">Introduction</h3><p>Integrating Kubernetes and Vault should meet a few requirements:</p><ul>
<li>Secrets should be securely communicated to running services. No sensitive tokens should be committed into source code repositories.</li>
<li>Sensitive data should be centralized in Vault and not duplicated in other datastores to both simplify auditing and reduce potential attack surfaces.</li>
</ul><p>Moreover, applications should be able to migrate to Kubernetes without dramatic changes in order to consume secrets - this is critical to aid in migration from traditional platforms like AWS EC2 to Kubernetes.</p><h4 id="background">Background</h4><p>Before diving into the implementation, there are a few details specific to our solution that are worth highlighting:</p><ul>
<li>We make heavy use of an internally hosted, private Docker registry. Aside from the obvious ability to reference images that should be internal-only, building custom images for many of these services (such as kubernetes-vault) ensures that they are deployed with audited code and that updates are more closely controlled.</li>
<li>Familiarity with Vault concepts is expected. In particular, the AppRole backend is heavily used in the following design pattern, which couples a specific role identifier with a securely distributed secret that kubernetes-vault pushes to a Kubernetes <a href="https://kubernetes.io/docs/concepts/workloads/pods/init-containers/">initContainer</a>.</li>
<li>Many best practices such as audit logging, policy design, and other considerations are not covered in this high-level overview. In a production deployment, these and other decisions should be reviewed, especially when deploying a service that manages sensitive information such as Vault.</li>
</ul><h3 id="implementation">Implementation</h3><p>There are a few steps that come together to achieve the goals outlined in the introduction: first, connecting Kubernetes to Vault; second, providing a mechanism to expose secrets to running applications; and third, creating reusable tools to aid in generic solutions for additional applications.</p><h4 id="connecting-kubernetes-and-vault">Connecting Kubernetes and Vault</h4><p>Fortunately, the <a href="https://github.com/Boostport/kubernetes-vault">kubernetes-vault</a> project provides a well-designed solution to this problem. <a href="https://www.vaultproject.io/docs/concepts/response-wrapping.html">Response wrapping</a> ensures that tokens are passed to applications securely in-transit, meeting the first requirement for integrating with Vault.</p><p>Our deployment of kubernetes-vault looks similar to the one outlined in the project&apos;s quick start guide, with the exception that Vault is run <em>outside</em> of Kubernetes as an independent service. This works equally well as running Vault within Kubernetes itself, which is important as a variety of services outside of our Kubernetes cluster also consumes Vault for different use cases.</p><p>One important consideration when deploying the kubernetes-vault deployment is to ensure that the token passed to the service is <em>periodic</em>. When interacting with Vault as a limited-privilege user, any generated tokens are subject to expiry when their parent token is revoked or expired.</p><h4 id="exposing-secrets-to-applications">Exposing Secrets to Applications</h4><p>Leveraging kubernetes-vault brings us to the point that a JSON-formatted file is available with several values including the Vault token. While consuming this file can be done with a little code for <em>new</em> applications, retrofitting existing applications can be challenging.</p><p>Fortunately, many applications can easily consume environment variables, which provides a convenient way to pass secret values without needing to write them out to a persistent file or in a Pod/Deployment spec. Another tool called <a href="https://github.com/channable/vaultenv">vaultenv</a> can automate this process.</p><p>To illustrate how this works, consider a <code>Deployment</code> for Logstash (which can <a href="https://www.elastic.co/guide/en/logstash/current/environment-variables.html">reference environment variables in its configuration file</a>). How can we pass an HTTP basic authentication username and password to the <a href="what">elasticsearch output plugin</a> to securely index events to an external, secured cluster?</p><p>First, we define a <code>ConfigMap</code> defining where to reference secrets from within Vault:</p><pre class="prettyprint">apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-secrets
data:
  logstash.secrets: |
    ELASTICSEARCH_USERNAME=elasticsearch/production#username
    ELASTICSEARCH_PASSWORD=elasticsearch/production#password</pre><p>This instructs <code>vaultenv</code> to expose a secret stored in the <a href="https://www.vaultproject.io/docs/secrets/generic/">generic secret backend</a> under the path <code>secret/elasticsearch/production</code> with the key <code>password</code> as the environment variable <code>ELASTICSEARCH_PASSWORD</code>. This <code>ConfigMap</code> is mounted into the pod at <code>/etc/secrets.d</code> as a de facto directory for vaultenv secrets.</p><p>In a similar fashion, we can define our Logstash config as a <code>ConfigMap</code>, interpolating the named variables into the configuration file directly:</p><pre class="prettyprint">apiVersion: v1
kind: ConfigMap
metadata:
  name: logstsah-config
data:
  example.conf: |
    input {
      exec {
        command =&gt; &quot;date&quot;
        interval =&gt; 5
      }
    }

    output {
        elasticsearch {
            index    =&gt; &quot;logstash-%{+YYYY}&quot;
            hosts    =&gt; [&quot;https://elasticsearch.cluster.url:9200&quot;]
            user     =&gt; &quot;${ELASTICSEARCH_USERNAME}&quot;
            password =&gt; &quot;${ELASTICSEARCH_PASSWORD}&quot;
            ssl      =&gt; true
        }
    }</pre><p>Ensuring that the <code>vaultenv</code> executable is present in the container can be done in a <code>Dockerfile</code> with some simple <code>RUN</code> commands or in a multi-stage <code>Dockerfile</code> to compile <code>vaultenv</code> locally and <code>COPY</code> it into the final container image. Additionally, installing <code>jq</code> makes consuming the <code>vault-token</code> file easier.</p><p>Finally, the container&apos;s command can be defined. In the case of <a href="https://www.docker.elastic.co/">the official Elastic Logstash Docker image</a>, the default <code>ENTRYPOINT</code> is <code>/usr/local/bin/docker-entrypoint</code>, so we simply wrap that script within a <code>vaultenv</code> invocation:</p><pre class="prettyprint">token=$(jq -r &apos;.clientToken&apos; /var/run/secrets/boostport.com/vault-token)
exec vaultenv \
      --host $VAULT_ADDR \
      --token $token \
      --secrets-file /etc/secrets.d/logstash.secrets \
      /usr/local/bin/docker-entrypoint</pre><p>Coupled with an <code>initContainer</code> as explained in the kubernetes-vault documentation, <code>vaultenv</code> will retrieve the configured secrets and invoke <code>/usr/local/bin/docker-entrypoint</code> with the <code>ELASTICSEARCH_USERNAME</code> and <code>ELASTICSEARCH_PASSWORD</code> environment variables.</p><p>This pattern permits the token to be securely passed to <code>vaultenv</code>, which lets us modify the original application container minimally to invoke its same command wrapped in <code>vaultenv</code>&apos;s provided environment variables.</p><h4 id="managing-periodic-tokens">Managing Periodic Tokens</h4><p>Like the token used for kubernetes-vault, defining a TTL for tokens issued to an <code>AppRole</code> is a best practice to ensure that access is revoked regularly for deleted pods and old tokens. While <code>vaultenv</code> only executes once before passing control to its command argument, if the pod&apos;s container should die for any reason, the container will be re-run, and the token may have expired in the intervening period. Fortunately, because the token JSON file is managed as an <a href="https://kubernetes.io/docs/concepts/storage/volumes/#emptydir"><code>emptyDir</code></a>, we can consume the token from a sidecar container to renew it out-of-band from the pod&apos;s main container.</p><p>In our environment, we host a small token renewal image on our Docker registry that has the simple task of reading the token JSON file and renewing it regularly. This permits us to re-use the image in <code>Deployment</code> definitions that require it, without needing to change the application container in any way.</p><p>As an example, the python daemon&apos;s renewal logic could include something as simple as this (using the <a href="https://github.com/ianunruh/hvac">hvac</a> library):</p><pre class="prettyprint">while True:
    lookup = vault.lookup_token()[&apos;data&apos;]
    interval = lookup[&apos;creation_ttl&apos;] / 2

    if lookup[&apos;ttl&apos;] &lt;= interval:
        print(&apos;renewing vault token&apos;)
        vault.renew_token()

    sleep(10)</pre><p>This loop simply renews the Vault token if it has less than half its TTL left, checking every ten seconds (the actual logic we use is slightly different, but this is simpler to illustrate). Connection exceptions to Vault are also important to consider.</p><p>Adding this automatic renewal to an existing <code>Deployment</code> becomes as simple as defining an additional container to the Kubernetes yaml configuration:</p><pre class="prettyprint">- name: token-renewer
  image: private.registry.url/token-renewer
  volumeMounts:
    - name: vault-token
      mountPath: /var/run/secrets/boostport.com</pre><p>If the main container is restarted for any reason, <code>vaultenv</code> will have a still-valid token to request secrets again to invoke the container command with newly requested secrets.</p><h3 id="an-example-kubernetes-definition">An Example Kubernetes Definition</h3><p>Yaml configuration files for kubernetes-vault and related specifications are likely deployment-specific, but <a href="https://gist.github.com/tylerjl/4b348c403c4a2e37769edcf4016a6f66">this example <code>Deployment</code> illustrates what an example application definition might look like</a>. Some specific items of note in this configuration file:</p><ul>
<li><code>VAULT_ROLE_ID</code> and <code>VAULT_ADDR</code> are environment-specific. Note that a Vault <code>AppRole</code> will need to be created for each <code>AppRole</code> that kubernetes-vault is expected to generate secret IDs and tokens for.</li>
<li>Our infrastructure team manages an access-controlled Docker registry, which serves as a way to host private images for many of these utilities. We build these images after verifying the code in order to ensure each component is run on a Docker image with known code. In an environment where security is paramount, doing so is important to ensure containers are running trusted code.</li>
<li>Though this example explicitly lists the <code>vaultenv</code> container <code>command</code>, additional <code>Dockerfile</code> directives can encapsulate this logic during the container building phase.</li>
</ul><p>With the necessary yaml additions, many <code>Deployment</code> specifications can re-use this pattern to securely reference Vault secrets with limited policies to ensure that <a href="https://en.wikipedia.org/wiki/Principle_of_least_privilege">the principle of least privilege</a> is followed.</p><h3 id="final-thoughts">Final Thoughts</h3><p>This example illustrates how to create applications that are easy to scale and manage thanks to Kubernetes, and can safely consume secrets from a highly secure store such as Vault.</p><p>In practice, this has proven to be a useful system design for our use case. In particular, using Vault as a cloud-agnostic secret store has permitted us to run Kubernetes in any arbitrary environment and consume Vault APIs regardless of where kubernetes-vault is running, whether in AWS, GCP, or otherwise. Relying on kubernetes-vault&apos;s response wrapping to securely deposit a token in a container alleviates some of the tension regarding how to distribute initial access credentials, and vaultenv has allowed us to provide an application-agnostic way of exposing those fetched secrets to applications that need them.</p><p><strong>Note</strong>: As of today, <a href="https://www.hashicorp.com/blog/hashicorp-vault-0-8-3/">Hashicorp has announced native integration with Kubernetes</a>. While the approach laid out in this post obviously predates this technique, some concepts in this post (such as leveraging vaultenv for services that do not natively consume Vault tokens) are still useful to consider.</p></body></html>

sailing-ship-659758_1920.jpg

Pods, Tokens, and a Little Glue: Integrating Kubernetes and Vault in Elastic Infrastructure

Curious about recent developments in Elastic configuration management modules and Puppet? Read on to hear about the latest features, fixes, and coming changes.

<html><head></head><body><p>The Elastic Stack is a far-reaching ecosystem with many interconnected parts, and the <a href="https://forge.puppet.com/elastic">official Puppet Modules</a> are no exception when it comes to the configuration management side of the house. Today, we want to highlight some developments taking place in the Elastic Puppet ecosystem to help spread awareness of new features and upcoming changes.</p><h3 id="modules-roadmap">Modules Roadmap</h3><p>First, some housekeeping items that pertain to all of our official modules.</p><h4 id="puppet-3">Puppet 3</h4><p>The most prominent upcoming change is the deprecation of Puppet 3 (if you&#x2019;re skimming this post, don&#x2019;t panic! You still have time). <a href="https://docs.puppet.com/upgrade">As noted in the upstream Puppet Labs page</a>, version 3 met its end in December 2016, which means that it is not only deprecated, but end-of-life, and past the cutoff date for any future updates. As systems operators ourselves, we recognize that keeping stride with software versions is one of the challenges that comes with the discipline, particularly when it comes to large enterprises.</p><p>With that in mind, we&#x2019;ll be keeping version 3.8 of Puppet in our list of supported versions for a period to ensure everyone has a window of time to work in, for those cases where upgrading is a challenge. Nevertheless, the Puppet ecosystem is moving on from version 3, and we can only hold on to old versions for so long. It&#x2019;s for this reason that we will be removing support for Puppet 3 with the 6.x release of the Elastic Stack, which grants many months of leeway for those operations engineers with a Puppet upgrade still on their plate. Version 6 of the&#xA0;Elastic Stack is likely to arrive in late 2017, and will be prefaced with additional announcements as its release draws closer.</p><h4 id="new-forge-namespace">New Forge Namespace</h4><p>What open source project doesn&#x2019;t like changing names every so often? <a href="https://forge.puppet.com/elastic">Our modules on the Puppet Forge</a> now properly live under the &#x201C;elastic&#x201D; namespace where all future module updates and release will land. While any references to old modules under the <code>elasticsearch</code> namespace will continue to work, the modules there are no longer updated. We suggest updating any automation or tooling that references released modules from the Forge to this new namespace (for example, to <code>elastic/elasticsearch</code>).</p><h3 id="official-puppet-kibana-module">Official Puppet Kibana Module</h3><p>One of the most common requests we hear from Puppet users is for other parts of our stack to ship official modules. At Elastic{ON} 2017 we released a beta version of <a href="https://forge.puppet.com/elastic/kibana">an official Puppet Kibana module</a>, which is now formally released at version 5.0 (major version synchronized with the rest of the Elastic Stack). The module supports everything you would expect from our Puppet modules, including Kibana plugin resources to make managing Kibana through Puppet a much easier experience.</p><p>As always, we&#x2019;re eager to hear from you regarding any bugs you&apos;ve found or features you&#x2019;d like to see and would encourage any bug reports or feature requests to be filed in the <a href="https://github.com/elastic/puppet-kibana">puppet-kibana GitHub repository</a>.</p><h3 id="elasticsearch-tasty-new-features">Elasticsearch: Tasty New Features</h3><p>The official Elasticsearch module hasn&#x2019;t stood still either. In addition to the steady march of bugfixes, <a href="https://forge.puppet.com/elastic/elasticsearch">the official Elasticsearch module</a> supports a couple of useful new resource types: indices and pipelines.</p><p>While the module already supported managing index templates (useful for auto-created indices from tools like Logstash or Beats), the <code>elasticsearch::index</code> type can help manage mappings and settings for <em>existing</em> indices in an Elasticsearch cluster. For example, to manage an Elasticsearch index called &#x201C;logs&#x201D; to retain only 1 replica per shard:</p><pre class="prettyprint">elasticsearch::index { &apos;logs&apos;:
  settings =&gt; {
    &apos;index&apos; =&gt; {
      &apos;number_of_replicas&apos; =&gt; 1
    }
  }
}</pre><p>Likewise, <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/pipeline.html">pipelines</a> are now supported as well. The ability to codify processing pipelines in Puppet can help centralize your workflow for processing logs and documents in Elasticsearch (see our <a href="https://www.elastic.co/blog/new-way-to-ingest-part-1">other blog posts about ingest node pipelines for more introductory content</a>). For example, to create a pipeline that will automatically parse dates, the following Puppet resource could be used:</p><pre class="prettyprint">elasticsearch::pipeline { &#x2018;parsedate&#x2019;:
  content =&gt; {
    &apos;description&apos; =&gt; &apos;Parse the date field into @timestamp&apos;,
    &apos;processors&apos; =&gt; [{
      &apos;date&apos; =&gt; {
        &apos;field&apos; =&gt; &apos;date&apos;,
        &apos;formats&apos; =&gt; [&apos;dd/MM/yyyy hh:mm:ss&apos;],
        &apos;timezone&apos; =&gt; &apos;US/Pacific&apos;
      }
    }]
  }
}</pre><h3 id="logstash-all-aboard-for-version-5">Logstash: All Aboard for Version 5</h3><p>Version 5 of Logstash brought huge improvements to the handling of the Logstash system service, adding full support for systemd, among other things. In the past, the Puppet module was responsible for much more of the work around service handling, which was a common source of problems as the module became progressively more complex in order to handle various edge-cases across distributions. With Logstash itself now handling the service, the Puppet module is much simpler and more stable when it comes to service management.</p><p>It&apos;s important to note, however, that the changes in service management were so comprehensive that it became impractical for the module to support Logstash 2.x and Logstash 5.x at the same time. Thus, module version 5 only supports Logstash 5. Of course, the legacy module <a href="https://forge.puppet.com/elasticsearch/logstash">elasticsearch/logstash</a> is still available if you need to run Logstash 2.</p><p>It&apos;s now possible to fully <a href="https://github.com/elastic/puppet-logstash#logstash-config-files">manage Logstash settings, startup options, and JVM options with the module</a>. Like this:</p><pre class="prettyprint">class { &apos;logstash&apos;:
  settings =&gt; {
    &apos;pipeline&apos; =&gt; {
      &apos;batch&apos; =&gt; {
        &apos;size&apos; =&gt; 25,
        &apos;delay =&gt; 5,
      }
    }
  }
}</pre><p>Since the settings are all defined as standard arrays and hashes, they work well with Hiera too.</p><p>Plugin support is now comprehensive, with support for remote locations like <code>puppet://</code> and <code>http://</code> URLs. The latter is essential for people choosing to install the new <a href="https://www.elastic.co/guide/en/x-pack/current/monitoring-logstash.html">X-Pack for Logstash</a>.</p><p>Finally, a lot of progress on the module is something you probably won&apos;t see, but hopefully you get to see the results. All new features, including those above, are now developed under testing from a <a href="https://github.com/elastic/puppet-logstash/blob/master/spec/acceptance/class_logstash_spec.rb">re-written test suite</a>. The tests are there to ensure everything in the module works reliably, <a href="https://travis-ci.org/elastic/puppet-logstash/builds/229800549">across every supported operating system and Puppet version</a>.</p></body></html>

wooden-mannequin-bleedimage.jpg

wooden-mannequin-thumb.jpg

Puppet State of the Union: May 2017

Learn best practices when using Elasticsearch plugins within Docker for easier container management.

<p>If you're running Elasticsearch within Docker containers, there are some important operational considerations to bear in mind. This is especially true when managing stateful services and daemons - when persisting data outside of an ephemeral container becomes important.</p><p>Using Elasticsearch plugins within containers is an example of this, both in terms of <strong>installing</strong> them in a repeatable, trackable manner and <strong>managing</strong> plugin configuration and data.</p><p>In this post, we'll explore some of the options to achieve sane plugin management within the context of Docker and Elasticsearch.</p><p><strong>Note</strong>: In this guide we will reference the Elasticsearch image found on the Docker Hub. The development and production of this Docker image is not affiliated with Elastic, Inc.</p><h3 id="a-docker-persistence-primer">A Docker Persistence Primer</h3><p>Like a file change in a version control system, changing the filesystem in a running container introduces differences in the running image. Steps need to be taken if data and changes need to persist permanently outside the scope of an impermanent container.</p><p>Although one could leverage some more complex storage schemes to achieve container persistence, the basic Docker mechanism of <em>bind mounts</em> is the most illustrative. This is achieved by, for example, keeping Elasticsearch indices long-term in <code>/data</code> by passing an option like <code>-v /data:/usr/share/elasticsearch/data</code> to a <code>docker run</code> command, which effectively stores your Elasticsearch data in the <em>host's</em> (not container's) <code>/data</code> directory.</p><p>If the changes are more permanent, that is, they are expected to be there without changing over time, codifying changes in a <code>Dockerfile</code> may make more sense. Both approaches are useful in different cases.</p><h3 id="managing-basic-plugins">Managing Basic Plugins</h3><p>In the following examples, I refer to "basic" plugins as those that do not require licenses or any other special components. Straightforward plugins like <a href="https://www.elastic.co/guide/en/elasticsearch/plugins/current/cloud-aws.html">cloud-aws</a> just need to place some files on the system to work.</p><p>In a case like this, managing the presence of the plugin is simplest by just extending the image using a <a href="https://docs.docker.com/engine/reference/builder/"><code>Dockerfile</code></a>. For example, considering the following <code>Dockerfile</code>:</p><pre class="prettyprint">FROM elasticsearch:2

RUN /usr/share/elasticsearch/bin/plugin install --batch cloud-aws</pre><p>This <code>Dockerfile</code> starts with the Elasticsearch image provided by maintainers at the Docker hub and runs a simple <code>plugin install</code> command. This image can then be built and referenced by a tag:</p><pre class="prettyprint">$ docker build -t elasticsearch-aws .</pre><p>When run in the same directory as the <code>Dockerfile</code>, the image name <code>elasticsearch-aws</code> is built and can now be referenced in future Docker commands when starting new containers with behavior inherited <code>FROM</code> the original.</p><h3 id="more-complex-plugins">More Complex Plugins</h3><p>Some plugins may require the presence of additional files (such as certificates when using <a href="https://www.elastic.co/what-is/elastic-stack-security">Shield</a>) for certain features. The aforementioned technique of building a custom image can handle the installation of these plugins, but managing configuration is a task better left to a different approach. This helps keep images generic for deployment re-use and maintains tighter control over secrets.</p><p><strong>Note</strong>: Some commercial plugins require the presence of a license. In the following examples, we simply rely on the temporary trial license present by default. When deploying in production, license management is performed through the Elasticsearch REST API, which stores the license in Elasticsearch's data path. As long as your data is persisted appropriately through a volume mount or otherwise, your license will be saved within your cluster.</p><h4 id="example-shield">Example: Shield</h4><p>As outlined in the <a href="https://www.elastic.co/guide/en/shield/current/installing-shield.html">Shield installation documentation</a>, installing the <code>license</code> and <code>shield</code> plugins is a prerequisite, which we can achieve by using the previous strategy to build a derived image:</p><pre class="prettyprint">FROM elasticsearch:2

RUN /usr/share/elasticsearch/bin/plugin install --batch license
RUN /usr/share/elasticsearch/bin/plugin install --batch shield</pre><p>Then build the image to use for future steps:</p><pre class="prettyprint">$ docker build -t elasticsearch-shield .</pre><p>At this point, if we volume mount a config directory into the container, Shield will pick up our settings. As an example configuration, consider the following directory structure:</p><pre class="prettyprint">$ tree config
config
├── elasticsearch.yml
├── logging.yml
├── scripts
└── shield
    ├── roles.yml
    ├── users
    └── users_roles

2 directories, 4 files</pre><p>The <code>logging.yml</code> contains a default logging configuration. In <code>elasticsearch.yml</code>, binding to the wildcard <code>0.0.0.0</code> ensures we can reach the container:</p><pre class="prettyprint">$ cat elasticsearch.yml
network.host: 0.0.0.0</pre><p>For the Shield configuration, we have defined a single role, admin, along with a user called "example" with a password of "password" and added it to the admin role (you'll obviously want a more secure configuration than this!):</p><pre class="prettyprint">$ cat shield/roles.yml
admin:
  cluster: all
  indices:
    '*':
    privileges: all
$ cat shield/users
example:$2a$10$ppZqjFEXgVE3yT/yQPsp4etGMdF4.RFCS9OOGwZGAp0l3lPh4/ALC
$ cat shield/users_roles
admin:example</pre><p><strong>Note</strong>: In this example we are using a password hash generated using the <code>esusers</code> Shield utility.</p><p>We then start the container, passing in the volume for our configuration and exposing the REST port:</p><pre class="prettyprint">$ docker run -d -p 9200:9200 -v "$PWD/config":/usr/share/elasticsearch/config elasticsearch-shield</pre><p>Elasticsearch should deny unauthenticated requests and permit access to the credentials used earlier (in this example it is assumed that Docker is exposing ports on the localhost):</p><pre class="prettyprint">$ curl -I -XGET -k https://localhost:9200/_cluster/health
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Basic realm="shield"
Content-Type: application/json; charset=UTF-8
Content-Length: 389
$ curl -I -XGET -k -u example:password https://localhost:9200/_cluster/health
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 389</pre><h5 id="adding-ssltls">Adding SSL/TLS</h5><p>Like mounting Shield configuration files, SSL and TLS certificates can be similarly managed. Most of the steps outlined in the <a href="https://www.elastic.co/guide/en/shield/current/ssl-tls.html">Shield SSL/TLS guide</a> can be followed, bearing in mind that the <code>CONFIG_DIR</code> directory is the path that we will be mounting into the container at runtime. The full extent of CA and certificate management is outside the scope of this tutorial, so we will assume here that you are using a correctly configured Java keystore file, referred to here as <code>node01.jks</code>.</p><p>Exposing the keystore file is simply a matter of including it within the configuration directory that is mounted into the container.</p><pre class="prettyprint">$ tree config
config
├── elasticsearch.yml
├── logging.yml
├── node01.jks
├── scripts
└── shield
    ├── roles.yml
    ├── users
    └── users_roles
$ file config/node01.jks
config/node01.jks: Java KeyStore</pre><p>Following the Shield user guide, we enable transport encryption:</p><pre class="prettyprint">$ cat config/elasticsearch.yml
network.host: 0.0.0.0
shield.ssl.keystore.path: /usr/share/elasticsearch/config/node01.jks
shield.ssl.keystore.password: password
shield.transport.ssl: true
shield.http.ssl: true</pre><p><strong>Note</strong>: In production, you may want tighter control over your keystore - in this example, we only locked the keystore with a generic password.</p><p>With the keystore file in place and SSL enabled, we can start the node and issue requests over HTTPS (in this case, passing <code>-k</code> to curl to bypass a self-signed certificate):</p><pre class="prettyprint">$ docker run -d -p 9200:9200 -v "$PWD/config":/usr/share/elasticsearch/config elasticsearch-shield
87e51d000cc11d63fbedb8a61d58ab1723f4a598b13614272a3b9d7f36a7b223
$ curl -I -XGET -k -u example:password https://localhost:9200/_cluster/health
HTTP/1.1 200 OK
Content-Type: text/plain; charset=UTF-8
Content-Length: 0</pre><p>When it comes time to distribute keystore files across the cluster, they could potentially be managed by a <a href="https://forge.puppetlabs.com/puppetlabs/java_ks">configuration management module</a> or a similar approach.</p><p><strong>Note</strong>: Pay close attention to the appropriate options to pass to the <code>-ext</code> option when generating certificates for your nodes that will run within Docker. DNS names and IP addresses should correctly reflect the hostname or IP address that nodes will use to communicate with one another.</p><h3 id="summary">Summary</h3><p>Although we've given a few concrete examples in this blog post, every deployment is different, and you should tailor your setup according to whatever promotes reliability, repeatability, and security in your environment. Generally speaking, following these guidelines should aid in a good plugin management scheme:</p><ul>
<li>Define your most generic steps in Docker images. By managing plugins early then running containers from those images, you can avoid re-running the same installation commands over and over.</li>
<li>Maintain persistent data within volume mounts. Storing your index data and plugin configuration separately from the container image ensures that your data is controlled and containers remain ephemeral.</li>
<li>Test! Before deploying any production infrastructure, ensure that Elasticsearch behaves as expected within Docker, especially in regards to network communication (including unicast and IP address binding behavior), JVM resource allocation, and plugin functionality.</li>
</ul>

elasticsearch-docker-plugin-management-full-bleed.jpg

elasticsearch-docker-plugin-management-thumbnail.jpg

Effective Elasticsearch Plugin Management with Docker

Unlike the usual Elasticsearch API endpoints, the cat API (as its name would imply) is especially suited for command-line tools.

<html><head></head><body><p>One of the most useful utilities for investigating Elasticsearch from the command line is the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/cat.html">_cat API</a>. Whereas the usual Elasticsearch API endpoints are ideal for consuming JSON from within a fully-fledged programming language, the cat API (as its name would imply) is especially suited for command-line tools.</p><p>In <a href="https://www.elastic.co/blog/introducing-cat-api">previous posts</a> we&apos;ve explored some of the different endpoints for the API. We can build upon that and the existing plethora of command-line utilities to build even more useful patterns that we can combine for simple (and effective) monitoring and debugging use cases.</p><h3 id="a-primer-for-your-cat">A Primer for your Cat</h3><p>A basic familiarity with the cat API is helpful before reading on. In particular:</p><ul>
<li>The <code>h</code> query string parameter allows us to ask the API to only return certain fields that we&apos;re interested in (i.e., the <strong>h</strong>eader)</li>
<li>Note that the <code>-s</code> argument is used in curl - this is to enable silent output, otherwise extraneous HTTP transfer data may get into the pipeline when chaining commands together.</li>
</ul><h3 id="diving-into-the-heap">Diving Into The Heap</h3><p>An oft-asked question is how to debug <code>OutOfMerroryError</code> messages. You&apos;ve taken <a href="https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html">the right measures to configure your heap</a>, but after some time of normal use, heap usage grows again and instability ensues. How can you dig further?</p><p>There are lots of good resources to track this sort of resource utilization: <a href="https://www.elastic.co/products/marvel">Marvel</a> offers commercial monitoring, and there&apos;s many of <a href="http://bigdesk.org/">open</a> <a href="https://github.com/lmenezes/elasticsearch-kopf">source</a> <a href="http://www.elastichq.org/">options</a> as well. However, when you&apos;re debugging a red cluster at the last minute, you need immediate options. What tools can you reach for easily?</p><p>The cat API offers many endpoints, and piping a <code>curl</code> command which retrieves heap metrics into <code>sort</code> can quickly answer the question, &quot;Which node is experiencing the most memory pressure right now?&quot;</p><pre class="prettyprint">$ curl -s &apos;localhost:9200/_cat/nodes?h=host,heap.percent&apos; | sort -r -n -k2
es02  71
es00  60
es01  59</pre><p>We can see that node <code>es02</code> is using 71% of the JVM heap. Following the pipeline:</p><ul>
<li>We ask for the <code>nodes</code> endpoint, querying the hostname and heap percentage in use, then</li>
<li>Pipe to <code>sort</code>, returning the second column (heap percentage in use) by <code>-r</code> (reverse) order to get the highest in use first.</li>
</ul><p>Coupled with other utilities like <code>head</code> and <code>tail</code>, we can find both over- and under-utilitized nodes in very large clusters very quickly.</p><p>This is useful, but we can do more. It would be nice if we could query heap usage at a more granular level in order to determine what, exactly, is using space on our nodes.</p><p>It turns out we can:</p><pre class="prettyprint">$ curl -s &apos;localhost:9200/_cat/nodes?h=name,fm,fcm,sm,qcm,im&amp;v&apos;
name       fm     fcm      sm
es01  781.4mb 675.6mb 734.5mb
es02    1.6gb 681.3mb 892.2mb
es00    1.4gb 620.1mb 899.4mb</pre><p><strong>Note</strong>: These are abbreviated column names for <code>fielddata.memory_size</code> (fm), <code>filter_cache.memory_size</code> (fcm), and <code>segments.memory</code> (sm). Other fields exist as well, consult <code>curl -s &apos;localhost:9200/_cat/nodes?help&apos; | grep -i memory</code> for additional information.</p><p>From this we can see that <code>fielddata.memory_size</code> is consuming a fairly large part of our node&apos;s memory. Armed with this knowledge, mitigations such as increased use of doc_values can aid in shrinking that aspect of heap usage.</p><h3 id="gazing-into-the-thread-pool">Gazing Into The Thread Pool</h3><p>Many Elasticsearch operations take place in thread pools, which are useful when inspecting what your cluster is busy doing. During peak times, the thread pool can be a useful reflection of what operations (searching, indexing, etc.) are keeping machines busy.</p><p>The cat API is useful here, too. By default it returns common thread pools&apos; active, queued, and rejected pools, which can often help pinpoint requests that are backing up into queued pools under heavy load. Consider this generic output:</p><pre class="prettyprint">$ curl -s &apos;localhost:9200/_cat/thread_pool&apos;
es03  10.xxx.xx.xxx 0 0 0 0 0 0 1 0 0
elk00 10.xx.xxx.xxx 0 0 0 0 0 0 1 0 0
es00  10.xx.xx.xxx  0 0 0 0 0 0 0 0 0</pre><p><strong>Note</strong>: the table headers are omitted here, but the numbers following node IPs are the <code>active</code>, <code>queue</code>, and <code>rejected</code> pools for the <code>bulk</code>, <code>index</code>, and <code>search</code> thread pools, respectively.</p><p>This cluster is serving a single search request, which isn&apos;t terribly exciting. However, what if your cluster is having problems and you need to closely watch operations? A <code>watch</code> command can help here:</p><pre class="prettyprint">$ watch &apos;curl -s localhost:9200/_cat/thread_pool | sort -n -k1&apos;</pre><p><code>watch</code> executes the command every 2 seconds by default. We sort on the first column to keep ordering consistent, and <code>watch</code> usefully highlights field values as they change so we can keep a close eye on thread pools as they change, so it&apos;s easy to spot problems such as a deluge of search requests getting queued if users are hitting the cluster hard.</p><h3 id="diffing-indices-for-fun-and-profit">Diffing Indices for Fun and Profit</h3><p>Another common use case is migrating data from one cluster to another: there are several ways to do this including snapshots and utilities like logstash. With large datasets, this can take quite some time, so how can we gain visibility into the process?</p><p>The cat API offers simple endpoints for index metrics through <code>_cat/indices</code> which includes information such as index disk usage and document count per index.</p><p>Given a scenario in which we&apos;re streaming documents from one index to another on a different cluster, we can perform some command-line gymnastics to watch a diff between index document counts. Consider the following command:</p><pre class="prettyprint">$ join &lt;(curl -s localhost:9200/_cat/indices | awk &apos;$3 ~ /foo-index/ { print $3 &quot; &quot; $6; }&apos;) &lt;(curl -s otherhost:9200/_cat/indices | awk &apos;$3 ~ /foo-index/ { print $3 &quot; &quot; $6; }&apos;) | awk &apos;{ print $1 &quot; &quot; $2 - $3; }&apos;
foo-index -231700</pre><p>This command makes use of bash process substitution in order to create temporary file descriptors from the output of commands between the <code>&lt;()</code> parenthesis. We use <code>awk</code> to find the index of interest, <code>foo-index</code>, and print the index name and document count for the local index and the remote one we&apos;re streaming to. Using the <code>join</code> command on the host field merges them into one line, and we pipe the results through another <code>awk</code> in order to calculate a difference between the two indices.</p><p>The results of the example command indicate that there&apos;s a disparity in document count that should converge as the stream nears completion and finishes replicating the data.</p><p>By placing this command into either a <code>watch</code> command or bash <code>for</code> loop, we can quickly whip up a small utility to watch the progress of our index import.</p><h3 id="going-further">Going Further</h3><p>These are just a few examples of how everyday command line utilities can be paired with the cat API to make lives easier for administering Elasticsearch. There are plenty of other potential applications to consider - for example:</p><ul>
<li>Use the <code>_cat/recovery</code> API to watch the recovery progress of a cluster in a shell loop</li>
<li>Consume metrics like thread pools or node statistics in tools like ganglia or nagios for simple alerting on Elasticsearch health</li>
<li>Use <code>_cat/shards</code> when in a pinch to find exactly which shard is causing your cluster to go into a red or yellow state</li>
</ul><p>Good luck, and may the cat be with you!</p></body></html>

Articles By Tyler Langlois

Pods, Tokens, and a Little Glue: Integrating Kubernetes and Vault in Elastic Infrastructure

Puppet State of the Union: May 2017

Effective Elasticsearch Plugin Management with Docker

Elasticsearch Command Line Debugging With The _cat API

Products & Solutions

Company

Resources

SaaS

ORCHESTRATION

Articles By Tyler Langlois

Pods, Tokens, and a Little Glue: Integrating Kubernetes and Vault in Elastic Infrastructure

Puppet State of the Union: May 2017

Effective Elasticsearch Plugin Management with Docker

Elasticsearch Command Line Debugging With The _cat API