Learn how to keep service and container credentials safe during autodiscovery with the new Metricbeat Kubernetes secrets keystore.

<p>In the world of containers and Kubernetes, <a href="/observability">observability</a> is crucial. Cluster administrators need visibility into the infrastructure and cluster operators need to know the status of their workloads at any given time. And in both cases, they need observability into moving objects.
</p>
<p>This is where <a href="/beats/metricbeat">Metricbeat</a> and its autodiscover feature do the hard part for you. Operators can set up fairly complex monitoring scenarios with a self-service approach following the principle of least privilege at the cluster level.
</p>
<p>The self-service approach is great, but what if the service I want to monitor is behind authentication? In this blog, we’ll take a look at how easy it is to configure observability for authenticated services with Elastic.
</p>
<h2>How autodiscover works</h2>
<p>Applications running on containers and pods become moving targets to the monitoring system. Autodiscover tracks changes at the container and pod level and dynamically adjusts the monitoring configuration using two strategies:
</p>
<ol>
	<li><strong>Template-based autodiscover:</strong> By defining configuration templates, the autodiscover subsystem can use them to monitor services as they start running.</li>
	<li><strong>Hints-based autodiscover:</strong> The <code>hints</code> system looks for hints in Kubernetes pod annotations or Docker labels which have the prefix <code>co.elastic.metrics</code>. As soon as the container starts, Metricbeat will check if it contains any hints and launch the proper configuration for it. Hints tell Metricbeat how to get metrics for the given container.</li>
</ol>
<p>Let’s look at an example with <em>template-based autodiscover</em>. Here is a configuration template for automatically detecting Redis pods and starting to collect metrics using the respective Redis Metricbeat module:
</p>
<pre class="prettyprint">metricbeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - condition:
            contains:
              kubernetes.labels.app: "redis"
          config:
            - module: redis
              metricsets: ["info"]
              hosts: "${data.host}:6379"
</pre>
<p>Defining autodiscover templates at Metricbeat’s configuration level is a valid approach, but users must be granted the permission to globally configure Metricbeat. In addition, a restart of Metricbeat is required every time the autodiscover template is updated. 
With hints-based autodiscover, we let workload owners enable monitoring for their services on the fly, avoiding over-granted permissions without restarting the Beats. The following example configures the same Redis monitoring from above using hints:
</p>
<pre class="prettyprint">metricbeat.autodiscover:
  providers:
    - type: kubernetes
      hints.enabled: true
</pre>
<p>In this case, Redis pods need to be <a href="https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/">annotated</a> accordingly:
</p>
<pre class="prettyprint">apiVersion: v1
kind: Pod
metadata:
  name: annotations-demo
  annotations:
    co.elastic.metrics/module: redis
    co.elastic.metrics/metricsets: info
    co.elastic.metrics/hosts: '${data.host}:6379'
</pre>
<p><strong></strong>
</p>
<h2><strong>Using passwords with autodiscover</strong></h2>
<p>In many cases, the target service requires password authentication in order to provide monitoring metrics. For instance, Redis can be configured to be password protected, meaning we have to provide a password to the Redis module in Metricbeat in order to collect metrics successfully. With template-based autodiscover, we would configure the template like this:
</p>
<pre class="prettyprint">metricbeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - condition:
            contains:
              kubernetes.labels.app: "redis"
          config:
            - module: redis
              metricsets: ["info"]
              hosts: "${data.host}:6379"
              password: 'myred1sp@ss'
</pre>
<p>Similarly, with hints-based autodiscover, we would annotate the pod with a string containing the password:
</p>
<pre class="prettyprint">apiVersion: v1
kind: Pod
metadata:
  name: annotations-demo
annotations:
  co.elastic.metrics/module: redis
  co.elastic.metrics/metricsets: info
  co.elastic.metrics/hosts: '${data.host}:6379'
  co.elastic.metrics/password: 'myred1sp@ss'
</pre>
<p>While we can use these setups for development or testing purposes, storing passwords in plain text poses a serious risk and should be avoided. We could put the password in an environment variable and let Metricbeat interpolate the manifest, but this wouldn’t make our security posture any better. Let’s see a preferable alternative.
</p> <strong>
<h3>The Metricbeat keystore</h3></strong>
<p>Metricbeat provides a feature called <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.8/keystore.html">keystore</a> that can be used to securely store sensitive information and reference it in the Metricbeat configuration as a secret.
Let’s see how we can leverage this feature in our autodiscover template configurations. First, we create the keystore and add the password for Redis under the name 
	<code>REDIS_PASSWORD</code>:
</p>
<pre class="prettyprint">metricbeat keystore create
metricbeat keystore add REDIS_PASSWORD
</pre>
<p>And provide “myred1sp@ss” when prompted.
</p>
<p>Then the configuration template will be able to reference the string <code>REDIS_PASSWORD</code> to access the actual password:
</p>
<pre class="prettyprint">metricbeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - condition:
            contains:
              kubernetes.labels.app: "redis"
          config:
            - module: redis
              metricsets: ["info"]
              hosts: "${data.host}:6379"
              password: "${REDIS_PASSWORD}"
</pre>
<p>And that’s all! We managed to avoid using plain text passwords or defining ENV variables, and instead we leveraged the Metricbeat keystore to securely store the password for the target Redis pod.
</p>
<p>Unfortunately, we can't use this strategy with hints-based autodiscovery — it wouldn’t be safe to expose the keystore to the hints mechanism because it could be a target for an attacker trying to exploit the whole keystore just using hints annotations. This is why we introduced a completely new Keystore, the <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.9/configuration-autodiscover.html#_kubernetes_secrets">Kubernetes secrets keystore</a>.
</p> <strong>
<h3>The Kubernetes Secrets Keystore</h3></strong>
<p>With the Kubernetes Secrets Keystore we allow users to consume <a href="https://kubernetes.io/docs/concepts/configuration/secret/">Kubernetes secrets</a> directly in autodiscover configuration. It can be used for both template-based and hints-based autodiscover.
</p>
<p>Users can refer a Kubernetes Secret with the following pattern:
</p>
<pre class="prettyprint">${kubernetes.&lt;namespace&gt;.&lt;secret_name&gt;.&lt;value&gt;}
</pre>
<p>For instance, <code>${kubernetes.default.redisPass.value}</code> where <code>kubernetes.default.redisPass.value</code> specifies a key stored as a Kubernetes secret with the following:
</p>
<ol>
	<li>Kubernetes Namespace: <code>default</code></li>
	<li>Secret Name: <code>redisPass</code></li>
	<li>Secret Data Key: <code>value</code></li>
</ol>
<p>Note that pods can only consume secrets that belong to the same Kubernetes namespace. For example if our Redis pod is running in the <code>staging</code> namespace, it cannot access a secret created in the <code>testing</code> namespace.
</p>
<p>Let's see how we could leverage the new Keystore backend. First we need to create the secret:
</p>
<pre class="prettyprint">cat &lt;&lt; EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: redisPass
type: Opaque
data:
  value: $(echo -n "myred1sp@ss" | base64)
EOF
</pre>
<p>Now we can consume the <code>redisPass</code> secret from within our hint based autodiscover configuration:
</p>
<pre class="prettyprint">apiVersion: v1
kind: Pod
metadata:
  name: annotations-demo
  app: redis
annotations:
  co.elastic.metrics/module: redis
  co.elastic.metrics/metricsets: info
  co.elastic.metrics/hosts: '${data.host}:6379'
  co.elastic.metrics/password: '${kubernetes.default.redisPass.value}'
</pre>
<p>Kubernetes secrets keystore also works with autodiscover templates and we can access Kuberentes secrets from our static configurations, too:
</p>
<pre class="prettyprint">metricbeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - condition:
            contains:
              kubernetes.labels.app: "redis"
          config:
            - module: redis
              metricsets: ["info", "keyspace"]
              hosts: "${data.host}:6379"
              password: "${kubernetes.default.redisPass.value}"
</pre> <strong>
<h2>Conclusions</h2></strong>
<p>If you find the idea of securing sensitive configurations in your Kubernetes environment of utmost importance, then you will love Metricbeat keystore and Kubernetes secrets keystore — they allow you to securely connect without leaking plain text passwords.
</p>
<p><a href="/downloads">Download Metricbeat</a> and start storing your passwords securely for your autodiscovered workloads without worrying about leaking sensitive data. And make sure your data stays safe by using Elasticsearch Service on <a href="/cloud">Elastic Cloud</a>, where security comes standard. <a href="/elasticsearch/service">Start a free trial today</a>.
</p>If you have any questions, we are always happy to help on the Discuss <a href="https://discuss.elastic.co/">forums</a>.

blog-banner-release-observability.png

blog-thumb-kubernetes-observability.png

Securely manage credentials while monitoring Kubernetes workloads with autodiscovery

Capture and analyze NATS logs and metrics in Elasticsearch using the new NATS modules, contributed, and battle-tested by real-world NATS users in production.

<p>In a world where stateless applications are optimized to run blazingly fast, message exchanging cannot be allowed to affect their performance. Having the ability to <a href="https://hackernoon.com/want-a-super-fast-messaging-system-c19c9f8cf560">process millions of messages per second</a>, NATS is the sprinter of messaging systems.
</p><p>While benchmarks are good indicators for choosing a tool, there is no way to confirm its value without monitoring its performance in production. As a team working together in the telecommunications sector, we're building NFV solutions on top of Kubernetes microservices, and we're using NATS as a messaging system to ensure the resiliency of the critical paths. These include the communication between the software components running on the clients’ data centers and our centralized platforms. It has allowed us to achieve high rates of message exchanging with great reliability. In this regard, we need to assure its health by providing actionable visibility in real-time. Given the fact that we use <a href="https://docs.fluentd.org/v/0.12/articles/kubernetes-fluentd">EFK</a> to monitor our <a href="https://github.com/fluent/fluentd-kubernetes-daemonset/blob/master/fluentd-daemonset-elasticsearch-rbac.yaml">Kubernetes</a> stack, monitoring of NATS had to be achieved by the exact same way.
</p><p>Everything started with a goal: Ship NATS monitoring data to Elasticsearch.&nbsp;
</p><p>What we achieved: Extending Beats, by providing NATS modules for both <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.0/metricbeat-module-nats.html">Metricbeat</a> and <a href="https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-module-nats.html">Filebeat</a> shippers. Get on board to find out how!
</p><h2>Motivated by an internal hackathon</h2><p>We wanted results and we wanted them fast, but our internal backlog was full of requirements forcing our time to be very limited. Luckily, our company was organizing its annual hackathon. Without hesitation, we grabbed the opportunity to work on creating our own NATS Beat by leveraging the <a href="https://github.com/nfvsap/natsbeat">Beats extension mechanism</a>. After two days of effort our NATS Beat succeeded into becoming part of the official list of <a href="https://www.elastic.co/guide/en/beats/libbeat/current/community-beats.html">community Beats</a>. And that was only the beginning. Intrigued by the idea, the Elastic Beats team encouraged us to put our code into the core Beats project. With their support, we managed to develop a Metricbeat module for metrics collection, and a Filebeat module, for log aggregation. Today, NATS monitoring is deeply integrated into the core of Beats upstream.
</p><h2>How to monitor NATS</h2><p>NATS is very helpful when it comes to providing <a href="https://nats.io/documentation/managing_the_server/monitoring/">monitoring data</a>. If requested, NATS server can serve such data in JSON format by exposing four different HTTP endpoints:
</p><ul>
	<li>/varz: reports general statistics such as CPU utilization, memory consumption, etc.
	</li>
	<li>/connz: reports detailed information on client connections
	</li>
	<li>/routez: reports information on routes between servers of the NATS cluster
	</li>
	<li>/subz: reports detailed information about current subscriptions and the routing data structure</li>
</ul><p>By querying these endpoints in a periodic way, user can understand a lot about the running state of his messaging queue. And so we did with the brand new NATS module of Metricbeat.
</p><p>But NATS can provide more. Another interesting place to find useful information regarding monitoring is the server’s logs. If requested, NATS server can provide <a href="https://nats.io/documentation/server/gnatsd-logging/">TRACE level logs</a> for every message received or transmitted. These log entries can be proven really valuable if handled properly. Our NATS module for Filebeat did exactly that. Every log is parsed and all its meaningful bits are squeezed out of it.
</p><h2>NATS as a Metricbeat module</h2><p>As it happens with any Metricbeat module, NATS needs to be activated and configured accordingly in order to fetch the data from the monitoring endpoints. Upon configured properly, it fetches data from the endpoints, curates them accordingly and ships them to Elasticsearch.
</p><h3>Configuration
</h3><p>The module can be configured as simple as this:
</p><pre class="prettyprint">metricbeat.modules:
- module: nats
  metricsets: ["connections", "routes", "stats", "subscriptions"]
  period: 10s
  hosts: ["localhost:8222"]
  stats.metrics_path: "/varz"
  connections.metrics_path: "/connz"
  routes.metrics_path: "/routez"
  subscriptions.metrics_path: "/subsz"
  enabled: true
</pre><p>The above configuration provides the location of the NATS server(s) we want to monitor, the query interval for metrics, which metrics to fetch and the URIs under which they can be accessed.
</p><h3>Let the metrics speak</h3><p>Fetching the data is one thing. Sending them to Elasticsearch is another. But what is the value of storing them if you cannot visualize them? With that in mind, this module ships with a pre-built dashboard which can be loaded automatically from Beats and give a good starting point to the user.
</p><p>Let’s take a sneak peak:
</p><p><img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/blt28de268fcb47839e/5cabacdbef14588e0283421b/download" data-sys-asset-uid="blt28de268fcb47839e" alt="nats-let-metrics-speak.png">
</p><p>Users&nbsp;can detect slow consumers which may affect NATS performance, monitor how NATS cache is performing, or even realize the utilization of resources made by our NATS servers. Additionally, users can correlate the stored data according to their needs and extend the dashboard. It is in their&nbsp;hands!
</p><h2>NATS as a Filebeat module</h2><p>After making Metricbeat module a reality, a brilliant idea was put on the table: Is there something we could do for NATS server logs? The answer was standing in front of us… Filebeat!
</p><p>And so we did. Filebeat is an agent dedicated to fetching logs, parsing them with a wide range of text processors, and creating docs out of them in an Elasticsearch index. But to go far you need to start simple. Our very first version of NATS module was just extracting log timestamp, level, and message, and putting them into the datastore. The second version though is taking it to the next level. After investigating all log patterns we processed the log messages even more. The outcome is far more interesting, it can be visualized and it can offer a much deeper understanding of the NATS cluster’s state
</p><h3>Configuration</h3><p>Filebeat module can be configured as the Metricbeat module by adding the following lines to the configuration file:
</p><pre class="prettyprint">filebeat.modules:
- module: nats  
  log:
    enabled: true
    var.paths:
      - /var/log/nats/nats.log*
</pre><p>As it can be easily observed, apart from enabling the module, the only configurable field a user has to set is a list of log path regular expressions. Given the location of the logs, NATS module has everything it needs to start scraping data from them.
</p><h3>Logs as a Dashboard (LAAD)</h3><p>As the title implies, NATS module gives the opportunity to visualize the logs that the user's NATS cluster is producing. With the pre-built dashboard accompanying the module and the log trace option enabled to NATS server, user can come up with a dashboard as interesting as the following:
</p><p><img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/blt3a5a663088e662c6/5cabacaa051f0ba57b53c4d5/download" data-sys-asset-uid="blt3a5a663088e662c6" alt="nats-logs-as-a-dashboard-filebeat-1.png">
</p><p>Filebeat dashboard can provide various forms of data and statistics. Users can observe the distribution of messages by its type, the heat received by the NATS subjects, and the type of logs produced in a timeline. Furthermore, users have a good indicator on messages’ traffic, both incoming and outgoing, as well as the IP range distribution of the clients. The most interesting part: All of the above can happen with a click of a button!
</p><h3>At the end of the day</h3><p>All in all, time invested on this effort was definitely worth it. Not only did we solve our problem of NATS monitoring, which could be achieved with only the community Beat, but we also gained a lot of experience on the way. We had the chance to work with excellent engineers from Elastic and we felt the joy of giving back to the community. So it goes without saying that we enjoyed that and we would suggest such kind of contributions to everyone who faces similar situations.
</p><p>Are you a NATS user?&nbsp; Download <a href="https://www.elastic.co/downloads/beats/metricbeat">Metricbeat</a> 7.0 and build <a href="https://www.elastic.co/downloads/beats/filebeat">Filebeat</a>&nbsp;master branch, enable the NATS modules, and start looking under the covers to see what you can discover about your NATS environment.&nbsp;
</p><hr><div>
	<p><img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/bltdd3ec6d6a9397204/5cabb848ef14588e0283427f/download" data-sys-asset-uid="bltdd3ec6d6a9397204" alt="stamatis-katsaounis.png" width="100" style="width: 100px; float: left; margin: 10px;"><strong><em>Stamatis Katsaounis&nbsp;</em></strong><em>is a software engineer, focused mainly on the Cloud. He is an active member of the OPNFV community with contributions to various OPNFV projects. He experiments with OpenStack and Kubernetes deployments and is a Linux enthusiast. He speaks C, Bash and Python. Lately, he is a frequent visitor of Go playground</em>.
	</p>
</div><div>
	<p><img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/bltb63f1699387ac50a/5cabb8978f70e39878691c08/download" data-sys-asset-uid="bltb63f1699387ac50a" alt="michael-katsoulis.jpg" width="100" style="width: 100px; float: left; margin: 10px;"><strong><em>Michael Katsoulis</em></strong><em> is a cloud software engineer, backed with working knowledge in telecommunications systems, currently focusing in SDN/NFV area. Leveraging Linux power and open source tools he always tries to combine work in internal projects with open source contributions.</em>
	</p>
</div><div>
	<p><br><strong><em>Christos Markou</em></strong><em> is a systems and software engineer that's passionate about distributed systems and cloud technologies. He has hands-on experience in web services, cloud computing and large system architecture, and is an open source enthusiast with various contributions in high class projects.</em>
	</p>
</div>

cloud-computing-nats-full-bleed.jpg

nats-logs-as-a-dashboard-filebeat-thumbnail.png

Monitoring a NATS messaging system with Beats

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Christos Markou

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Communication Preferences - ONLY for /communication-preferences

<p>提交即表示您确认您已阅读并同意我们的<a href="/cn/legal/elastic-cloud-account-terms" target="_self">服务条款</a>，并且您同意通过您提供的上述联系方式接收<a href="/cn/legal/privacy-statement#how-we-use-the-information" target="_self">所选的通信</a>。如需了解更多信息，或想随时取消订阅，请参阅 <a href="/cn/legal/privacy-statement/" target="_self">Elastic 的隐私声明</a>。</p>

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Serverless GDPR Text

<p>提交即表示您确认您已阅读并同意我们的<a href="/cn/legal/serverless-preview-terms" target="_self">服务条款</a>，并且 Elastic 可以使用您提供的上述详细信息与您联系，以向您介绍我们相关的<a href="/cn/legal/privacy-statement#how-we-use-the-information" target="_self">产品和服务</a>。如需了解更多信息，或想随时取消订阅，请参阅 <a href="/cn/legal/privacy-statement/" target="_self">Elastic 的隐私声明</a>。</p>

<p>By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.</p>

Cloud Trial GDPR Text

<p>提交即表示您同意 <a href="/cn/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud 服务条款</a>。我们将会按照 <a href="/cn/legal/privacy-statement">Elastic 的隐私声明</a>处理您的个人数据。
</p>

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Newsletter GDPR Text

<p>提交即表示您确认您已阅读并同意我们的<a href="/cn/legal/terms-of-use" target="_self">服务条款</a>，并且 Elastic 可以使用您提供的上述详细信息与您联系，以向您介绍我们相关的<a href="/cn/legal/privacy-statement#how-we-use-the-information" target="_self">产品和服务</a>。如需了解更多信息，或想随时取消订阅，请参阅 <a href="/cn/legal/privacy-statement/" target="_self">Elastic 的隐私声明</a>。</p>

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

本页内容尚不支持所选语言。Elastic 正在不断努力，以实现对多种语言内容的支持。感谢您在此期间给予的耐心与陪伴！

Author

Articles by

Learn more

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

查看本次网络研讨会在我的时区的开始时间

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

无法实时参加？注册后我们会给您发送录像。您还将收到相关内容的电子邮件。

Author

Articles by Christos Markou

Securely manage credentials while monitoring Kubernetes workloads with autodiscovery

Monitoring a NATS messaging system with Beats

Sign up for Elastic Cloud free trial