Prometheus plays a significant role in observability, but it has its limitations. Learn how to use the Elastic Stack with Prometheus to boost scalability and durability, as well as open the door for more use cases.

<p>Prometheus plays a significant role in the observability area. An increasing number of applications use Prometheus exporters to expose performance and monitoring data, which is later scraped by a Prometheus server.
</p>
<p>However, when it comes to storage, Prometheus faces some limitations in its scalability and durability since its local storage is limited by single nodes. In order to tackle this, users should either go with clustered storage in Prometheus itself, or use the Prometheus interfaces that allow integrating with remote storage systems. In this blog, we're going to look at how simple and effective it is to integrate Prometheus with Elasticsearch for long-term storage.
</p>
<p>Elasticsearch is a powerful, scalable, and efficient storage system that can store various types of data (text, numerics, geo, structured, unstructured, etc.) in the long term. The scaling capabilities and search performance of Elasticsearch make it a prime option when it comes to storing monitoring data. Plus, combining Elasticsearch with Kibana gives users dynamic insights into their data.
</p>
<h2>Integrating Prometheus with Elasticsearch for long-term storage</h2>
<p>Prometheus integrates with remote storage systems in two ways:
</p>
<ul>
	<li>Prometheus can write samples that it ingests to a remote URL in a standardized format</li>
	<li>Prometheus can read (back) sample data from a remote URL in a standardized format</li>
</ul>
<p>The read and write protocols both use a snappy-compressed protocol buffer encoding over HTTP. For details on the request and response messages, see the remote storage <a href="https://github.com/prometheus/prometheus/blob/master/prompb/remote.proto">protocol buffer definitions</a>.
</p>
<p>In order to use Elasticsearch as a remote storage system for Prometheus, the user can employ the official Metricbeat module and more specifically the <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.7/metricbeat-metricset-prometheus-remote_write.html">remote_write metricset</a>. In this, the user can configure Metricbeat like:
</p>
<pre class="prettyprint"># Metrics sent by a Prometheus server using remote_write option 
- module: prometheus 
  metricsets: ["remote_write"] 
  host: "localhost" 
  port: "9201"
</pre>
<p>And Prometheus:
</p>
<pre class="prettyprint">remote_write: 
  - url: "http://localhost:9201/write"
</pre>
<p>In addition, the user can set up secure settings for the server using TLS/SSL.
</p>
<p>With this type of configuration, Metricbeat, on startup, will start an http server that will wait for metrics from a Prometheus server. When metrics are received they will be processed accordingly by Metricbeat and will be efficiently stored in Elasticsearch. Then the user can retrieve and analyze the metrics using Kibana's visualizations and dashboards.
</p>
<p>It’s also worth mentioning that Metricbeat, Elasticsearch, and Kibana can run out of the box in Kubernetes environments. This can be achieved by using <a href="https://www.elastic.co/guide/en/cloud-on-k8s/current/index.html">Elastic Cloud on Kubernetes (ECK)</a>, the official Elasticsearch and Kibana operator, along with the predefined Metricbeat <a href="https://github.com/elastic/cloud-on-k8s/tree/1.0/config/recipes/beats">manifests</a>. Putting all these together gives the user the opportunity to quickly set up an Elastic-based remote storage integration for their Prometheus servers that run on Kubernetes.
</p>
<p>As a result, users will benefit from doubling down on observability for their cloud-native applications and workloads. By creating a long-term storage solution for their data with Elasticsearch, they’ll also have the opportunity to efficiently query them back and further analyze them at any time. Moreover, users can leverage Kibana to build visualizations.
</p>
<p>Below is an example of how we can build a dashboard visualizing Prometheus metrics from node exporters and cAdvisor:
</p>
<p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt8e72ab55c29b5702/5ea36f2ddd109257057d68a6/long-term-prometheus-blog-stats.png" data-sys-asset-uid="blt8e72ab55c29b5702" alt="" style="display: block; margin: auto;">
</p><figcaption>Node statistics
</figcaption>
<p>And it goes without saying that we would like to monitor how remote storage statistics evolve by visualizing how many samples have been successfully sent to remote storage or how many have been retried, failed, or dropped. Another significant metric to monitor here is the number of shards, which shows the number of concurrency/parallelism with which Prometheus server sends metrics to the remote storage endpoint.
</p>
<p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt0871b8297a8451f0/5ea36f18ae23d30cd1dfb35c/long-term-prometheus-blog-metrics.png" data-sys-asset-uid="blt0871b8297a8451f0" alt="" style="display: block; margin: auto;">
</p><figcaption>Prometheus remote storage statistics
</figcaption>
<h2>Implement long-term storage for Prometheus today</h2>
<p>Are you already considering a long-term storage solution for your Prometheus metrics? <a href="https://www.elastic.co/downloads">Download</a> Metricbeat 7.7 and start storing your data in Elasticsearch without worrying about scalability and query efficiency. The fastest way to get your cluster deployed is to spin up a <a href="https://www.elastic.co/elasticsearch/service">free trial of Elasticsearch Service</a>. Of course, if you have any questions, remember that we are always happy to help on the Discuss <a href="https://discuss.elastic.co/">forums</a>.
</p>

observability-prometheus-swiss-army-knife-blog-banner.png

observability-prometheus-swiss-army-knife-blog-thumb.png

How to implement Prometheus long-term storage using Elasticsearch

Capture and analyze NATS logs and metrics in Elasticsearch using the new NATS modules, contributed, and battle-tested by real-world NATS users in production.

<p>In a world where stateless applications are optimized to run blazingly fast, message exchanging cannot be allowed to affect their performance. Having the ability to <a href="https://hackernoon.com/want-a-super-fast-messaging-system-c19c9f8cf560">process millions of messages per second</a>, NATS is the sprinter of messaging systems.
</p><p>While benchmarks are good indicators for choosing a tool, there is no way to confirm its value without monitoring its performance in production. As a team working together in the telecommunications sector, we're building NFV solutions on top of Kubernetes microservices, and we're using NATS as a messaging system to ensure the resiliency of the critical paths. These include the communication between the software components running on the clients’ data centers and our centralized platforms. It has allowed us to achieve high rates of message exchanging with great reliability. In this regard, we need to assure its health by providing actionable visibility in real-time. Given the fact that we use <a href="https://docs.fluentd.org/v/0.12/articles/kubernetes-fluentd">EFK</a> to monitor our <a href="https://github.com/fluent/fluentd-kubernetes-daemonset/blob/master/fluentd-daemonset-elasticsearch-rbac.yaml">Kubernetes</a> stack, monitoring of NATS had to be achieved by the exact same way.
</p><p>Everything started with a goal: Ship NATS monitoring data to Elasticsearch.&nbsp;
</p><p>What we achieved: Extending Beats, by providing NATS modules for both <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.0/metricbeat-module-nats.html">Metricbeat</a> and <a href="https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-module-nats.html">Filebeat</a> shippers. Get on board to find out how!
</p><h2>Motivated by an internal hackathon</h2><p>We wanted results and we wanted them fast, but our internal backlog was full of requirements forcing our time to be very limited. Luckily, our company was organizing its annual hackathon. Without hesitation, we grabbed the opportunity to work on creating our own NATS Beat by leveraging the <a href="https://github.com/nfvsap/natsbeat">Beats extension mechanism</a>. After two days of effort our NATS Beat succeeded into becoming part of the official list of <a href="https://www.elastic.co/guide/en/beats/libbeat/current/community-beats.html">community Beats</a>. And that was only the beginning. Intrigued by the idea, the Elastic Beats team encouraged us to put our code into the core Beats project. With their support, we managed to develop a Metricbeat module for metrics collection, and a Filebeat module, for log aggregation. Today, NATS monitoring is deeply integrated into the core of Beats upstream.
</p><h2>How to monitor NATS</h2><p>NATS is very helpful when it comes to providing <a href="https://nats.io/documentation/managing_the_server/monitoring/">monitoring data</a>. If requested, NATS server can serve such data in JSON format by exposing four different HTTP endpoints:
</p><ul>
	<li>/varz: reports general statistics such as CPU utilization, memory consumption, etc.
	</li>
	<li>/connz: reports detailed information on client connections
	</li>
	<li>/routez: reports information on routes between servers of the NATS cluster
	</li>
	<li>/subz: reports detailed information about current subscriptions and the routing data structure</li>
</ul><p>By querying these endpoints in a periodic way, user can understand a lot about the running state of his messaging queue. And so we did with the brand new NATS module of Metricbeat.
</p><p>But NATS can provide more. Another interesting place to find useful information regarding monitoring is the server’s logs. If requested, NATS server can provide <a href="https://nats.io/documentation/server/gnatsd-logging/">TRACE level logs</a> for every message received or transmitted. These log entries can be proven really valuable if handled properly. Our NATS module for Filebeat did exactly that. Every log is parsed and all its meaningful bits are squeezed out of it.
</p><h2>NATS as a Metricbeat module</h2><p>As it happens with any Metricbeat module, NATS needs to be activated and configured accordingly in order to fetch the data from the monitoring endpoints. Upon configured properly, it fetches data from the endpoints, curates them accordingly and ships them to Elasticsearch.
</p><h3>Configuration
</h3><p>The module can be configured as simple as this:
</p><pre class="prettyprint">metricbeat.modules:
- module: nats
  metricsets: ["connections", "routes", "stats", "subscriptions"]
  period: 10s
  hosts: ["localhost:8222"]
  stats.metrics_path: "/varz"
  connections.metrics_path: "/connz"
  routes.metrics_path: "/routez"
  subscriptions.metrics_path: "/subsz"
  enabled: true
</pre><p>The above configuration provides the location of the NATS server(s) we want to monitor, the query interval for metrics, which metrics to fetch and the URIs under which they can be accessed.
</p><h3>Let the metrics speak</h3><p>Fetching the data is one thing. Sending them to Elasticsearch is another. But what is the value of storing them if you cannot visualize them? With that in mind, this module ships with a pre-built dashboard which can be loaded automatically from Beats and give a good starting point to the user.
</p><p>Let’s take a sneak peak:
</p><p><img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/blt28de268fcb47839e/5cabacdbef14588e0283421b/download" data-sys-asset-uid="blt28de268fcb47839e" alt="nats-let-metrics-speak.png">
</p><p>Users&nbsp;can detect slow consumers which may affect NATS performance, monitor how NATS cache is performing, or even realize the utilization of resources made by our NATS servers. Additionally, users can correlate the stored data according to their needs and extend the dashboard. It is in their&nbsp;hands!
</p><h2>NATS as a Filebeat module</h2><p>After making Metricbeat module a reality, a brilliant idea was put on the table: Is there something we could do for NATS server logs? The answer was standing in front of us… Filebeat!
</p><p>And so we did. Filebeat is an agent dedicated to fetching logs, parsing them with a wide range of text processors, and creating docs out of them in an Elasticsearch index. But to go far you need to start simple. Our very first version of NATS module was just extracting log timestamp, level, and message, and putting them into the datastore. The second version though is taking it to the next level. After investigating all log patterns we processed the log messages even more. The outcome is far more interesting, it can be visualized and it can offer a much deeper understanding of the NATS cluster’s state
</p><h3>Configuration</h3><p>Filebeat module can be configured as the Metricbeat module by adding the following lines to the configuration file:
</p><pre class="prettyprint">filebeat.modules:
- module: nats  
  log:
    enabled: true
    var.paths:
      - /var/log/nats/nats.log*
</pre><p>As it can be easily observed, apart from enabling the module, the only configurable field a user has to set is a list of log path regular expressions. Given the location of the logs, NATS module has everything it needs to start scraping data from them.
</p><h3>Logs as a Dashboard (LAAD)</h3><p>As the title implies, NATS module gives the opportunity to visualize the logs that the user's NATS cluster is producing. With the pre-built dashboard accompanying the module and the log trace option enabled to NATS server, user can come up with a dashboard as interesting as the following:
</p><p><img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/blt3a5a663088e662c6/5cabacaa051f0ba57b53c4d5/download" data-sys-asset-uid="blt3a5a663088e662c6" alt="nats-logs-as-a-dashboard-filebeat-1.png">
</p><p>Filebeat dashboard can provide various forms of data and statistics. Users can observe the distribution of messages by its type, the heat received by the NATS subjects, and the type of logs produced in a timeline. Furthermore, users have a good indicator on messages’ traffic, both incoming and outgoing, as well as the IP range distribution of the clients. The most interesting part: All of the above can happen with a click of a button!
</p><h3>At the end of the day</h3><p>All in all, time invested on this effort was definitely worth it. Not only did we solve our problem of NATS monitoring, which could be achieved with only the community Beat, but we also gained a lot of experience on the way. We had the chance to work with excellent engineers from Elastic and we felt the joy of giving back to the community. So it goes without saying that we enjoyed that and we would suggest such kind of contributions to everyone who faces similar situations.
</p><p>Are you a NATS user?&nbsp; Download <a href="https://www.elastic.co/downloads/beats/metricbeat">Metricbeat</a> 7.0 and build <a href="https://www.elastic.co/downloads/beats/filebeat">Filebeat</a>&nbsp;master branch, enable the NATS modules, and start looking under the covers to see what you can discover about your NATS environment.&nbsp;
</p><hr><div>
	<p><img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/bltdd3ec6d6a9397204/5cabb848ef14588e0283427f/download" data-sys-asset-uid="bltdd3ec6d6a9397204" alt="stamatis-katsaounis.png" width="100" style="width: 100px; float: left; margin: 10px;"><strong><em>Stamatis Katsaounis&nbsp;</em></strong><em>is a software engineer, focused mainly on the Cloud. He is an active member of the OPNFV community with contributions to various OPNFV projects. He experiments with OpenStack and Kubernetes deployments and is a Linux enthusiast. He speaks C, Bash and Python. Lately, he is a frequent visitor of Go playground</em>.
	</p>
</div><div>
	<p><img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/bltb63f1699387ac50a/5cabb8978f70e39878691c08/download" data-sys-asset-uid="bltb63f1699387ac50a" alt="michael-katsoulis.jpg" width="100" style="width: 100px; float: left; margin: 10px;"><strong><em>Michael Katsoulis</em></strong><em> is a cloud software engineer, backed with working knowledge in telecommunications systems, currently focusing in SDN/NFV area. Leveraging Linux power and open source tools he always tries to combine work in internal projects with open source contributions.</em>
	</p>
</div><div>
	<p><br><strong><em>Christos Markou</em></strong><em> is a systems and software engineer that's passionate about distributed systems and cloud technologies. He has hands-on experience in web services, cloud computing and large system architecture, and is an open source enthusiast with various contributions in high class projects.</em>
	</p>
</div>

Articles By Christos Markou

How to implement Prometheus long-term storage using Elasticsearch

Monitoring a NATS messaging system with Beats

Products & Solutions

Company

Resources

SaaS

ORCHESTRATION

Articles By Christos Markou

How to implement Prometheus long-term storage using Elasticsearch

Monitoring a NATS messaging system with Beats