Find out how you can use the Elastic Stack to monitor applications built on your Hyperledger Fabric blockchain networks.

<p><a href="https://github.com/hyperledger/fabric#releases">Hyperledger Fabric</a> is a blockchain framework by the Linux Foundation that uses some open source projects to build a modular blockchain application. And do you know what? You can monitor your Hyperledger Fabric applications with the Elastic Stack. :)
</p><p>We are going to assume a specific Hyperledger Fabric setup. With Hyperledger Fabric you can deploy your application using different "blocks" to achieve the same result. For example, you can use a Hyperledger Fabric internal implementation, Kafka, or <a href="https://github.com/etcd-io/etcd">etcd</a> for consensus operations, and you can use GoLevelDB or CouchDB as database.
</p><p>The services deployed for this example are:
</p><ul>
	<li>Apache Kafka for ordering, monitored with the <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.2/metricbeat-module-kafka.html">Metricbeat Kafka module</a></li>
	<li>Apache Zookeeper as a requirement of Apache Kafka, monitored with the <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.2/metricbeat-module-zookeeper.html">Metricbeat Zookeeper module</a></li>
	<li>CouchDB as the source of truth database, monitored with the <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.2/metricbeat-module-couchdb.html">Metricbeat CouchDB module</a></li>
	<li>Peers monitored with the <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.2/metricbeat-module-prometheus.html">Metricbeat Prometheus module</a></li>
	<li>Orderer service written in Go, monitored with the <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.2/metricbeat-module-golang.html">Metricbeat Golang module</a></li>
</ul><p>The result is a very rich dashboard with an overall view of a Hyperledger Fabric network:
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt0c455b80c85170db/5d3e0f08165116375f8ad5e0/hyperledger-observability-overall-dashboard-blog.jpg" data-sys-asset-uid="blt0c455b80c85170db" alt="Dashboard with overall view" style="display: block; margin: auto;">
</p><p>In the screenshot above we see a portion of a dashboard specifically built for Hyperledger Fabric. We can also see Metricbeat, Filebeat, and Heartbeat working in harmony. Each peer inside the Hyperledger Fabric network is monitored by configuring them to expose Prometheus metrics. <a href="https://www.elastic.co/guide/en/beats/heartbeat/7.2/heartbeat-getting-started.html">Heartbeat</a> is also used; in this case, Heartbeat supports ICMP, TCP, and HTTP protocols so we have configured it to monitor all HTTP endpoints that were available.
</p><p>The network is deployed in Docker so the <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.2/metricbeat-module-docker.html">Metricbeat Docker module</a> has been configured to monitor it. But also, a Filebeat configured with <a href="https://www.elastic.co/guide/en/beats/filebeat/7.2/filebeat-input-container.html">Container as input</a> (previously called Docker input) and with the <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.2/add-docker-metadata.html">add_docker_metadata</a> processor allow us to extract and enrich Docker generated logs. Because we are not using any Filebeat module to process Docker log lines, the configuration can’t be simpler:
</p><pre class="prettyprint">filebeat.inputs:
  - type: container
    enabled: true
    paths:
      - /path_to_your_docker_logs/*/*.log
    json.keys_under_root: true
output.elasticsearch:
  hosts: ["your_elasticsearch:9200"]
processors:
  - add_host_metadata: ~
  - add_docker_metadata:
      host: "unix:///var/run/docker.sock"
</pre><p>The config above points the container input in Filebeat to the logs path (in this case, they were Docker containers so it points to the Docker logs path). Docker outputs log lines as JSON so we set to true <code>json.keys_under_root</code> so that they are not “stringified.” We configure our Elasticsearch cluster and add two very useful processors: <a href="https://www.elastic.co/guide/en/beats/filebeat/7.2/add-host-metadata.html">add_host_metadata</a>, which enriches the JSON with useful data coming from the host machine where Filebeat is running; and <a href="https://www.elastic.co/guide/en/beats/filebeat/7.2/add-docker-metadata.html">add_docker_metadata</a> which, in turn, enriches the JSON with useful information about the Docker engine that is running the containers.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc0b18b2bb3e8e634/5d3e0f32165116375f8ad5e6/hyperledger-observability-container-json-data-blog.jpg" data-sys-asset-uid="bltc0b18b2bb3e8e634" alt="JSON enriched with useful Docker data" style="display: block; margin: auto;">
</p><p>Apart from the Discover tab, you can also use the Logs UI to have a never-ending stream of logs with customized columns and data:
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt88ca3aff539e63e1/5d3e0f8965a8bf360426aea8/hyperledger-observability-logs-ui-blog.jpg" data-sys-asset-uid="blt88ca3aff539e63e1" alt="Logs UI with customized columns and data" style="display: block; margin: auto;">
</p><p>Finally, if logs, metrics, and uptime data playing together weren’t enough, you can also use the Infrastructure app. There you can see a different view of the infrastructure; in this case, we grouped the containers by image name to see their CPU usage.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltb54a114ddfb52861/5d3e0fcc1f1c20368b7349af/hyperledger-observability-infra-ui-blog.jpg" data-sys-asset-uid="bltb54a114ddfb52861" alt="Infrastructure UI with containers grouped by image name" style="display: block; margin: auto;">
</p><p>The Elastic Stack for observability is critical for any complex distributed system, but don't forget that you must also instrument your apps with Elastic APM solution to "close the circle" of observability inside your product: logs, metrics, uptime, and tracing.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blte7cf30b41ea796bf/5d3e1015cc2cb63e6348cb4d/hyperledger-observability-apm-blog.png" data-sys-asset-uid="blte7cf30b41ea796bf" alt="APM dashboard showing transaction sample" style="display: block; margin: auto;">
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt17ddfa9b106db882/5d3e104527d09d3764c6df44/hyperledger-observability-apm-txns-blog.jpg" data-sys-asset-uid="blt17ddfa9b106db882" alt="APM dashboard showing transaction graphs" style="display: block; margin: auto;">
</p><p>Bonus! Did you know that Kibana now has dark mode? If you want to give it a try, just go to Management - Advanced Settings and activate it! :) Here we can see the Logs UI in dark mode:
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltbc31b85a4e1952cd/5d3e10a18d62573e6d6c6f15/hyperledger-observability-dark-mode.jpg" data-sys-asset-uid="bltbc31b85a4e1952cd" alt="Kibana in dark mode" style="display: block; margin: auto;">
</p><h2>Want to see it in action?</h2><p>You can start monitoring your Hyperledger Fabric today by signing up for a free trial of the <a href="https://www.elastic.co/cloud/elasticsearch-service">Elasticsearch Service</a> on Elastic Cloud, or you can <a href="https://www.elastic.co/downloads/">download</a> it as part of the default distribution of the Elastic Stack. Grab the <a href="https://www.elastic.co/downloads/beats/metricbeat">Metricbeat</a>&nbsp;and&nbsp;<a href="https://www.elastic.co/downloads/beats/filebeat">Filebeat</a> executables appropriate for your platform and start monitoring!
</p>

hyperledger-observability-blog-banner.jpg

hyperledger-observability-blog-thumb.jpg

Observability on blockchain and the Hyperledger project

With two new metricsets and a pre-configured Kibana dashboard, monitoring your Microsoft SQL Servers with the Elastic Stack has never been easier.

<p>We are pleased to introduce support for Microsoft SQL Server as part of our generally available Metricbeat modules to capture database metrics. Microsoft SQL Server has been a popular and successful SQL database over the years, and we are really excited about this new addition to the Metricbeat family.
</p><p>Two metricsets are launching as beta with Metricbeat 7.0: <a href="/guide/en/beats/metricbeat/7.0/metricbeat-metricset-mssql-performance.html">performance</a> and <a href="/guide/en/beats/metricbeat/7.0/metricbeat-metricset-mssql-transaction_log.html">transaction_log</a>. In addition, a curated dashboard was also developed to help navigate Microsoft SQL Server metrics in Kibana.
</p><h2>Performance metric set</h2><p>The first thing that any user will want to look at is, as you have probably guessed, performance. It is very important is track the number of transactions, connected users, and cache hits, and all of this is included in this metricset. These metrics are retrieved from Performance Counters in Microsoft SQL Server
</p><ul>
	<li><code>user_connections</code>: Total number of user connections</li>
	<li><code>logins_per_sec</code>: Total number of logins started per second</li>
	<li><code>connections_reset_per_sec</code>: Total number of logins started from the connection pool</li>
	<li><code>transactions</code>: Total number of transactions</li>
	<li><code>batch_requests_per_sec</code>: Number of Transact-SQL command batches received per second</li>
	<li><code>buffer.cache_hit.pct</code>: The ratio of total number of cache hits divided by the total number of cache lookups over the last few thousand page accesses. After a long period of time, this ratio moves very little. Because reading from the cache is much less expensive than reading from disk, you’ll want this ratio to be high.</li>
	<li><code>compilations_per_sec</code>: Number of SQL compilations per second. This indicates the number of times the compile code path is entered. This includes compiles caused by statement-level recompilations in SQL Server. After SQL Server user activity is stable, this value reaches a steady state.</li>
</ul><h2>Transaction Log metric set</h2><p>This is another key metricset to include in your monitoring, because it helps you understand the shape and size of the server log. Some of the key metrics included here are:
</p><ul>
	<li><code>space_usage</code>: This shows space usage information about the transaction log, a key metric to understand the performance of a Microsoft SQL Server. It also shows the time passed since the last backup of the log, useful for disaster recovery</li>
	<li><code>stats</code>: This returns summary level attributes and information on transaction log files of databases. Use this information for monitoring and diagnostics of transaction log health. Here you can see active size, total size, or backup time for example.</li>
</ul><h2>Getting started with SQL Server monitoring using Metricbeat</h2><p>The easiest way to configure this module can be found on the "Add Data UI" screen in Kibana. This UI is a <a href="/subscriptions">Basic feature</a>, which means it's&nbsp;part of the free, default distribution of Kibana.
</p><p>Here you will find instructions to set up the module as shown on the following screen:
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt147f2792b1eda5ac/5cc0a5de887e04ba6916ea8c/mssql-metricbeat-1.jpg" data-sys-asset-uid="blt147f2792b1eda5ac" alt="Configure your Microsoft SQL metricset" width="650" style="display: block; margin: auto;">
</p><p>Following the instructions, you will see how easy it is to activate the Microsoft SQL Server Metricbeat module by setting up the addresses of the SQL Servers, filling username and password, and enabling the module. Once enabled, you can start Metricbeat and set up the dashboards:
</p><pre class="prettyprint">./metricbeat modules enable mssql
./metricbeat -E setup.dashboards.enabled=true -e
</pre><h2>Microsoft SQL Server dashboards</h2><p>Two curated dashboards are included with the module — one for the <em>Transaction Log</em>, and the other for the <em>Performance</em> metricset. In the following screenshot you can see the "Transaction Log" dashboard.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt98e7947302d40c04/5cc0a606ee4bf29d69c58014/mssql-metricbeat-2.jpg" data-sys-asset-uid="blt98e7947302d40c04" alt="Microsoft SQL Server metric dashboard in Kibana" "="">
</p><p>As shown in the screenshot, you can also select a specific database in the server to see a detailed view of it as well as comparing them in a single View.
</p><h2>Try it out today </h2><p>Deploy a cluster on our <a href="/cloud/elasticsearch-service">Elasticsearch Service</a> or <a href="/downloads/">download</a> the stack, spin up the new Metricbeat module (you can also add more modules from the list of <a href="/guide/en/beats/metricbeat/7.0/metricbeat-modules.html">supported modules</a>), open the curated dashboards in Kibana and start monitoring your Microsoft SQL Server today!
</p> <span></span>

blog-generic-teal-banner.png

mssql-metricbeat-thumb.jpg

Monitoring Microsoft SQL Server using Metricbeat and Elasticsearch

Take advantage of Hacktoberfest and become a contributor of Beats or Logstash projects at Elastic.

<html><head></head><body><p>So, let&apos;s start from the beginning, what&apos;s Hacktoberfest? Hacktoberfest is a way to celebrate open source during an entire month by contributing. Open source projects tag issues on their Github boards with the tag <a href="https://github.com/search?q=label%3Ahacktoberfest+state%3Aopen+type%3Aissue&amp;type=Issues">Hacktoberfest</a> which are issues&#xA0;low on complexity and a really good way to start contributing to any open source project.<br>
</p><h2>It&apos;s all about open source</h2><p>At Elastic, we embrace open source software so, for us, it&apos;s a really exciting event and month to get closer relationships with open source contributors from the community.
</p><p>The open source projects at Elastic gained a lot of members in Hacktoberfest of last year and the feedback and the help received from those members has push forward our widely used open source projects. We had communication with people who were already active contributors as well as people who were doing their first pull request (PR) in an open source project.
</p><h2>It&apos;s all about lowering the barrier to OSS</h2><p>Contributing to OSS can seem daunting at first, and that&#x2019;s completely understandable. But it is not daunting at all... it just looks like it is. Nobody is born knowing how to ride a bike, as well as nobody is born knowing how to open a PR. It takes practice to know what information to add and which tests to write.
</p><p>But one thing to keep in mind is this: maintainers know that there&apos;s a lot of people out there that have never contributed and want some help with their first issues, so here&apos;s the guideline about Hacktoberfest:
</p><h2>Making your first OSS contribution:</h2><p>To make your first contribution to a OSS project, we suggest the following steps:
</p><h3>1. Choose a project </h3><p>Do you like to write Java and or Ruby? Peek at the widely used and incredibly popular <strong>Logstash</strong> project: <a href="https://github.com/elastic/logstash">https://github.com/elastic/logstash</a>.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blte7c56322b9497aa8/5c3050065a7621286e3e90b5/hacktoberfest_1.png" data-sys-asset-uid="blte7c56322b9497aa8" alt="hacktoberfest_1.png" style="display: block; margin: auto;">
</p><p>What about Go? The entire Beats project is waiting you: <a href="https://github.com/elastic/beats">https://github.com/elastic/beats</a>. In Beats, you can contribute to several packages: <strong>Filebeat</strong> to collect logs, <strong>Winlogbeat</strong> to get Windows event logs, <strong>Metricbeat</strong> for system monitoring, or the networking and security related projects of <strong>Packetbeat</strong> and <strong>Auditbeat</strong>, respectively. Everything is written in Go, and some projects like Filebeat have configuration based modules where you don&apos;t need to know Go even!
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltfa4ecba560907d35/5c3050036d4494e30b7b5501/hacktoberfest_2.png" data-sys-asset-uid="bltfa4ecba560907d35" alt="hacktoberfest_2.png" style="display: block; margin: auto;">
</p><p>Do you like Python more? If so, try Rally (<a href="https://github.com/elastic/rally">https://github.com/elastic/rally</a>) to contribute to this terrific tool for Elasticsearch benchmarking. Or try Curator (<a href="https://github.com/elastic/curator">https://github.com/elastic/curator</a>),an incredibly useful and popular tool for index management.
</p><h3>2. Read the contribution guidelines</h3><p>Each project has its own contribution guidelines intended to help setting up the development environment as well as give some clues about the contribution process. &#xA0;For example, here are the <a href="https://www.elastic.co/guide/en/beats/devguide/current/beats-contributing.html">Beats contribution guidelines</a>.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltd505d1a00b7639bf/5c304ffe3fdf8ce26f0c086f/hacktoberfest_3.png" data-sys-asset-uid="bltd505d1a00b7639bf" alt="hacktoberfest_3.png" style="display: block; margin: auto;">
</p><h3>3. Search for &quot;Hacktoberfest&quot; tag in the Issues board</h3><p>You&apos;ll find several issues, read the contents of any of them and try to find one that seems easy for you. Most of the issues tagged in Hacktoberfest will be a good first-time issue, or simply tasks that don&apos;t need a lot of context to understand and that doesn&apos;t involve much complexity.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltb5a1f6823e486548/5c304ffa61a626fb0b6300b6/hacktoberfest_4.png" data-sys-asset-uid="bltb5a1f6823e486548" alt="hacktoberfest_4.png" style="display: block; margin: auto;">
</p><p>Choose any issue and start peeking at the code. You don&apos;t really need to warn that you are starting with that issue if you are not sure if you&apos;ll finish it so don&apos;t feel pressed. It&apos;s not as common as you might think that two persons are tackling the same issue at the same time.
</p><h3>4. Ask!</h3><p>Asking questions is way more important than it might seem. Contributing is not usually as simple as reading the issue description and opening a PR.
</p><p>You are probably not the only one that has your specific question, so by asking, you are already contributing by giving more context to an issue. So don&apos;t worry about asking, we&apos;ll answer you or redirect you to any issue that could solve your question.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt297d6a6427b657f0/5c304ff6b7017c456eff09ad/hacktoberfest_5.png" data-sys-asset-uid="blt297d6a6427b657f0" alt="hacktoberfest_5.png" style="display: block; margin: auto;">
</p><p>Even if you don&#x2019;t manage the &#xA0;PR, the question will remain there in case someone in the future wants to try so.
</p><p>Don&apos;t be shy to ask.
</p><h3>4. Make some changes and open your first PR</h3><p>For big PRs, make some small changes and open the PR, don&apos;t wait until you have a lot of code developed. When you submit, you&#x2019;ll receive some guidance from the maintainer, so better to only have a few changes instead of a lot.
</p><p>That is why it is important to ask and to have an open communication channel with the maintainers of each project.<br>
</p><h3>5. Maintain focus.</h3><p>Contributing to OSS is not a difficult task, nor does it need to be a time consuming task, but it does take some time to merge even the smallest change. Don&apos;t expect to do it all in a day. It usually takes days even for the smallest changes.
</p><p>Remember that people are in different time zones and that you won&apos;t receive feedback immediately, so try to stay focused on one task during a week by spending some time every day on it.<br>
</p><p>Contributing to OSS is a marathon, not a sprint. Just get used to asking a question, waiting some time until getting a response, updating your code, pushing the new code, and waiting a few hours until more feedback is received. Slow and steady.<br>
</p><h2>Congratulations: You have become an open source software contributor.</h2><p>That&apos;s all. Personally, I think that the most important point is to &quot;maintain focus&quot; because it&apos;s easy (and normal) to feel enthusiastic about contributing, but it&apos;s also very easy (and normal) to lose focus after the first evening. Just don&apos;t forget that it is not possible to do everything in a single evening. Feedback is necessary (and desirable) and very easy way to learn new things.
</p><p>And don&apos;t forget that Hacktoberfest lasts for the entire month of October, so you have plenty of opportunities to contribute! Enjoy!
</p></body></html>

46093389-0e93dc00-c16c-11e8-9b53-28459be3bafc.png

46093518-64688400-c16c-11e8-8a44-251b15c164a2.png

Hacktoberfest 2018 for Beats and Logstash

Articles By Mario Castro

Observability on blockchain and the Hyperledger project

Monitoring Microsoft SQL Server using Metricbeat and Elasticsearch

Hacktoberfest 2018 for Beats and Logstash

Products & Solutions

Company

Resources

SaaS

ORCHESTRATION

Articles By Mario Castro

Observability on blockchain and the Hyperledger project

Monitoring Microsoft SQL Server using Metricbeat and Elasticsearch

Hacktoberfest 2018 for Beats and Logstash