<p>In this article, we'll be looking at how we can monitor NGINX using the various components of the Elastic Stack. We'll use Metricbeat and Filebeat to collect data. This data will be shipped off to and stored within Elasticsearch. Finally, we'll view that data with Kibana.</p><p>Metricbeat will collect data related to connections (active, handled, accepted, etc.) and the total number of client requests. Filebeat will gather data related to access and error logs. Mileage will vary between different setups, but for the most part we can use this information to deduce certain things, for example:</p><ul><li>A spike in error logs for a certain resource may mean we have deleted a resource that is still needed.</li><li>Access logs can show when a services peak times are (and therefore when it might be best to perform things like maintenance).</li><li>A sudden spike in client requests may point to something malicious (like a DDoS attack).</li></ul><p>The core focus of this article is the monitoring functionality, so things like Elasticsearch setup will be brief. I'm using a Mac and will therefore be using the Mac distributions of Elasticsearch, Metricbeat, Filebeat, and Kibana. Distributions for other platforms <a href="/downloads/">can be acquired from the downloads page</a>.</p><h2>Installing Elasticsearch</h2><p><a href="/products/beats/metricbeat">Metricbeat</a> and <a href="/products/beats/filebeat">Filebeat</a> (in fact, all <a href="/products/beats/">Beats</a>) need an Elasticsearch cluster to store data. We'll set up a simple Elasticsearch cluster locally. Since we're not exposing Elasticsearch externally, we won't spend time <a href="/blog/getting-started-with-elasticsearch-security">configuring security</a>.</p><h4>Step 1: Download Elasticsearch</h4><pre class="prettyprint">curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.1.0-darwin-x86_64.tar.gz<br /></pre><h4>Step 2: Extract the archive</h4><pre class="prettyprint">tar -xvf elasticsearch-7.1.0-darwin-x86_64.tar.gz<br /></pre><h4>Step 3: Change to the bin directory</h4><pre class="prettyprint">cd elasticsearch-7.1.0/bin<br /></pre><h4>Step 4: Start our single node cluster</h4><pre class="prettyprint">./elasticsearch<br /></pre><p>By default, your cluster will be running at localhost:9200.</p><h2>Running NGINX</h2><p>There are many ways we can run NGINX — standalone on the host, via a Docker container, within a Kubernetes setup, etc. Beats has an Autodiscover feature that can listen to container API events to track our server containers as they spin up or are removed.</p><p>Given the variety of ways NGINX can be set up and configured, this article doesn't make too many assumptions and will show host configuration as well as Autodiscover (Docker, in this case) examples.</p><h2>Configuring Metricbeat and Filebeat</h2><p>Now we'll configure Beats to start collecting and shipping our data.</p><h3>Metricbeat</h3><p>We'll use Metricbeat to collect metrics.</p><h4>Step 1: Download Metricbeat:</h4><pre class="prettyprint">curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-7.1.0-darwin-x86_64.tar.gz<br /></pre><h4>Step 2: Extract the archive</h4><pre class="prettyprint">tar -xvf metricbeat-7.1.0-darwin-x86_64.tar.gz<br /></pre><h4>Step 3: Change directories</h4><pre class="prettyprint">cd metricbeat-7.1.0-darwin-x86_64<br /></pre><p>Configuration time!</p><p>By default our configuration file (metricbeat.yml) will be looking for an Elasticsearch cluster running at localhost:9200, which is what we configured earlier.</p><h4>Step 4: Enable relevant Metricbeat modules</h4><p>By default only system level metrics are collected:</p><pre class="prettyprint">./metricbeat modules enable nginx<br /></pre><h4>Step 5: Change file ownerships</h4><p>We need to change a couple of file ownerships, as we'll be running Metricbeat as root (alternatively, you may choose to use the --strict.perms=false option from the command line to disable strict permission checks):</p><pre class="prettyprint">sudo chown root metricbeat.yml <br />sudo chown root modules.d/system.yml <br />sudo chown root modules.d/nginx.yml<br /></pre><h4>Step 6: Enable metricsets</h4><p>If we open our modules.d/nginx.yml file we can enable the stubstatus metricset, which involves uncommenting the following:</p><pre class="prettyprint">#metricsets:<br />#  - stubstatus<br /></pre><p>This metricset collects data from the NGINX <a href="https://nginx.org/en/docs/http/ngx_http_stub_status_module.html">ngx_http_stub_status</a> module, it is important that this NGINX module is configured for the metricset to work.</p><p>You may also change the hosts being monitored here, by default &lt;a href="http://127.0.0.1"&gt;http://127.0.0.1&lt;/a&gt; is monitored. For host based setups this is all we need.</p><h4>Step 7: Autodiscover setups (alternative)</h4><p>With <a href="/guide/en/beats/metricbeat/7.1/configuration-autodiscover.html">Autodiscover</a> setups, the equivalent configuration will vary between the three providers, Docker, Kubernetes, and Jolokia. A Docker example would look like the following, where we add a metricbeat.autodiscover to the bottom of the metricbeat.yml configuration file.</p><pre class="prettyprint">metricbeat.autodiscover:<br />  providers:<br />    - type: docker<br />      templates:<br />        - condition:<br />            contains:<br />              docker.container.image: nginx<br />          config:<br />            - module: nginx<br />              metricsets: ["stubstatus"]<br />              hosts: "${data.host}:${data.port}"<br /></pre><p>With this configuration all Docker containers using an image with NGINX in the name (<a href="/guide/en/beats/metricbeat/7.1/defining-processors.html#conditions">contains will perform a substring match, rather than an exact match</a>) will be matched, and will use a config enabling the NGINX module with the stubstatus metricset.</p><h4>Step 8: Run Metricbeat</h4><p>Great, now we're configured, we can run Metricbeat with a flag that sends the logs to stderr and disables syslog/file output:</p><pre class="prettyprint">sudo ./metricbeat -e<br /></pre><p>If everything has worked, you should see some initial output, and then further output each time Metricbeat posts data back to your cluster.</p><h3>Filebeat</h3><p>We'll use Filebeat to collect logs.</p><h4>Step 1: Download Filebeat</h4><pre class="prettyprint">curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.1.0-darwin-x86_64.tar.gz<br /></pre><h4>Step 2: Extract the archive</h4><pre class="prettyprint">tar -xvf filebeat-7.1.0-darwin-x86_64.tar.gz<br /></pre><h4>Step 3: Change directories</h4><pre class="prettyprint">cd filebeat-7.1.0-darwin-x86_64<br /></pre><h4>Step 4: Enable the NGINX module</h4><pre class="prettyprint">./filebeat modules enable nginx<br /></pre><p>By default, access and error logs will be collected. As for log file paths Filebeat will try to determine these based on your OS, if you need to set different paths explicitly you can set these in the modules.d/nginx.yml configuration file.</p><h4>Step 5: Change file ownerships</h4><pre class="prettyprint">sudo chown root filebeat.yml<br />sudo chown root modules.d/nginx.yml<br />sudo chown root module/nginx/access/manifest.yml<br />sudo chown root module/nginx/error/manifest.yml<br /></pre><p>For a host-based setup this is all we need.</p><h4>Step 6: Autodiscover setup (alternative)</h4><p>As before, additional configuration is needed to use Autodiscover. Similar to the previous section, the Docker example would need an autodiscover section added to the config, and the filebeat.yml would look like the following:</p><pre class="prettyprint">filebeat.autodiscover:<br /> providers:<br />   - type: docker<br />     templates:<br />       - condition:<br />         contains:<br />           docker.container.image: nginx<br />         config:<br />           - module: nginx<br />             access:<br />               input:<br />                 type: docker<br />                 containers.ids:<br />                   - "${data.docker.container.id}"<br />             error:<br />               input:<br />                 type: docker<br />                 containers.ids:<br />                   - "${data.docker.container.id}"<br /></pre><p>Here we configure the access and error options to use a <a href="/guide/en/beats/filebeat/7.1/filebeat-input-docker.html">Docker input</a>. A Docker input will search for container logs under its path (the base path where Docker logs are located, by default this is /var/lib/docker/containers). We specify containers.ids using the <a href="/guide/en/beats/filebeat/7.1/configuration-autodiscover.html#_docker_2">id(s) of the containers that matched our template condition</a>. Put together, the logs (by default) will be sourced from /var/lib/docker/containers/ac29b98ad83ca43bb4c15ae8f0d03aff8c7d57bf5dee9024124374b92b14b0f2/ (ids will differ).</p><h4>Step 7: Run Filebeat</h4><pre class="prettyprint">sudo ./filebeat -e<br /></pre><h2>Installing Kibana</h2><p>Now that we have Metricbeat and Filebeat shipping off data about our NGINX server(s) we need a way to view this data, which is where Kibana comes in.</p><h4>Step 1: Download Kibana</h4><pre class="prettyprint">curl -O https://artifacts.elastic.co/downloads/kibana/kibana-7.1.0-darwin-x86_64.tar.gz<br /></pre><h4>Step 2: Extract the archive and change directory</h4><pre class="prettyprint">tar -xzf kibana-7.1.0-darwin-x86_64.tar.gz<br /></pre><pre class="prettyprint">cd kibana-7.1.0-darwin-x86_64<br /></pre><h4>Step 3: Run Kibana</h4><pre class="prettyprint">./bin/kibana<br /></pre><p>By default Kibana will use an Elasticsearch host of http://localhost:9200, and will be accessible at http://localhost:5601.</p><p>If you navigate to <a href="http://localhost:5601">http://localhost:5601</a> you should be greeted by the following:</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltad0c6331722acdc1/5d2dba9c9f0a7673f0580236/nginx-monitoring-image1.png" data-sys-asset-uid="bltad0c6331722acdc1" alt="Welcome to Kibana screen" style="display: block;margin: auto;" width="420" height="530"/></p><p>You can optionally try out the sample data, but we'll just click <strong>Explore on my own</strong>.</p><h2>Visualizing NGINX data in Kibana</h2><p>Next we'll take a look at how we can use Kibana to view and analyze our data.</p><h3>Infrastructure</h3><p>If we navigate to the Infrastructure app on the Kibana sidebar we'll be able to see a snapshot (last minute) view of our infrastructure. Depending on the configuration that was used, data will be accessible under Hosts for non-Autodiscover based setups, for Autodiscover setups the Docker and Kubernetes buttons (based on the provider that was used) will lead you to the correct dataset:</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt1873b9142f8f4dc9/5d2d374e9f0a7673f057fe6c/nginx-monitoring-image2.png" data-sys-asset-uid="blt1873b9142f8f4dc9" alt="Hosts, Kubernetes, and Docker buttons" style="display: block;margin: auto;" width="420" height="100"/></p><p>As I was using a Hosts based setup, I can click the Hosts button and see the following data being shipped from Metricbeat:</p><h2><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt2594e5bbb16134f9/5d2d37839f0a7673f057fe72/nginx-monitoring-image3.png" style="background-color: initial;font-family: proximaNovaRegular, Arial, Helvetica, sans-serif;font-size: 14px;color: rgb(116, 133, 144);display: block;margin: auto;" data-sys-asset-uid="blt2594e5bbb16134f9" alt="Click Hosts button to see data being shippped"/></h2><p>If we select one of the nodes and choose to <strong>View Metrics</strong> we can see a page with detailed metrics for that single NGINX server.</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt134bbe8e86663c4b/5d2d37bdae880b03aa40661e/nginx-monitoring-image4.png" data-sys-asset-uid="blt134bbe8e86663c4b" alt="Kibana graphs showing server metrics" style="display: block;margin: auto;"/></p><h3>Logs</h3><p>If we navigate to the Logs app instead we'll be able to see our access and error logs that have been shipped from Filebeat.</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt79961499be8a9f3b/5d2d3803ae880b03aa406624/nginx-monitoring-image5.jpg" data-sys-asset-uid="blt79961499be8a9f3b" alt="Kibana Logs app showing error and access logs" style="display: block;margin: auto;"/></p><h3>Pre-configured NGINX dashboards</h3><p>Now that we have a running and accessible instance of Kibana we can load some premade dashboards.</p><p>To load the Metricbeat dashboards run the following in your Metricbeat directory:</p><pre class="prettyprint">sudo ./metricbeat setup --dashboards<br /></pre><p>And to do the same for Filebeat:</p><pre class="prettyprint">sudo ./filebeat setup --dashboards<br /></pre><p>Now if we navigate to the Dashboards tab within Kibana we should see the following:</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltf6e6bf414a9c13c8/5d2d383eb3c3b6032396a3d7/nginx-monitoring-image6.jpg" data-sys-asset-uid="bltf6e6bf414a9c13c8" alt="Kibana page showing available dashboards" style="display: block;margin: auto;"/></p><p>If we narrow this down to an "nginx" search, we have these dashboards available:</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltcf3ccffbb71a48aa/5d2d38bb2eb7167479193ecf/nginx-monitoring-image7.png" data-sys-asset-uid="bltcf3ccffbb71a48aa" alt="Kibana page showing available NGINX dashboards" style="display: block;margin: auto;" width="720" height="390"/></p><p>The <em><strong>[Filebeat NGINX] Access and error logs ECS</strong></em> dashboard, as an example, looks like the following:</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt51e7d57d2a6b575d/5d2d38f88e6f797b0da30462/nginx-monitoring-image8.jpg" data-sys-asset-uid="blt51e7d57d2a6b575d" alt="Filebeat NGINX access and error logs ECS" style="display: block;margin: auto;"/></p><p>And with that we've learned how to use the Elastic Stack to monitor NGINX servers. From here there are various options to play with (different grouping and filtering options for example) to drill down to the information that really matters to you.</p><h2>Try it out yourself</h2>Getting started with Elastic Stack is easy — you can download the various products via the curl commands provided, or navigate to our <a href="/downloads/">downloads</a> page and grab them from there. Or even easier, spin up 14-day free trial of the <a href="/cloud/elasticsearch-service/signup">Elasticsearch Service</a>, and go from there. If you run into any questions, reach out on the <a href="https://discuss.elastic.co/">Discuss</a> forums. It might also help to <span><a href="https://www.elastic.co/webinars/powering-search-using-the-elastic-stack" target="_self">understand why Elastic Stack is a top choice for application search in the most diverse industries</a>.</span>

monitor-nginx-blog-banner.jpg

monitor-nginx-blog-thumb.jpg

How to monitor NGINX web servers with the Elastic Stack

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Blog Footer

Kerry Gallagher

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content.

You'll also receive an email with related content.

You'll also receive an email with related content

Load More

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Newsletter GDPR Text

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

Elasticsearch Platform

ELK Stack

Elastic Cloud

Observability

Security

Search

By industry

By solution

Customer spotlight

Developers

Connect

Learn

Help

See what's happening at Elastic

Author

Articles by Kerry Gallagher

How to monitor NGINX web servers with the Elastic Stack

Follow us

About us

Join us

Press

Partners

Trust & Security

Investor relations

EXCELLENCE AWARDS