Engineering

How to monitor NGINX web servers with the Elastic Stack

In this article, we'll be looking at how we can monitor NGINX using the various components of the Elastic Stack. We'll use Metricbeat and Filebeat to collect data. This data will be shipped off to and stored within Elasticsearch. Finally, we'll view that data with Kibana.

Metricbeat will collect data related to connections (active, handled, accepted, etc.) and the total number of client requests. Filebeat will gather data related to access and error logs. Mileage will vary between different setups, but for the most part we can use this information to deduce certain things, for example:

  • A spike in error logs for a certain resource may mean we have deleted a resource that is still needed.
  • Access logs can show when a services peak times are (and therefore when it might be best to perform things like maintenance).
  • A sudden spike in client requests may point to something malicious (like a DDoS attack).

The core focus of this article is the monitoring functionality, so things like Elasticsearch setup will be brief. I'm using a Mac and will therefore be using the Mac distributions of Elasticsearch, Metricbeat, Filebeat, and Kibana. Distributions for other platforms can be acquired from the downloads page.

Installing Elasticsearch

Metricbeat and Filebeat (in fact, all Beats) need an Elasticsearch cluster to store data. We'll set up a simple Elasticsearch cluster locally. Since we're not exposing Elasticsearch externally, we won't spend time configuring security.

Step 1: Download Elasticsearch

curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.1.0-darwin-x86_64.tar.gz

Step 2: Extract the archive

tar -xvf elasticsearch-7.1.0-darwin-x86_64.tar.gz

Step 3: Change to the bin directory

cd elasticsearch-7.1.0/bin

Step 4: Start our single node cluster

./elasticsearch

By default, your cluster will be running at localhost:9200.

Running NGINX

There are many ways we can run NGINX — standalone on the host, via a Docker container, within a Kubernetes setup, etc. Beats has an Autodiscover feature that can listen to container API events to track our server containers as they spin up or are removed.

Given the variety of ways NGINX can be set up and configured, this article doesn't make too many assumptions and will show host configuration as well as Autodiscover (Docker, in this case) examples.

Configuring Metricbeat and Filebeat

Now we'll configure Beats to start collecting and shipping our data.

Metricbeat

We'll use Metricbeat to collect metrics.

Step 1: Download Metricbeat:

curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-7.1.0-darwin-x86_64.tar.gz

Step 2: Extract the archive

tar -xvf metricbeat-7.1.0-darwin-x86_64.tar.gz

Step 3: Change directories

cd metricbeat-7.1.0-darwin-x86_64

Configuration time!

By default our configuration file (metricbeat.yml) will be looking for an Elasticsearch cluster running at localhost:9200, which is what we configured earlier.

Step 4: Enable relevant Metricbeat modules

By default only system level metrics are collected:

./metricbeat modules enable nginx

Step 5: Change file ownerships

We need to change a couple of file ownerships, as we'll be running Metricbeat as root (alternatively, you may choose to use the --strict.perms=false option from the command line to disable strict permission checks):

sudo chown root metricbeat.yml 
sudo chown root modules.d/system.yml 
sudo chown root modules.d/nginx.yml

Step 6: Enable metricsets

If we open our modules.d/nginx.yml file we can enable the stubstatus metricset, which involves uncommenting the following:

#metricsets:
#  - stubstatus

This metricset collects data from the NGINX ngx_http_stub_status module, it is important that this NGINX module is configured for the metricset to work.

You may also change the hosts being monitored here, by default <a href="http://127.0.0.1">http://127.0.0.1</a> is monitored. For host based setups this is all we need.

Step 7: Autodiscover setups (alternative)

With Autodiscover setups, the equivalent configuration will vary between the three providers, Docker, Kubernetes, and Jolokia. A Docker example would look like the following, where we add a metricbeat.autodiscover to the bottom of the metricbeat.yml configuration file.

metricbeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains:
              docker.container.image: nginx
          config:
            - module: nginx
              metricsets: ["stubstatus"]
              hosts: "${data.host}:${data.port}"

With this configuration all Docker containers using an image with NGINX in the name (contains will perform a substring match, rather than an exact match) will be matched, and will use a config enabling the NGINX module with the stubstatus metricset.

Step 8: Run Metricbeat

Great, now we're configured, we can run Metricbeat with a flag that sends the logs to stderr and disables syslog/file output:

sudo ./metricbeat -e

If everything has worked, you should see some initial output, and then further output each time Metricbeat posts data back to your cluster.

Filebeat

We'll use Filebeat to collect logs.

Step 1: Download Filebeat

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.1.0-darwin-x86_64.tar.gz

Step 2: Extract the archive

tar -xvf filebeat-7.1.0-darwin-x86_64.tar.gz

Step 3: Change directories

cd filebeat-7.1.0-darwin-x86_64

Step 4: Enable the NGINX module

./filebeat modules enable nginx

By default, access and error logs will be collected. As for log file paths Filebeat will try to determine these based on your OS, if you need to set different paths explicitly you can set these in the modules.d/nginx.yml configuration file.

Step 5: Change file ownerships

sudo chown root filebeat.yml
sudo chown root modules.d/nginx.yml
sudo chown root module/nginx/access/manifest.yml
sudo chown root module/nginx/error/manifest.yml

For a host-based setup this is all we need.

Step 6: Autodiscover setup (alternative)

As before, additional configuration is needed to use Autodiscover. Similar to the previous section, the Docker example would need an autodiscover section added to the config, and the filebeat.yml would look like the following:

filebeat.autodiscover:
 providers:
   - type: docker
     templates:
       - condition:
         contains:
           docker.container.image: nginx
         config:
           - module: nginx
             access:
               input:
                 type: docker
                 containers.ids:
                   - "${data.docker.container.id}"
             error:
               input:
                 type: docker
                 containers.ids:
                   - "${data.docker.container.id}"

Here we configure the access and error options to use a Docker input. A Docker input will search for container logs under its path (the base path where Docker logs are located, by default this is /var/lib/docker/containers). We specify containers.ids using the id(s) of the containers that matched our template condition. Put together, the logs (by default) will be sourced from /var/lib/docker/containers/ac29b98ad83ca43bb4c15ae8f0d03aff8c7d57bf5dee9024124374b92b14b0f2/ (ids will differ).

Step 7: Run Filebeat

sudo ./filebeat -e

Installing Kibana

Now that we have Metricbeat and Filebeat shipping off data about our NGINX server(s) we need a way to view this data, which is where Kibana comes in.

Step 1: Download Kibana

curl -O https://artifacts.elastic.co/downloads/kibana/kibana-7.1.0-darwin-x86_64.tar.gz

Step 2: Extract the archive and change directory

tar -xzf kibana-7.1.0-darwin-x86_64.tar.gz
cd kibana-7.1.0-darwin-x86_64

Step 3: Run Kibana

./bin/kibana

By default Kibana will use an Elasticsearch host of http://localhost:9200, and will be accessible at http://localhost:5601.

If you navigate to http://localhost:5601 you should be greeted by the following:

Welcome to Kibana screen

You can optionally try out the sample data, but we'll just click Explore on my own.

Visualizing NGINX data in Kibana

Next we'll take a look at how we can use Kibana to view and analyze our data.

Infrastructure

If we navigate to the Infrastructure app on the Kibana sidebar we'll be able to see a snapshot (last minute) view of our infrastructure. Depending on the configuration that was used, data will be accessible under Hosts for non-Autodiscover based setups, for Autodiscover setups the Docker and Kubernetes buttons (based on the provider that was used) will lead you to the correct dataset:

Hosts, Kubernetes, and Docker buttons

As I was using a Hosts based setup, I can click the Hosts button and see the following data being shipped from Metricbeat:

Click Hosts button to see data being shippped

If we select one of the nodes and choose to View Metrics we can see a page with detailed metrics for that single NGINX server.

Kibana graphs showing server metrics

Logs

If we navigate to the Logs app instead we'll be able to see our access and error logs that have been shipped from Filebeat.

Kibana Logs app showing error and access logs

Pre-configured NGINX dashboards

Now that we have a running and accessible instance of Kibana we can load some premade dashboards.

To load the Metricbeat dashboards run the following in your Metricbeat directory:

sudo ./metricbeat setup --dashboards

And to do the same for Filebeat:

sudo ./filebeat setup --dashboards

Now if we navigate to the Dashboards tab within Kibana we should see the following:

Kibana page showing available dashboards

If we narrow this down to an "nginx" search, we have these dashboards available:

Kibana page showing available NGINX dashboards

The [Filebeat NGINX] Access and error logs ECS dashboard, as an example, looks like the following:

Filebeat NGINX access and error logs ECS

And with that we've learned how to use the Elastic Stack to monitor NGINX servers. From here there are various options to play with (different grouping and filtering options for example) to drill down to the information that really matters to you.

Try it out yourself

Getting started with Elastic Stack is easy — you can download the various products via the curl commands provided, or navigate to our downloads page and grab them from there. Or even easier, spin up 14-day free trial of the Elasticsearch Service, and go from there. If you run into any questions, reach out on the Discuss forums.