Monitoring NGINX web server with Prometheus and Grafana

Monitoring and Observability are some of the most important aspects of software engineering. It helps us with debugging issues, getting insightful information about infrastructure and/or application performance, security, & availability, planning for effective utilization of resources and scaling of infrastructure to handle any number of client requests.

In this chapter, we will discuss about monitoring of open-source NGINX server by exposing various metrics data using a module called ngx_http_stub_status_module. This module is enabled by default and exposes basic status information about the server. Using this module, we can determine following metrics:

• Active connections - Current number  of active client connections including waiting connections
• accepts - Total number of accepted client connections
• handled - Total number of handled connections (Generally, it's value is same as accepts unless some resource limits have been breached - ex: worker connections limit)
• requests - Total number of client requests
• Reading - Current number of connections where nginx is reading the request header
• Writing - Current number of connections where nginx is writing response back to the client
• Waiting - Current number of idle client connections waiting for a request

With NGINX Plus, there is a module called ngx_http_api_module, which exposes an extensive list of status information, including the ones we discussed above. If you have not already installed NGINX in your system, go to the following link:

https://www.nodexplained.com/deploy-web-applications-with-nginx-web-server/

Now that the NGINX web server is successfully installed, check the version of it.

   
   	 nginx -v
   

For successful installation, it should give following output:

   
   	 nginx version: nginx/1.22.0
   

To view the list of installed modules, issue following command:

   
nginx -V 2>&1 | tr -- - '\n' | grep  _module
   

As we can see, ngx_http_stub_status_module is already installed and enabled.

Let's expose basic status information using a location directive, which matches any requests starting with /metrics path (can be any other meaningful name for the path). We don't want status information to be public. Since, we are using NGINX prometheus exporter to collect metrics data, which will be also installed in the same machine, we can make the endpoint only accessible from within a same server and deny every other requests for the endpoint from outside.

stub_status is a directive which exposes the basic status information. Let's define a location directive fulfilling those requirements in a following way:

   
location /metrics {
    stub_status;
	allow 127.0.0.1;
	deny all;
}
   

Now, let's modify default.conf file, located at conf.d directory and add above location block. Finally, default.conf file looks like below:

   
server {
    listen       80;
    server_name  localhost;
    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }

    location /metrics {
        stub_status;
        allow 127.0.0.1;
        deny all;
    }

    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }
}
   

Check the correctness of recent configuration file changes using following command:

   
   	sudo nginx -t
   

Reload the NGINX configuration using following command:

   
   	sudo nginx -s reload
   

Let's check if NGINX server is exposing metrics data or not using cURL command:

   
	curl http://localhost/metrics
   

As we can see from below image, various status information are being exposed. By default, these nginx metrics are not supported by Prometheus. We need to convert these metrics into a prometheus compliant format using NGINX prometheus exporter.

Convert metrics to Prometheus compliant format using NGINX Prometheus Exporter

Prometheus uses exporters to collect metrics from various data sources. There are many official exporters as well as exporters contributed by the community. One of such exporter is NGINX Prometheus exporter, which makes it possible to monitor NGINX using Prometheus. It fetches the metrics data from a single NGINX, converts the metrics into appropriate Prometheus metrics types and finally exposes them via HTTP server to be collected by Prometheus. Prometheus server will be scraping for these metrics data in a regular time interval.

At the time of writing this article, latest version of NGNIX Prometheus Exporter is v0.10.0. Download the latest version and extract it using following command:

   
mkdir $HOME/workspace/ && cd $HOME/workspace/
wget https://github.com/nginxinc/nginx-prometheus-exporter/releases/download/v0.10.0/nginx-prometheus-exporter_0.10.0_linux_amd64.tar.gz
tar xzf nginx-prometheus-exporter_*.tar.gz
   

Now, we need to copy nginx-prometheus-exporter binary file to /usr/local/bin directory. For running the exporter in a secure manner, we will be using a separate user. Issue following commands:

   
sudo groupadd nginx_prometheus_exporter --system
sudo useradd -g nginx_prometheus_exporter --system --shell /sbin/nologin nginx_prometheus_exporter
sudo cp $HOME/workspace/nginx-prometheus-exporter /usr/local/bin/
sudo chown nginx_prometheus_exporter:nginx_prometheus_exporter /usr/local/bin/nginx-prometheus-exporter
   

Create systemd service file

Usually in linux, we run software processes using systemctl command. To run NGINX prometheus exporter using systemctl command, we need to create a service file for it as well. To create a service file, issue following command:

   
sudo vi /etc/systemd/system/nginx-prometheus-exporter.service
   

Paste the following content in above service file:

   
[Unit]
Description=Prometheus exporter for NGINX server metrics
Wants=network-online.target
After=network-online.target

[Service]
User=nginx_prometheus_exporter
Group=nginx_prometheus_exporter
Type=simple
ExecStart=/usr/local/bin/nginx-prometheus-exporter -nginx.scrape-uri=http://localhost/metrics

Restart=always
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

   

To reload the modification in system configurations, issue following command:

   
	sudo systemctl daemon-reload
   

To check the version:

   
	nginx-prometheus-exporter --version
   

Output is as follows:

   
NGINX Prometheus Exporter version=0.10.0 commit=7a03d0314425793cf4001f0d9b0b2cfd19563433 date=2021-12-21T19:24:34Z

   

To start the exporter:

   
	sudo systemctl start nginx-prometheus-exporter
   

To restart the exporter:

   
	sudo systemctl restart nginx-prometheus-exporter
   

To check the status of exporter, issue following command:

   
	sudo systemctl status nginx-prometheus-exporter
   

To make sure that exporter starts after each system reboot, issue following command:

   
	sudo systemctl enable nginx-prometheus-exporter
   

Note:

If you are using cloud server instances for installing NGINX prometheus exporter, you have to open tcp port 9113 for inbound connections, so that prometheus can connect to it and start scraping nginx metrics.

We will be using prometheus server to collect the NGINX metrics data and visualize it using dashboards in a grafana server. If you haven't already installed prometheus and grafana, please click on the following link:

https://www.nodexplained.com/prometheus-getting-started-with-installation-and-configuration-details/

https://www.nodexplained.com/grafana-getting-started-with-installation-and-configuration-details/

Monitor NGINX metrics using Prometheus

Now that, we have both prometheus and grafana server installed and running, let's modify prometheus configuration file, so that it can connect with NGINX prometheus exporter service and start scraping for metrics data.

   
	sudo vi /etc/prometheus/prometheus.yml
   

Under scrape_configs configuration parameter, we need to add another job which will be connecting to the NGINX prometheus exporter server and then start scraping for metrics data regularly in 15 seconds interval.

• job_name
  
  It can be any meaningful name for the job. Here, we provide name of the job as nginx-exporter
• scrape_interval
  
  We can put any time duration that satisfies our scraping requirements. It can be in seconds, minutes etc. Here, we use 15s as the interval time.
• targets
  
  We can add as many scraping targets to the array as we want and prometheus server will scrape metrics data from all the specified targets.

It looks like following:

   

scrape_configs:
  ...
  ...
  - job_name: 'nginx-exporter'
    scrape_interval: 15s
    static_configs:
      - targets: ['NGINX_PROMETHEUS_EXPORTER_IP:9113']
   

Replace NGINX_PROMETHEUS_EXPORTER_IP with an ip address of a NGINX prometheus exporter server. If you are using your local machine for everything, use localhost as NGINX_PROMETHEUS_EXPORTER_IP.

Final version of prometheus.yml file looks like below:

   
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=` to any timeseries scraped from this config.
  - job_name: "prometheus"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["localhost:9090"]

  - job_name: 'nginx-exporter'
    scrape_interval: 15s
    static_configs:
      - targets: ['35.89.16.109:9113']
   

For reflecting these configuration changes, restart the prometheus server using following command:

   
	sudo systemctl restart prometheus
   

Go to the following url to browse the prometheus web UI:

http://localhost:9090/graph

As we can see from below image, nginx exporter is shown in the list of targets, from where metrics data are scraped.

The open-source version of NGINX server exposes following metrics data

Let's see a graph visualizations for one of the metric exposed by NGINX server called nginx_http_requests_total(the total number of http requests).

Visualize NGINX metrics using Grafana

We can use grafana server to display NGINX metrics in an even more sophisticated dashboard containing one or more visualizations. We can either create the dashboard manually or use one of the pre-configured dashboards, available for free of charge. We will use pre-configured dashboard to show NGINX metrics data. Click on the following link and copy id of the dashboard. Here, id is 12708

https://grafana.com/grafana/dashboards/12708-nginx/

To import this dashboard, click on the menu item Import as shown by red arrow.

It will show following page, from where we need to choose a data source for this dashboard. Select Prometheus as a data source for displaying NGINX metrics data.

Click on Import button to create a dashboard. The dashboard looks like below:

As we can see, this dashboard is extremely helpful for monitoring the NGINX server. It shows whether the server is running or not, and also displays other various helpful metrics data. We will expand this dashboard with more visualizations in later chapters.

That's it for this chapter. In our next chapter, we will look into visualizing metrics data collected from access logs and error logs file in grafana server.


Other tutorials on NGINX:

Chapter - 1: Deploy Web applications with NGINX web server

Chapter 2 - Set Up Load Balancing using Nginx web server

Chapter - 3: Block visitors from certain countries using NGINX

Chapter - 4: Responsive Images using Nginx Image-Filter module