Prometheus - Getting started with Installation and Configuration

Prometheus is an open-source software for monitoring and alerting purposes. It collect and store its metrics as time series data, a series of data that is collected over consistent intervals of time. It collect metrics from configured targets at given intervals, evaluates rule expressions, display the results, and can trigger alerts when specified conditions are met.  To collect metrics data, it uses a HTTP endpoint. Written in Golang, it is highly scalable and performant.

Originally developed by SoundCloud, it is now a Cloud Native Computing Foundation (CNCF) graduated project. It joined the Cloud Native Computing Foundation in 2016 as the second hosted project, after Kubernetes. With the help of prometheus, we can monitor various system infrastructures, databases, application services and many others.

Some of the notable features that it offers are as follows:

• Implements a multi-dimensional data model (time series are identified by a metric name and a set of key/value pairs)
• PromQL, a powerful and flexible query language to leverage this dimensionality
• An HTTP pull model for time series collection
• Targets are discovered via service discovery or static configuration
• Precise alerting - Alerts are defined using PromQL and maintain dimensional information. An alert manager handles notifications and silencing.

To install Prometheus, we can use precompiled binaries which can be downloaded using following  link:

https://prometheus.io/download/

To start installation process, let's update all the packages to their latest available versions and also remove all the obsolete packages. Also, let's install wget package:

Install Prometheus on Amazon Linux 2

   
   	sudo yum -y update && sudo yum -y upgrade
    sudo yum install wget -y
   

Install Prometheus on Ubuntu

   
   	sudo apt-get update -y && sudo apt-get upgrade -y
	sudo apt-get install wget -y
   

As of writing this article, latest version of prometheus is 2.38.0.  To check for latest releases, go to following url.

https://github.com/prometheus/prometheus/releases

We need to download the latest release of Prometheus using following command:

   
mkdir workspace && cd workspace
wget https://github.com/prometheus/prometheus/releases/download/v2.38.0/prometheus-2.38.0.linux-amd64.tar.gz
   

It's a good practice to verify the integrity of a downloaded package using following command:

   
	sha256sum prometheus-2.38.0.linux-amd64.tar.gz
   

Output is as follows:

   
	43ddb515e1f3c28566aed0eba296bae136064e164ef0245986cfa9c497b839ce  prometheus-2.38.0.linux-amd64.tar.gz
   

If output from above command matches with checksum shown on the Prometheus download page, then we can move ahead with the installation. Else, we need to remove downloaded file and then repeat the download process for fetching prometheus zip file.

Let's extract the zip file using following command:

   
tar xvfz prometheus-2.38.0.linux-amd64.tar.gz
   

This will extract all the files to prometheus-2.38.0.linux-amd64 folder. Since the folder name is very long, it will be easier to work if we rename it to something smaller like only prometheus.

   
mv prometheus-2.38.0.linux-amd64 $HOME/workspace/prometheus
cd prometheus
   

Here, we can see several files and folders being extracted to the prometheus folder. There are two binary files - prometheus & promtool, which, we can use to start interacting with the prometheus server. Also, there is a main configuration YAML file, called prometheus.yml, which we need to configure for starting the server.

Let's check the CLI options that can be used with prometheus binary package using following command:

   
	./prometheus --help
   

Some of the most commonly used CLI parameters with prometheus binary are as follows:

Check the version of prometheus using following command:

   
	./prometheus --version
   

Output is as follows:

   
prometheus, version 2.38.0 (branch: HEAD, revision: 818d6e60888b2a3ea363aee8a9828c7bafd73699)
build user:       root@e6b781f65453
build date:       20220816-13:23:14
go version:       go1.18.5
platform:         linux/amd64
   

Let's start a prometheus server with very basic configurations:

   
./prometheus --config.file=$HOME/workspace/prometheus/prometheus.yml \
	--web.listen-address="0.0.0.0:9090" \
    --web.console.templates="$HOME/workspace/prometheus/consoles" \
    --web.console.libraries="$HOME/workspace/prometheus/console_libraries" \
    --storage.tsdb.path="$HOME/workspace/prometheus/" \
    --log.level=info
   

As you can see, we have to navigate to the prometheus folder, every time we want to interact with prometheus binary. There is a much better and standardized way of running binaries in linux machines. We usually store binary files in /usr/local/bin/ and configuration files in /etc directory. Let's implement the standard way of running software binaries in linux:

Create system user

In terms of security, the best practice is to use a separate user for running software processes. If you are using root user to run software processes, then you are exposing the system to higher security risks.

Let's create a system group called prometheus and assign that group to a newly created system user called prometheus by issuing following command:

   
sudo groupadd prometheus --system
sudo useradd -g prometheus --system --shell /sbin/nologin prometheus
   

Copy binary files to /usr/local/bin directory

Let's copy both prometheus and promtool binaries to /usr/local/bin/ directory. Also, issue chown command to change the user and group ownership of both binaries to prometheus user.

   
sudo cp $HOME/workspace/prometheus/prometheus /usr/local/bin/
sudo cp $HOME/workspace/prometheus/promtool /usr/local/bin/

sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool
   

Copy configuration files to /etc directory

Next, copy all of the configuration files to /etc/prometheus directory. Issue following commands to copy configuration files, as well as, also change the user and group ownership of configuration files to prometheus user.

   
sudo mkdir /etc/prometheus
sudo cp $HOME/workspace/prometheus/prometheus.yml  /etc/prometheus/prometheus.yml
sudo cp -R $HOME/workspace/prometheus/consoles /etc/prometheus
sudo cp -R $HOME/workspace/prometheus/console_libraries /etc/prometheus

sudo chown prometheus:prometheus /etc/prometheus/*
   

Create data directory

We also need to create a directory for storing metrics data collected by prometheus server. Usually, in linux machines, we use /var/lib folder to store databases of softwares. Also, change the user and group ownership of the data folder to prometheus user.

   
sudo mkdir /var/lib/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus
   

Basic configuration file for prometheus looks like below:

   
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=` to any timeseries scraped from this config.
  - job_name: "prometheus"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["localhost:9090"]
   

Create systemd service file

Usually in linux, we run software processes using systemctl command. To run prometheus server using systemctl command, we need to create a service file for it as well. To create a service file, issue following command:

   
sudo vi /etc/systemd/system/prometheus.service
   

Paste the following content in above service file:

   
[Unit]
Description=Prometheus Server
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
  --config.file /etc/prometheus/prometheus.yml \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --storage.tsdb.path /var/lib/prometheus/ \
  --web.listen-address=0.0.0.0:9090 \
  --log.level=info

Restart=always
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target
   

To reload the modification in system configurations, issue following command:

   
	sudo systemctl daemon-reload
   

To start prometheus server, issue following command:

   
	sudo systemctl start prometheus
   

To restart prometheus server, issue following command:

   
	sudo systemctl restart prometheus
   

To check the status of prometheus server, issue following command:

   
	sudo systemctl status prometheus
   

To make sure that prometheus server starts after each system reboot, issue following command:

   
	sudo systemctl enable prometheus
   

Check the version of prometheus using following command:

   
	prometheus --version
   

Install Prometheus in macOS

We can install prometheus in macOS using following command:

   
    brew update &&  brew upgrade
   	brew install prometheus
   

To start a prometheus service in macOS:

   
	brew services start prometheus 
   

To stop a prometheus service in macOS:

   
	brew services stop prometheus 
   

Now that the prometheus server is successfully installed, we can access it in the browser using following url:

http://localhost:9090/graph

Following image shows prometheus graph page, from where we can query for several metrics and show the metrics data either in table or graph visualizations. By default, prometheus is collecting metrics data about itself using its own HTTP metrics endpoint.

Navigate to following url to see the list of metrics exposed by prometheus.

http://localhost:9090/metrics

Let's see a graph visualizations for one of the metric exposed by prometheus itself called promhttp_metric_handler_requests_total(the total number of /metrics requests, the Prometheus server has served).

Also, we can see the list of registered targets in prometheus.yml file, by clicking on the menu items shown by red arrow:

Following image shows the list of registered targets as well as the health status of those targets. By default, only registered target is the prometheus server itself.

Note:

If you are using cloud server, you need to open port 9090 for inbound connections. For security reasons, since this is an internal monitoring tool, we can install it in a private server instance and make it accessible to the internet using proxy web server like NGINX. Also, we can further secure it by making it accessible using VPN connection only.

In our next chapter, we will discuss about the installation and configuration of grafana server.

                                                                                                              Next Chapter