Prometheus - Getting started with Installation and Configuration

Prometheus is an open-source software for monitoring and alerting purposes. It collect and store its metrics as time series data, a series of data that is collected over consistent intervals of time. It collect metrics from configured targets at given intervals, evaluates rule expressions, display the results, and can trigger alerts when specified conditions are met.  To collect metrics data, it uses a HTTP endpoint. Written in Golang, it is highly scalable and performant.

Originally developed by SoundCloud, it is now a Cloud Native Computing Foundation (CNCF) graduated project. It joined the Cloud Native Computing Foundation in 2016 as the second hosted project, after Kubernetes. With the help of prometheus, we can monitor various system infrastructures, databases, application services and many others.

Some of the notable features that it offers are as follows:

  • Implements a multi-dimensional data model (time series are identified by a metric name and a set of key/value pairs)
  • PromQL, a powerful and flexible query language to leverage this dimensionality
  • An HTTP pull model for time series collection
  • Targets are discovered via service discovery or static configuration
  • Precise alerting - Alerts are defined using PromQL and maintain dimensional information. An alert manager handles notifications and silencing.

To install Prometheus, we can use precompiled binaries which can be downloaded using following  link:

To start installation process, let's update all the packages to their latest available versions and also remove all the obsolete packages. Also, let's install wget package:

Install Prometheus on Amazon Linux 2

   	sudo yum -y update && sudo yum -y upgrade
    sudo yum install wget -y

Install Prometheus on Ubuntu

   	sudo apt-get update -y && sudo apt-get upgrade -y
	sudo apt-get install wget -y

As of writing this article, latest version of prometheus is 2.38.0.  To check for latest releases, go to following url.

We need to download the latest release of Prometheus using following command:

mkdir workspace && cd workspace

It's a good practice to verify the integrity of a downloaded package using following command:

	sha256sum prometheus-2.38.0.linux-amd64.tar.gz

Output is as follows:

	43ddb515e1f3c28566aed0eba296bae136064e164ef0245986cfa9c497b839ce  prometheus-2.38.0.linux-amd64.tar.gz
prometheus sha256 checksum

If output from above command matches with checksum shown on the Prometheus download page, then we can move ahead with the installation. Else, we need to remove downloaded file and then repeat the download process for fetching prometheus zip file.

Let's extract the zip file using following command:

tar xvfz prometheus-2.38.0.linux-amd64.tar.gz

This will extract all the files to prometheus-2.38.0.linux-amd64 folder. Since the folder name is very long, it will be easier to work if we rename it to something smaller like only prometheus.

mv prometheus-2.38.0.linux-amd64 $HOME/workspace/prometheus
cd prometheus
prometheus zip file contents

Here, we can see several files and folders being extracted to the prometheus folder. There are two binary files - prometheus & promtool, which, we can use to start interacting with the prometheus server. Also, there is a main configuration YAML file, called prometheus.yml, which we need to configure for starting the server.

Let's check the CLI options that can be used with prometheus binary package using following command:

	./prometheus --help
prometheus help configuration parameters

Some of the most commonly used CLI parameters with prometheus binary are as follows:

CLI Parameter Description
--version Show application version.
--config.file="prometheus.yml" Prometheus configuration file path.
--web.listen-address="" Address to listen on for client requests
--web.config.file="" [EXPERIMENTAL] Path to configuration file that can enable TLS or authentication.
--web.external-url= The URL under which Prometheus is externally reachable(if served by web server using reverse proxy)
--web.console.templates="consoles" Path to the console template directory, available at /consoles.
--web.console.libraries="console_libraries" Path to the console library directory.
--storage.tsdb.path="data/" Base path for metrics storage. Use with server mode only.
--log.level=info Only log messages with the given severity or above. One of: [debug, info, warn, error]
--log.format=logfmt Output format of log messages. One of: [logfmt, json]

Check the version of prometheus using following command:

	./prometheus --version

Output is as follows:

prometheus, version 2.38.0 (branch: HEAD, revision: 818d6e60888b2a3ea363aee8a9828c7bafd73699)
build user:       root@e6b781f65453
build date:       20220816-13:23:14
go version:       go1.18.5
platform:         linux/amd64

Let's start a prometheus server with very basic configurations:

./prometheus --config.file=$HOME/workspace/prometheus/prometheus.yml \
	--web.listen-address="" \
    --web.console.templates="$HOME/workspace/prometheus/consoles" \
    --web.console.libraries="$HOME/workspace/prometheus/console_libraries" \
    --storage.tsdb.path="$HOME/workspace/prometheus/" \
prometheus server start

As you can see, we have to navigate to the prometheus folder, every time we want to interact with prometheus binary. There is a much better and standardized way of running binaries in linux machines. We usually store binary files in /usr/local/bin/ and configuration files in /etc directory. Let's implement the standard way of running software binaries in linux:

Create system user

In terms of security, the best practice is to use a separate user for running software processes. If you are using root user to run software processes, then you are exposing the system to higher security risks.

Let's create a system group called prometheus and assign that group to a newly created system user called prometheus by issuing following command:

sudo groupadd prometheus --system
sudo useradd -g prometheus --system --shell /sbin/nologin prometheus

Copy binary files to /usr/local/bin directory

Let's copy both prometheus and promtool binaries to /usr/local/bin/ directory. Also, issue chown command to change the user and group ownership of both binaries to prometheus user.

sudo cp $HOME/workspace/prometheus/prometheus /usr/local/bin/
sudo cp $HOME/workspace/prometheus/promtool /usr/local/bin/

sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool

Copy configuration files to /etc directory

Next, copy all of the configuration files to /etc/prometheus directory. Issue following commands to copy configuration files, as well as, also change the user and group ownership of configuration files to prometheus user.

sudo mkdir /etc/prometheus
sudo cp $HOME/workspace/prometheus/prometheus.yml  /etc/prometheus/prometheus.yml
sudo cp -R $HOME/workspace/prometheus/consoles /etc/prometheus
sudo cp -R $HOME/workspace/prometheus/console_libraries /etc/prometheus

sudo chown prometheus:prometheus /etc/prometheus/*

Create data directory

We also need to create a directory for storing metrics data collected by prometheus server. Usually, in linux machines, we use /var/lib folder to store databases of softwares. Also, change the user and group ownership of the data folder to prometheus user.

sudo mkdir /var/lib/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

Basic configuration file for prometheus looks like below:

  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
    - static_configs:
        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
  # The job name is added as a label `job=` to any timeseries scraped from this config.
  - job_name: "prometheus"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

      - targets: ["localhost:9090"]

Create systemd service file

Usually in linux, we run software processes using systemctl command. To run prometheus server using systemctl command, we need to create a service file for it as well. To create a service file, issue following command:

sudo vi /etc/systemd/system/prometheus.service

Paste the following content in above service file:

Description=Prometheus Server

ExecStart=/usr/local/bin/prometheus \
  --config.file /etc/prometheus/prometheus.yml \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --storage.tsdb.path /var/lib/prometheus/ \
  --web.listen-address= \

ExecReload=/bin/kill -HUP $MAINPID


To reload the modification in system configurations, issue following command:

	sudo systemctl daemon-reload

To start prometheus server, issue following command:

	sudo systemctl start prometheus

To restart prometheus server, issue following command:

	sudo systemctl restart prometheus

To check the status of prometheus server, issue following command:

	sudo systemctl status prometheus
prometheus status check

To make sure that prometheus server starts after each system reboot, issue following command:

	sudo systemctl enable prometheus

Check the version of prometheus using following command:

	prometheus --version
prometheus version check

Install Prometheus in macOS

We can install prometheus in macOS using following command:

    brew update &&  brew upgrade
   	brew install prometheus

To start a prometheus service in macOS:

	brew services start prometheus 

To stop a prometheus service in macOS:

	brew services stop prometheus 

Now that the prometheus server is successfully installed, we can access it in the browser using following url:


Following image shows prometheus graph page, from where we can query for several metrics and show the metrics data either in table or graph visualizations. By default, prometheus is collecting metrics data about itself using its own HTTP metrics endpoint.

prometheus dashboard

Navigate to following url to see the list of metrics exposed by prometheus.


prometheus metrics endpoint

Let's see a graph visualizations for one of the metric exposed by prometheus itself called promhttp_metric_handler_requests_total(the total number of /metrics requests, the Prometheus server has served).

prometheus metrics about prometheus_target_interval_length_seconds visualizations screen

Also, we can see the list of registered targets in prometheus.yml file, by clicking on the menu items shown by red arrow:

prometheus dashboard - target menu

Following image shows the list of registered targets as well as the health status of those targets. By default, only registered target is the prometheus server itself.

prometheus dashboard - list of targets


If you are using cloud server, you need to open port 9090 for inbound connections. For security reasons, since this is an internal monitoring tool, we can install it in a private server instance and make it accessible to the internet using proxy web server like NGINX. Also, we can further secure it by making it accessible using VPN connection only.

In our next chapter, we will discuss about the installation and configuration of grafana server.

                                                                                                              Next Chapter