Prometheus is an open-source software for monitoring and alerting purposes. It collect and store its metrics as time series data, a series of data that is collected over consistent intervals of time. It collect metrics from configured targets at given intervals, evaluates rule expressions, display the results, and can trigger alerts when specified conditions are met. To collect metrics data, it uses a HTTP endpoint. Written in Golang, it is highly scalable and performant.
Originally developed by SoundCloud, it is now a Cloud Native Computing Foundation (CNCF) graduated project. It joined the Cloud Native Computing Foundation in 2016 as the second hosted project, after Kubernetes. With the help of prometheus, we can monitor various system infrastructures, databases, application services and many others.
Some of the notable features that it offers are as follows:
- Implements a multi-dimensional data model (time series are identified by a metric name and a set of key/value pairs)
- PromQL, a powerful and flexible query language to leverage this dimensionality
- An HTTP pull model for time series collection
- Targets are discovered via service discovery or static configuration
- Precise alerting - Alerts are defined using PromQL and maintain dimensional information. An alert manager handles notifications and silencing.
To install Prometheus, we can use precompiled binaries which can be downloaded using following link:
To start installation process, let's update all the packages to their latest available versions and also remove all the obsolete packages. Also, let's install wget package:
Install Prometheus on Amazon Linux 2
sudo yum -y update && sudo yum -y upgrade sudo yum install wget -y
Install Prometheus on Ubuntu
sudo apt-get update -y && sudo apt-get upgrade -y sudo apt-get install wget -y
As of writing this article, latest version of prometheus is 2.38.0. To check for latest releases, go to following url.
We need to download the latest release of Prometheus using following command:
mkdir workspace && cd workspace wget https://github.com/prometheus/prometheus/releases/download/v2.38.0/prometheus-2.38.0.linux-amd64.tar.gz
It's a good practice to verify the integrity of a downloaded package using following command:
Output is as follows:
If output from above command matches with checksum shown on the Prometheus download page, then we can move ahead with the installation. Else, we need to remove downloaded file and then repeat the download process for fetching prometheus zip file.
Let's extract the zip file using following command:
tar xvfz prometheus-2.38.0.linux-amd64.tar.gz
This will extract all the files to prometheus-2.38.0.linux-amd64 folder. Since the folder name is very long, it will be easier to work if we rename it to something smaller like only prometheus.
mv prometheus-2.38.0.linux-amd64 $HOME/workspace/prometheus cd prometheus
Here, we can see several files and folders being extracted to the prometheus folder. There are two binary files - prometheus & promtool, which, we can use to start interacting with the prometheus server. Also, there is a main configuration YAML file, called prometheus.yml, which we need to configure for starting the server.
Let's check the CLI options that can be used with prometheus binary package using following command:
Some of the most commonly used CLI parameters with prometheus binary are as follows:
|--version||Show application version.|
|--config.file="prometheus.yml"||Prometheus configuration file path.|
|--web.listen-address="0.0.0.0:9090"||Address to listen on for client requests|
|--web.config.file=""||[EXPERIMENTAL] Path to configuration file that can enable TLS or authentication.|
||The URL under which Prometheus is externally reachable(if served by web server using reverse proxy)|
|--web.console.templates="consoles"||Path to the console template directory, available at /consoles.|
|--web.console.libraries="console_libraries"||Path to the console library directory.|
|--storage.tsdb.path="data/"||Base path for metrics storage. Use with server mode only.|
|--log.level=info||Only log messages with the given severity or above. One of: [debug, info, warn, error]|
|--log.format=logfmt||Output format of log messages. One of: [logfmt, json]|
Check the version of prometheus using following command:
Output is as follows:
prometheus, version 2.38.0 (branch: HEAD, revision: 818d6e60888b2a3ea363aee8a9828c7bafd73699) build user: root@e6b781f65453 build date: 20220816-13:23:14 go version: go1.18.5 platform: linux/amd64
Let's start a prometheus server with very basic configurations:
./prometheus --config.file=$HOME/workspace/prometheus/prometheus.yml \ --web.listen-address="0.0.0.0:9090" \ --web.console.templates="$HOME/workspace/prometheus/consoles" \ --web.console.libraries="$HOME/workspace/prometheus/console_libraries" \ --storage.tsdb.path="$HOME/workspace/prometheus/" \ --log.level=info
As you can see, we have to navigate to the prometheus folder, every time we want to interact with prometheus binary. There is a much better and standardized way of running binaries in linux machines. We usually store binary files in /usr/local/bin/ and configuration files in /etc directory. Let's implement the standard way of running software binaries in linux:
Create system user
In terms of security, the best practice is to use a separate user for running software processes. If you are using root user to run software processes, then you are exposing the system to higher security risks.
Let's create a system group called prometheus and assign that group to a newly created system user called prometheus by issuing following command:
sudo groupadd prometheus --system sudo useradd -g prometheus --system --shell /sbin/nologin prometheus
Copy binary files to /usr/local/bin directory
Let's copy both prometheus and promtool binaries to /usr/local/bin/ directory. Also, issue chown command to change the user and group ownership of both binaries to prometheus user.
sudo cp $HOME/workspace/prometheus/prometheus /usr/local/bin/ sudo cp $HOME/workspace/prometheus/promtool /usr/local/bin/ sudo chown prometheus:prometheus /usr/local/bin/prometheus sudo chown prometheus:prometheus /usr/local/bin/promtool
Copy configuration files to /etc directory
Next, copy all of the configuration files to /etc/prometheus directory. Issue following commands to copy configuration files, as well as, also change the user and group ownership of configuration files to prometheus user.
sudo mkdir /etc/prometheus sudo cp $HOME/workspace/prometheus/prometheus.yml /etc/prometheus/prometheus.yml sudo cp -R $HOME/workspace/prometheus/consoles /etc/prometheus sudo cp -R $HOME/workspace/prometheus/console_libraries /etc/prometheus sudo chown prometheus:prometheus /etc/prometheus/*
Create data directory
We also need to create a directory for storing metrics data collected by prometheus server. Usually, in linux machines, we use /var/lib folder to store databases of softwares. Also, change the user and group ownership of the data folder to prometheus user.
sudo mkdir /var/lib/prometheus sudo chown prometheus:prometheus /var/lib/prometheus
Basic configuration file for prometheus looks like below:
global: scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute. evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute. # scrape_timeout is set to the global default (10s). # Alertmanager configuration alerting: alertmanagers: - static_configs: - targets: # - alertmanager:9093 # Load rules once and periodically evaluate them according to the global 'evaluation_interval'. rule_files: # - "first_rules.yml" # - "second_rules.yml" # A scrape configuration containing exactly one endpoint to scrape: # Here it's Prometheus itself. scrape_configs: # The job name is added as a label `job=
` to any timeseries scraped from this config. - job_name: "prometheus" # metrics_path defaults to '/metrics' # scheme defaults to 'http'. static_configs: - targets: ["localhost:9090"]
Create systemd service file
Usually in linux, we run software processes using systemctl command. To run prometheus server using systemctl command, we need to create a service file for it as well. To create a service file, issue following command:
sudo vi /etc/systemd/system/prometheus.service
Paste the following content in above service file:
[Unit] Description=Prometheus Server Wants=network-online.target After=network-online.target [Service] User=prometheus Group=prometheus Type=simple ExecStart=/usr/local/bin/prometheus \ --config.file /etc/prometheus/prometheus.yml \ --web.console.templates=/etc/prometheus/consoles \ --web.console.libraries=/etc/prometheus/console_libraries \ --storage.tsdb.path /var/lib/prometheus/ \ --web.listen-address=0.0.0.0:9090 \ --log.level=info Restart=always ExecReload=/bin/kill -HUP $MAINPID [Install] WantedBy=multi-user.target
To reload the modification in system configurations, issue following command:
sudo systemctl daemon-reload
To start prometheus server, issue following command:
sudo systemctl start prometheus
To restart prometheus server, issue following command:
sudo systemctl restart prometheus
To check the status of prometheus server, issue following command:
sudo systemctl status prometheus
To make sure that prometheus server starts after each system reboot, issue following command:
sudo systemctl enable prometheus
Check the version of prometheus using following command:
Install Prometheus in macOS
We can install prometheus in macOS using following command:
brew update && brew upgrade brew install prometheus
To start a prometheus service in macOS:
brew services start prometheus
To stop a prometheus service in macOS:
brew services stop prometheus
Now that the prometheus server is successfully installed, we can access it in the browser using following url:
Following image shows prometheus graph page, from where we can query for several metrics and show the metrics data either in table or graph visualizations. By default, prometheus is collecting metrics data about itself using its own HTTP metrics endpoint.
Navigate to following url to see the list of metrics exposed by prometheus.
Let's see a graph visualizations for one of the metric exposed by prometheus itself called promhttp_metric_handler_requests_total(the total number of /metrics requests, the Prometheus server has served).
Also, we can see the list of registered targets in prometheus.yml file, by clicking on the menu items shown by red arrow:
Following image shows the list of registered targets as well as the health status of those targets. By default, only registered target is the prometheus server itself.
If you are using cloud server, you need to open port 9090 for inbound connections. For security reasons, since this is an internal monitoring tool, we can install it in a private server instance and make it accessible to the internet using proxy web server like NGINX. Also, we can further secure it by making it accessible using VPN connection only.
In our next chapter, we will discuss about the installation and configuration of grafana server.