Prometheus

Prometheus - an application that allows you to customize the process of collecting, storing and processing metrics.

Set up Prometheus before you start working.

Customize Prometheus

You can configure an agent to get metrics into your infrastructure.

Add a service user.
Configure the agent to retrieve metrics.

1. Add a service user

Add a service user with permission in the access area Projects and role:

metrics.admin;
I'm a member;
or reader.

Users can be added by the Account Owner or users with the role of iam.admin.

2. Configure the agent to receive metrics

Open the CLI.
Create the prometheus.yaml configuration file:
```
nano /etc/prometheus/prometheus.yaml
```
In the prometheus.yaml configuration file, fill in the scrape_configs block:
```
scrape_configs:
  - job_name: test_scrape
    scrape_interval: 1m
    metrics_path: /projects/<project_id>/namespaces/<namespace>/prometheus/metrics
    static_configs:
      - targets:
          - <base_url>
    basic_auth:
      username: <user_id>
      password: <password>
```
Specify:
- <project_id> - Project ID. Can be copied in control panel: in the top menu click Products and select any product → open the projects menu → in the line of the required project click ;
- <namespace> - the name of the selected namespace, e.g. compute;
- <base_url> - URL for accessing the API of the Metrics service. The list of URLs can be found in the instructions List of URLs;
- <user_id> - ID of the user you added in step 1;
- <password> - password of the user you you added in step 1.
Optional: you can configure metrics collection from multiple pools or projects. To add a new pool or project, repeat step 3.
Optional: add sending metrics to external storage or processing systems, such as VictoriaMetrics. To do this, add and complete the section remote_write to the prometheus.yaml configuration file.
Exit the nano text editor with your changes saved: press Ctrl+X and then Y+Enter.

Start Prometheus:

docker run \
  --name prometheus \
  --rm \
  -v ${PWD}/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro \
  -p 9090:9090 \
  prom/prometheus:latest \
  --config.file=/etc/prometheus/prometheus.yaml

Optional: make sure that metrics are collected. To do this, query all collected metrics in Prometheus:
```
curl localhost:9090/federate -X GET -d 'match[]={__name__=~".+"}'
```