Monitoring of cluster, nodes and PostgreSQL databases

Last update: October 08 2024

Monitoring of cluster, nodes and PostgreSQL databases

In cloud databases PostgreSQL you can monitor the status of the cluster.

To assess the overall state of the cluster check his status..

For more detailed analysis, some metrics can be viewed as graphs in the dashboard:

• node cluster metrics;
• database metrics;
• coupling pooler metrics.

The full set of available metrics can be export in Prometheus format.

You can find out if your disk is about to run out of space by using disk fill notifications.

View cluster status⁠

1. In control panels go to Cloud platform → Databases.
2. In the cluster row, look at the status.

ACTIVE	The cluster is available
CREATING	A cluster is created
UPDATING	Changes are applied to the cluster
RESIZING	The cluster is scalable
ERROR	There's been a mistake, file a ticket
DISK FULL	The disk is full and the cluster is read-only. To make the cluster read and write, disk cleanup or scale the cluster and select a configuration with larger disk size
DEGRADED	Some nodes in the cluster are unavailable
DELETING	The cluster is being deleted

View the status of the node cluster⁠

1. In control panels go to Cloud platform → Databases.
2. Open the cluster page → tab Monitoring.
3. In the block Cluster monitoring click Nodes of the cluster.
4. In the field Server select the node whose metrics you want to view.
5. See what's available node cluster metrics.

Cluster node metrics in the control panel⁠

vCPU	How many percent of the node cluster cores are utilized
CPU iowait	How much percent of the processor's time was spent waiting for I/O
Memory	Memory utilization excluding cache and operating system buffers in percent or gigabytes
Disk	Used disk space in percent or gigabytes. The file system reserves 4% of the disk space for service needs, so the space occupied is greater than the actual space used by this 4%
Network load	The number of bits or packets sent and received over the network interface
Disk, read and write speed	Data read speed in B/s and data write speed in KB/s
Disk, number of IOPS	Number of read and write operations per second
OOM	Number of processes that ended with an error Out of Memory due to lack of RAM
Load Average	The average value of system load over a period of time. Shows how many processes are processed by the cluster cores. The indicator is presented in the form of three values — for one minute, five minutes and 15 minutes. These values should not be greater than the number of cores on the node

View the status of the databases⁠

1. In control panels go to Cloud platform → Databases.
2. Open the cluster page → tab Monitoring.
3. In the block Cluster monitoring click Databases.
4. In the field Server select the node whose metrics you want to view.
5. See what's available database metrics.

Database metrics in the dashboard⁠

Connections	Number of connections to each cluster database and total number of connections to all databases
Execution time of the longest query	Execution time of the longest query in each database of the cluster for a period of time
Transactions	Number of transactions per second in each cluster database
Cache hit	What percentage of the data in the query was read from the cache is a ratio of blks_hit to the sum of blks_hit and blks_read
Locks	Number of locks in each cluster database
Deadlocks	Number of mutual locks in each database
Database size	Total size of the selected database in megabytes
Statistics file size	Total size of the statistics file in kilobytes
Size of WAL files	Total size of WAL files in megabytes
Temporary file size	Total size of temporary files in kilobytes
String operations	The number of operations in the selected database per second: tup_deleted — number of rows deleted by queries in the database per second; tup_fetched — the number of rows retrieved by queries in the database per second; tup_inserted — the number of rows inserted by queries in the database per second; tup_returned — the number of rows returned by queries in the database per second; tup_updated — number of rows changed by queries in the database per second

View the status of the connection pooler⁠

1. In control panels go to Cloud platform → Databases.
2. Open the cluster page → tab Monitoring.
3. In the block Cluster monitoring click Connection Pooler.
4. In the field Server select the node whose metrics you want to view.
5. See what's available node cluster metrics.

Connection pooler metrics in the control panel⁠

Client connections to the pool	The number of client connections to the pool in the selected database: pools_client_active_connections — the number of client connections associated with server connections or idle without requests; pools_client_waiting_connections — number of client connections where the request has been sent, but there is no connection to the node yet
Active connections to the server	Number of server connections associated with clients in the selected database
Waiting time for a response from the server	Time to wait for a response from a node in the selected database in seconds
Maximum waiting time of the client in the queue	Maximum waiting time of the client in the queue in the selected database in seconds

Export metrics in Prometheus format⁠

Historical information for clusters is not available — metrics are requested only in real time. The list of all metrics that are supported in cloud databases and their description can be seen in the following table Metrics in Prometheus format.

1. Get a token.
2. Get metrics in Prometheus format.

Get a token⁠

The token gives access to the metrics of all clusters project in one bullet.

1. In control panels go to Cloud platform → Databases.

2. Open the cluster page → tab Monitoring.

3. In the block Tokens for Prometheus click Add token. The token will be generated automatically.

4. Copy the token. To do this, in the token row, click .

Get metrics in Prometheus format⁠

Configuration file

1. Add to the Prometheus configuration file:

   scrape_configs:
     - job_name: get-metrics-from-dbaas
       scrape_interval: 1m
       static_configs:
         - targets:
           - '<pool>.dbaas.selcloud.ru'
       scheme: https
       authorization:
         type: Bearer
         credentials: <monitoring_token>

   Specify:

   • <pool> — pool in which the token is valid, e.g. ru-3. The address (URL) depends on the region and pool, you can look in the URL list;
   • <monitoring_token> — a monitoring token that you received Previously.

2. Open a page in your browser where Prometheus-formatted metrics will be available:

   http://<ip_address>:9090/targets

   Specify <ip_address> — IP address where Prometheus is installed.

3. Independently configure monitoring and alerts for database clusters.

CLI

1. Open the CLI.

2. To get the metrics, submit a request:

   curl -L "https://<pool>.dbaas.selcloud.ru/metrics" -H "Authorization: Bearer <monitoring_token>"

   Specify:

   • <pool> — pool in which the token is valid, e.g. ru-3. The address (URL) depends on the region and pool, you can look in the URL list;
   • <monitoring_token> — a monitoring token that you received Previously.

   Available metrics in Prometheus format will appear in the response.

3. Independently configure monitoring and alerts for database clusters.

Metrics in Prometheus format⁠

Metrics in Prometheus format are provided for all clusters. A specific cluster can be found by the database cluster identifier in the label ds_id.

Infrastructure level metrics

dbaas_memory_percent	Memory utilization excluding cache and operating system buffers (RAM) in percent
dbaas_memory_bytes	Occupied memory excluding cache and operating system buffers (RAM) in bytes
dbaas_oom_count	Number of processes that ended with an error Out of Memory due to lack of RAM
dbaas_cpu	Percent vCPU utilization on database cluster nodes
dbaas_cpu_iowait	I/O waiting time in percent
dbaas_disk_percent	Percentage of disk space occupied. The file system reserves 4% of disk space for service needs, so the occupied space is larger than the actual space used by this 4%
dbaas_disk_bytes	Occupied disk space in bytes. The file system reserves 4% of the disk space for service needs, so the occupied space is larger than the actual space used by this 4%
dbaas_disk_read_iops	Number of read operations per second
dbaas_disk_write_iops	Number of recording operations per second
dbaas_disk_read_bytes	Disk read speed in bytes per second
dbaas_disk_write_bytes	Data write speed to disk in bytes per second
dbaas_node_load1	The average value of system load in one minute. Shows how many processes are processed by the cluster cores
dbaas_node_load5	The average system utilization over five minutes. Shows how many processes are processed by the cluster cores
dbaas_node_load15	The average value of system utilization over 15 minutes. Shows how many processes are processed by the cluster cores
dbaas_network_receive_bytes	Number of bytes received through the network interface
dbaas_network_transmit_bytes	Number of bytes sent through the network interface
dbaas_network_receive_packets	Number of packets received through the network interface per second
dbaas_network_transmit_packets	Number of packets sent through the network interface per second
dbaas_role	Role of the node: 0 — role unknown; 1 — master; 2 — replica

Application level metrics

dbaas_connections	Number of active process connections PostgreSQL. For example, you can use labels: ds_name — database cluster name; datname — database name.
dbaas_total_connections	Total number of established connections to the process PostgreSQL
dbaas_max_tx_duration	Execution time of the longest request in seconds
dbaas_xact_commit_rollback	The number of transactions per second in each database in the cluster. For example, you can use labels: ds_name — database cluster name; datname — database name.
dbaas_tup_deleted	Number of rows per second removed by queries in the database
dbaas_tup_fetched	Number of rows per second retrieved by queries in the database
dbaas_tup_inserted	Number of rows per second inserted by queries in the database
dbaas_tup_returned	Number of rows per second returned by queries in the database
dbaas_tup_updated	Number of rows per second changed by queries in the database
dbaas_xact_commit	Number of recorded transactions per second in the database
dbaas_xact_rollback	Number of transactions per second in the database for which rollback was performed
dbaas_cache_hit_ratio	Percentage of data in the query that was read from the cache — ratio of blks_hit to the sum of blks_hit and blks_read
dbaas_deadlocks	The number of mutual locks per second in each database. For example, you can use labels: ds_name — database cluster name; datname — database name.
dbaas_locks	The number of locks per second in each cluster database. For example, you can use labels: ds_name — database cluster name; datname — database name.
dbaas_pg_pgss_query_texts_size_bytes	File size with statistics from pg_stat_statements in bytes
dbaas_pg_total_wals_size_bytes	Directory size with WAL files in bytes
dbaas_pg_tmp_size_bytes	Temporary file size PostgreSQL in bytes
dbaas_databases_size_bytes	The total size of the database in bytes. For example, you can use the label datname — database name
dbaas_pgbouncer_pools_client_maxwait_seconds	Maximum waiting time of the client in the queue in seconds
dbaas_pgbouncer_pools_client_waiting_connections	Number of client connections where the request has been sent but the connection to the node is not yet available
dbaas_pgbouncer_stats_client_wait_seconds_total	Time to wait for a response from a node in microseconds
dbaas_pgbouncer_pools_client_active_connections	The number of client connections associated with server connections or idle without requests. For example, you can use labels: ds_id — database cluster identifier; data_base — database name.
dbaas_pgbouncer_pools_server_active_connections	Number of server connections associated with clients
dbaas_pg_replication_slot_active	Replication slot status: 0 — the slot is not used. The slot has no consumer, the data is not transferred to the receiving database, accumulates in the replication slot and occupies additional disk space; 1 — slot is used. The slot has a consumer, the data is transmitted to the receiving database
dbaas_pg_replication_slot_lag	The size of accumulated WAL files in megabytes. Indicates how much transactional information the receiving database needs to process to catch up with the source database

Disk fullness notifications⁠

If the cluster disk is 80% full, a notification will appear in the dashboard and will be sent to the email of the Account Owner and those users subscribed to the the "Services and services" notification category.

If the cluster disk is 95% full or more, the cluster will go into status DISK_FULL and will be read-only. To make the cluster read-write, disk cleanup or scale the cluster and select a configuration with a larger disk size.