Skip to main content

Metrics Reference

This instruction describes the product metrics collected by the service.

The list of products that support metrics will be expanded.

Dedicated Servers

The dedicated servers namespace is dedicated.

Monitoring is supported by servers with a KVM console, which is available for every dedicated server, except servers of the Chipcore product line and servers in a dedicated rack — for details, see the Servers in a dedicated rack page on selectel.ru.

The set of IPMI metrics may differ depending on the server model.

Each metric includes a timestamp — the metric collection time. You can convert the value using online services, for example, on the unixtimestamp.com website.

Metric name
Type, unit of measurement
DescriptionLabels
ipmi_bmc_info
Gauge, UNIT
BMC (Baseboard Management Controller) metadata — firmware, manufacturer, and more. The metric value is always 1
  • datacenter — name of the data center where the server is located;
  • location_id — pool where the server is located;
  • resource_id — server UUID;
  • bmc_url — BMC address;
  • firmware_revision — BMC firmware version;
  • manufacturer_id — manufacturer (vendor) ID;
  • system_firmware_version — system firmware version (BIOS, UEFI)
up
Gauge, bool

Check metric source availability:

  • 1 — success;
  • 0 — failure
  • datacenter — name of the data center where the server is located;
  • location_id — pool where the server is located;
  • resource_id — server UUID
ipmi_up
Gauge, bool

Check IPMI availability:

  • 1 — success;
  • 0 — failure
  • datacenter — name of the data center where the server is located;
  • collector — name of the exporter's internal collector responsible for gathering a specific group of metrics;
  • location_id — pool where the server is located;
  • resource_id — server UUID
ipmi_scrape_duration_seconds
Gauge, seconds
Time taken to collect IPMI exporter metrics
  • datacenter — name of the data center where the server is located;
  • location_id — pool where the server is located;
  • resource_id — server UUID
ipmi_sel_events_count_by_state
Gauge, count
Number of events from the SEL (System Event Log) journal, grouped by event severity level
  • datacenter — name of the data center where the server is located;
  • location_id — pool where the server is located;
  • resource_id — server UUID;
  • state — event severity level
ipmi_chassis_cooling_fault_state
Gauge, bool

Server cooling system state:

  • 1 — healthy;
  • 0 — failure
  • datacenter — name of the data center where the server is located;
  • location_id — pool where the server is located;
  • resource_id — server UUID
ipmi_chassis_drive_fault_state
Gauge, bool

Drive state:

  • 1 — healthy;
  • 0 — failure
  • datacenter — name of the data center where the server is located;
  • location_id — pool where the server is located;
  • resource_id — server UUID
ipmi_chassis_power_state
Gauge, bool

Server power state:

  • 1 — powered on;
  • 0 — powered off
  • datacenter — name of the data center where the server is located;
  • location_id — pool where the server is located;
  • resource_id — server UUID
ipmi_fan_speed_state
Gauge, enum

Fan state:

  • 0 — healthy;
  • 1 — deviation;
  • 2 — failure
  • datacenter — name of the data center where the server is located;
  • location_id — pool where the server is located;
  • resource_id — server UUID;
  • id — sensor identifier;
  • name — sensor name
ipmi_power_state
Gauge, enum

Server component power state:

  • 0 — healthy;
  • 1 — deviation;
  • 2 — failure
  • datacenter — name of the data center where the server is located;
  • location_id — pool where the server is located;
  • resource_id — server UUID;
  • id — sensor ID;
  • name — sensor name
ipmi_temperature_state
Gauge, enum

Server component temperature state:

  • 0 — healthy;
  • 1 — deviation;
  • 2 — failure
  • datacenter — name of the data center where the server is located;
  • location_id — pool where the server is located;
  • resource_id — server UUID;
  • id — sensor ID;
  • name — sensor name
ipmi_voltage_state
Gauge, enum

Voltage state:

  • 0 — healthy;
  • 1 — deviation;
  • 2 — failure
  • datacenter — name of the data center where the server is located;
  • location_id — pool where the server is located;
  • resource_id — server UUID;
  • id — sensor ID;
  • name — sensor name
ipmi_sensor_state
Gauge, enum

Server component state:

  • 0 — healthy;
  • 1 — deviation;
  • 2 — failure
  • datacenter — name of the data center where the server is located;
  • location_id — pool where the server is located;
  • resource_id — server UUID;
  • id — sensor ID;
  • name — sensor name;
  • type — sensor type
ipmi_fan_speed_rpm
Gauge, RPM
Fan rotation speed
  • datacenter — name of the data center where the server is located;
  • location_id — pool where the server is located;
  • resource_id — server UUID;
  • id — sensor ID;
  • name — sensor name
ipmi_power_watts
Gauge, watts
Power consumption
  • datacenter — name of the data center where the server is located;
  • location_id — pool where the server is located;
  • resource_id — server UUID;
  • id — sensor ID;
  • name — sensor name
ipmi_temperature_celsius
Gauge, degrees °C
Server component temperature
  • datacenter — name of the data center where the server is located;
  • location_id — pool where the server is located;
  • resource_id — server UUID;
  • name — sensor name;
  • id — sensor ID
ipmi_voltage_volts
Gauge, volts
Voltage
  • datacenter — name of the data center where the server is located;
  • location_id — pool where the server is located;
  • resource_id — server UUID;
  • id — sensor ID;
  • name — sensor name
ipmi_sensor_value
Gauge
Data from the IPMI sensor. The unit of measurement depends on the sensor type
  • datacenter — name of the data center where the server is located;
  • location_id — pool where the server is located;
  • resource_id — server UUID;
  • id — sensor ID;
  • name — sensor name;
  • type — sensor type

Cloud servers

The cloud servers namespace is compute.

Metrics are collected at the hypervisor level. Metrics from the hypervisor side and metrics inside the cloud server OS may differ in values:

  • hypervisor metrics reflect the physical state of cloud servers and their resources — CPU, memory, disks, and network, including disk and network operation maintenance work and virtual machine service operations;
  • in-OS metrics reflect only computations seen by the OS kernel and do not take background work into account.

Under high disk or network load, metric values may differ — this is expected behavior for a virtualized environment, not a metric collection error.

Metric collection does not depend on the OS state. Metrics are collected even if the cloud server is booting, is under heavy load, or is not responding over the network. Metrics are not collected only when the cloud server is powered off.

Metric name
Type, unit of measurement
DescriptionLabels
server_meta_info
Gauge, UNIT
Cloud server metadata
  • project_name — project name;
  • server_uuid — cloud server ID;
  • server_name — cloud server name
server_state_info
Gauge, UNIT

Current cloud server status:

  • 0 (no state) — server state is unknown or not defined;
  • 1 (running) — server is running;
  • 2 (blocked) — server is blocked (waiting for resources or completion of a disk read/write operation);
  • 3 (paused) — server is paused;
  • 4 (shutdown) — server is shutting down;
  • 5 (shut off) — server is stopped and powered off;
  • 6 (crashed) — server crashed;
  • 7 (pmsuspended) — server is suspended and has entered sleep mode
  • server_uuid — cloud server ID
cpu_count
Gauge, count
Number of cloud server vCPUs
  • server_uuid — cloud server ID
vcpu_usage_ratio
Gauge, %
Utilization of each cloud server vCPU in %
  • server_uuid — cloud server ID;
  • vcpu — virtual processor number
cpu_usage_ratio
Gauge, %
Cloud server CPU utilization in %
  • server_uuid — cloud server ID
memory_total_bytes
Gauge, bytes
Cloud server memory size
  • server_uuid — cloud server ID
memory_usage_bytes
Gauge, bytes
Memory size used by the cloud server
  • server_uuid — cloud server ID
memory_usage_ratio
Gauge, %
percentage of RAM used by the cloud server
  • server_uuid — cloud server ID
memory_major_faults_total
Counter, count
Number of major page faults (major page faults) that have occurred on the cloud server since its last startup
  • server_uuid — cloud server ID
disk_meta_info
Gauge, UNIT
Cloud server disk metadata
  • server_uuid — cloud server ID;
  • disk_uuid — cloud server disk ID;
  • disk_type="network" / "file" — disk type
disk_write_requests_rate
Gauge, operations/s
Number of write operations per second on the cloud server disk
  • server_uuid — cloud server ID;
  • disk_uuid — cloud server disk ID
disk_read_requests_rate
Gauge, operations/s
Number of read operations per second from the cloud server disk
  • server_uuid — cloud server ID;
  • disk_uuid — cloud server disk ID
disk_read_requests_limit
Gauge, operations/s
Maximum number of read operations for the cloud server disk
  • server_uuid — cloud server ID;
  • disk_uuid — cloud server disk ID
disk_write_requests_limit
Gauge, operations/s
Maximum number of write operations for the cloud server disk
  • server_uuid — cloud server ID;
  • disk_uuid — cloud server disk ID
disk_flush_requests_rate
Gauge, ops/s
Number of disk cache flush operations for the cloud server disk
  • server_uuid — cloud server ID;
  • disk_uuid — cloud server disk ID
disk_read_bytes_rate
Gauge, bytes/s
Number of bytes per second read from the cloud server disk
  • server_uuid — cloud server ID;
  • disk_uuid — cloud server disk ID
disk_write_bytes_rate
Gauge, bytes/s
Number of bytes per second written to the cloud server disk
  • server_uuid — cloud server ID;
  • disk_uuid — cloud server disk ID
network_meta_info
Gauge, UNIT
Cloud server network interface metadata
  • server_uuid — cloud server ID;
  • virtual_interface — cloud server port ID
network_receive_packets_rate
Gauge, packets/s
Number of packets per second received on the cloud server network interface
  • server_uuid — cloud server ID;
  • virtual_interface — cloud server port ID
network_transmit_packets_rate
Gauge, packets/s
Number of packets per second transmitted through the cloud server network interface
  • server_uuid — cloud server ID;
  • virtual_interface — cloud server port ID
network_receive_bits_rate
Gauge, bits/s
Number of bits per second received on the cloud server network interface
  • server_uuid — cloud server ID;
  • virtual_interface — cloud server port ID
network_transmit_bits_rate
Gauge, bits/s
Number of bits per second sent through the cloud server network interface
  • server_uuid — cloud server ID;
  • virtual_interface — cloud server port ID

Dedicated Inference

Dedicated Inference namespace — inference.

Metric name
Type, unit of measurement
DescriptionLabels
info
Gauge, UNIT

Inference service metadata. The metric value is always 1

  • service_id — inference service ID;
  • name — inference service name;
  • status — inference service status;
  • availability_zone — pool segment where the inference service is located;
  • region — pool where the inference service is located;
  • gpu_count — number of GPUs;
  • gpu_name — GPU model;
  • ram_mb — RAM size in MB;
  • vcpu — number of vCPUs;
  • disk_size_gb — disk size in GB;
  • disk_type — disk type;
  • vm_type — configuration line of the cloud server on which the inference service is deployed:
config_info
Gauge, UNIT

Configuration of the inference service. The metric value is always 1

  • service_id — inference service ID;
  • max_concurrent_requests — maximum number of concurrent requests;
  • public_access_enabled — connection type to the inference service:
    • true — public access;
    • false — private access;
  • logs_enabled — logging:
    • true — enabled;
    • false — disabled
autoscaling_config_info
Gauge, UNIT

Autoscaling settings of the inference service. The metric value is always 1

  • service_id — inference service ID;
  • min_replicas — minimum number of inference instances;
  • max_replicas — maximum number of inference instances
model_config_info
Gauge, UNIT

Inference service model configuration. The metric value is always 1

  • service_id — inference service ID;
  • model_id — model name;
  • model_quantization — model quantization type;
  • cache_quantization — data type for KV cache (Key-Value Cache);
  • context_window_size — maximum context length
vllm_cache_config_info
Gauge, UNIT

vLLM engine KV cache settings. The metric value is always 1

  • service_id — inference service ID;
  • region — pool where the inference service is located;
  • pod — inference instance;
  • block_size — KV cache block size in tokens;
  • cache_dtype — KV cache data type;
  • cpu_kvcache_space_bytes — CPU space for KV cache, bytes;
  • cpu_offload_gb — additional CPU memory size for KV cache in GB;
  • enable_prefix_caching — request prefix caching:
    • true — enabled;
    • false — disabled;
  • gpu_memory_utilization — allowable GPU memory utilization fraction;
  • kv_cache_memory_bytes — GPU memory size for KV cache in bytes;
  • kv_offloading_size — size of the area for temporarily offloading KV cache to RAM in bytes;
  • kv_sharing_fast_prefill — accelerated processing of cached request prefixes:
    • true — enabled;
    • false — disabled;
  • num_cpu_blocks — number of KV cache blocks in CPU memory;
  • num_gpu_blocks — number of KV cache blocks in GPU memory;
  • num_gpu_blocks_override — explicitly set number of KV cache blocks in GPU memory;
  • prefix_caching_hash_algo — request prefix hashing algorithm;
  • sliding_window — sliding context window size in tokens;
  • swap_space — KV cache swap space size in bytes
vllm_kv_cache_usage_perc
Gauge, fraction

Fraction of used KV cache blocks (from 0 to 1)

  • engine — unique identifier of the vLLM engine;
  • model_name — model name;
  • service_id — inference service ID;
  • region — pool where the inference service is located;
  • pod — inference instance
vllm_prefix_cache_hits_total
Gauge, pcs

Number of prefix cache hits

  • engine — unique identifier of the vLLM engine;
  • model_name — model name;
  • service_id — inference service ID;
  • region — pool where the inference service is located;
  • pod — inference instance
vllm_prefix_cache_queries_total
Gauge, pcs

Number of queries to the prefix cache

  • engine — unique identifier of the vLLM engine;
  • model_name — model name;
  • service_id — inference service ID;
  • region — pool where the inference service is located;
  • pod — inference instance
vllm_prompt_tokens_total
Gauge, tokens

Total number of prompt tokens processed by the model

  • engine — unique identifier of the vLLM engine;
  • model_name — model name;
  • service_id — inference service ID;
  • region — pool where the inference service is located;
  • pod — inference instance
vllm_generation_tokens_total
Gauge, tokens

Total number of generated tokens

  • engine — unique identifier of the vLLM engine;
  • model_name — model name;
  • service_id — inference service ID;
  • region — pool where the inference service is located;
  • pod — inference instance
vllm_num_requests_running
Gauge, pcs

Number of requests currently being processed by the model

  • engine — unique identifier of the vLLM engine;
  • model_name — model name;
  • service_id — inference service ID;
  • region — pool where the inference service is located;
  • pod — inference instance
vllm_num_requests_waiting
Gauge, pcs

Number of requests waiting in the queue

  • engine — unique identifier of the vLLM engine;
  • model_name — model name;
  • service_id — inference service ID;
  • region — pool where the inference service is located;
  • pod — inference instance
vllm_request_success_total
Gauge, pcs

Number of successfully processed requests

  • engine — unique identifier of the vLLM engine;
  • model_name — model name;
  • service_id — inference service ID;
  • region — pool where the inference service is located;
  • pod — inference instance;
  • finished_reason — reason for completing the request processing
vllm_engine_sleep_state
Gauge, bool

Inference instance load state. The metric exports three lines at once — one line for each value of the sleep_state label. The value 1 is assigned to the line with the current state of the inference instance. The other lines are assigned the value 0:

  • sleep_state="awake" — inference instance is active and ready to process requests;

  • sleep_state="weights_offloaded" — model weights are offloaded from GPU memory, inference instance is partially freed;

  • sleep_state="discard_all" — model weights and KV cache data are discarded, inference instance is completely freed

  • engine — unique identifier of the vLLM engine;
  • model_name — model name;
  • service_id — inference service ID;
  • region — pool where the inference service is located;
  • pod — inference instance;
  • sleep_state — inference instance state: awake, weights_offloaded, or discard_all
vllm_e2e_request_latency_seconds
Histogram, seconds

Request processing latency from receiving the prompt to full completion of the response (end-to-end)

  • le — upper bound of the histogram bucket;
  • engine — unique identifier of the vLLM engine;
  • model_name — model name;
  • service_id — inference service ID;
  • region — pool where the inference service is located;
  • pod — inference instance
vllm_request_queue_time_seconds
Histogram, seconds

Request wait time in the queue

  • le — upper bound of the histogram bucket;
  • engine — unique identifier of the vLLM engine;
  • model_name — model name;
  • service_id — inference service ID;
  • region — pool where the inference service is located;
  • pod — inference instance
vllm_request_prefill_time_seconds
Histogram, seconds

Processing time of the input prompt

  • le — upper bound of the histogram bucket;
  • engine — unique identifier of the vLLM engine;
  • model_name — model name;
  • service_id — inference service ID;
  • region — pool where the inference service is located;
  • pod — inference instance
vllm_request_decode_time_seconds
Histogram, seconds

Response generation time for a request

  • le — upper bound of the histogram bucket;
  • engine — unique identifier of the vLLM engine;
  • model_name — model name;
  • service_id — inference service ID;
  • region — pool where the inference service is located;
  • pod — inference instance
vllm_request_inference_time_seconds
Histogram, seconds

Total prompt processing and response generation time

  • le — upper bound of the histogram bucket;
  • engine — unique identifier of the vLLM engine;
  • model_name — model name;
  • service_id — inference service ID;
  • region — pool where the inference service is located;
  • pod — inference instance
vllm_request_time_per_output_token_seconds
Histogram, seconds

Average time to generate each token

  • le — upper bound of the histogram bucket;
  • engine — unique identifier of the vLLM engine;
  • model_name — model name;
  • service_id — inference service ID;
  • region — pool where the inference service is located;
  • pod — inference instance
vllm_time_to_first_token_seconds
Histogram, seconds

Time from receiving the request to generating the first token

  • le — upper bound of the histogram bucket;
  • engine — unique identifier of the vLLM engine;
  • model_name — model name;
  • service_id — inference service ID;
  • region — pool where the inference service is located;
  • pod — inference instance
vllm_inter_token_latency_seconds
Histogram, seconds

Inter-token latency

  • le — upper bound of the histogram bucket;
  • engine — unique identifier of the vLLM engine;
  • model_name — model name;
  • service_id — inference service ID;
  • region — pool where the inference service is located;
  • pod — inference instance
vllm_request_prompt_tokens
Histogram, tokens

Number of input tokens per request

  • le — upper bound of the histogram bucket;
  • engine — unique identifier of the vLLM engine;
  • model_name — model name;
  • service_id — inference service ID;
  • region — pool where the inference service is located;
  • pod — inference instance
vllm_request_generation_tokens
Histogram, tokens

Number of generated tokens per request

  • le — upper bound of the histogram bucket;
  • engine — unique identifier of the vLLM engine;
  • model_name — model name;
  • service_id — inference service ID;
  • region — pool where the inference service is located;
  • pod — inference instance
vllm_iteration_tokens_total
Histogram, tokens

Number of tokens processed by the model per iteration across all active requests

  • le — upper bound of the histogram bucket;
  • engine — unique identifier of the vLLM engine;
  • model_name — model name;
  • service_id — inference service ID;
  • region — pool where the inference service is located;
  • pod — inference instance
cpu_usage_cores
Gauge, cores

Current CPU consumption by the inference instance

  • service_id — inference service ID;
  • region — pool where the inference service is located;
  • pod — inference instance
cpu_limit_cores
Gauge, cores

Configured CPU limit for the inference instance

  • service_id — inference service ID;
  • region — pool where the inference service is located;
  • pod — inference instance
memory_usage_bytes
Gauge, bytes

Current memory consumption by the inference instance

  • service_id — inference service ID;
  • region — pool where the inference service is located;
  • pod — inference instance
memory_limit_bytes
Gauge, bytes

Configured memory limit for the inference instance

  • service_id — inference service ID;
  • region — pool where the inference service is located;
  • pod — inference instance
http_requests_total
Gauge, pcs

Total number of HTTP requests to the inference service

  • service_id — inference service ID;
  • region — pool where the inference service is located;
  • code — HTTP request response code
http_requests_bytes_total
Gauge, bytes

Total size of HTTP requests

  • service_id — inference service ID;
  • region — pool where the inference service is located
http_responses_bytes_total
Gauge, bytes

Total size of HTTP responses

  • service_id — inference service ID;
  • region — pool where the inference service is located
network_receive_bytes_total
Gauge, bytes/s

Number of bytes per second received by the inference instance

  • service_id — inference service ID;
  • region — pool where the inference service is located;
  • pod — inference instance
network_receive_packets_total
Gauge, packets/s

Number of packets per second received by the inference instance

  • service_id — inference service ID;
  • region — pool where the inference service is located;
  • pod — inference instance
network_receive_packets_dropped_total
Gauge, packets/s

Number of packets per second dropped on receive by the inference instance

  • service_id — inference service ID;
  • region — pool where the inference service is located;
  • pod — inference instance
network_transmit_bytes_total
Gauge, bytes/s

Number of bytes per second transmitted by the inference instance

  • service_id — inference service ID;
  • region — pool where the inference service is located;
  • pod — inference instance
network_transmit_packets_total
Gauge, packets/s

Number of packets per second transmitted by the inference instance

  • service_id — inference service ID;
  • region — pool where the inference service is located;
  • pod — inference instance
network_transmit_packets_dropped_total
Gauge, packets/s

Number of packets per second dropped on transmit by the inference instance

  • service_id — inference service ID;
  • region — pool where the inference service is located;
  • pod — inference instance