Inference Service model performance metrics

Last update: September 07 2026Open in Markdown

Using metrics, you can compare model performance across different configurations and select the one you need. Metrics are displayed in the card for each configuration when creating an inference service.

Avg Time to First Token	Average time from request receipt to generation of the first token in milliseconds
Avg Request Throughput	Average number of requests processed per second
Output Token Throughput	Average number of generated tokens per second
Request Latency	Average time from request receipt to full response completion in seconds