Inference service model performance metrics

Last update: October 13 2026Open in Markdown

You can compare model performance metrics across different configurations and choose the one you need. Performance metrics are displayed in the card of each configuration when creating an inference service.

Avg Time to First Token	Average time from receiving a request to generating the first token, in milliseconds
Avg Request Throughput	Average number of processed requests per second
Output Token Throughput	Average number of generated tokens per second
Request Latency	Average time from receiving a request to full response completion, in seconds