Skip to main content

Dedicated Inference product overview

For your information

The product is in public preview.

Dedicated Inference is a service for deploying pre-configured ML models with a ready-made API in the Selectel infrastructure. Available models are deployed as isolated inference services with dedicated resources (GPU, vCPU, RAM, disk).

To work with the product, the Selectel access control model is supported: user types, roles, projects, and project limits and quotas.

Use cases

  • work with models without having to deploy infrastructure yourself;

  • selecting ready-made infrastructure for expected or fluctuating loads. You can compare model performance indicators on different inference service configurations and select the required one, as well as configure autoscaling;

  • testing and selecting different models for your projects. You can deploy multiple models and compare which one best suits your tasks;

  • integrating models into your own projects via a dedicated endpoint.

How it works

Dedicated Inference uses Cloud Platform resources. Each model is deployed as an isolated inference service based on a Managed Kubernetes cluster with GPU. Model weights are stored in Selectel S3.

For Dedicated Inference fault tolerance, inference services are deployed in clusters across different locations. All clusters with inference services are centrally managed by a separate cluster via Flux CD. Multiple inference services can run in a single cluster — each isolated from one another and scaled independently.

Each inference service uses a suitable inference server depending on the model type. For example, the vLLM inference server is used for large language models (LLMs).

Requests to models are executed via a dedicated endpoint compatible with the OpenAI API.

How to work with Dedicated Inference

You can work with Dedicated Inference in the control panel or via API. To get started with Dedicated Inference, use the Dedicated Inference: Quick Start guide.

When creating an inference service, you can select its configuration. The list of available configurations depends on the selected model and its parameters. When selecting a configuration, you can view the expected model performance indicators.

Also, when creating an inference service, you can select the connection type to the inference service:

  • public — access from the internet;

  • private — access from a private network. Access can only be configured via a device in the inference service subnet — for example, via a cloud server.

The endpoint is automatically generated after the inference service is created and will be available in the control panel on the card and page of the inference service.

Dedicated Inference supports only synchronous mode on the server side. On the client side, you can send synchronous and asynchronous requests. For more details, see the Inference service interaction modes guide.

Access to the inference service is carried out via API keys. API keys are individual for each inference service. You can manage API keys.

You can scale the inference service depending on the number of requests to the model. To scale an inference service, change the number of inference instances — deployed model instances. For more details, see the Scale inference service guide.

You can track inference service metrics using the Metrics service.

Available models

You can view the current list of models in the control panel: in the top menu, click Products → Dedicated Inference.

Areas of responsibility

Selectel provides

  • infrastructure for creating inference services;

  • access to models via a public API compatible with the OpenAI API;

  • the ability to scale inference services;

  • a monitoring system for inference services in the control panel;

  • data storage security in compliance with 152-FZ requirements;

  • integration with other Selectel services;

  • technical support.

Selectel is not responsible for

  • integrating models into your projects;

  • business logic of models in your projects.

Cost

Dedicated Inference is billed under the pay-as-you-go payment model. Funds are deducted from the balance every hour for the previous hour of using Cloud Platform resources. The number of tokens does not affect the cost. For more details, see the Dedicated Inference billing model and prices guide.

What is included in the cost

  • unlimited tokens;

  • a free domain name for public access to the model.

Limitations

Dedicated Inference does not support: