Skip to main content

Dedicated Inference: Quickstart

  1. Create an inference service.
  2. Send a test request to the model.

1. Create an inference service

  1. Select a model.
  2. Configure the infrastructure.
  3. Configure the inference service.
  4. Confirm the configuration.

1. Select a model

  1. In the control panel, in the top menu, click Products and select Dedicated Inference.

  2. Select a location. You cannot change the location after creating the inference service.

  3. On the model card, click Create.

  4. Enter an inference service name.

  5. To filter inference services in the list, add tags. A tag with the model name is added automatically. To add a new tag, in the Tags field, enter a tag and press Enter.

  6. Optional: enter a description of the inference service. For example, specify its purpose.

  7. Click Next.

2. Configure the infrastructure

  1. Set the model parameters.

    1.1. Select the data type for the KV cache.

    1.2. Select the maximum context length.

  2. Select the inference service configuration. When choosing, consider the expected model performance indicators.

    You cannot change the configuration after creating the inference service.

  3. Click Next.

3. Configure the inference service

  1. Configure the number of inference instances.

    1.1. To have a fixed number of instances in the service, open the Fixed tab and specify the number of instances.

    1.2. To use autoscaling in the service, open the Autoscaling tab and set the minimum and maximum number of instances. The number of instances will automatically change only within the specified range depending on the load on the inference service.

    You can change the number of inference instances after creating the inference service. For more information, see the Scale inference service instructions.

  2. Select the disk type for the inference instance.

  3. Select the connection type to the inference service:

    • public — access via a public endpoint from the Internet;

    • private — access via a private endpoint only from Cloud Platform devices created in the subnet of the inference service.

    You cannot change the connection type after creating the inference service.

  4. Click Next.

4. Confirm the configuration

  1. Check the final configuration of the inference service.

  2. Check the price of the inference service.

  3. Click Create Inference Service. Creating an inference service can take about 15 minutes.

2. Send a test request to the model

The request data structure depends on the model, API type, and request type. You can copy test requests to the model in the control panel: in the top menu, click Products → Inference services → inference service page → Quickstart tab → in the Test request block, click .

For more information about request kinds and API types, see the Inference service interaction modes instructions.