Skip to main content

Foundation Models Catalog: Quick Start

Last update:
  1. Create an inference service.
  2. Connect to the inference service.

1. Create an inference service

  1. Select a model.
  2. Configure infrastructure.
  3. Configure the inference service.
  4. Confirm configuration.

1. Select a model

  1. In the control panel, on the top menu, click Products and select Foundation Models Catalog.

  2. In the model card, click Create.

  3. Enter an inference service name.

  4. To filter inference services in the list, add tags. A tag with the model name is added automatically. To add a new tag, enter it in the Tags field and press Enter.

  5. Optional: enter a description for the inference service. For example, specify its purpose.

  6. Click Continue.

2. Configure infrastructure

  1. Specify the model parameters.

    1.1. Select the model parameter data type.

    1.2. Select the data type for the KV cache.

    1.3. Select the maximum context length.

  2. Select the inference service configuration. When choosing, consider the expected model performance metrics.

    You cannot change the configuration after the inference service is created.

  3. Click Continue.

3. Configure the inference service

  1. Configure the number of inference instances.

    1.1. To have a fixed number of instances in the service, open the Fixed tab and specify the number of instances.

    1.2. To use autoscaling in the service, open the With autoscaling tab and set the minimum and maximum number of instances. The number of instances will change automatically only within the specified range depending on the inference service load.

    You can change the number of inference instances after the inference service is created. Learn more in the Scale an inference service guide.

  2. Select the disk type for the inference instance.

  3. Click Continue.

4. Confirm configuration

  1. Check the final inference service configuration.

  2. Check the price of the inference service.

  3. Click Create Inference Service. Creating an inference service may take about 15 minutes.

2. Connect to the inference service

To connect to the inference service, send a test request via the Completions API or Chat API.

Use the Completions API to generate text based on a single prompt, without dialogue support or message history. For example, to continue a phrase, generate text based on a template, or perform a one-time generation.

Use the Chat API to maintain a dialogue in chatbot mode, taking into account roles and message history.

  1. Open the CLI.

  2. Send a test curl request:

curl <endpoint>/v1/completions \
-H "Authorization: Bearer <api_key>" \
-H "Content-Type: application/json" \
-d '{
"model": "<model>",
"prompt": "<prompt>",
"temperature": 0,
"max_tokens": 7
}'

Specify:

  • <endpoint> — inference service endpoint. You can copy the endpoint in the control panel: from the top menu, click Products → Inference Services → inference service page → Quick Start tab → in the Endpoint block, click ;

  • <api_key> — API key. You can copy it in the control panel: from the top menu, click Products → Inference Services → inference service page → API Keys tab → in the API key row, click , then ;

  • <model> — model name. You can view it in the control panel: from the top menu, click Products → Inference Services → inference service page → tab Service → row Model;

  • <prompt> — prompt, for example:

    Объясни, что такое промт

You will receive a response in the OpenAI API format.