---
title: "Autoscaling in a Managed Kubernetes cluster"
sidebar_label: "Autoscaling"
sidebar_position: 7
description: "Autoscaling tools in a Managed Kubernetes cluster, their operating principles, and configuration"
---

import Formbricks from '@theme/MDXComponents/Formbricks';
import MoreVerticalIcon from '@selectel/docux/icons/more-vertical';
import { CustomTable } from '@selectel/docux/components';

# Autoscaling in a Managed Kubernetes cluster

:::info

Autoscaling is unavailable:

* for node groups with GPU without drivers;
* node groups on dedicated servers.

:::

In a Managed Kubernetes cluster, you can use [Cluster Autoscaler](#autoscaling-with-cluster-autoscaler) or [Karpenter](#autoscaling-with-karpenter) to autoscale node groups. They help to optimize cluster resource usage—the number of nodes in a group will automatically decrease or increase depending on the cluster load. Cluster Autoscaler is installed automatically when a cluster is created; to start using it, you just need to enable it. Karpenter requires prior installation and configuration.

When using autoscaling tools, keep the [recommendations](#recommendations) in mind.

For pod autoscaling in Managed Kubernetes, [Metrics Server](/managed-kubernetes/clusters/metrics-server.mdx) is used.

## Recommendations \{#recommendations}

For optimal autoscaling performance, we recommend:

* do not use more than one autoscaling tool at the same time;
* make sure the project has [quotas](/access-control/projects/quotas.mdx) for vCPU, RAM, GPU, and disk capacity to create the maximum number of nodes in the group;
* specify resource requests in the manifests for pods. For more details, see the [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) instruction in the Kubernetes documentation;
* configure a [PodDisruptionBudget](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/#pod-disruption-budgets) for pods that cannot be stopped. This will help avoid downtime during migration between nodes;
* do not manually modify node resources through the control panel. Cluster Autoscaler  and Karpenter will not account for these changes;
*  when using Cluster Autoscaler  check that nodes in the group have the same configuration and labels.

## Autoscaling using Cluster Autoscaler \{#autoscaling-with-cluster-autoscaler}

Cluster Autoscaler does not need to be installed in the cluster—it is installed automatically when a cluster is created. To use Cluster Autoscaler in a cluster, [enable autoscaling for the node group](#enable-autoscaling). Once autoscaling is enabled, default settings are used, but you can [configure Cluster Autoscaler](#configure-cluster-autoscaler) for each node group.

### Operating principle \{#principle-of-operation}

Cluster Autoscaler works with existing node groups and preselected configurations. If a node group is in the `ACTIVE` status, Cluster Autoscaler checks every 10 seconds whether there are pods in the `PENDING` status and analyzes the load—requests from pods for vCPU, RAM, and GPU. Depending on the check results, nodes are [added](#adding-node) or [deleted](#deleting-node). During this time, the node group switches to `PENDING_SCALE_UP` or `PENDING_SCALE_DOWN`. The cluster status during autoscaling is `ACTIVE`. For more details on cluster statuses, see the [View cluster status](/managed-kubernetes/clusters/cluster-statuses.mdx) instruction.

The minimum and maximum number of nodes in a group can be set when [enabling autoscaling](#enable-autoscaling) —Cluster Autoscaler will scale the number of nodes only within these limits. If at least two working nodes remain in other node groups of the cluster, you can configure [autoscaling to zero nodes](#autoscaling-down-to-zero).

#### Adding a node \{#adding-node}

If there are pods in the `PENDING` status and the cluster lacks free resources to host them, the required number of nodes is added to the cluster. In a cluster with Kubernetes version 1.28 and higher, Cluster Autoscaler will work across multiple groups simultaneously and distribute nodes evenly.

:::note

For example, you have two node groups with autoscaling enabled. The load on the cluster has increased and requires adding four nodes. Two new nodes will be created in each node group simultaneously.

:::

In a cluster with Kubernetes version 1.27 or lower, nodes are added one at a time per check cycle.

#### Deleting a node \{#deleting-node}

If there are no pods in the `PENDING` status, Cluster Autoscaler checks the amount of resources requested by the pods.

If the requested resource amount for pods on a single node is less than 50% of its total resources, Cluster Autoscaler flags the node as unnecessary. If the resource request volume does not increase on the node within 10 minutes, Cluster Autoscaler checks whether the pods can be migrated to other nodes.

Cluster Autoscaler will not move pods and, consequently, will not delete a node if one of the following conditions is met:

* pods use [PodDisruptionBudget](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/#pod-disruption-budgets);
* In kube-system pods there is no PodDisruptionBudget;
* pods are created without a controller — for example, Deployment, ReplicaSet, StatefulSet;
* pods use local storage;
* there are no resources on other nodes for the pod requests;
* there is a mismatch of nodeSelector, affinity and anti-affinity rules, or other parameters.

You can allow such pods to be moved — to do this, add the following annotation:

```yaml
cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
```

If there are no restrictions, the pods will be migrated, and the underutilized nodes will be removed. Nodes are removed one by one per check cycle.

#### Autoscaling to zero nodes \{#autoscaling-down-to-zero}

You can configure autoscaling to zero nodes for a node group—all nodes in the group are removed during low load. The node group card with all settings is not removed. When the load increases, nodes can be added to this node group again.

Autoscaling to zero nodes only works if at least two working nodes remain in other cluster node groups. The cluster must have working nodes for hosting system components required for the cluster to function.

:::note

For example, autoscaling to zero nodes will not work if the cluster has:

* two node groups with one working node in each;
* one node group with two working nodes.

:::

When there are no nodes in a group, you do not pay for unused resources.

### Enable autoscaling using Cluster Autoscaler \{#enable-autoscaling}

:::info

If you set the minimum number of nodes in a group higher than the current number, it will not increase to the lower limit immediately. The node group will only scale after pods appear in the `PENDING` status. Similarly, for the upper limit of nodes—if the current number of nodes is higher than the upper limit, removal will begin only after checking the pods.

:::

You can enable autoscaling using Cluster Autoscaler in the control panel, via the [Managed Kubernetes API](/api/managed-kubernetes/) or via [Terraform](/terraform/selectel-provider-reference/resources/mks_nodegroup_v1/).

1. In the [control panel](https://my.selectel.ru/mks/), on the top menu, click **Products** and select **Managed Kubernetes**.
2. In the **Clusters** section, open the cluster page → **Cluster Composition** tab.
3. In the <MoreVerticalIcon /> node group menu, select **Change node count**.
4. In the **Node count** field, open the **With autoscaling** tab.
5. Set the minimum and maximum number of nodes in a group; the value will change only within this range. For fault-tolerant operation of system components, we recommend using at least two working nodes in the cluster. Nodes can be in different groups.
6. Click **Save**.

### Configure Cluster Autoscaler \{#configure-cluster-autoscaler}

You can configure Cluster Autoscaler separately for each node group.

You can view the parameters, their descriptions, and default values in the [Cluster Autoscaler parameters](#cluster-autoscaler-parameters) table. If you do not specify a parameter in the manifest, the default value will be used.

Example manifest:

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
    name: cluster-autoscaler-nodegroup-options
    namespace: kube-system
data:
    config.yaml: |
        150da0a9-6ea6-4148-892b-965282e195b0:
          scaleDownUtilizationThreshold: 0.55
          scaleDownUnneededTime: 7m
          zeroOrMaxNodeScaling: true
        e3dc24ca-df9d-429c-bcd5-be85f8d28710:
          scaleDownGpuUtilizationThreshold: 0.25
          ignoreDaemonSetsUtilization: true
```

Here, `150da0a9-6ea6-4148-892b-965282e195b0` and `e3dc24ca-df9d-429c-bcd5-be85f8d28710` are unique identifiers (UUIDs) of node groups in the cluster. You can view them in the [control panel](https://my.selectel.ru/mks/): in the top menu, click **Products** ⟶ **Managed Kubernetes** ⟶ **Kubernetes** section ⟶ cluster page ⟶ copy the UUID above the node group card, next to the pool segment.

#### Cluster Autoscaler parameters \{#cluster-autoscaler-parameters}

<CustomTable>
  <table data-sticky>
    <thead>
      <tr>
        <th />

        <th>Description</th><th>Default value</th>
      </tr>
    </thead>

    <tbody>
      <tr>
        <th>scaleDownUtilizationThreshold</th><td>The minimum vCPU and RAM load of a node at which the system can remove the node. If the node uses less than the specified percentage of vCPU and RAM, for example less than 50% with a value of `0.5`, the system removes the node</td><td>0.5</td>
      </tr>

      <tr>
        <th>scaleDownGpuUtilizationThreshold</th><td>The minimum GPU load at which the system can remove the node. If the node uses less than the specified percentage of GPU, for example less than 50% with a value of `0.5`, the system removes the node</td><td>0.5</td>
      </tr>

      <tr>
        <th>scaleDownUnneededTime</th><td>Waiting time before deleting a low-load node. The system does not delete a node immediately when the node load drops — it waits for the specified time to ensure that the load reduction is stable</td><td>10m</td>
      </tr>

      <tr>
        <th>scaleDownUnreadyTime</th><td>The timeout before removing a node in the `NotReady` status. The system will not leave a node in the `NotReady` status in the cluster; it will wait for the specified time to ensure that the node is frozen and will not recover, and then it will remove it.</td><td>20m</td>
      </tr>

      <tr>
        <th>maxNodeProvisionTime</th><td>Waiting time for adding a new node. If an error occurs and the node is not added within the specified time, the system will restart the node addition process</td><td>15m</td>
      </tr>

      <tr>
        <th>zeroOrMaxNodeScaling</th><td>Allows the number of nodes to be automatically changed only to zero or the maximum you have set. Useful if you need the system to deploy all nodes in a group at once when load appears, and remove all nodes when there is no load</td><td>false</td>
      </tr>

      <tr>
        <th>ignoreDaemonSetsUtilization</th><td>Allows ignoring system services (DaemonSets) when the system determines whether to reduce the number of nodes in a group. If the value is `true`, system services are ignored.</td><td>false</td>
      </tr>
    </tbody>
  </table>
</CustomTable>

## Autoscaling using Karpenter \{#autoscaling-with-karpenter}

### Operating principle \{#principle-of-operation}

Karpenter is a cluster autoscaling tool with flexible settings. Unlike Cluster Autoscaler, Karpenter not only uses existing node groups but can also create new node groups.

Karpenter can select an optimal node configuration—taking into account not only technical parameters but also cost. Karpenter chooses the cheapest option suitable for the current load. This is possible because Karpenter integrates directly with the OpenStack API used to create cloud platform resources.

If a cluster is in the `ACTIVE` status, Karpenter checks if there are pods ([Pod](https://kubernetes.io/docs/concepts/workloads/pods/)) in the `PENDING` status and analyzes the load — requests from pods for vCPU, RAM, and GPU. Depending on the check results, node groups and nodes are added or removed. Karpenter can only remove nodes and node groups that it created.

During this time, the cluster switches to the `PENDING_SCALE_UP` or `PENDING_SCALE_DOWN` status. The cluster status during autoscaling is `ACTIVE`. Read more about cluster statuses in the [View cluster status](/managed-kubernetes/clusters/cluster-statuses.mdx) instructions.

### Install Karpenter \{#install-karpenter}

1. In the [control panel](https://my.selectel.ru/mks/), in the top menu, click **Products** and select **Managed Kubernetes**.

2. In the **Clusters** section, open the cluster page → **Settings** tab.

3. Click **Download kubeconfig**. Downloading the kubeconfig file is unavailable if the cluster status is `PENDING_CREATE`, `PENDING_ROTATE_CERTS`, `PENDING_DELETE` or `ERROR`.

4. Export the path to the kubeconfig file to the `KUBECONFIG` environment variable:

   ```bash
   export KUBECONFIG=<path>
   ```

   Specify `<path>` — the path to the `cluster_name.yaml` kubeconfig file, where `<cluster_name>` is the cluster name.

5. Export the Managed Kubernetes cluster ID to the `ClusterID` environment variable:

   ```bash
   export ClusterID=<cluster_id>
   ```

   Specify `<cluster_id>` — the Managed Kubernetes cluster ID; it can be viewed in the [control panel](https://my.selectel.ru/mks/): in the top menu, click **Products** → **Managed Kubernetes** → cluster page → copy the ID under the cluster name, next to the region and pool.

6. Install Karpenter using Helm:

   ```bash

   helm install karpenter-helmrelease oci://ghcr.io/selectel/mks-charts/karpenter: \
   --namespace kube-system \
   --set controller.settings.clusterID=$ClusterID
   ```

   By default, the latest available version is installed. Available versions can be viewed in the [mks-charts repository](https://github.com/selectel/mks-charts/releases) of the Selectel documentation on GitHub.

### Configure Karpenter \{#configure-karpenter}

To configure autoscaling using Karpenter, set up NodePool and NodeClass objects.

A NodePool describes the rules for node selection and scaling. For example:

* what types of nodes can be created;
* with what configurations (flavors) and resources;
* when these nodes can be deleted or recreated.

Each NodePool refers to a specific NodeClass. In configurations (flavors) with network volumes, the NodeClass defines the infrastructure parameters of the network volumes to be used by the nodes in the cluster. A single cluster can have several different NodeClasses, for example, those that differ in the type or size of the network volume. Configurations (flavors) with a local boot disk can also be used, but the local disk parameters are determined by the selected configuration.

Read more about NodePools in the [NodePools](https://karpenter.sh/docs/concepts/nodepools/) article in the Karpenter documentation.

1. [Check the cluster for compliance with the requirements](#check-compliance-with-requirements).
2. [Create a NodeClass](#create-nodeclass).
3. [Create a NodePool](#create-nodepool).

#### 1. Check cluster compliance with requirements \{#check-compliance-with-requirements}

1. Ensure that the Kubernetes version is 1.28 or higher. You can [upgrade the cluster version](/managed-kubernetes/clusters/upgrade-version.mdx).

2. Ensure that the cluster has at least one node with at least 2 vCPU and 4 GiB of RAM. For optimal operation of Karpenter, we recommend adding two nodes to the cluster, each with at least 2 vCPU and 4 GiB of RAM;

3. Ensure that [autoscaling](/managed-kubernetes/node-groups/autoscaling.mdx) is disabled.

4. Ensure that [auto-healing](/managed-kubernetes/node-groups/reinstall-nodes.mdx) is disabled.

#### 2. Create a NodeClass \{#create-nodeclass}

1. Create a `nodeclass.yaml` YAML file with a manifest for the NodeClass object.

   Example of a NodeClass manifest for a network volume of type Universal:

   ```yaml
   apiVersion: karpenter.k8s.selectel/v1alpha1
   kind: SelectelNodeClass
   metadata:
       name: default
   spec:
       disk:
           categories:
               - universal
           sizeGiB: 30
   ```

   Where:

   * `universal` — [network volume type](/cloud-servers/volumes/about-network-volumes.mdx#network-volume-types-list);
   * `30` — network volume size in GB.

2. Apply the manifest:

   ```bash
   kubectl apply -f nodeclass.yaml
   ```

#### 3. Create a NodePool \{#create-nodepool}

1. Create a `nodepool.yaml` YAML file with a manifest for the NodePool object. Descriptions of all parameters, except for the `requirements` block parameters, can be found in the [NodePools](https://karpenter.sh/docs/concepts/nodepools/) instructions of the Karpenter documentation. Descriptions of the `requirements` block parameters are in the [Parameters of the requirements block in NodePool](#requirements-in-nodepool).

   <details>
     <summary>Example of a NodePool manifest when using the Calico network plugin</summary>

     ```yaml
     apiVersion: karpenter.sh/v1
     kind: NodePool
     metadata:
         name: default
     spec:
         template:
             spec:
                 nodeClassRef:
                     name: default
                     kind: SelectelNodeClass
                     group: karpenter.k8s.selectel
                 requirements:
                     - key: topology.kubernetes.io/zone
                       operator: In
                       values: ['ru-7a', 'ru-7b']
                     - key: node.kubernetes.io/instance-type
                       operator: In
                       values: ['SL1.1-2048', 'SL1.2-4096', 'SL1.2-8192']
                     - key: karpenter.sh/capacity-type
                       operator: In
                       values: ['on-demand']
                 expireAfter: 720h
         disruption:
             consolidationPolicy: WhenEmptyOrUnderutilized
             consolidateAfter: 0s
         limits:
             cpu: '1000'
             memory: 1000Gi
     ```
   </details>

   <details>
     <summary>Example of a NodePool manifest when using the Cilium network plugin</summary>

     ```yaml
     apiVersion: karpenter.sh/v1
     kind: NodePool
     metadata:
         name: default
     spec:
         template:
             spec:
                 nodeClassRef:
                     name: default
                     kind: SelectelNodeClass
                     group: karpenter.k8s.selectel
                 startupTaints:
                     - key: node.cilium.io/agent-not-ready
                       value: "true"
                       effect: NoExecute
                     - key: node.cilium.io/agent-not-ready
                       effect: NoSchedule
                 requirements:
                     - key: topology.kubernetes.io/zone
                       operator: In
                       values: ['ru-7a', 'ru-7b']
                     - key: node.kubernetes.io/instance-type
                       operator: In
                       values: ['SL1.1-2048', 'SL1.2-4096', 'SL1.2-8192']
                     - key: karpenter.sh/capacity-type
                       operator: In
                       values: ['on-demand']
                 expireAfter: 720h
         disruption:
             consolidationPolicy: WhenEmptyOrUnderutilized
             consolidateAfter: 0s
         limits:
             cpu: '1000'
             memory: 1000Gi
     ```
   </details>

2. Apply the manifest:

   ```bash
   kubectl apply -f nodepool.yaml
   ```

#### Parameters of the requirements block in NodePool \{#requirements-in-nodepool}

In the NodePool object, the `requirements` block describes the requirements for the nodes to be created.

<CustomTable>
  <table>
    <tbody>
      <tr>
        <th>Parameter</th><td>Description</td>
      </tr>

      <tr>
        <th>karpenter.sh/capacity-type</th>

        <td>
          Type of nodes being created:

          * `on-demand` — non-interruptible nodes;
          * or `spot` — interruptible nodes.
        </td>
      </tr>

      <tr>
        <th>karpenter.k8s.selectel/instance-category</th><td>The configuration line. For example, Standard Line (`SL`) or GPU Line (`GL`). Read more about configurations in the [Configurations](/cloud-servers/create/configurations.mdx#server-flavors-list) instructions</td>
      </tr>

      <tr>
        <th>karpenter.k8s.selectel/instance-family</th><td>The configuration line and line generation. For example, `SL1` or `GL1`</td>
      </tr>

      <tr>
        <th>karpenter.k8s.selectel/instance-generation</th><td>Line generation. For example, `1` or `2`</td>
      </tr>

      <tr>
        <th>node.kubernetes.io/instance-type</th><td>Configurations (flavors). For example, `["SL1.1-2048", "SL1.2-4096", "SL1.2-8192"]`. You can view the configurations in the [List of fixed configuration flavors in all pools](/cloud-servers/create/configurations.mdx#server-flavors-full-list) subsection of the [Configurations](/cloud-servers/create/configurations.mdx) instructions</td>
      </tr>

      <tr>
        <th>karpenter.k8s.selectel/instance-cpu</th><td>The number of vCPUs. For example, `Gt: "4"` — more than 4.</td>
      </tr>

      <tr>
        <th>karpenter.k8s.selectel/instance-memory</th><td>The amount of RAM in GiB. For example, `Gt: "8"` — more than 8 GiB</td>
      </tr>

      <tr>
        <th>karpenter.k8s.selectel/instance-gpu-manufacturer</th><td>The GPU manufacturer. The available value is `["NVIDIA"`]</td>
      </tr>

      <tr>
        <th>karpenter.k8s.selectel/instance-gpu-name</th><td>The GPU name for configurations with GPU. For example, `["A100", "H100"]`. Available GPUs can be viewed in the [Available GPUs](/managed-kubernetes/create/create-cloud-gpu-cluster.mdx#available-gpu) subsection of the [Creating a Managed Kubernetes cluster with GPU](/managed-kubernetes/create/create-cloud-gpu-cluster.mdx) instructions</td>
      </tr>

      <tr>
        <th>karpenter.k8s.selectel/instance-gpu-count</th><td>The number of GPUs. For example, `Gt: "0"` — more than one graphics processor.</td>
      </tr>

      <tr>
        <th>karpenter.k8s.selectel/instance-local-disk</th><td>Specifies whether a [local disk](/cloud-servers/volumes/about-local-disks.mdx) of the cloud platform is used as a boot disk. The available values are `["true"]` and `["false"`]</td>
      </tr>

      <tr>
        <th>topology.kubernetes.io/zone</th><td>Pool segments where node groups can be created. For example, `["ru-7a", "ru-7b"]`. Available values can be viewed in the [Managed Kubernetes](/infrastructure/availability-matrix.mdx#managed-kubernetes) subsection of the [Availability Matrix](/infrastructure/availability-matrix.mdx) instructions</td>
      </tr>
    </tbody>
  </table>
</CustomTable>

<Formbricks />