Autoscaling in a Managed Kubernetes cluster

For your information

Autoscaling is unavailable:

for node groups with GPU without drivers;
node groups on dedicated servers.

In a Managed Kubernetes cluster, you can use Cluster Autoscaler or Karpenter to autoscale node groups. They help you use cluster resources optimally—depending on the cluster load, the number of nodes in a group will be automatically increased or decreased. Cluster Autoscaler is installed automatically when a cluster is created; to start working, you just need to enable it. Karpenter must be installed and configured in advance.

When using autoscaling tools, keep the recommendations in mind.

For pod autoscaling in Managed Kubernetes, Metrics Server is used.

Recommendations

For optimal autoscaling performance, we recommend:

do not use more than one autoscaling tool at the same time;
make sure the project has quotas for vCPU, RAM, GPU, and disk capacity to create the maximum number of nodes in the group;
specify resource requests in the manifests for pods. For more information, see the Resource Management for Pods and Containers instruction in the Kubernetes documentation;
configure PodDisruptionBudget for pods that cannot be interrupted. This will help avoid downtime during node migration;
do not manually modify node resources through the control panel. Cluster Autoscaler and Karpenter will not account for these changes;
when using Cluster Autoscaler check that nodes in the group have the same configuration and labels.

Autoscaling using Cluster Autoscaler

Cluster Autoscaler does not need to be installed in the cluster—it is installed automatically when the cluster is created. To use Cluster Autoscaler in a cluster, enable autoscaling for the node group. After enabling autoscaling, default settings are used, but you can configure Cluster Autoscaler for each node group.

Operating principle

Cluster Autoscaler works with existing node groups and pre-selected configurations. If a node group is in the ACTIVE status, Cluster Autoscaler checks every 10 seconds to see if there are any pods in the PENDING status and analyzes the load—pod requests for vCPU, RAM, and GPU. Depending on the check results, nodes are added or removed. At this time, the node group switches to the PENDING_SCALE_UP or PENDING_SCALE_DOWN. The cluster status during autoscaling is ACTIVE. For more information about cluster statuses, see the View cluster status.

The minimum and maximum number of nodes in a group can be specified when enabling autoscaling; Cluster Autoscaler will only change the number of nodes within these limits. If there are at least two working nodes remaining in other node groups in the cluster, you can configure autoscaling to zero nodes.

Adding a node

If there are pods in the PENDING status and the cluster lacks free resources to place them, the required number of nodes is added to the cluster. In a cluster with Kubernetes version 1.28 and higher, Cluster Autoscaler will work in several groups simultaneously and distribute nodes evenly.

note

For example, you have two node groups with autoscaling enabled. The cluster load has increased and requires four nodes to be added. Two new nodes will be created in each node group simultaneously.

In a cluster with Kubernetes version 1.27 or lower, nodes are added one at a time per check cycle.

Deleting a node

If there are no pods in the PENDING status, Cluster Autoscaler checks the amount of resources requested by the pods.

If the requested amount of resources for pods on a node is less than 50% of its resources, Cluster Autoscaler marks the node as unneeded. If the number of resource requests on the node does not increase within 10 minutes, Cluster Autoscaler will check if the pods can be migrated to other nodes.

Cluster Autoscaler will not move pods and, consequently, will not delete a node if one of the following conditions is met:

pods use PodDisruptionBudget;
In kube-system pods there is no PodDisruptionBudget;
pods are created without a controller — for example, Deployment, ReplicaSet, StatefulSet;
pods use local storage;
there are no resources on other nodes for the pod requests;
there is a mismatch of nodeSelector, affinity and anti-affinity rules, or other parameters.

You can allow such pods to be moved — to do this, add the following annotation:

cluster-autoscaler.kubernetes.io/safe-to-evict: "true"

If there are no restrictions, the pods will be migrated, and the underloaded nodes will be removed. Nodes are removed one by one per check cycle.

Autoscaling to zero nodes

In a node group, you can configure autoscaling to zero nodes—if the load is low, all nodes in the group are removed. The node group card with all settings is not removed. When the load increases, nodes can be added to this node group again.

Autoscaling to zero nodes works only if there are at least two working nodes remaining in other node groups in the cluster. Working nodes must remain in the cluster to host system components required for cluster operation.

note

For example, autoscaling to zero nodes will not work if the cluster has:

two node groups with one working node in each;
one node group with two working nodes.

When there are no nodes in a group, you do not pay for unused resources.

Enable autoscaling using Cluster Autoscaler

For your information

If you set the minimum number of nodes in a group to be higher than the current number, it will not increase to the lower bound immediately. The node group will only scale after pods appear in the PENDING status. The same applies to the upper bound of nodes in a group—if the current number of nodes is higher than the upper bound, removal will only begin after checking the pods.

You can enable autoscaling using Cluster Autoscaler in the control panel, via the Managed Kubernetes API or via Terraform.

In the control panel, on the top menu, click Products and select Managed Kubernetes.
In the Clusters section, open the cluster page → Cluster Composition tab.
In the menu of the node group, select Change number of nodes.
In the Number of nodes field, open the With autoscaling.
Set the minimum and maximum number of nodes in the group; the number of nodes will only change within this range. For fault-tolerant operation of system components, we recommend using at least two working nodes in the cluster. Nodes can be located in different groups.
Click Save.

Configure Cluster Autoscaler

You can configure Cluster Autoscaler separately for each node group.

Parameters, their descriptions, and default values can be viewed in the Cluster Autoscaler parameters table. If you do not specify a parameter in the manifest, the default value will be used.

Example manifest:

apiVersion: v1
kind: ConfigMap
metadata:
    name: cluster-autoscaler-nodegroup-options
    namespace: kube-system
data:
    config.yaml: |
        150da0a9-6ea6-4148-892b-965282e195b0:
          scaleDownUtilizationThreshold: 0.55
          scaleDownUnneededTime: 7m
          zeroOrMaxNodeScaling: true
        e3dc24ca-df9d-429c-bcd5-be85f8d28710:
          scaleDownGpuUtilizationThreshold: 0.25
          ignoreDaemonSetsUtilization: true

Here, 150da0a9-6ea6-4148-892b-965282e195b0 and e3dc24ca-df9d-429c-bcd5-be85f8d28710 are the unique identifiers (UUIDs) of the node groups in the cluster. You can view them in the control panel: in the top menu, click Products ⟶ Managed Kubernetes ⟶ Kubernetes section ⟶ cluster page ⟶ copy the UUID above the node group card, next to the pool segment.

Cluster Autoscaler parameters

	Description	Default value
scaleDownUtilizationThreshold	Minimum vCPU and RAM load of the node at which the system can remove the node. If the node uses less than the specified percentage of vCPU and RAM, for example less than 50% with a value of `0.5`, the system removes the node	0.5
scaleDownGpuUtilizationThreshold	Minimum GPU load at which the system can remove the node. If the node uses less than the specified percentage of GPU, for example less than 50% with a value of `0.5`, the system removes the node	0.5
scaleDownUnneededTime	Waiting time before deleting a low-load node. The system does not delete a node immediately when the node load drops — it waits for the specified time to ensure that the load reduction is stable	10m
scaleDownUnreadyTime	Waiting time before removing a node in the `NotReady` status. The system will not leave a node in the `NotReady` status in the cluster—it will wait for the specified time to ensure that the node has frozen and will not recover, and then it will remove it	20m
maxNodeProvisionTime	Waiting time for adding a new node. If an error occurs and the node is not added within the specified time, the system will restart the node addition process	15m
zeroOrMaxNodeScaling	Allows the number of nodes to be automatically changed only to zero or the maximum you have set. Useful if you need the system to deploy all nodes in a group at once when load appears, and remove all nodes when there is no load	false
ignoreDaemonSetsUtilization	Allows you to ignore system services (DaemonSets) when the system determines whether to reduce the number of nodes in a group. If the value is `true`, system services are ignored	false

Autoscaling using Karpenter

Operating principle

Karpenter is a cluster autoscaling tool with flexible settings. Unlike Cluster Autoscaler, Karpenter does not just use existing node groups; it can also create new node groups.

Karpenter can select the optimal node configuration—taking into account not only technical parameters but also cost. Karpenter selects the cheapest option that fits the current load. This is possible because Karpenter integrates directly with the OpenStack API, which is used to create cloud platform resources.

If the cluster is in the ACTIVE status, Karpenter checks if there are any pods (Pod) in the PENDING status and analyzes the load—pod requests for vCPU, RAM, and GPU. Depending on the check results, node groups and nodes are added or removed. Karpenter can only remove the nodes and node groups that it created.

At this time, the cluster switches to the PENDING_SCALE_UP or PENDING_SCALE_DOWN. The cluster status during autoscaling is ACTIVE. For more information about cluster statuses, see the View cluster status.

Install Karpenter

In the control panel, in the top menu, click Products and select Managed Kubernetes.
In the Clusters section, open the cluster page → Settings.
Click Download kubeconfig. Downloading the kubeconfig file is not available if the cluster has the PENDING_CREATE, PENDING_ROTATE_CERTS, PENDING_DELETE or ERROR.
Export the path to the kubeconfig file to the KUBECONFIG environment variable:
```
export KUBECONFIG=<path>
```
Specify <path> —the path to the cluster_name.yaml kubeconfig file, where <cluster_name> is the cluster name.
Export the Managed Kubernetes cluster ID to the ClusterID environment variable:
```
export ClusterID=<cluster_id>
```
Specify <cluster_id> —the Managed Kubernetes cluster ID. You can view it in the control panel: in the top menu, click Products → Managed Kubernetes → cluster page → copy the ID under the cluster name, next to the region and pool.
Install Karpenter using Helm:
```
helm install karpenter-helmrelease oci://ghcr.io/selectel/mks-charts/karpenter: \
--namespace kube-system \
--set controller.settings.clusterID=$ClusterID
```
The latest available version is installed by default. Available versions can be viewed in the mks-charts repository of the Selectel documentation on GitHub.

Configure Karpenter

To configure autoscaling using Karpenter, set up NodePool and NodeClass objects.

NodePool describes the rules for selecting and scaling nodes. For example:

what types of nodes can be created;
with what configurations (flavors) and resources;
when these nodes can be deleted or recreated.

Each NodePool refers to a specific NodeClass. In configurations (flavors) with network volumes, the NodeClass defines the infrastructure parameters of the network volumes to be used by the nodes in the cluster. A single cluster can have several different NodeClasses, such as those that differ in type or size of network volume. Configurations (flavors) with a local boot disk can also be used, but the local disk parameters are determined by the selected configuration.

For more information about NodePool, see the NodePools article in the Karpenter documentation.

Check cluster compliance with requirements.
Create a NodeClass.
Create a NodePool.

1. Check cluster compliance with requirements

Ensure that the Kubernetes version is 1.28 or higher. You can upgrade the cluster version.
Ensure that there is at least one node in the cluster with at least 2 vCPU and 4 GiB of RAM. For optimal Karpenter performance, we recommend adding two nodes to the cluster, each with at least 2 vCPU and 4 GiB of RAM;
Ensure that autoscaling is disabled.
Ensure that auto-recovery is disabled.

2. Create a NodeClass

Create a nodeclass.yaml YAML file with a manifest for the NodeClass object.

Example of a NodeClass manifest for a network volume of type Universal:
```
apiVersion: karpenter.k8s.selectel/v1alpha1
kind: SelectelNodeClass
metadata:
    name: default
spec:
    disk:
        categories:
            - universal
        sizeGiB: 30
```
Where:
- universal — network volume type;
- 30 — network volume size in GB.
Apply the manifest:
```
kubectl apply -f nodeclass.yaml
```

3. Create a NodePool

Create a YAML file nodepool.yaml with a manifest for the NodePool object. The description of all parameters, except for the requirements block parameters, can be found in the NodePools instruction in the Karpenter documentation. The description of the requirements block parameters — in the Parameters of the requirements block in NodePool.

Example of a NodePool manifest when using the Calico network plugin

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
    name: default
spec:
    template:
        spec:
            nodeClassRef:
                name: default
                kind: SelectelNodeClass
                group: karpenter.k8s.selectel
            requirements:
                - key: topology.kubernetes.io/zone
                  operator: In
                  values: ['ru-7a', 'ru-7b']
                - key: node.kubernetes.io/instance-type
                  operator: In
                  values: ['SL1.1-2048', 'SL1.2-4096', 'SL1.2-8192']
                - key: karpenter.sh/capacity-type
                  operator: In
                  values: ['on-demand']
            expireAfter: 720h
    disruption:
        consolidationPolicy: WhenEmptyOrUnderutilized
        consolidateAfter: 0s
    limits:
        cpu: '1000'
        memory: 1000Gi

Example of a NodePool manifest when using the Cilium network plugin

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
    name: default
spec:
    template:
        spec:
            nodeClassRef:
                name: default
                kind: SelectelNodeClass
                group: karpenter.k8s.selectel
            startupTaints:
                - key: node.cilium.io/agent-not-ready
                  value: "true"
                  effect: NoExecute
                - key: node.cilium.io/agent-not-ready
                  effect: NoSchedule
            requirements:
                - key: topology.kubernetes.io/zone
                  operator: In
                  values: ['ru-7a', 'ru-7b']
                - key: node.kubernetes.io/instance-type
                  operator: In
                  values: ['SL1.1-2048', 'SL1.2-4096', 'SL1.2-8192']
                - key: karpenter.sh/capacity-type
                  operator: In
                  values: ['on-demand']
            expireAfter: 720h
    disruption:
        consolidationPolicy: WhenEmptyOrUnderutilized
        consolidateAfter: 0s
    limits:
        cpu: '1000'
        memory: 1000Gi

Apply the manifest:
```
kubectl apply -f nodepool.yaml
```

Parameters of the requirements block in NodePool

In the NodePool object, the requirements block describes the requirements for the nodes being created.

Parameter	Description
karpenter.sh/capacity-type	Type of nodes being created: `on-demand` — non-interruptible nodes; or `spot` — interruptible nodes
karpenter.k8s.selectel/instance-category	Configuration line. For example, Standard Line (`SL`) or GPU Line (`GL`). For more information about configurations, see the Configurations
karpenter.k8s.selectel/instance-family	Configuration line and line generation. For example, `SL1` or `GL1`
karpenter.k8s.selectel/instance-generation	Line generation. For example, `1` or `2`
node.kubernetes.io/instance-type	Configurations (flavors). For example, `["SL1.1-2048", "SL1.2-4096", "SL1.2-8192"]`. Configurations can be viewed in the List of fixed configuration flavors in all pools subsection of the Configurations
karpenter.k8s.selectel/instance-cpu	Number of vCPUs. For example, `Gt: "4"` — more than 4
karpenter.k8s.selectel/instance-memory	Amount of RAM in GiB. For example, `Gt: "8"` — more than 8 GiB
karpenter.k8s.selectel/instance-gpu-manufacturer	GPU manufacturer. The available value is `["NVIDIA"`]
karpenter.k8s.selectel/instance-gpu-name	GPU name for GPU configurations. For example, `["A100", "H100"]`. Available GPUs can be viewed in the Available GPUs subsection of the Create a Managed Kubernetes cluster with GPU
karpenter.k8s.selectel/instance-gpu-count	Number of GPUs. For example, `Gt: "0"` — more than one GPU
karpenter.k8s.selectel/instance-local-disk	Specifies whether a local disk of the cloud platform is used as a boot disk. Available values are `["true"]` and `["false"`]
topology.kubernetes.io/zone	Pool segments where node groups can be created. For example, `["ru-7a", "ru-7b"]`. Available values can be viewed in the Managed Kubernetes subsection of the Availability of products by location