A three-node cluster with Apache Kafka

Last update: June 09 2026

Apache Kafka is a distributed software message broker for real-time data stream processing from The Apache Software Foundation. The broker allows you to collect, store, and process large volumes of data from various sources and transfer them to data processing and analysis systems. Apache Kafka uses a three-node distributed cluster architecture for fault tolerance.

You can create a three-node cluster with a pre-installed Apache Kafka application. In Russia, the cluster nodes run with a configured SelectOS 1 64-bit operating system. In other countries — Ubuntu 24.04.

Before creating a cloud server with an application, review the software license agreements included in the image.

Minimum resource requirements

	Test environment	Production environment
Number of vCPUs	4	8
RAM	8 GB	32 GB
Boot volume	16 GB	from 16 GB *

* The boot volume size for a production environment depends on your needs.

Create a three-node cluster with Apache Kafka

In Selectel, to create a cluster, you must first create a temporary cloud server with Apache Kafka. Creating the temporary server initiates the creation of a three-node cluster. Each node is a cloud server with the configuration that was selected when creating the temporary cloud server.

The name of each server in the cluster will be in the format cs-<server_name>-<cluster_number>, where <server_name> is the server name and <cluster_number> is the server number in the cluster from 1 to 3.

The three created cloud servers are in the same subnet as the temporary cloud server.

Next, an Apache Kafka cluster will be configured on the three cloud servers. If the cluster is configured successfully, the temporary cloud server is deleted. If the temporary cloud server is not deleted, an issue occurred while configuring the Apache Kafka cluster. To identify the problem, view the logs.

The cluster cost will be calculated as the cost of three cloud servers. Learn more about the Cloud Platform payment model in the Payment model and prices article.

To configure Apache Kafka, you must specify user data during creation — custom configuration parameters used for connecting to the OpenStack API.

1. Create a service user.

2. Create a cloud server with Apache Kafka.

1. Create a service user

Create a service user, select the member or reader role under permissions, and set the access scope to Projects. Users can be created by the Account Owner or users with the iam.admin role.

2. Create a cloud server with Apache Kafka

1. In the Control panel, on the top menu, click Products and select Cloud Servers.

2. Click Create server.

3. Fill in the blocks:

   • Name and placement
   • Source
   • Configuration
   • Volumes
   • Network
   • Security
   • Access
   • Additional settings

4. Check the cloud server price. Note that after configuration, the price for a three-node cluster will be calculated using the formula: cloud server price * 3.

5. Click Create.

Name and placement

1. Enter the server name. It will be set as the hostname in the operating system.

2. Select a location where the server will be created. The available server configurations and resource costs depend on the location. You cannot change the location after the server is created.

Source

1. Open the Apps tab.

2. Select Cloud Kafka.

3. Optional: if you need a different current or archived application version, select the required version in the Version field.

Configuration

Choose a configuration with at least 2 vCPUs and 8 GB of RAM if the server is intended for development and testing. If you need the server for a moderate workload in a production environment, choose a configuration with at least 8 vCPUs and 32 GB of RAM. The boot disk size for a production environment depends on your needs.

For all lines, except Shared and Dedicated, two types of configuration are available:

• prebuilt configurations — range configurations with different specifications and fixed resource ratios;
• custom configurations — configurations in which you can specify any resource ratio.

Configurations use different processors depending on the line and pool segment. You can customize the selected configuration. After the server is created, you will be able to change the configuration.

Prebuilt configuration

1. Open the tab with a line.

2. Click Prebuilt.

3. Optional: you can adjust the configuration if you are creating a server in the multi-zone pool segment ru-6 or pool segments ru-3b, ru-7a, and ru-7b:

   3.1. Expand the block with the configuration settings description.

   3.2. Optional: select a processor manufacturer. Manufacturer selection is not available in all pools.

   3.3. Optional: if you do not want physical processor cores to be pinned to the cloud server vCPUs, uncheck the Dedicated Cores checkbox. For more information, see the Dedicated Cores tutorial.

   3.4. Optional: if you want to disable Hyper-Threading for a server with dedicated cores, uncheck the Hyper-Threading (SMT) checkbox.

   3.5. Optional: if you are creating a server with dedicated cores and want to host a multiprocessor server on a single NUMA node, check the Mandatory placement on one NUMA node checkbox. You can place a server with 4 vCPUs or more on a single NUMA node. If the cloud server resources cannot be placed on one node, it will not be created. For more information, see the Placement on a single NUMA node section of the Dedicated Cores tutorial.

4. Select a configuration.

5. If both local and network volumes are available in the selected configuration, select the volume to be used as a boot volume:

   • local volume — check the Local SSD NVMe volume checkbox. A server with a local volume can only be created from images and apps;
   • network volume — do not check the Local SSD NVMe disk checkbox.

   The amount of RAM allocated to the server may be less than specified in the configuration — the operating system kernel reserves a portion of the RAM depending on the kernel version and distribution. You can check the allocated volume on the server using the sudo dmesg | grep Memory command.

Custom configuration

1. Open the tab with a line.

2. Click Custom.

3. Optional: you can adjust the configuration if you are creating a server in the multi-zone pool segment ru-6 or pool segments ru-3b, ru-7a, and ru-7b:

   3.1. Expand the block with the configuration settings description.

   3.2. Optional: select a processor manufacturer. Manufacturer selection is not available in all pools.

   3.3. Optional: if you do not want physical processor cores to be pinned to the cloud server vCPUs, uncheck the Dedicated Cores checkbox. For more information, see the Dedicated Cores tutorial.

   3.4. Optional: if you want to disable Hyper-Threading for a cloud server with dedicated cores, uncheck the Hyper-Threading (SMT) checkbox.

   3.5. Optional: if you are creating a cloud server with dedicated cores and want to host a multiprocessor server on a single NUMA node, check the Mandatory placement on one NUMA node checkbox. You can place a cloud server with 4 vCPUs or more on a single NUMA node. If the cloud server resources cannot be placed on one node, it will not be created. For more information, see the Placement on a single NUMA node section of the Dedicated Cores tutorial.

4. Specify the number of vCPUs.

5. Specify the RAM size.

6. If you need to add GPUs to the server, click Add GPU, select the GPU type, and specify the number of GPUs. The selected line will change to the GPU line. If you selected an image without GPU optimization in the Source block and the Auto-select GPU image toggle is on, the selected image will automatically change to a GPU-optimized image.

7. If both local and network volumes are available in the selected configuration, select the volume to be used as a boot volume:

   • local disk — check the Local SSD NVMe disk checkbox. A server with a local disk can only be created from images and apps;
   • network volume — do not check the Local SSD NVMe disk checkbox.

   The amount of RAM allocated to the server may be less than specified in the configuration — the operating system kernel reserves a portion of the RAM depending on the kernel version and distribution. You can check the allocated volume on the server using the sudo dmesg | grep Memory command.

Volumes

1. If you did not check the Local SSD NVMe disk checkbox when setting up the configuration, the first specified network volume will be used as the server boot disk. To configure it:

   1.1. Select the network boot volume type.

   1.2. Specify the network boot volume size in GB or TB. Mind the network volume limits for the maximum size.

   1.3. If you selected the Universal v2 or Fast SSD v2 disk type, specify the total number of read and write operations in IOPS. After creating the volume, you can change the IOPS count — reduce or increase it. The number of IOPS changes is unlimited.

2. Optional: add an additional network volume to the server:

   2.1. Click Add.

   2.2. Select the network volume type.

   2.3. Specify the network volume size in GB or TB. Mind the network volume limits for the maximum size.

   2.4. If you selected the Universal v2 or Fast SSD v2 volume type, specify the total IOPS. After the volume is created, you can adjust the number of IOPS — decrease or increase them. The number of IOPS changes is unlimited.

   After the server is created, you will be able to attach new additional volumes.

Network

The server can be added to a new private subnet or an existing one.

Private — a subnet without internet access or with a single static public IP address. The public IP address connects to a server that will be accessible from the internet.

Existing private subnet

1. Click Private subnet.

2. Expand the private subnet settings block.

3. In the Subnet field, select an existing subnet that is connected to a cloud router with internet access.

4. In the Private IP field, specify the server private IP address.

New private subnet

1. Click Private subnet.

2. Expand the private subnet settings block.

3. If you have existing networks in the project, select New subnet in the Subnet field.

4. Optional: change the subnet CIDR.

5. Optional: turn on the DHCP toggle.

6. Optional: in the Gateway field, change the default gateway IP address.

7. In the Network field, select the existing network where the subnet will be created, or New network. For a private subnet with a public IP address, a router-<network_name> router will be created automatically, where <network_name> is the network name.

8. If you selected New network, enter the network name.

Security

Select security groups to filter traffic on server ports. Without security groups, traffic will be denied. If the block is missing, traffic filtering (port security) is disabled in the server network. With traffic filtering disabled, all traffic is allowed.

Access

1. Add an SSH key for the project to the server for secure connection:

   1.1. If an SSH key for the project is not added to the cloud platform, click Add SSH key, enter a key name, paste the public key in OpenSSH format, and click Add.

   1.2. If an SSH key for the project has been added to the Cloud Platform, select an existing key in the SSH key field. The SSH key is only available in the pool where it is placed.

2. Optional: in the Password for “root” field:

   2.1. Copy the password for the root user — the user with unrestricted privileges for all system actions.

   2.2. Keep the password in a safe place and do not share it in plain text.

Additional settings

1. Optional: if you plan to create multiple servers and want to increase infrastructure fault tolerance, add the server to a placement group:

   1.1. To create a new group, click Create in the Placement group field.

   1.2. Select New group and enter the group name.

   1.3. Select the placement policy on different hosts:

   • preferred — soft-anti-affinity. The system will attempt to place servers on different hosts. If no suitable host is available when creating the server, it will be created on the same host;
   • mandatory — anti-affinity. Servers in the group must be located on different hosts. If no suitable host is available when creating the server, the server will not be created.

   1.4. If the group is created, select the placement group in the Placement group field.

2. Optional: to add additional information or filter servers in the list, add server tags. OS and configuration tags are added automatically. To add a new tag, enter it in the Tags field.

3. To add a script that will be executed by the cloud-init agent at the first operating system startup, in the Automation block in the User data field:

   • open the Text tab and paste the script as text;
   • or open the File tab and upload the script file.

   #cloud-config
   
   write_files:
   - path: "/opt/gomplate/values/user-values.yaml"
     permissions: "0644"
     content: |
       ServiceUsername: <username>
       ServicePassword: <password>
       ProjectId: <project_id>
       CloudRegion: <pool>
       CloudZone: <pool_segment>
       AccountNumber: <account_id>

   Specify:

   • <username> — service user name. You can view it in the control panel: in the top menu, click IAM and go to the Service users section (this section is available only to the Account Owner and users with the iam.admin role);
   • <password> — service user password. You can view it when creating the user or change it to a new one.
   • <project_id> — project ID. You can copy it in the control panel: in the top menu, click Products → Cloud Servers → open the projects menu → in the project line, click ;
   • <pool> — the pool where the cloud server will be created, for example, ru-3;
   • <pool_segment> — the pool segment where the cloud server will be created, for example, ru-3b;
   • <account_id> — account ID in the control panel. You can view it in the control panel in the top right corner.

View logs

If the temporary cloud server is not deleted, an issue occurred while configuring the Apache Kafka cluster. For example, the service user does not have sufficient permissions, an incorrect password was entered, or the user data settings contain errors.

1. Connect to the server that you created earlier.

2. To view the error log, open the cloud-init-output.log file in the vi text editor:

   vi /var/log/cloud-init-output.log

3. Depending on the error log, grant permissions to the service user or change the user data settings.

4. Recreate the server.

5. If you cannot fix the issue on your own, create a ticket and attach the logs with errors.