Cluster of three nodes with Apache Kafka

Apache Kafka is a distributed software message broker for processing real-time data streams from The Apache Software Foundation. The broker allows you to collect, store, and process large amounts of data from a variety of sources and feed it into data processing and analysis systems.Apache Kafka uses a distributed cluster architecture with three nodes for resilience to failures.

It is possible to create a cluster of three nodes with a ready-made Apache Kafka application.In Russia, the cluster nodes work with the configured SelectOS 1.1 operating system .In other countries — Ubuntu 24.04.

In Selectel, to create a cluster, you must first create a temporary cloud server with Apache Kafka.Creating a temporary server initiates the creation of a cluster of three nodes.

Each node is a cloud server with the configuration that was selected when the temporary cloud server was created.The three cloud servers created are on the same subnet as the temporary cloud server.

Next, a Kafka cluster will be configured on the three cloud servers.If the cluster is configured, the temporary cloud server is removed.If the temporary cloud server is not removed, there was a problem when configuring the Kafka cluster.To identify the problem, look at the logs.

The cost of the cluster will be calculated as the cost of three cloud servers.Read more about the cloud platform payment model in the article Payment Model and Pricing.

Create a cluster of three nodes with Apache Kafka

To configure Apache Kafka when creating a server, specify user data, which is the user configuration settings used to connect to the OpenStack API.

Create a service user.
Create a cloud server with Apache Kafka.

1. Create a service user

Create a service user with the role of Project Administrator or Project Supervisor. role Account Owner or User Administrator.

2. Create a cloud server with Apache Kafka

In the dashboard, on the top menu, click Products and select Cloud Servers.
Click Create Server.
In the Name and Location block:

3.1. enter the server name in the Name field.It will be used in the names of each server in the cluster in the format cs-<server_name>-<cluster_number>.

Here:
- <server_name> — server name;
- <cluster_number> — number of the server in the cluster from 1 to 3.
3.2 In the Region and Pool fields, select the region and the pool segment in which the server will be created.The list of available server configurations and resource costs depends on the pool segment.Once the server is created, the pool segment cannot be changed.
In the Source block, open the Applications tab.
Select Cloud Apache Kafka Cluster.
Optional: if you need a lower version of the application, in the Version field, select the desired version.
In the Configuration block, select the server configuration depending on the amount of data to be processed, the number of tops and partitions, and the number of clients.If the server is needed for development and testing, select a configuration of 2 vCPUs and RAM of 8 GB or more.If the server is needed for moderate workloads in a productive environment, select a configuration of 8 vCPUs and RAM of 32 GB or more.You can select:
- fixed configuration — rulers in which the ratio of resources is fixed;
- or an arbitrary configuration in which any resource ratio can be specified.
The configurations use different processors depending on the line and pool segment.

7.1 To select a fixed configuration, click Fixed, open the tab with the desired ruler and select the configuration.

7.2 To select an arbitrary configuration, click Arbitrary, specify the number of vCPUs and the RAM size.

7.3 To select a local disk as the server boot disk, check the Local SSD NVMe disk checkbox . To select a network disk as the boot disk, do not check the checkbox.

The amount of RAM allocated to the server may be less than the amount specified in the configuration — the operating system kernel reserves some RAM depending on the kernel version and distribution. You can check the allocated amount on the server using the sudo dmesg | grep Memory command.

After the server is created, you can change the configuration.
If you did not check the Local SSD NVMe disk checkbox in step 7.3, the first specified network disk will be used as the server boot disk.In the Disks block:

8.1. In the Disk Type field, select the type of network boot disk.

8.2 Specify the size of the network boot disk in GB or TB.Observe the maximum size limits of network disks.
In the Network block, connect an existing private subnet with the cloud router or create a new one:
- Existing private subnet
- New private subnet
9.1 Click Private Subnet.
9.2. In the Public IP Address for Internet Access field, select New Public IP Address.
9.3. Expand the block with private subnet settings.
9.4. In the Subnet field, select an existing subnet.
9.5. In the Private IP field, specify the private IP address of the server. The public IP address will be automatically connected to the private address.
In the Access block:

10.1 Place an SSH key on the server for secure connection.

To add a new SSH key to the cloud platform, click Add SSH Key, enter the key name, paste the public SSH key in OpenSSH format, and click Add.

If an SSH key is added to the cloud platform, select the existing key in the SSH Key field.

10.2 Optionally: in the Password for "root" field, copy the password of the root user (a user with unlimited rights to all actions on the system).Save the password in a safe place and do not transmit it in the public domain.

In the Automation block, in the User data field, insert the script that will be executed when the system boots:

#cloud-config

write_files:
- path: "/opt/gomplate/values/user-values.yaml"
  permissions: "0644"
  content: |
    ServiceUsername: <username>
    ServicePassword: <password>
    ProjectId: <project_id>
    CloudRegion: <pool>
    CloudZone: <pool_segment>
    AccountNumber: <account_id>

Specify:

<username> — service user name. You can view it in the control panel: in the top menu, click Account and go to the section Users → tab Service Users (the section is available only to the Account Owner and User Administrator);
<password> — service user password. You can view it when creating a user or change it to a new one. change it to a new one.
<project_id> — Project ID.Can be copied in control panel: in the top menu click Products and select Cloud Servers → open the projects menu → in the project line, click ;
<pool> — pool in which the cloud server will be created, e.g. ru-3;
<pool_segment> — pool segment in which the cloud server will be created, e.g. ru-3b;
<account_id> — account number in the control panel.Can be viewed in the control panel in the upper right corner.

Check the price of the cloud server.Note that once the three-node cluster is configured, the price will be calculated using the formula:
```
Цена кластера из трех нод = Цена за облачный сервер * 3
```
Click Create.

View logs

If the temporary cloud server is not removed, there is a problem when configuring the Apache Kafka cluster, for example, the service user does not have sufficient rights, the password is incorrect, or the settings in user data are incorrect.

Connect to the server that you you created earlier.
To see the error log, open the cloud-init-output.log file in a vi text editor:
```
vi /var/log/cloud-init-output.log
```
Depending on the error log, grant service user access or change the user data settings.
Re-create the server.
If you cannot fix the problem yourself, create a ticket and attach logs with errors.