Cluster of three nodes with Apache Kafka

Apache Kafka is a distributed software message broker for processing real-time data streams from The Apache Software Foundation. The broker allows you to collect, store, and process large amounts of data from a variety of sources and feed it into data processing and analysis systems.Apache Kafka uses a distributed cluster architecture with three nodes for resilience to failures.

It is possible to create a cluster of three nodes with a ready-made Apache Kafka application. In Russia, the cluster nodes work with a customized SelectOS 1.1 operating system. In other countries — Ubuntu 22.04.

In Selectel, to create a cluster, you must first create a temporary cloud server with Apache Kafka.Creating a temporary server initiates the creation of a cluster of three nodes.

Each node is a cloud server with the configuration that was selected when the temporary cloud server was created.The three cloud servers created are on the same subnet as the temporary cloud server.

Next, a Kafka cluster will be configured on the three cloud servers.If the cluster is configured, the temporary cloud server is removed.If the temporary cloud server is not removed, there was a problem when configuring the Kafka cluster.To identify the problem, look at the logs.

The cost of the cluster will be calculated as the cost of three cloud servers.Read more about the cloud platform payment model in the article Payment Model and Pricing.

Create a cluster of three nodes with Apache Kafka⁠

To configure Apache Kafka when creating a server, you must specify user data, which is the user configuration settings used to connect to the OpenStack API.

Create a service user.
Create a cloud server with Apache Kafka.

1. Create a service user⁠

Create a service user with the role of Project Administrator or Project Supervisor. role Account Owner or User Administrator.

2. Create a cloud server with Apache Kafka⁠

In the dashboard, on the top menu, click Products and select Cloud Servers.
Click Create Server.
In the Name and Location block:

3.1. enter the server name in the Name field.It will be used in the names of each server in the cluster in the format cs-<server_name>-<cluster_number>.
```
Здесь:

* `<server_name>` — имя сервера;
* `<cluster_number>` — номер сервера в кластере от 1 до 3.
```
3.2 In the Region and Pool fields, select the region and the pool segment in which the server will be created.The list of available server configurations and resource costs depends on the pool segment.Once the server is created, the pool segment cannot be changed.
In the Source block, select the source from which the server will be created.

Click the default source name, open the Applications tab, select Cloud Apache Kafka Cluster <version> 64-bit, and click Select.

In the Configuration block, select the server configuration depending on the amount of data to be processed, the number of tops and partitions, and the number of clients.If the server is needed for development and testing, select a configuration of 2 vCPUs and RAM of 8 GB or more.If the server is needed for moderate workloads in a productive environment, select a configuration of 8 vCPUs and RAM of 32 GB or more.You can select:

*   [фиксированную](/cloud-servers/create/configurations.mdx#fixed-configurations) конфигурацию — линейки, в которых зафиксировано соотношение ресурсов;
*   или [произвольную](/cloud-servers/create/configurations.mdx#custom-configurations) конфигурацию, в которой можно указать любое соотношение ресурсов.

В конфигурациях используются разные [процессоры](/cloud-servers/create/configurations.mdx#available-processors) в зависимости от линейки и сегмента пула.

5.1. Чтобы выбрать фиксированную конфигурацию, нажмите **Фиксированная**, откройте вкладку с нужной линейкой и выберите конфигурацию.

5.2. Чтобы выбрать произвольную конфигурацию, нажмите **Произвольная**, укажите количество vCPU и размер RAM.

5.3. Чтобы в качестве загрузочного диска сервера выбрать [локальный диск](/cloud-servers/volumes/about-local-disks.mdx), отметьте чекбокс **Локальный SSD NVMe диск**. Чтобы в качестве загрузочного диска выбрать [сетевой диск](/cloud-servers/volumes/about-network-volumes.mdx), не отмечайте чекбокс.

Объем оперативной памяти, который выделяется серверу, может быть меньше указанного в конфигурации, — ядро операционной системы резервирует часть оперативной памяти в зависимости от версии ядра и дистрибутива. Выделенный объем на сервере можно проверить с помощью команды `sudo dmesg | grep Memory`.

После создания сервера можно [изменить конфигурацию](/cloud-servers/manage/change-server-configuration.mdx).

If you did not check the Local SSD NVMe disk checkbox in step 5.3, the first specified network disk will be used as the server boot disk.In the Disks block:

6.1. In the Disk Type field, select the type of network boot disk.

6.2 Specify the size of the network boot disk in GB or TB.Observe the maximum size limits of network disks.
In the Network block, connect an existing private subnet with the cloud router or create a new one:
- Existing private subnet
- New private subnet
7.1 Click Private Subnet.
7.2. In the Public IP Address for Internet Access field, select New Public IP Address.
7.3. Expand the block with private subnet settings.
7.4. In the Subnet field, select an existing subnet.
7.5. In the Private IP field, specify the private IP address of the server. The public IP address will be automatically connected to the private address.
In the Access block:

8.1 Place an SSH key on the server for secure connection.

To add a new SSH key to the cloud platform, click Add SSH Key, enter the key name, paste the public SSH key in OpenSSH format, and click Add.

If an SSH key is added to the cloud platform, select the existing key in the SSH Key field.

8.2 Optionally: in the Password for "root" field, copy the password of the root user (a user with unlimited rights to all actions on the system).Save the password in a safe place and do not transmit it in the public domain.
In the Automation block, in the User data field, insert the script that will be executed when the system boots:
```
#cloud-config

write_files:
- path: "/opt/gomplate/values/user-values.yaml"
    permissions: "0644"
    content: |
    ServiceUsername: <username>
    ServicePassword: '<password>'
    ProjectId: '<project_id>'
    CloudRegion: <pool>
    CloudZone: <pool_segment>
    AccountNumber: <account_id>
```
Specify:
- <username> — service user name. You can look it up in the control panel: in the top menu, click Account and go to the section Users → tab Service Users (the section is available only to the Account Owner and User Administrator);
- <password> — service user password. You can view it when creating a user or change it to a new one. change it to a new one.
- <project_id> — Project ID.Can be copied in control panel: in the top menu click Products and select Cloud Servers → open the projects menu → in the project line, click ;
- <pool> — pool in which the cloud server will be created, e.g. ru-3;
- <pool_segment> — pool segment in which the cloud server will be created, e.g. ru-3b;
- <account_id> — account number in the control panel.Can be viewed in the control panel in the upper right corner.
Check the price of the cloud server.Note that once the three-node cluster is configured, the price will be calculated using the formula:
```
Цена кластера из трех нод = Цена за облачный сервер * 3
```
Click Create.

View logs⁠

If the temporary cloud server is not removed, there is a problem when configuring the Apache Kafka cluster, for example, the service user does not have sufficient rights, the password is incorrect, or the settings in user data are incorrect.

Connect to the server that you you created earlier.
To see the error log, open the cloud-init-output.log file in a vi text editor:
```
vi /var/log/cloud-init-output.log
```
Depending on the error log, grant service user access or change the user data settings.
Re-create the server.
If you cannot fix the problem yourself, create a ticket and attach logs with errors.