--- title: "A three-node cluster with Apache Kafka" sidebar_label: "Apache Kafka" sidebar_position: 12 toc_max_heading_level: 3 description: "How to create a three-node cluster with an Apache Kafka message broker for real-time data stream processing" --- import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import { TabItemLabel } from 'docs-kit/components'; import {CustomTable} from 'docs-kit/components' import Formbricks from '@theme/MDXComponents/Formbricks'; import CopyIcon from 'docs-kit/icons/copy'; # A three-node cluster with Apache Kafka Apache Kafka is a distributed software message broker for real-time data stream processing from [The Apache Software Foundation](https://kafka.apache.org/). The broker allows you to collect, store, and process large volumes of data from various sources and transfer them to data processing and analysis systems. Apache Kafka uses a three-node distributed cluster architecture for fault tolerance. You can [create a three-node cluster with a pre-installed Apache Kafka application](#create-cloud-server-with-apache-kafka). In Russia, the cluster nodes run on a configured [SelectOS 1 64-bit](https://selectos.selectel.ru/news/mega/) operating system. In other [countries](/infrastructure/locations.mdx#country) — Ubuntu 24.04. Before creating a cloud server with an application, review the [software license agreements](https://423.selcdn.ru/kb/license-agreements-for-application-images-LANG.pdf) included in the image. ## Minimum resource requirements \{#minimum-requirements}
Test environmentProduction environment
Number of vCPUs48
RAM8 GB32 GB
Boot volume16 GBfrom 16 GB `*`
`*` The boot volume size for a production environment depends on your needs. ## Create a three-node cluster with Apache Kafka \{#create-cloud-server-with-apache-kafka} In Selectel, to create a cluster, you must first create a temporary cloud server with Apache Kafka. Creating a temporary server initiates the creation of a three-node cluster. Each node is a cloud server with the configuration chosen when creating the temporary cloud server. The name of each server in the cluster will be in the format `cs--`, where `` is the server name and `` is the server number in the cluster from 1 to 3. The three created cloud servers are in the same subnet as the temporary cloud server. Next, a Kafka cluster will be configured on three cloud servers. If the cluster is configured successfully, the temporary cloud server is deleted. If the temporary cloud server was not deleted, there was a problem during the Kafka cluster configuration. To identify the problem, [view the logs](#view-logs). The cost of the cluster will be calculated based on the cost of three cloud servers.For more information about the cloud platform payment model, see the [Payment model and prices](/cloud-servers/about/payment.mdx) article. To configure Apache Kafka, you must specify user data during creation — custom configuration parameters used for connecting to the OpenStack API. 1. [Create a service user](#add-service-user). 2. [Create a public floating IP address](#create-floating-ip). 3. [Create a cloud server with Apache Kafka](#create-apache-kafka-server). ### 1. Create a service user \{#add-service-user} [Create a service user](/access-control/manage/add-user.mdx#add-service-user), select the [`member`](/access-control/role-reference.mdx#member) or [`reader`](/access-control/role-reference.mdx#reader) role in the permissions and **Projects** as the access scope. Users can be created by the [Account Owner](/access-control/user-types.mdx#account-owner) or users with the [`iam.admin`](/access-control/role-reference.mdx#iam-admin) role. ### 2. Create a public floating IP address \{#create-floating-ip} Create a public floating IP address so the cloud server with Zabbix is accessible from the internet. Use the [Create a public floating IP address](/cloud-servers/cloud-networks/public-floating-ip-addresses.mdx#create-public-floating-ip) section of the [Public floating IP addresses](/cloud-servers/cloud-networks/public-floating-ip-addresses.mdx) guide. ### 3. Create a cloud server with Apache Kafka \{#create-apache-kafka-server} 1. In the [control panel](https://my.selectel.ru/vpc/default/servers/), on the top menu, click **Products** and select **Cloud Servers**. 2. Click **Create server**. 3. Fill in the blocks: * [Name and placement](#name-and-placement); * [Source](#source); * [Configuration](#configuration); * [Volumes](#disks); * [Internet](#internet); * [Private network](#private-network); * [Security](#security); * [Access](#access); * [Additional settings](#additional-settings). 4. Check the cloud server price. Note that after configuration, the price for a three-node cluster will be calculated using the formula: cloud server price \* 3. 5. Click **Create**. #### Name and placement \{#name-and-placement} 1. Enter the server name. It will be set as the hostname in the operating system. 2. Select a [location](/infrastructure/locations.mdx) where the server will be created. The available server configurations and resource costs depend on the location. You cannot change the location after the server is created. #### Source \{#source} 1. Open the **Applications** tab. 2. Select **Cloud Kafka**. 3. Optional: if you need a different current or archived application version, select it in the **Version** field. #### Configuration \{#configuration} Choose a configuration with at least 2 vCPUs and at least 8 GB of RAM if the server is intended for development and testing. If the server is intended for moderate workloads in a production environment, choose a configuration with at least 8 vCPUs and at least 32 GB of RAM. The boot disk size for a production environment depends on your needs. For all [lines](/cloud-servers/create/configurations.mdx#lines), except [Shared](/cloud-servers/create/configurations.mdx#shared) and [Dedicated](/cloud-servers/create/configurations.mdx#dedicated), two types of server configurations are available: * fixed configurations — configurations of lines with different specifications, in which the resource ratio is fixed; * custom configurations — configurations in which you can specify any resource ratio. Different [processors](/cloud-servers/create/configurations.mdx#available-processors) are used in the configurations, depending on the line and pool segment. You can customize the selected configuration. After the server is created, you will be able to [change the configuration](/cloud-servers/manage/change-server-configuration.mdx). Fixed configuration 1. Open the tab with the line. 2. Click **Fixed**. 3. Optional: you can adjust the configuration if you are creating a server in the ru-6 [multi-zone pool](/infrastructure/locations.mdx#multi-az-pool) or in the ru-3b, ru-7a, and ru-7b pool segments: 3.1. Expand the block with the configuration settings description. 3.2. Optional: select the processor vendor. Vendor selection is not available in all pools. 3.3. Optional: if you do not want physical processor cores reserved for the cloud server vCPUs, uncheck the **Dedicated cores** checkbox. For more information, see the [Dedicated cores](/cloud-servers/about/dedicated-cores.mdx) instructions. 3.4. Optional: if you want to disable [Hyper-Threading](/cloud-servers/about/dedicated-cores.mdx#hyper-threading) for a server with dedicated cores, uncheck the **Hyper-Threading (SMT)** checkbox. 3.5. Optional: if you are creating a server with dedicated cores and want to host a multiprocessor server on a single NUMA node, select the **Mandatory placement on a single NUMA node** checkbox. You can host a server with 4 vCPUs or more on one NUMA node. If the cloud server resources cannot be placed on one node, the server will not be created. For more information, see the [Placement on a single NUMA node](/cloud-servers/about/dedicated-cores.mdx#placing-on-one-numa-node) subsection of the [Dedicated cores](/cloud-servers/about/dedicated-cores.mdx) instructions. 4. Select a [configuration](/cloud-servers/create/configurations.mdx). 5. If both local and network volumes are available in the selected configuration, select the volume to be used as the boot volume: * [local volume](/cloud-servers/volumes/about-local-disks.mdx) — check the **Local SSD NVMe volume** checkbox. A server with a local volume can only be created from images and applications; * [network volume](/cloud-servers/volumes/about-network-volumes.mdx) — do not check the **Local SSD NVMe disk** checkbox. The amount of RAM allocated to the server may be less than specified in the configuration — the operating system kernel reserves a portion of RAM depending on the kernel version and distribution. You can check the allocated amount on the server using the `sudo dmesg | grep Memory` command. Custom configuration 1. Open the tab with the line. 2. Click **Custom**. 3. Optional: you can adjust the configuration if you are creating a server in the ru-6 [multi-zone pool](/infrastructure/locations.mdx#multi-az-pool) or in the ru-3b, ru-7a, and ru-7b pool segments: 3.1. Expand the block with the configuration settings description. 3.2. Optional: select the processor vendor. Vendor selection is not available in all pools. 3.3. Optional: if you do not want physical processor cores reserved for the cloud server vCPUs, uncheck the **Dedicated cores** checkbox. For more information, see the [Dedicated cores](/cloud-servers/about/dedicated-cores.mdx) instructions. 3.4. Optional: if you want to disable [Hyper-Threading](/cloud-servers/about/dedicated-cores.mdx#hyper-threading) for a cloud server with dedicated cores, uncheck the **Hyper-Threading (SMT)** checkbox. 3.5. Optional: if you are creating a cloud server with dedicated cores and want to host a multiprocessor server on a single NUMA node, select the **Mandatory placement on a single NUMA node** checkbox. You can host a cloud server with 4 vCPUs or more on one NUMA node. If the cloud server resources cannot be placed on one node, the server will not be created. For more information, see the [Placement on a single NUMA node](/cloud-servers/about/dedicated-cores.mdx#placing-on-one-numa-node) subsection of the [Dedicated cores](/cloud-servers/about/dedicated-cores.mdx) instructions. 4. Specify the number of vCPUs. 5. Specify the RAM size. 6. If you need to add graphics processors to the server, click **Add GPU**, select the [GPU type](/cloud-servers/create/gpus.mdx#available-gpus), and specify the number of GPUs. The selected line will change to the GPU line. If you selected an image without GPU optimization in the **Source** block and the **Auto-select GPU image** toggle is on, the selected image will automatically change to a GPU-optimized image. 7. If both local and network volumes are available in the selected configuration, select the volume to be used as the boot volume: * [local volume](/cloud-servers/volumes/about-local-disks.mdx) — check the **Local SSD NVMe volume** checkbox. A server with a local volume can only be created from images and applications; * [network volume](/cloud-servers/volumes/about-network-volumes.mdx) — do not check the **Local SSD NVMe disk** checkbox. The amount of RAM allocated to the server may be less than specified in the configuration — the operating system kernel reserves a portion of RAM depending on the kernel version and distribution. You can check the allocated amount on the server using the `sudo dmesg | grep Memory` command. #### Volumes \{#disks} 1. If you did not check the **Local SSD NVMe disk** checkbox when configuring, the first specified network volume will be used as the server boot disk. To configure it: 1.1. Select the [network boot volume type](/cloud-servers/volumes/about-network-volumes.mdx#network-volume-types). 1.2. Specify the network boot volume size in GB or TB. Take into account the [maximum size limits for network volumes](/cloud-servers/volumes/about-network-volumes.mdx#network-volume-limits). 1.3. If you chose the Universal v2 or SSD Fast v2 disk type, specify the total number of read and write operations in IOPS. After creating the disk, you can [change the number of IOPS](/cloud-servers/volumes/edit-volume.mdx#update-iops) — increase or decrease. You can change IOPS an unlimited number of times. 2. Optional: add an additional server [network volume](/cloud-servers/volumes/about-network-volumes.mdx): 2.1. Click **Add**. 2.2. Select the [network volume type](/cloud-servers/volumes/about-network-volumes.mdx#network-volume-types). 2.3. Specify the network volume size in GB or TB. Take into account the [maximum size limits for network volumes](/cloud-servers/volumes/about-network-volumes.mdx#network-volume-limits). 2.4. If you selected the Universal v2 or Fast SSD v2 volume type, specify the total IOPS for read and write operations. After the volume is created, you can [change the IOPS value](/cloud-servers/volumes/edit-volume.mdx#update-iops) — increase or decrease it. There is no limit to the number of IOPS changes. After the server is created, you will be able to [attach new additional volumes](/cloud-servers/volumes/attach-detach-volume.mdx). #### Internet \{#internet} Configure public access to the server. The cloud server will be added to a [private subnet](/cloud-servers/cloud-networks/private-networks-and-subnets.mdx) that is connected to a [cloud router](/cloud-servers/cloud-networks/cloud-routers.mdx) with 1:1 NAT and internet access. Access to and from the internet will be through the cloud router. The server will be accessible from the internet via a [public floating IP address](/cloud-servers/cloud-networks/public-floating-ip-addresses.mdx). 1. In the **Internet access** field, select the **Public floating IP address** access type. 2. Select the public floating IP address you [created in step 2](#create-floating-ip). #### Private network \{#private-network} The cloud server can be added to an existing or new private subnet. Existing private subnet 1. In the **Subnet** field, select a private subnet. 2. Optional: in the **IP address** field, change the default IP address. 3. In the **Router** field, select an existing router or create a new one. If the router is not connected to the internet, it will be automatically connected after the server is created. New private subnet 1. In the **Subnet** field, select **New private subnet**. 2. In the **Network** field, select an existing network or create a new one. 3. Optional: change the subnet CIDR. 4. Optional: in the **IP address** field, change the default IP address. 5. In the **Router** field, select an existing router or create a new one. If the router is not connected to the internet, it will be automatically connected after the server is created. #### Security \{#security} Select [security groups](/cloud-servers/security-groups/) to filter traffic on server ports. Without security groups, all traffic is denied. If this block is missing, [traffic filtering (port security](/cloud-servers/cloud-networks/about-networks.mdx#traffic-filtering-port-security)) is disabled in the server network. With filtering disabled, all traffic is permitted. #### Access \{#access} 1. Place an [SSH key for the project](/cloud-servers/manage/create-and-place-ssh-key.mdx#place-public-ssh-key) on the server for secure connection: 1.1. If an SSH key for the project has not been added to the cloud platform, click **Add SSH key**, enter the key name, paste the public key in OpenSSH format, and click **Add**. 1.2. If a project SSH key has been added to the cloud platform, select an existing key in the **SSH key** field. The SSH key is available only in the pool where it is hosted. 2. Optional: in the **Password for root** field: 2.1. Copy the password for the `root` user—a user with unrestricted privileges for any system action. 2.2. Save the password in a secure place and do not share it in plain text. #### Additional settings \{#additional-settings} 1. Optional: if you plan to create multiple servers and want to increase infrastructure reliability, add the server to a [placement group](/cloud-servers/placement-groups/about-placement-groups.mdx): 1.1. To create a new group, click **Create** in the **Placement group** field. 1.2. Select **New group** and enter the group name. 1.3. Select the placement policy for different hosts: * preferred — soft-anti-affinity. The system will try to place servers on different hosts. If there is no suitable host when creating the server, it will be created on the same host; * mandatory — anti-affinity. Servers in the group must be placed on different hosts. If there is no suitable host when creating the server, the server will not be created. 1.4. If the group has been created, select the required placement group in the **Placement group** field. 2. Optional: to add additional information or filter servers in the list, add server [tags](/cloud-servers/manage/tags.mdx). OS and configuration tags are added automatically. To add a new tag, enter it in the **Tags** field. 3. To add a script to be executed by the [cloud-init](https://cloud-init.io/) agent on the first boot of the operating system, in the **Automation** block, in the **User data** field: * open the **Text** tab and paste the script as text; * or open the **File** tab and upload the script file. ```yaml #cloud-config write_files: - path: "/opt/gomplate/values/user-values.yaml" permissions: "0644" content: | ServiceUsername: ServicePassword: ProjectId: CloudRegion: CloudZone: AccountNumber: ``` Specify: * `` — service user name. You can view it in the [control panel](https://my.selectel.ru/iam/service-users): in the top menu, click **IAM** and go to the **Service Users** section (this section is only available to the account owner and users with the `iam.admin` role); * `` — service user password. You can view it when creating the user or [change it to a new one](/access-control/manage/edit-user-data-or-role.mdx#change-service-user-password). * `` — project ID. You can copy it in the [control panel](https://my.selectel.ru/vpc/): in the top menu, click **Products** → **Cloud Servers** → open the projects menu → in the project line, click ; * `` — the [pool](/infrastructure/locations.mdx#pool) where the cloud server will be created, for example, `ru-3`; * `` — the [pool segment](/infrastructure/locations.mdx#pool) where the cloud server will be created, for example, `ru-3b`; * `` — account number in the control panel. You can view it in the [control panel](https://my.selectel.ru/) in the top right corner. ## View logs \{#view-logs} If the temporary cloud server was not deleted, an issue occurred during the Apache Kafka cluster configuration. For example, the service user may have insufficient permissions, a wrong password was entered, or the user data settings contain errors. 1. [Connect to the server](/cloud-servers/manage/connect-to-server.mdx) that you [created earlier](#create-apache-kafka-server). 2. To view the error log, open the `cloud-init-output.log` file in the `vi` text editor. ```bash vi /var/log/cloud-init-output.log ``` 3. Depending on the error log, grant access to the service user or change the user data settings. 4. Recreate the server. 5. If you cannot fix the issue yourself, [create a ticket](https://my.selectel.ru/tickets/create) and attach the logs with errors.