Cloud server with ELK

Last update: October 05 2026Open in Markdown

ELK (Elasticsearch, Logstash, Kibana) is a set of open-source tools for analyzing and visualizing logs and other data.

In Selectel, a cloud server with ELK includes:

• Elasticsearch — an open distributed data management and search system based on the Apache Lucene search engine. The system provides powerful capabilities for indexing, storing, searching, and analyzing large volumes of information in real time;
• Kibana — a web interface for monitoring, analyzing Elasticsearch logs, and presenting data on dashboards;
• Filebeat — an agent for collecting logs and transferring them to Elasticsearch;
• Metricbeat — an agent for collecting metrics and transferring them to Elasticsearch.

You can create a cloud server with a pre-installed ELK app. In Russia, the server runs the configured operating system SelectOS 1 64-bit. In other countries — Ubuntu 22.04.

Before creating a cloud server with the app, review the software license agreements included in the image.

Minimum resource requirements

Number of vCPUs	4
RAM	10 GB
Boot volume	30 GB

Create a cloud server with ELK

For ELK to work, the cloud server must be accessible from the internet. To do this, when creating a server, you need to create a private subnet and attach a public floating IP address. To configure ELK, specify application parameters when creating the server.

After creating the server with ELK, a free TLS certificate from Let’s Encrypt® will be automatically issued for the domain you specify. To issue the certificate, add an A record for the domain and specify the server's public floating IP address in the record value. You can add the domain to Selectel DNS hosting (actual).

1. Create a public floating IP address.

2. Add an A record for the domain.

3. Create a cloud server with ELK.

1. Create a public floating IP address

Create a public floating IP address so that the cloud server with ELK is accessible from the internet.

Use the Create a public floating IP address section in Public floating IP addresses.

2. Add an A record for the domain

Add a DNS record to access ELK via domain.

Specify:

• DNS record type in the group — A;
• record value — the public floating IP address you created in step 1.

3. Create a cloud server with ELK

1. In the control panel, in the top menu, click Products and select Cloud Servers.

2. Click Create Server.

3. Fill in the blocks:

   • Name and location;
   • Source;
   • Configuration;
   • Disks;
   • Internet;
   • Private network;
   • Security;
   • Access;
   • Additional settings.

4. Check the cloud server price.

5. Click Create.

Name and location

1. Enter the server name. It will be set as the hostname in the operating system.

2. Select the location where the server will be created. The list of available server configurations and resource cost depend on the location. You cannot change the location after creating the server.

Source

1. Open the Applications tab.

2. Select Cloud ELK.

3. Optional: if you need a different current or archived application version, in the Version field, select the desired version.

4. Fill in the application parameters:

   4.1. In the Domain zone field, enter the domain for accessing ELK that you added in step 2.

   4.2. Enter the administrator email address to issue the Let’s Encrypt® certificate.

   4.3. Enter the Kibana administrator username. You cannot use the name elastic — it is reserved by the system.

   4.4. Enter the Kibana administrator password. The minimum password length is eight characters.

Configuration

Select a configuration from 4 vCPUs, RAM from 10 GB, and boot disk size from 30 GB. For all lines except Shared and Dedicated, two types of server configurations are available:

• fixed configurations — configurations of lines with different technical specifications, in which the ratio of resources is fixed;
• custom configurations — configurations in which you can specify any ratio of resources.

Configurations use different processors depending on the line and pool segment. You can customize the selected configuration. After creating the server, you will be able to change the configuration.

Fixed configuration

1. Open the tab with the line.

2. Click Fixed.

3. Optional: you can customize the configuration if you are creating a server in the ru-6 multi-AZ pool segment or pool segments ru-3b, ru-7a, and ru-7b:

   3.1. Expand the block with the configuration settings description.

   3.2. Optional: select a processor manufacturer. Selecting a manufacturer is not available in all pools.

   3.3. Optional: if you do not want physical processor cores to be assigned to the cloud server vCPUs, uncheck the Dedicated cores checkbox. For more information, see the Dedicated Cores instructions.

   3.4. Optional: if you want to disable Hyper-Threading for a server with dedicated cores, uncheck the Hyper-Threading (SMT) checkbox.

   3.5. Optional: if you are creating a server with dedicated cores and want to place a multi-processor server on a single NUMA node, check the Mandatory placement on a single NUMA node checkbox. A server with 4 vCPUs or more can be placed on a single NUMA node. If the cloud server resources cannot be placed on a single node, it will not be created. For more information, see the Placement on a Single NUMA Node section of the Dedicated Cores instructions.

4. Select a configuration.

5. If both local and network disks are available in the selected configuration, select the disk to be used as the boot disk:

   • local disk — check the Local SSD NVMe disk checkbox. A server with a local disk can only be created from images and applications;
   • network disk — do not check the Local SSD NVMe disk checkbox.

   The amount of RAM allocated to the server may be less than specified in the configuration — the operating system kernel reserves part of the RAM depending on the kernel version and distribution. You can check the allocated amount on the server using the sudo dmesg | grep Memory command.

Custom configuration

1. Open the tab with the line.

2. Click Custom.

3. Optional: you can customize the configuration if you are creating a server in the ru-6 multi-AZ pool segment or pool segments ru-3b, ru-7a, and ru-7b:

   3.1. Expand the block with the configuration settings description.

   3.2. Optional: select a processor manufacturer. Selecting a manufacturer is not available in all pools.

   3.3. Optional: if you do not want physical processor cores to be assigned to the cloud server vCPUs, uncheck the Dedicated cores checkbox. For more information, see the Dedicated Cores instructions.

   3.4. Optional: if you want to disable Hyper-Threading for a cloud server with dedicated cores, uncheck the Hyper-Threading (SMT) checkbox.

   3.5. Optional: if you are creating a cloud server with dedicated cores and want to place a multi-processor server on a single NUMA node, check the Mandatory placement on a single NUMA node checkbox. A cloud server with 4 vCPUs or more can be placed on a single NUMA node. If the cloud server resources cannot be placed on a single node, it will not be created. For more information, see the Placement on a Single NUMA Node section of the Dedicated Cores instructions.

4. Specify the number of vCPUs.

5. Specify the RAM size.

6. If you need to add graphics processing units to the server, click Add GPU, select the GPU type, and specify the number of GPUs. The selected line will change to the GPU line. If you selected an image without GPU optimization in the Source block and the Auto-select GPU image toggle switch is enabled, the selected image will automatically switch to a GPU-optimized image.

7. If both local and network disks are available in the selected configuration, select the disk to be used as the boot disk:

   • local disk — check the Local SSD NVMe disk checkbox. A server with a local disk can only be created from images and applications;
   • network disk — do not check the Local SSD NVMe disk checkbox.

   The amount of RAM allocated to the server may be less than specified in the configuration — the operating system kernel reserves part of the RAM depending on the kernel version and distribution. You can check the allocated amount on the server using the sudo dmesg | grep Memory command.

Disks

1. If you did not check the Local SSD NVMe disk checkbox when customizing the configuration, the first specified network disk will be used as the server boot disk. To configure it:

   1.1. Select the boot network disk type.

   1.2. Specify the network boot disk size in GB or TB. Take into account the network disk limits on maximum size.

   1.3. If you selected the Universal v2 or Fast SSD v2 disk type, specify the total number of read and write operations in IOPS. After creating the disk, you can change the number of IOPS — decrease or increase it. The number of IOPS changes is unlimited.

2. Optional: add an additional network disk of the server :

   2.1. Click Add.

   2.2. Select the network disk type.

   2.3. Specify the network disk size in GB or TB. Take into account the network disk limits on maximum size.

   2.4. If you selected the Universal v2 or Fast SSD v2 disk type, specify the total number of read and write operations in IOPS. After creating the disk, you can change the number of IOPS — decrease or increase it. The number of IOPS changes is unlimited.

   After creating the server, you will be able to attach new additional disks.

Internet

Configure public access to the server.

The cloud server will be added to the private subnet connected to the cloud router with 1:1 NAT and internet access. Internet access to and from the internet will be carried out via the cloud router. The server will be accessible from the internet via a public floating IP address.

1. In the Connecting from the internet field, select the Public floating IP address access type.

2. Select the public floating IP address you created in Step 1.

Private network

A cloud server can be added to an existing or new private subnet.

Existing private subnet

1. In the Subnet field, select a private subnet.

2. Optional: in the IP address field, change the default IP address.

3. In the Router field, select an existing router or create a new one.

   If the router is not connected to the internet, it will be automatically connected to the internet after the server is created.

New private subnet

1. In the Subnet field, select New private subnet.

2. In the Network field, select an existing network or create a new one.

3. Optional: change the subnet CIDR.

4. Optional: in the IP address field, change the default IP address.

5. In the Router field, select an existing router or create a new one.

   If the router is not connected to the internet, it will be automatically connected to the internet after the server is created.

Security

Select security groups to filter traffic on server ports. Without security groups, traffic will be blocked. If the block is missing, traffic filtering (port security) is disabled in the server network. With traffic filtering disabled, all traffic will be allowed.

Access

1. Place a project SSH key on the server for secure connection:

   1.1. If the project SSH key has not been added to the cloud platform, click Add SSH key, enter the key name, paste the public key in OpenSSH format, and click Add.

   1.2. If the project SSH key has been added to the cloud platform, in the SSH key field, select the existing key. The SSH key is only available in the pool where it is placed.

2. Optional: in the Password for "root" field:

   2.1. Copy the password of the root user — a user with unrestricted permissions for all actions on the system.

   2.2. Save the password in a safe place and do not share it in plain text.

Additional settings

1. Optional: if you plan to create multiple servers and want to increase infrastructure fault tolerance, add the server to a placement group:

   1.1. To create a new group, in the Placement group field, click Create.

   1.2. Select New group and enter the group name.

   1.3. Select a placement policy on different hosts:

   • desirable — soft-anti-affinity. The system will try to place the servers on different hosts. If there is no suitable host when creating the server, it will be created on the same host;
   • mandatory — anti-affinity. Servers in the group must be located on different hosts. If there is no suitable host when creating the server, the server will not be created.

   1.4. If the group has been created, in the Placement group field, select the placement group.

2. Optional: to add additional information or filter servers in the list, add server tags. Operating system and configuration tags are added automatically. To add a new tag, in the Tags field, enter the tag.

3. To add a script that will be executed using the cloud-init agent on the first boot of the operating system, in the Automation block, in the User data field:

   • open the Text tab and paste the script as text;
   • or open the File tab and upload the file with the script.

Install Filebeat on the observed servers

Filebeat is installed on the servers from which you want to collect monitoring data. Filebeat collects log messages from log files and forwards them to the cloud server with Elasticsearch for indexing.

1. Create a Kibana user in advance with the required permissions and roles in the ELK web interface. For more information, see Create a user in the Elastic documentation.

2. Install Filebeat. For more information, see Install Filebeat in the Elastic documentation.

   To install on Ubuntu, you can download the distribution package from the mirror.

3. Download the CA public certificate from the cloud server to connect Filebeat via the HTTPS protocol:

   cd /etc/filebeat && wget https://<example.com>/ca.crt

   Specify <example.com> — the domain for accessing Elasticsearch that you added in step 2.

4. Open the configuration file /etc/filebeat/filebeat.yml in a text editor.

5. Add the Elasticsearch connection parameters to the configuration file:

   outputs:
     default:
       type: elasticsearch
       hosts: [https://<example.com>:9200]
       username: "<username>"
       password: "<password>"
       ssl.enabled: true
       ssl.certificate_authorities: "/etc/filebeat/ca.crt"

   Specify:

   • <example.com> — the domain for accessing ELK that you added in step 2;
   • <username> — the Kibana username you created in step 1;
   • <password> — the Kibana user password you created in step 1.

6. Restart the agent:

   systemctl restart filebeat

Install Elastic Agent on the observed servers

Elastic Agent is installed on the servers from which you want to collect metrics, logs, and security events. The agent collects information and forwards it to the cloud server with Elasticsearch.

1. Create a Kibana user in advance with the required permissions and roles in the ELK web interface. For more information, see Create a user in the Elastic documentation.

2. Install Elastic Agent. For more information, see Install Elastic Agents in the Elastic documentation.

   To install on Ubuntu, you can download the distribution package from the mirror.

3. Download the CA public certificate from the cloud server to connect Elastic Agent via the HTTPS protocol:

   cd /etc/elastic-agent && wget https://<example.com>/ca.crt

   Specify <example.com> — the domain for accessing Elasticsearch that you added in step 2.

4. Open the configuration file /etc/elastic-agent/elastic-agent.yml in a text editor.

5. Add the Elasticsearch connection parameters to the configuration file:

   outputs:
     default:
       type: elasticsearch
       hosts: [https://<example.com>:9200]
       username: "<username>"
       password: "<password>"
       ssl.enabled: true
       ssl.certificate_authorities: "/etc/elastic-agent/ca.crt"

   Specify:

   • <example.com> — the domain for accessing ELK that you added in step 2;
   • <username> — the Kibana username you created in step 1;
   • <password> — the Kibana user password you created in step 1.

6. Restart the agent:

   systemctl restart elastic-agent