Skip to main content

Using disk space in a Kafka cluster

Last update:

In Managed Databases, part of the disk space is reserved for service needs. The total amount of reserved space depends on the Kafka version.

Kafka 3.5Kafka 4.1
File system reservation4% of disk capacity4% of disk capacity

Reservation for the operating system,
service components, and logs

8 GB15 GB

The reserved portion of the disk space is not available for hosting databases. Take this into account when selecting a configuration lineup.

You can track disk occupancy using disk occupancy notifications and metrics. For more information about metrics, see the Monitoring Kafka clusters and nodes guide.

When a cluster disk is 95% full or more, the cluster will enter the DISK_FULL status and will function in read-only mode. This is necessary to prevent complete blocking or corruption of the cluster due to lack of free space. To make the cluster work in read-write mode, free up disk space or scale the cluster and select a configuration with a larger disk size than the previous configuration. is in the previous configuration.

Disk occupancy notifications

Disk occupancy notifications are sent to the email address of the Account Owner and users who are subscribed to the "Services" notification category. Notifications are sent when the disk reaches 80% and 95% occupancy.

Free up disk space

To free up disk space, you can:

Delete a topic

  1. In the Dashboard, on the top menu, click Products and select Managed Databases.
  2. Open the Active tab.
  3. Open the cluster page → Topics tab.
  4. In the menu of the topic, select Delete.
  5. Enter the topic name to confirm deletion.
  6. Click Delete.

Configure a data cleanup policy

warning

After changing the log.retention.minutes and log.retention.hours parameters in the DBMS settings, the cluster will be restarted and will be unavailable during the restart.

To speed up disk clearing, change the data retention policy in the cluster settings. Kafka will automatically delete data according to the new rules.

  1. In the Dashboard, on the top menu, click Products and select Managed Databases.
  2. Open the Active tab.
  3. Open the cluster page → Settings tab.
  4. In the DBMS settings block, click Edit.
  5. Specify the new values for the parameters for configuring a data cleanup policy.
  6. Click Save.

Parameters for configuring a data cleanup policy

log.segment.bytesThe maximum size of a single segment in a topic partition in bytes. When the current segment is full, Kafka will create a new one. The default value is 1 073 741 824
log.retention.bytesThe maximum size of a topic partition in bytes. When the partition is full, Kafka will start deleting old segments. The default value is -1 (maximum size is unlimited)
log.retention.msThe segment retention time in milliseconds. If not set, the log.retention.minutes parameter is used. If -1 is specified, the retention time is unlimited, even if the log.retention.minutes or log.retention.hours parameters are set
log.retention.minutesThe segment retention time in minutes. If not set, the log.retention.hours parameter is used
log.retention.hoursThe segment retention time in hours. Used if the log.retention.ms and log.retention.minutes parameters are not specified. The default value is 168