Skip to main content

Using disk space in a Kafka cluster

In Managed Databases, some disk space is reserved for service needs. The total reserved space depends on the Kafka version.

Kafka 3.5Kafka 4.1
File system reservation4% of disk capacity4% of disk capacity

Reservation for the operating system,
service components, and logs

8 GB15 GB

The reserved portion of the disk space is not available for hosting databases. Take this into account when choosing a configuration lineup.

You can monitor disk occupancy using disk occupancy notifications and metrics. Learn more about metrics in the Monitoring Kafka clusters and nodes guide.

When a cluster disk is 95% full or more, the cluster will enter the DISK_FULL status and will be set to read-only mode. This is necessary to prevent the cluster from becoming completely locked or corrupted due to a lack of free space. To restore read and write operations, free up disk space or scale the cluster and select a configuration with a larger disk size than the previous one.

Disk occupancy notifications

Disk occupancy notifications are sent to the email addresses of the Account Owner and panel users who are subscribed to the notification category "Services". Notifications are sent when the disk is 80% and 95% full.

Free up disk space

To free up disk space, you can:

Delete a topic

  1. In the Dashboard, on the top menu, click Products and select Managed Databases.
  2. Open the Active tab.
  3. Open the cluster page → Topics tab.
  4. In the menu of the topic, select Delete.
  5. Enter the topic name to confirm deletion.
  6. Click Delete.

Configure a data cleanup policy

warning

After changing the log.retention.minutes and log.retention.hours parameters in the DBMS settings, the cluster will be restarted and will be unavailable during the restart.

To speed up disk cleanup, change the data retention policy in the cluster settings. Kafka will automatically delete data based on the new rules.

  1. In the Dashboard, click Products in the top menu and select Managed Databases.
  2. Open the Active tab.
  3. Open the cluster page → Settings tab.
  4. In the DBMS settings block, click Edit.
  5. Specify the new values for the data retention policy parameters.
  6. Click Save.

Parameters for configuring a data cleanup policy

log.segment.bytesThe maximum size of a single segment in a topic partition in bytes. Once the current segment is full, Kafka will create a new one. The default value is 1 073 741 824
log.retention.bytesThe maximum size of a topic partition in bytes. Once the partition is full, Kafka will start deleting old segments. The default value is -1 (no size limit)
log.retention.msSegment retention time in milliseconds. If not set, the log.retention.minutes parameter is used. If -1 is specified, there is no retention limit, even if log.retention.minutes or log.retention.hours are set
log.retention.minutesSegment retention time in minutes. If not set, the log.retention.hours parameter is used
log.retention.hoursSegment retention time in hours. Used if the log.retention.ms and log.retention.minutes parameters are not set. The default value is 168