---
title: "Fluent Bit"
sidebar_label: "Fluent Bit"
description: "Fluent Bit capabilities, how to configure Fluent Bit to receive events into your infrastructure and add events"
sidebar_position: 3
---

import Tabs from '@theme/Tabs'
import TabItem from '@theme/TabItem'
import {TabItemLabel} from '@selectel/docux/components'
import Formbricks from '@theme/MDXComponents/Formbricks'
import IssueS3Key from '@site/i18n/en/docusaurus-plugin-content-docs/current/_partials/common/issue-s3-key.mdx'

# Fluent Bit

[Fluent Bit](https://docs.fluentbit.io/manual) is an agent that allows you to configure the process of collecting, processing, and exporting logs, as well as importing logs from third-party storage. Fluent Bit is extensible with output and input plugins. You can view the list of plugins on the Fluent Bit website ([Outputs](https://docs.fluentbit.io/manual/pipeline/outputs), [Inputs](https://docs.fluentbit.io/manual/pipeline/inputs)).

With Fluent Bit, you can manage logs in the Logs service: receive logs or add logs from your own storage.

Before you begin, [configure Fluent Bit](#configure-fluent-bit).

## Configure Fluent Bit \{#configure-fluent-bit}

1. [Add a service user](#add-service-user).
2. [Issue an S3 key to the user](#issue-s3-key).
3. [Configure the agent](#configure-agent) to receive or add events.

### 1. Add a service user \{#add-service-user}

[Add a service user](/access-control/manage/add-user.mdx#add-service-user) with access to the **Projects** area and the following role:

* [`member`](/access-control/role-reference.mdx#member) or [`logs.admin`](/logs/manage-access.mdx#logs-admin) for log read and write access;
* [`logs.write`](/logs/manage-access.mdx#logs-writer) for log write access;
* [`reader`](/access-control/role-reference.mdx#reader) or [`logs.viewer`](/logs/manage-access.mdx#logs-viewer) for log read-only access.

Users can be added by the [Account Owner](/access-control/user-types.mdx#account-owner) or users with the [`iam.admin`](/access-control/role-reference.mdx#iam-admin) role.

### 2. Issue an S3 key to the user \{#issue-s3-key}

<IssueS3Key />

### 3. Configure the agent \{#configure-agent}

<Tabs queryString="configure-agent">
  <TabItem value="get-logs">
    <TabItemLabel>
      Retrieving logs
    </TabItemLabel>

    Using the built-in plugin [fluent-bit-cloudwatch-input-plugin](https://github.com/selectel/fluent-bit-cloudwatch-input-plugin), you can configure exporting events from the Logs service to your infrastructure, for example to [Elasticsearch](https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch), [Loki](https://docs.fluentbit.io/manual/pipeline/outputs/loki), or [Kafka](https://docs.fluentbit.io/manual/pipeline/outputs/kafka).

    1. Download the configuration file [`config/fluent-bit.yaml`](https://github.com/selectel/fluent-bit-cloudwatch-input-plugin/blob/master/master/config/fluent-bit.yaml).

    2. Open the configuration file `config/fluent-bit.yaml` in a text editor.

    3. In the configuration file parameters, specify:

       * `region` — [pool](/infrastructure/locations.mdx#pool), for example  `ru-9`;
       * `endpoint` — URL for accessing the Logs API. The list of URLs can be found in the [Logs](/api/urls/#logs) subsection of the [List of URLs](/api/urls/);
       * `log_group_name` — the log group name, for example `s/lbaas/Chromie-lb`;
       * `log_stream_name` — the event stream name (stream), for example `http-21380357`;
       * `sqlite_path` — the path to the SQLite database that stores information about the last received events. One database can be used for several groups and streams.

       The list of events from the [GetLogEvents](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html) Amazon CloudWatch API method is split into separate records using a Lua filter. Each received event is converted into a separate, independent Fluent Bit record (record). This allows you to process events individually, tag them, and send them to different output plugins. The configuration file uses the [stdout](https://docs.fluentbit.io/manual/pipeline/outputs/standard-output) output plugin, but you can use any other, for example [Elasticsearch](https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch) or [Kafka](https://docs.fluentbit.io/manual/pipeline/outputs/kafka).

    4. Optional: to configure data collection from multiple groups and streams within a single configuration file, in the `pipeline` block add the corresponding `inputs`.

    5. Run Fluent Bit:

       ```bash
       docker run \
         --name fluent-bit-cloudwatch \
         --rm \
         -v ${PWD}/config/fluent-bit.yaml:/fluent-bit/etc/fluent-bit.yaml:ro \
         -v ${PWD}/sqlite:/var/lib/fluent-bit/cloudwatch/sqlite:rw \
         -e AWS_ACCESS_KEY_ID=<access_key> \
         -e AWS_SECRET_ACCESS_KEY=<secret_key> \
         ghcr.io/selectel/fluent-bit-cloudwatch-input-plugin:latest
       ```

       Specify:

       * `<access_key>` — the value of the **Access key** field from the S3 key that you [issued to the user](#issue-s3-key);
       * `<secret_key>` — the value of the **Secret key** field from the S3 key that you [issued to the user](#issue-s3-key).

       You can add `<access_key>` and `<secret_key>` to the container in any way:

       * via environment variables. More details can be found in the [Using environment variables to globally configure AWS SDKs and tools](https://docs.aws.amazon.com/sdkref/latest/guide/environment-variables.html) guide in the Amazon documentation;
       * or via `config` and `credentials` files. More details can be found in the [Using shared config and credentials files to globally configure AWS SDKs and tools](https://docs.aws.amazon.com/sdkref/latest/guide/file-format.html) guide in the Amazon documentation.

       Example of output in stdout in MessagePack format:

       ```bash
       ...
       [1029] cloudwatch-input.0: [[1751577177.000000000, {}], {"ingestion_time"=>1750606362344, "message"=>"{"level":"System", "msg":"InnoDB initialization has started."}", "timestamp"=>1750606335369}]
       [1031] cloudwatch-input.0: [[1751577177.000000000, {}], {"ingestion_time"=>1750608768923, "message"=>"{"level":"System", "msg":"InnoDB initialization has ended."}", "timestamp"=>1750608747624}]
       ...
       ```
  </TabItem>

  <TabItem value="add-logs">
    <TabItemLabel>
      Adding logs
    </TabItemLabel>

    1. Install the Fluent Bit agent. Use the [Download and install Fluent Bit](https://docs.fluentbit.io/manual/installation/downloads) guide in the official Fluent Bit documentation.

    2. Open the `config/fluent-bit.yaml` configuration file in a text editor.

    3. Add the configuration for adding logs to the `fluent-bit.yaml` file. Configuration example:

       ```yaml
       service:
         log_level: debug

       pipeline:
         inputs:
           - name: tail
             path: /tmp/fluentbit.log

         outputs:
           - name: stdout
             match: '*'

           - name: cloudwatch_logs
             match: '*'
             region: <pool>
             endpoint: <log_endpoint>
             log_group_name: <log_group_name>
             log_stream_name: <log_stream_names>
             auto_create_group: true
       ```

       Specify:

       * `<pool>` — [pool](/infrastructure/locations.mdx#pool), for example  `ru-9`;
       * `<log_endpoint>` — URL for accessing the Logs service API. The list of URLs can be found in the [List of URLs](/api/urls/);
       * `<log_group_name>` — the name of the log group to which you want to add events, for example `user-log-group`;
       * `<log_stream_name>` — the name of the stream to which you want to add events, for example `user-log-stream`;

    4. Add the values of the **Access Key** and **Secret Key** fields from the [S3 key](#issue-s3-key) using one of the following methods:

       * via environment variables. More details can be found in the [Using environment variables to globally configure AWS SDKs and tools](https://docs.aws.amazon.com/sdkref/latest/guide/environment-variables.html) guide in the Amazon documentation;
       * or via `config` and `credentials` files. More details can be found in the [Using environment variables to globally configure AWS SDKs and tools](https://docs.aws.amazon.com/sdkref/latest/guide/file-format.html) guide in the Amazon documentation

    5. Run Fluent Bit:

       ```bash
       sudo /opt/fluent-bit/bin/fluent-bit -c /etc/fluent-bit/fluent-bit.conf -vv
       ```
  </TabItem>
</Tabs>

<Formbricks />