Skip to main content

Fluent Bit

Last update:

Fluent Bit — an agent that allows you to configure the process of collecting, processing, and offloading logs, as well as importing logs from third-party storage. Fluent Bit is extensible using output and input plugins. You can view the list of plugins on the Fluent Bit website (Outputs, Inputs).

With Fluent Bit, you can manage logs in the Logs service: receive logs or add logs from your own storage.

Before you begin, configure Fluent Bit.

Configure Fluent Bit

  1. Add a service user.
  2. Issue an S3 key to the user.
  3. Configure the agent to receive or add events.

1. Add a service user

Add a service user with access to the Projects area and the following role:

Users can be added by the Account Owner or users with the iam.admin role.

2. Issue an S3 key to the user

Users with access to the control panel can issue S3 keys for themselves, but we recommend creating service users and issuing S3 keys to them.

Only the Account Owner or a user with the iam.admin role can issue S3 keys to other users. A service user cannot get an S3 key independently because they do not have access to the control panel — the Account Owner or iam.admin.

You must create a separate key for each project. You can issue multiple keys for one project.

  1. In the control panel, click IAM.

  2. Go to the section for the required user type:

    • Users — for users with access to the control panel;
    • Service users — for service users.

  3. Open the user page → in the Access tab.

  4. In the S3 keys block, click Add key.

  5. Enter a key name.

  6. Select a project for which the key will work.

  7. Click Generate. Two values will be generated:

    • Access key — Access Key ID, a key identifier;
    • Secret key — Secret Access Key, a secret key.

  8. Click Copy and save the key — you will not be able to view it after closing the window.

3. Configure the agent

Using the built-in plugin fluent-bit-cloudwatch-input-plugin, you can configure exporting events from the Logs service to your infrastructure, for example to Elasticsearch, Loki, or Kafka.

  1. Download the configuration file config/fluent-bit.yaml.

  2. Open the configuration file config/fluent-bit.yaml in a text editor.

  3. In the configuration file parameters, specify:

    • region — pool, for example ru-9;
    • endpoint — URL for accessing the Logs API. The list of URLs can be found in the URL list;
    • log_group_name — log group name, for example s/lbaas/Chromie-lb;
    • log_stream_name — event stream name (stream), for example http-21380357;
    • sqlite_path — path to the SQLite database that stores information about the last received events. One database can be used for several groups and streams.

    The list of events from the GetLogEvents Amazon CloudWatch API method is split into individual entries using a Lua filter. Each received event is converted into a separate independent Fluent Bit record. This allows you to process events individually, tag them, and send them to different output plugins. The configuration file uses the stdout output plugin, but you can use any other, for example, Elasticsearch or Kafka.

  4. Optional: to configure data collection from multiple groups and streams within one configuration file, in the pipeline block add the corresponding inputs blocks.

  5. Run Fluent Bit:

    docker run \
      --name fluent-bit-cloudwatch \
      --rm \
      -v ${PWD}/config/fluent-bit.yaml:/fluent-bit/etc/fluent-bit.yaml:ro \
      -v ${PWD}/sqlite:/var/lib/fluent-bit/cloudwatch/sqlite:rw \
      -e AWS_ACCESS_KEY_ID=<access_key> \
      -e AWS_SECRET_ACCESS_KEY=<secret_key> \
      ghcr.io/selectel/fluent-bit-cloudwatch-input-plugin:latest

    Specify:

    • <access_key> — value of the Access key field from the S3 key that you issued to the user;
    • <secret_key> — value of the Secret key field from the S3 key that you issued to the user.

    You can add <access_key> and <secret_key> to the container using any method:

    Example of output in stdout in MessagePack format:

    ...
    [1029] cloudwatch-input.0: [[1751577177.000000000, {}], {"ingestion_time"=>1750606362344, "message"=>"{"level":"System", "msg":"InnoDB initialization has started."}", "timestamp"=>1750606335369}]
    [1031] cloudwatch-input.0: [[1751577177.000000000, {}], {"ingestion_time"=>1750608768923, "message"=>"{"level":"System", "msg":"InnoDB initialization has ended."}", "timestamp"=>1750608747624}]
    ...