Skip to main content

Fluent Bit

Last update:

Fluent Bit - agent that allows you to customize log collection, processing and uploading, as well as import logs from third-party storages. Fluent Bit is extensible with output and input plugins. A list of plugins can be found on the Fluent Bit website (Outputs, Inputs).

With Fluent Bit you can work with logs in the Logs service: get logs or add logs from your own storage.

Set up Fluent Bit before you start working.

Customize Fluent Bit

  1. Add a service user.
  2. Issue an S3 key to the user.
  3. Configure the agent to receive or add events.

1. Add a service user

Add a service user with permission in the access area Projects and role:

Users can be added by the Account Owner or users with the role of iam_admin.

2. Issue an S3 key to the user

Users with access to the control panel can issue themselves S3 keys but we recommend to create service users and issue S3 keys to them.

S3 keys can only be issued to other users by the Account Owner or a user with the role iam_admin. A service user cannot get an S3 key by himself, because he does not have access to the control panel - he must be issued a key by the Account Owner or iam_admin.

A separate key must be created for each project. Multiple keys can be issued for one project.

  1. In the control panel, on the top menu, click Account.

  2. Go to the section with the desired user type:

  3. Open the user page → Access tab.

  4. In the S3 keys block, click Add Key.

  5. Enter the name of the key.

  6. Select the project for which the key will work.

  7. Click Generate. Two values will be generated:

    • Access key - Access Key ID, key identifier;
    • Secret key - Secret Access Key, secret key.

  8. Click Copy and save the key - it cannot be viewed after the window is closed.

3. Customize the agent

With the built-in fluent-bit-cloudwatch-input-plugin, you can configure to offload events from the Logi service to your infrastructure, such as Elasticsearch, Loki, or Kafka.

  1. Download the configuration file config/fluent-bit.yaml.

  2. Open the config/fluent-bit.yaml configuration file in a text editor.

  3. In the parameters of the configuration file, specify:

    • region - pool for example ru-9;
    • endpoint - URL to access the logging API. The list of URLs can be found in the instructions List of URLs;
    • log_group_name - log group name, e.g. s/lbaas/Chromie-lb;
    • log_stream_name - name of the event stream, e.g. http-21380357;
    • sqlite_path - address of the SQLite database that stores information about the last received events. One database can be used for several groups and streams.

    The list of events from the GetLogEvents method of Amazon CloudWatch API is split into separate records using a Lua filter. Each received event is converted into a separate independent Fluent Bit record. This allows the events to be processed separately, tagged and sent to different output plugins. The configuration file uses the stdout output plugin, but you can use any plugin, such as Elasticsearch or Kafka.

  4. Optional: to customize data collection from multiple groups and streams within a single configuration file, add the appropriate inputs blocks in the pipeline block.

  5. Start Fluent Bit:

    docker run \
      --name fluent-bit-cloudwatch \
      --rm \
      -v ${PWD}/config/fluent-bit.yaml:/fluent-bit/etc/fluent-bit.yaml:ro \
      -v ${PWD}/sqlite:/var/lib/fluent-bit/cloudwatch/sqlite:rw \
      -e AWS_ACCESS_KEY_ID=<access_key> \
      -e AWS_SECRET_ACCESS_KEY=<secret_key> \
      ghcr.io/selectel/fluent-bit-cloudwatch-input-plugin:latest

    Specify:

    You can add <access_key> and <secret_key> to the container in any way you want:

    Example of output to stdout in MessagePack format:

    ...
    [1029] cloudwatch-input.0: [[1751577177.000000000, {}], {"ingestion_time"=>1750606362344, "message"=>"{"level":"System", "msg":"InnoDB initialization has started."}", "timestamp"=>1750606335369}]
    [1031] cloudwatch-input.0: [[1751577177.000000000, {}], {"ingestion_time"=>1750608768923, "message"=>"{"level":"System", "msg":"InnoDB initialization has ended."}", "timestamp"=>1750608747624}]
    ...