Skip to main content

Customize Fluent Bit

Last update:

Fluent Bit — is an application that allows you to customize the process of collecting, processing and uploading logs.Fluent Bit can be extended with output and input plugins.A list of plugins can be found on the Fluent Bit website (Outputs, Inputs).

With the built-in fluent-bit-cloudwatch-input-plugin, you can configure offloading events from the logging platform to your infrastructure, such as Elasticsearch, Loki, or Kafka.Sending events to the logging platform is currently not available.

  1. Create a service user.
  2. Issue an S3 key to the user.
  3. Configure the receipt of events.

1. Create a service user⁠

Create a service user with the role of Project Administrator or Project Supervisor. role Account Owner or User Administrator.

2. Issue an S3 key to the user⁠

Control panel users can issue their own S3 keys on their own, but we recommend to create service users and use keys together with them.

Only the Account Owner or User Administrator can issue S3 keys to other users.A service user cannot get an S3 key by himself because he does not have access to the control panel — he must be issued a key by the Account Owner or User Administrator.

A separate key must be created for each project.Multiple keys can be issued for one project.

  1. In the control panel, on the top menu, click Account.

  2. Go to the section with the desired user type:

    • Users — for the users of the control panel;
    • Service users — For service users.

  3. Open the user page → Access tab.

  4. In the S3 keys block, click Add Key.

  5. Enter the name of the key.

  6. Select the project for which the key will work.

  7. Click Generate. Two values will be generated:

    • Access key — Access Key ID, key identifier;
    • Secret key — Secret Access Key, secret key.

  8. Click Copy and save the key — it cannot be viewed after the window is closed.

3. Configure the receipt of events⁠

  1. Copy the configuration file config/fluent-bit.yaml. In the file, specify:

    • region — pool for example ru-1;
    • endpoint — the address of the logging platform in the selected pool, e.g. https://ru-1.logs.selcloud.ru;
    • log_group_name — name of the log group, e.g. s/lbaas/Chromie-lb;
    • log_stream_name — event stream name (stream), e.g. http-21380357;
    • sqlite_path — address of the SQLite database that stores information about the last received events. One database can be used for several groups and threads.

    The Lua filter is used to split the list of events from the GetLogEvents method of Amazon CloudWatch API into separate records.This will convert each event into a separate independent Fluent Bit record, allowing them to be processed separately, tagged and sent to different output plugins.The configuration file uses the stdout output plugin, but you can use any, such as Elasticsearch or Kafka.

  2. Optional: to customize data collection from multiple groups and streams within one configuration file, add the corresponding inputs blocks in the pipeline block.

  3. Start Fluent Bit:

    docker run \
      --name fluent-bit-cloudwatch \
      --rm \
      -v ${PWD}/config/fluent-bit.yaml:/fluent-bit/etc/fluent-bit.yaml:ro \
      -v ${PWD}/sqlite:/var/lib/fluent-bit/cloudwatch/sqlite:rw \
      -e AWS_ACCESS_KEY_ID=<access_key> \
      -e AWS_SECRET_ACCESS_KEY=<secret_key> \
      ghcr.io/selectel/fluent-bit-cloudwatch-input-plugin:latest

    Specify:

    • <access_key> — field value Access key from S3 key;
    • <secret_key> — field value Secret key from S3 key.

    You can add <access_key> and <secret_key> to the container in any way you want:

    Example of output to stdout in MessagePack format:

    ...
    [1029] cloudwatch-input.0: [[1751577177.000000000, {}], {"ingestion_time"=>1750606362344, "message"=>"{"level":"System", "msg":"InnoDB initialization has started."}", "timestamp"=>1750606335369}]
    [1031] cloudwatch-input.0: [[1751577177.000000000, {}], {"ingestion_time"=>1750608768923, "message"=>"{"level":"System", "msg":"InnoDB initialization has ended."}", "timestamp"=>1750608747624}]
    ...