Skip to main content

Fluent Bit

Fluent Bit is an agent that allows you to configure the process of collecting, processing, and exporting logs, as well as importing logs from third-party storage. Fluent Bit is extensible with output and input plugins. You can view the list of plugins on the Fluent Bit website (Outputs, Inputs).

With Fluent Bit, you can manage logs in the Logs service: receive logs or add logs from your own storage.

Before you begin, configure Fluent Bit.

Configure Fluent Bit

  1. Add a service user.
  2. Issue an S3 key to the user.
  3. Configure the agent to receive or add events.

1. Add a service user

Add a service user with access to the Projects area and the following role:

Users can be added by the Account Owner or users with the iam.admin role.

2. Issue an S3 key to the user

Control panel users can generate their own S3 keys, but we recommend creating service users and issuing S3 keys to them.

Only the Account Owner or a user with the iam.admin role can issue S3 keys to other users. A Service User cannot obtain an S3 key independently as they do not have access to the control panel — the key must be issued by the Account Owner or iam.admin.

You must create a separate key for each project. You can issue multiple keys for a single project.

  1. In the control panel, click IAM.

  2. Go to the section for the required user type:

    • Control panel users — for users with access to the control panel;
    • Service users — for users with programmatic access who do not have access to the control panel.

  3. Open the user page → the Access tab.

  4. In the S3 keys block, click Add key.

  5. Enter a key name.

  6. Select the project for which the key will work.

  7. Click Generate. Two values will be generated:

    • Access key — Access Key ID, a key identifier;
    • Secret key — Secret Access Key, a secret key.

  8. Click Copy and save the key — you will not be able to view it after closing the window.

3. Configure the agent

Using the built-in plugin fluent-bit-cloudwatch-input-plugin, you can configure exporting events from the Logs service to your infrastructure, for example to Elasticsearch, Loki, or Kafka.

  1. Download the configuration file config/fluent-bit.yaml.

  2. Open the configuration file config/fluent-bit.yaml in a text editor.

  3. In the configuration file parameters, specify:

    • region — pool, for example ru-9;
    • endpoint — URL for accessing the Logs API. The list of URLs can be found in the Logs subsection of the List of URLs;
    • log_group_name — the log group name, for example s/lbaas/Chromie-lb;
    • log_stream_name — the event stream name (stream), for example http-21380357;
    • sqlite_path — the path to the SQLite database that stores information about the last received events. One database can be used for several groups and streams.

    The list of events from the GetLogEvents Amazon CloudWatch API method is split into separate records using a Lua filter. Each received event is converted into a separate, independent Fluent Bit record (record). This allows you to process events individually, tag them, and send them to different output plugins. The configuration file uses the stdout output plugin, but you can use any other, for example Elasticsearch or Kafka.

  4. Optional: to configure data collection from multiple groups and streams within a single configuration file, in the pipeline block add the corresponding inputs.

  5. Run Fluent Bit:

    docker run \
      --name fluent-bit-cloudwatch \
      --rm \
      -v ${PWD}/config/fluent-bit.yaml:/fluent-bit/etc/fluent-bit.yaml:ro \
      -v ${PWD}/sqlite:/var/lib/fluent-bit/cloudwatch/sqlite:rw \
      -e AWS_ACCESS_KEY_ID=<access_key> \
      -e AWS_SECRET_ACCESS_KEY=<secret_key> \
      ghcr.io/selectel/fluent-bit-cloudwatch-input-plugin:latest

    Specify:

    • <access_key> — the value of the Access key field from the S3 key that you issued to the user;
    • <secret_key> — the value of the Secret key field from the S3 key that you issued to the user.

    You can add <access_key> and <secret_key> to the container in any way:

    Example of output in stdout in MessagePack format:

    ...
    [1029] cloudwatch-input.0: [[1751577177.000000000, {}], {"ingestion_time"=>1750606362344, "message"=>"{"level":"System", "msg":"InnoDB initialization has started."}", "timestamp"=>1750606335369}]
    [1031] cloudwatch-input.0: [[1751577177.000000000, {}], {"ingestion_time"=>1750608768923, "message"=>"{"level":"System", "msg":"InnoDB initialization has ended."}", "timestamp"=>1750608747624}]
    ...