Connect Selectel S3 to ClearML

In ClearML, you can connect Selectel S3 to store datasets, results, and experiment artifacts.

Open the ClearML configuration file clearml.conf. Edit the api, aws, and development blocks:

api {
  ...
  files_server: s3://s3.<pool>.storage.selcloud.ru:443/<container_name>
  ...
}

...
sdk {
...
    aws {
      s3 {
        host: "s3.storage.selcloud.ru:443"
        region: "ru-1"
        key: "<access_key>"
        secret: "<secret_key>"
        use_credentials_chain: false
        credentials: [{
            bucket: "<container_name>"
            secure: true
        }]
      }
      boto3 {
        pool_connections: 512
        max_multipart_concurrency: 16
      }
    }
...
}
...

...
development {
  ...
  default_output_uri: "s3://s3.<pool>.storage.selcloud.ru:443/<container_name>/<path>"
  ...
}
...

Specify:

<container_name> — name of the S3 bucket where datasets and artifacts will be stored. You can find the name in the Control Panel: in the top menu, click Products → S3 → Buckets;
<access_key> — Access Key ID from the S3 key issued to the user;
<secret_key> — Secret Access Key from the S3 key issued to the user;
<path> — prefix in S3;
<pool> — pool where S3 is located (for example, ru-1).

To upload datasets to ClearML Server, run a python script.

Sample script for loading a single dataset:

# Create dataset via Dataset class
from clearml import Dataset
dataset = Dataset.create(
    dataset_name="<dataset_name>",
    dataset_project="<project_name>",
    output_uri="s3://s3.storage.selcloud.ru:443/<container_name>/<path>",
)

# Add files to the dataset
dataset.add_files(
    path="<local_path_to_dataset>",
)

# Upload the dataset to ClearML Server
dataset.upload()

# Commit changes in the dataset
dataset.finalize()

Specify:

<dataset_name> — dataset name, will be displayed in the WebApp;
<project_name> — project name, will be displayed in the WebApp;
<container_name>/<path> — S3 prefix from step 1;
<local_path_to_dataset> — path to the dataset on the local machine.