Connect S3 Selectel to ClearML

S3 Selectel can be connected to ClearML to store datasets, results, and artifacts of experiments.

Open the ClearML clearml.conf configuration file. Change the api, aws and development blocks:

api {
  ...
  files_server: s3://s3.<pool>.storage.selcloud.ru:443/<container_name>
  ...
}

...
sdk {
...
    aws {
      s3 {
        host: "s3.storage.selcloud.ru:443"
        region: "ru-1"
        key: "<access_key>"
        secret: "<secret_key>"
        use_credentials_chain: false
        credentials: [{
            bucket: "<container_name>"
            secure: true
        }]
      }
      boto3 {
        pool_connections: 512
        max_multipart_concurrency: 16
      }
    }
...
}
...

...
development {
  ...
  default_output_uri: "s3://s3.<pool>.storage.selcloud.ru:443/<container_name>/<path>"
  ...
}
...

Specify:

<container_name> - name of the container in S3 where datasets and artifacts will be stored. The name can be viewed in control panel: in the top menu, click Products → S3 → Buckets;
<access_key> - Access Key ID from S3 key issued to the user;
<secret_key> - Secret Access Key from S3 key issued to the user.
<path> - prefix in S3;
<pool> - pool in which S3 is located (e.g, ru-1).

To load the datasets into ClearML Server, run the python script.

An example script to load a single dataset:

# Create dataset via Dataset class
from clearml import Dataset
dataset = Dataset.create(
    dataset_name="<dataset_name>",
    dataset_project="<project_name>",
    output_uri="s3://s3.storage.selcloud.ru:443/<container_name>/<path>",
)

# Add files to dataset
dataset.add_files(
    path="<local_path_to_dataset>",
)

# Upload dataset to ClearML Server
dataset.upload()

# Commit dataset changes
dataset.finalize()

Specify:

<dataset_name> - dataset name, will be displayed in WebApp;
<project_name> - project name, will be displayed in the WebApp;
<container_name>/<path> - prefix in S3 from step 1;
<local_path_to_dataset> - path to the dataset on the local machine.