---
title: "Connect S3 to ClearML"
sidebar_position: 2
description: "How to connect Selectel S3 to ClearML for storing datasets, results, and experiment artifacts"
---

import Formbricks from '@theme/MDXComponents/Formbricks'

# Connect Selectel S3 to ClearML

In ClearML, you can connect [Selectel S3](/s3/) to store datasets, results, and experiment artifacts.

1. Open the ClearML configuration file `clearml.conf`. Update the `api`, `aws` and `development:` blocks:

   ```yaml
   api {
     ...
     files_server: s3://s3.<pool>.storage.selcloud.ru:443/<container_name>
     ...
   }
   ```

   ```yaml
   ...
   sdk {
   ...
       aws {
         s3 {
           host: "s3.storage.selcloud.ru:443"
           region: "ru-1"
           key: "<access_key>"
           secret: "<secret_key>"
           use_credentials_chain: false
           credentials: [{
               bucket: "<container_name>"
               secure: true
           }]
         }
         boto3 {
           pool_connections: 512
           max_multipart_concurrency: 16
         }
       }
   ...
   }
   ...
   ```

   ```yaml
   ...
   development {
     ...
     default_output_uri: "s3://s3.<pool>.storage.selcloud.ru:443/<container_name>/<path>"
     ...
   }
   ...
   ```

   Specify:

   * `<container_name>` — the name of the S3 bucket where datasets and artifacts will be stored. You can find the name in the [control panel](https://my.selectel.ru/storage/): in the top menu, click **Products** → **S3** → **Buckets**;
   * `<access_key>` — Access Key ID from the [S3 key](/access-control/manage/edit-user-data-or-role.mdx#issue-s3-key) issued to the user;
   * `<secret_key>` — Secret Access Key from the [S3 key](/access-control/manage/edit-user-data-or-role.mdx#issue-s3-key) issued to the user;
   * `<path>` — the prefix in S3;
   * `<pool>` — the [pool](/infrastructure/locations.mdx#pool) where S3 is located (for example, `ru-1`).

2. To upload datasets to ClearML Server, run a python script.

   Sample script for loading a single dataset:

   ```python
   # Create dataset via Dataset class
   from clearml import Dataset
   dataset = Dataset.create(
       dataset_name="<dataset_name>",
       dataset_project="<project_name>",
       output_uri="s3://s3.storage.selcloud.ru:443/<container_name>/<path>",
   )

   # Add files to dataset
   dataset.add_files(
       path="<local_path_to_dataset>",
   )

   # Upload dataset to ClearML Server
   dataset.upload()

   # Commit dataset changes
   dataset.finalize()
   ```

   Specify:

   * `<dataset_name>` — the name of the dataset, will be displayed in the [WebApp](https://clear.ml/docs/latest/docs/webapp/webapp_home);
   * `<project_name>` — the name of the project, will be displayed in the [WebApp](https://clear.ml/docs/latest/docs/webapp/webapp_home);
   * `<container_name>/<path>` — the S3 prefix from step 1;
   * `<local_path_to_dataset>` — the path to the dataset on the local machine.

<Formbricks />