Skip to main content
Migration to object storage
Last update:

Migration to object storage

You can migrate data to object storage from another provider's storage (e.g., AWS S3, Azure, GCP) via Rclone.

  1. Prepare the site for migration.
  2. Migrate data from source storage to Selectel object storage (target storage).

Prepare the site for migration⁠

  1. Create a container in object storage.
  2. Connect to the vault through Rclone.
  3. Create two new configurations via S3 API — for target and source storage. For the source storage use the parameters of the current provider.

Data migration⁠

  1. Perform copying of objects from the source storage to the target storage:

    rclone copy -v -P --log-file=rclone.log --transfers=<transfers> <source_provider>:<container_name> <destination_provider>:<container_name>

    Specify:

    • <transfers> — number of parallel copying threads. The recommended value for this parameter is 10. If a large number of timeout errors occur, reduce the number of threads;
    • <source_provider> — name of the source storage configuration;
    • <destination_provider> — name of the target storage configuration;
    • <container_name> — the name of the container in the target storage.

  2. Start synchronization. Only changed or new objects from the original storage will be copied:

    rclone sync -v -P --log-file=rclone.log --transfers=<transfers> <source_provider>:<container_name> <destination_provider>:<container_name>

  3. Stop those services that use the source storage. This is necessary for data consistency.

  4. Prohibit modification of data in the source storage, leave only the read function.

  5. Synchronize the repositories (previously copied objects will be skipped):

    rclone sync -v -P --log-file=rclone.log --transfers=<transfers> <source_provider>:<container_name> <destination_provider>:<container_name>

  6. When synchronization is complete, switch the services to object storage.

  7. Optional: delete the data from the source storage.