Skip to main content

Migration to S3

Last update:

You can migrate data to S3 storage from another provider's storage (e.g. AWS S3, Azure, GCP) via Rclone.

  1. Prepare the site for migration.
  2. Perform data migration from source storage to S3 Selectel (target storage).

1. Prepare the site for migration

  1. Create a bucket in S3.
  2. Connect to the repository via Rclone.
  3. Create two new configurations via S3 API — for target and source storage. For the source storage use the parameters of the current provider.

2. Data migration

  1. Perform copying of objects from the source storage to the target storage:

    rclone copy -v -P --log-file=rclone.log --transfers=<transfers> <source_provider>:<bucket_name> <destination_provider>:<bucket_name>

    Specify:

    • <transfers> — number of parallel copying threads. The recommended value of the parameter is 10. If a large number of timeout errors occur, reduce the number of threads;
    • <source_provider> — name of the source storage configuration;
    • <destination_provider> — name of the destination storage configuration;
    • <bucket_name> — name of the bucket in the target storage.

  2. Start synchronization. Only changed or new objects from the original storage will be copied:

    rclone sync -v -P --log-file=rclone.log --transfers=<transfers> <source_provider>:<bucket_name> <destination_provider>:<bucket_name>

  3. Stop those services that use the source storage. This is necessary for data consistency.

  4. Prohibit modification of data in the source storage, leave only the read function.

  5. Synchronize the repositories (previously copied objects will be skipped):

    rclone sync -v -P --log-file=rclone.log --transfers=<transfers> <source_provider>:<bucket_name> <destination_provider>:<bucket_name>

  6. When synchronization is complete, switch the operation of the services to S3.

  7. Optional: delete the data from the source storage.