Skip to main content

Migrating to S3

You can migrate data to S3 storage from another provider's storage (for example, AWS S3, Azure, GCP) using Rclone.

  1. Prepare the site for migration.
  2. Migrate data from the source storage to Selectel S3 (destination).

1. Prepare the site for migration

  1. Create a bucket in S3.
  2. Connect to the storage using Rclone.
  3. Create two new configurations via the S3 API — for the target and source storage. Use the parameters of the current provider for the source storage.

2. Data migration

  1. Perform the copying of objects from the source storage to the target:

    rclone copy -v -P --log-file=rclone.log --transfers=<transfers> <source_provider>:<bucket_name> <destination_provider>:<bucket_name>

    Specify:

    • <transfers> — number of parallel copy streams. The recommended parameter value is 10. If a large number of timeout errors occur, reduce the number of streams;
    • <source_provider> — source storage configuration name;
    • <destination_provider> — destination storage configuration name;
    • <bucket_name> — bucket name in the destination storage.

  2. Run the synchronization. Only changed or new objects from the source storage will be copied:

    rclone sync -v -P --log-file=rclone.log --transfers=<transfers> <source_provider>:<bucket_name> <destination_provider>:<bucket_name>

  3. Stop the services that use the source storage. This is necessary for data consistency.

  4. Disable data modification in the source storage; leave read-only access only.

  5. Synchronize the storages (previously copied objects will be skipped):

    rclone sync -v -P --log-file=rclone.log --transfers=<transfers> <source_provider>:<bucket_name> <destination_provider>:<bucket_name>

  6. When the synchronization is complete, switch the services to S3.

  7. Optional: delete the data from the source storage.