Skip to main content

Migration of cloud storage from Servers.ru to S3

Last update:

You can migrate your data from Servers.ru (Swift) cloud storage to S3 from Selectel by yourself.

You can use different S3 tools for migration, for example:

  • Rclone (recommended method) - supports direct data transfer without intermediate copying, as well as segmented download at high data transfer rates. Suitable for transferring large amounts of data;
  • Cyberduck graphical client - does not support copying and direct transfer of data from Servers.ru to S3. You will need to download data from Servers.ru storage and upload to S3 manually;
  • FTP protocol - does not support copying and direct transfer of data from Servers.ru to S3. You will need to download data from Servers.ru storage and upload to S3 manually. Works only with S3 bucket pools ru-1 and ru-7.

If difficulties arise during the migration process:

Perform migration

  1. Prepare S3.
  2. Configure access to S3.
  3. Get data to connect to Servers.ru.
  4. Customize the tool and transfer the data.
  5. Check the results of the migration.

1. Prepare S3

  1. If you do not already have an account in the my.selectel.ru control panel:

    1.1 Register an account in the control panel.

    1.2 Refill your balance.

  2. Create an S3 bucket in the desired bullet. We recommend creating the bucket in pool ru-7.

2. Configure access to S3

Access can be configured by the Account Owner or a user with the role of iam.admin.

  1. Create a service user with role with access to S3. If you use a service user with the role s3.user, object_storage_user or s3.bucket.user, the bucket must have a configured access policy and its rules must allow access to this user.
  2. Issue an S3 key to the user.

3. Get data for connection to Servers.ru

  1. In the Servers.ru control panel, open the Cloud Storage tab.

  2. In the Access Details block, copy and save the data:

    • login and password for access via Swift API to Servers.ru;

    • domain for Keystone v3 (Swift) in the format https://auth.files.<region>.cloud.servers.ru:5000/v3/, where you can find.files.<region>.cloud.Servers.<domain_zone> is the address of the server where:

      • <region> - region where the Servers.ru storage is located;
      • <domain_zone> - domain zone: ru or com.

    • if you will use FTP, FTP host in the format ftp.files.<region>.cloud.servers.ru, where <region> is the region where the Servers.ru repository is located.

4. Customize the tool and transfer the data

Learn more about working with Rclone in the Rclone manual.

  1. Install Rclone.

  2. Create a configuration to connect to Servers.ru:

    2.1 Open the configuration mode:

    rclone config

    A dialog box will appear in the console:

    No remotes found, make a new one?
    n) New remote
    s) Set configuration password
    q) Quit config

    2.2 Create a new configuration:

    n/s/q> n

    2.3. Enter the name of the remote storage connection, for example serversru:

    name> serversru

    2.4. enter the OpenStack Swift storage type:

    Storage> swift

    2.5. Select the method for obtaining credentials that is listed after the Enter AWS credentials in the next step line:

    env_auth> false

    2.6 Enter login:

    user> <login>

    Specify <login> - the login you saved when receiving data to connect to Servers.ru in step 3.

    2.7 Enter the password:

    key> <password>

    Specify <password> - the password you saved when receiving data to connect to Servers.ru in step 3.

    2.8 Enter the URL to access the Swift API:

    endpoint> <swift_domain>

    Specify <swift_domain> - enter the server address that you saved when receiving data to connect to Servers.ru in step 3.

    2.9. In the auth_version parameter, enter 3:

    auth_version> 3

    2.10. Leave the optional parameters blank.

    2.11. In the Option domain parameter, enter default.

    2.12. Leave the Edit advanced config parameter blank or enter n.

    2.13. Check the configuration. If the configuration is correct, press Enter. If not, enter e and edit the configuration.

    2.14. Confirm the creation of the configuration:

    y/e/d> y

    2.15. Exit the settings dialog box:

    e/n/d/r/c/s/q> q

  3. Create a configuration to connect to S3.

  4. Optional: see the data to be copied without performing a copy:

    rclone copy serversru:<container_name> selectel:<bucket_name> --dry-run

    Specify:

    • <serversru_name> - name of connection to Servers.ru, which you specified in step 2.3;
    • <container_name> - name of the Servers.ru container;
    • <selectel_name> - the name of the connection to S3 that you specified when creating the configuration creating the configuration in step 4;
    • <bucket_name> - bucket name in S3.

  5. Transfer the data in one of the following ways:

    • copy all data from the Servers.ru container to the S3 bucket:

      rclone copy <serversru_name>:<container_name> <selectel_name>:<bucket_name> --progress

      Specify:

      • <serversru_name> - name of connection to Servers.ru, which you specified in step 2.3;
      • <container_name> - name of the Servers.ru container;
      • <selectel_name> - the name of the connection to S3 that you specified when creating the configuration creating the configuration in step 4;
      • <bucket_name> - bucket name in S3;
      • optional: --progress - option that enables displaying copy progress.

    • or synchronize data - in S3 the data that is not in Servers.ru will be deleted:

      rclone sync <serversru_name>:<container_name> <selectel_name>:<bucket_name> --progress

5. Check the migration results

It is necessary to check the migration results before deleting data from the Servers.ru storage.

  1. Verify that all data appears in the S3 binnacle.

  2. Update the settings of applications that use storage data:

    2.1 Specify S3 domains instead of Servers.ru domains.

    2.2 Specify the S3 key you issued when setting up access to S3 in step 2 instead of the data for accessing Servers.ru.

    2.3 If the name of the bucket in S3 differs from the name of the container that was in the Servers.ru repository, specify the name of the S3 bucket.

  3. Check how the apps work after updating the settings.