Migrate data from ClickHouse® to a ClickHouse® Managed Database using Trino
You can migrate tables from one or more ClickHouse® databases to a ClickHouse® Managed Database using Cloud for Trino. This method is suitable if you need to migrate individual tables, filter, or transform data during migration.
The server with the source database and the ClickHouse® cluster with the target database connect to Cloud for Trino as two catalogs using the ClickHouse connector. Each database is displayed in the catalog as a schema. Trino reads data from the source database and writes it to the target database in a single SQL query, without intermediate files or dumps.
When migrating data using Trino, limitations apply.
Limitations
Migrate data
- Prepare the server with the source ClickHouse® database.
- Prepare the target ClickHouse® cluster.
- Create a Trino cluster with catalogs.
- Connect to the Trino cluster.
- Check the catalogs.
- Migrate data using Trino.
- Verify the result.
- Switch services.
1. Prepare the server with the source ClickHouse® database
To protect data from modification in the source database, Trino will connect to it as a dedicated read-only user.
-
Connect to the source ClickHouse® database as a user with user management privileges.
-
Create a new user and grant them read-only permissions using the
SELECTprivilege:CREATE USER <user_name> IDENTIFIED WITH sha256_password BY '<password>';GRANT SELECT ON <source_database>.* TO <user_name>;Specify:
<user_name>— username;<password>— user password;<source_database>— name of the database from which to migrate data.
If you need to migrate data from multiple databases, grant the
SELECTprivilege on each database. -
Make sure that the server is accessible from the internet via a public IP address or DNS name.
-
Make sure that the ClickHouse® HTTP interface is enabled on the server: the
https_portparameter (usually port 8443) is set in the configuration and a TLS certificate is configured. Trino connects to ClickHouse® via the HTTP interface. -
If traffic filtering is used on the server, allow incoming connections to the ClickHouse® HTTPS port in the firewall or cloud provider security groups.
2. Prepare the target ClickHouse® cluster
Cloud for Trino is available only in the ru-7 pool. If you want to migrate data from Trino to ClickHouse® Managed Database via a private subnet, create the ClickHouse® cluster in the ru-7 pool and use the same subnet for both ClickHouse® and Trino clusters.
If the ClickHouse® and Trino clusters are located in different pools, Trino will only be able to connect to ClickHouse® over the internet via a public IP address.
-
Create a ClickHouse® cluster. When creating a cluster:
-
if you want to migrate data via a private subnet, select the location — ru-7 pool;
-
in the Node group configuration block, select a configuration with a disk size sufficient for the amount of data being migrated.
-
-
If you want to migrate data over the internet, assign a public IP address to the cluster shard. Public IP addresses are connected to all nodes of the shard simultaneously — a dedicated address for each node.
3. Create a Trino cluster with catalogs
Follow the Create Trino cluster instructions. When creating a cluster:
-
in the Network block:
-
if the target ClickHouse® cluster is created in the ru-7 pool, select the target cluster subnet;
-
to connect to Trino from the internet, select the External GUI access checkbox;
-
-
in the Catalogs block, add two catalogs with the ClickHouse connector type for the source and target databases. When adding catalogs, use the connection parameters:
4. Connect to the Trino cluster
You can connect to the cluster via the Trino GUI, the Trino CLI console client, and DBeaver.
Trino GUI
Trino CLI
DBeaver
-
In the Control panel, in the top menu, click Products and select Trino.
-
In the cluster card, in the GUI column, click .
-
Enter the login —
admin. -
Enter the administrator password — set when creating the cluster. After cluster creation, the password cannot be viewed in the Control panel, but it can be changed.
-
Click Log In.
5. Check the catalogs
List catalogs, schemas, and tables of the source database and schemas of the target database:
SHOW CATALOGS;
SHOW SCHEMAS FROM <source_catalog>;
SHOW TABLES FROM <source_catalog>.<source_database>;
SHOW SCHEMAS FROM <target_catalog>;
Specify:
<source_catalog>— catalog name of the server with the source database;<source_database>— name of the source database;<target_catalog>— catalog name of the target cluster.
Both created catalogs should appear in the catalog list, and the tables of the source database should appear in the <source_catalog>.<source_database> schema.
6. Migrate data using Trino
Trino creates ClickHouse® tables only with the Log and MergeTree engines and without some source table settings. Therefore, you first need to create tables in the target cluster with the same structure as the source, and then migrate data into them. We recommend executing standard SQL queries to the target cluster via the system.execute procedure — in this case, separate access to the cluster is not required.
-
Create a database in the target cluster:
CALL <target_catalog>.system.execute(query => 'CREATE DATABASE IF NOT EXISTS <target_database>');Specify:
<target_catalog>— catalog name of the target cluster;<target_database>— name of the target database.
-
Get the definition with the structure of the source table:
SHOW CREATE TABLE <source_database>.<source_table>;Specify:
<source_database>— name of the source database;<source_table>— name of the source table.
Copy the query result.
-
Create a table in the target cluster:
CALL <target_catalog>.system.execute(query => '<create_table_query>');Specify:
<target_catalog>— catalog name of the target cluster;<create_table_query>— source table structure definition you obtained and copied in step 2.
If the table structure contains single quotes (for example, in
DEFAULTvalues or in theEnumtype), change them to paired single quotes:''. Remove settings from the structure that are not available in the target cluster, such asstorage_policy. If a shard in the target cluster contains multiple nodes, use the replicated engineReplicatedMergeTree. -
Optional: to speed up data migration, disable writing via a temporary table:
SET SESSION <target_catalog>.non_transactional_insert = true;Specify
<target_catalog>— catalog name of the target cluster. -
Migrate the data:
INSERT INTO <target_catalog>.<target_database>.<target_table>SELECT * FROM <source_catalog>.<source_database>.<source_table>;Specify:
<target_catalog>— catalog name of the target cluster;<target_database>— name of the target database;<target_table>— name of the target table;<source_catalog>— catalog name of the server with the source database;<source_database>— name of the source database;<source_table>— name of the source table.
The output will show the number of written rows.
-
If you disabled writing via a temporary table in step 4 and an error occurred while writing data to the target table, it will only contain part of the rows. To re-migrate all data:
6.1. Clear the target table:
TRUNCATE TABLE <target_catalog>.<target_database>.<target_table>;Specify:
<target_catalog>— catalog name of the target cluster;<target_database>— name of the target database;<target_table>— name of the target table.
6.2. Repeat step 5.
-
Repeat steps 2–6 for each table.
7. Verify the result
-
Compare the number of rows in the source and target tables:
SELECT(SELECT count(*) FROM <source_catalog>.<source_database>.<source_table>) AS source_rows,(SELECT count(*) FROM <target_catalog>.<target_database>.<target_table>) AS target_rows;Specify:
<source_catalog>— catalog name of the server with the source database;<source_database>— name of the source database;<source_table>— name of the source table;<target_catalog>— catalog name of the target cluster;<target_database>— name of the target database;<target_table>— name of the target table.
-
Find dates where the number of rows does not match:
SELECT <column>, count(*) AS rows FROM <source_catalog>.<source_database>.<source_table> GROUP BY <column>EXCEPTSELECT <column>, count(*) AS rows FROM <target_catalog>.<target_database>.<target_table> GROUP BY <column>;Specify:
<column>— date column against which the number of rows is verified;<source_catalog>— catalog name of the server with the source database;<source_database>— name of the source database;<source_table>— name of the source table;<target_catalog>— catalog name of the target cluster;<target_database>— name of the target database;<target_table>— name of the target table.
An empty output means that the number of rows matches for all days.
-
List the columns of the target table:
SHOW COLUMNS FROM <target_catalog>.<target_database>.<target_table>;Specify:
<target_catalog>— catalog name of the target cluster;<target_database>— name of the target database;<target_table>— name of the target table.
-
Compare the target table columns obtained in step 3 with the source table structure obtained during data migration in step 2.
8. Switch services
-
Switch services to the target cluster — learn more in the Connect to ClickHouse® cluster instructions.
-
If you assigned a public IP address in the ClickHouse® cluster and no longer need it, disconnect it.
-
If you no longer need the Trino cluster, delete it.
-
If you no longer need the server with the source ClickHouse® database, delete it.
ClickHouse® is a registered trademark of ClickHouse®, Inc. https://clickhouse.com.