Migrate data from PostgreSQL to Managed PostgreSQL using Trino
You can migrate tables from one or more PostgreSQL databases to a PostgreSQL Managed Database using Cloud for Trino. This method is suitable if you need to migrate individual tables, filter, or transform data during migration.
The server with the source database and the PostgreSQL cluster with the target database connect to Cloud for Trino as two catalogs using the PostgreSQL connector. A PostgreSQL catalog connects to one database: if you are migrating multiple databases, you need to add a pair of catalogs for each. Trino reads data from the source database and writes it to the target database in a single SQL query, without intermediate files and dumps.
When migrating data using Trino, limitations apply.
Limitations
Migrate data
- Prepare the server with the source PostgreSQL database.
- Prepare the target PostgreSQL cluster.
- Create a Trino cluster with catalogs.
- Connect to the Trino cluster.
- Check catalogs.
- Migrate data using Trino.
- Check the result.
- Switch services.
1. Prepare the server with the source PostgreSQL database
To protect data from modification in the source database, Trino will connect to it using a dedicated read-only user.
-
Connect to the source PostgreSQL database as a user with privileges to create roles.
-
Create a new user and grant read-only permissions using the
SELECTprivilege:CREATE USER <user_name> WITH PASSWORD '<password>';GRANT CONNECT ON DATABASE <source_database> TO <user_name>;GRANT USAGE ON SCHEMA public TO <user_name>;GRANT SELECT ON ALL TABLES IN SCHEMA public TO <user_name>;Specify:
<user_name>— username;<password>— user password;<source_database>— name of the database from which to migrate data.
If you are migrating multiple schemas, grant
USAGEandSELECTprivileges on each schema. -
Allow connections to the source database — add the following line to the
pg_hba.conffile:host <source_database> <user_name> 0.0.0.0/0 <authentication_method>Specify:
<source_database>— name of the database from which to migrate data;<user_name>— username;<authentication_method>— authentication method used on the server, usuallyscram-sha-256. If the server usesmd5, specify it instead ofscram-sha-256.
-
Make sure that PostgreSQL accepts connections on the external interface: in the
postgresql.conffile, the value of thelisten_addressesparameter must contain the public IP address of the server or*. Changes to thelisten_addressesparameter take effect only after restarting PostgreSQL. -
Apply the changes to the
pg_hba.conffile:SELECT pg_reload_conf(); -
Make sure that the server is accessible from the internet via a public IP address or DNS address.
-
If traffic filtering is used on the server, allow incoming connections to the PostgreSQL port (usually 5432) in the firewall or cloud provider security groups.
2. Prepare the target PostgreSQL cluster
Cloud for Trino is only available in pool ru-7. If you want to migrate data from Trino to a PostgreSQL Managed Database via a private subnet, create a PostgreSQL cluster in the ru-7 pool and use the same subnet for PostgreSQL and Trino clusters.
If the PostgreSQL and Trino clusters are in different pools, Trino will be able to connect to PostgreSQL only over the internet via a public floating IP address.
-
Create a PostgreSQL cluster. When creating the cluster:
-
if you want to migrate data via a private subnet, select location — pool ru-7;
-
select a PostgreSQL version not lower than the version of the source database;
-
select a node configuration with a disk size sufficient for the amount of data being migrated.
-
-
Grant user access to the created database as an owner. The database owner can create schemas and tables in it — this is required for Trino to write data.
-
If you want to migrate data over the internet, attach a public floating IP address to the cluster.
3. Create a Trino cluster with catalogs
A PostgreSQL catalog connects to one database: if you are migrating multiple databases, add a pair of catalogs for each.
Use the Create Trino cluster instructions. When creating the cluster:
-
in the Network block:
-
if the target PostgreSQL cluster is created in the ru-7 pool, select the target cluster subnet;
-
to connect to Trino from the internet, select the External access to GUI checkbox;
-
-
in the Catalogs block, add two catalogs with the PostgreSQL connector type for the source and target databases. When adding catalogs, use the connection parameters:
4. Connect to the Trino cluster
You can connect to the cluster via the Trino GUI, the Trino CLI console client, and DBeaver.
Trino GUI
Trino CLI
DBeaver
-
In the Control panel, in the top menu, click Products and select Trino.
-
In the cluster card, in the GUI column, click .
-
Enter the login —
admin. -
Enter the administrator password — set when creating the cluster. After creating the cluster, the password cannot be viewed in the control panel, but can be changed.
-
Click Log In.
5. Check catalogs
Display the list of catalogs, schemas, and tables of the source database, and schemas of the target database:
SHOW CATALOGS;
SHOW SCHEMAS FROM <source_catalog>;
SHOW TABLES FROM <source_catalog>.public;
SHOW SCHEMAS FROM <target_catalog>;
Specify:
<source_catalog>— catalog name of the server with the source database;<target_catalog>— catalog name of the target cluster.
The catalog list must contain both created catalogs, and the <source_catalog>.public schema must contain the tables of the source database.
6. Migrate data using Trino
-
Optional: to migrate array columns and columns with unsupported data types, configure data type mapping. By default, the PostgreSQL connector skips such columns — they will not be added to the target table:
SET SESSION <source_catalog>.array_mapping = 'AS_ARRAY';SET SESSION <source_catalog>.unsupported_type_handling = 'CONVERT_TO_VARCHAR';<source_catalog>— catalog name of the server with the source database.The
AS_ARRAYvalue migrates fixed-dimension arrays; for arrays of arbitrary dimensions, use theAS_JSONvalue. TheCONVERT_TO_VARCHARvalue migrates columns with unsupported types as text. Learn more in the Type mapping section of the Trino documentation. -
If you are migrating a schema other than public, create it in the target database:
CREATE SCHEMA IF NOT EXISTS <target_catalog>.<schema>;Specify:
<target_catalog>— catalog name of the target cluster;<schema>— schema name.
-
Migrate the table:
CREATE TABLE <target_catalog>.public.<target_table> ASSELECT * FROM <source_catalog>.public.<source_table>;Specify:
<target_catalog>— catalog name of the target cluster;<target_table>— target table name;<source_catalog>— catalog name of the server with the source database;<source_table>— source table name.
To migrate a subset of data or modify it during migration, add filters and expressions to the query — for example, a
WHEREcondition by date or thelower()function to convert column values to lowercase.The output will show the number of rows written.
-
Optional: load additional rows that appeared in the source database after migration:
INSERT INTO <target_catalog>.public.<target_table>SELECT * FROM <source_catalog>.public.<source_table>WHERE <column> > (SELECT coalesce(max(<column>), 0) FROM <target_catalog>.public.<target_table>);Specify:
<target_catalog>— catalog name of the target cluster;<target_table>— target table name;<source_catalog>— catalog name of the server with the source database;<source_table>— source table name;<column>— column with an incrementing row identifier.
The query is suitable for tables where rows are inserted only with an incrementing identifier. Row modifications and deletions in the source database cannot be migrated this way.
-
Repeat steps 1–4 for each table.
7. Check the result
-
Compare the number of rows in the source and target tables:
SELECT(SELECT count(*) FROM <source_catalog>.public.<source_table>) AS source_rows,(SELECT count(*) FROM <target_catalog>.public.<target_table>) AS target_rows;Specify:
<source_catalog>— catalog name of the server with the source database;<source_table>— source table name;<target_catalog>— catalog name of the target cluster;<target_table>— target table name.
-
Optional: find rows in the source table that are missing from the target table:
SELECT * FROM <source_catalog>.public.<source_table>EXCEPTSELECT * FROM <target_catalog>.public.<target_table>;Specify:
<source_catalog>— catalog name of the server with the source database;<source_table>— source table name;<target_catalog>— catalog name of the target cluster;<target_table>— target table name.
An empty output means that all rows have been migrated. On large tables, the query creates a significant load on both databases. Columns of type
jsoncannot be compared inEXCEPT— list the columns explicitly or limit yourself to comparing row counts. -
Display the list of columns of the target table:
SHOW COLUMNS FROM <target_catalog>.public.<target_table>;Specify:
<target_catalog>— catalog name of the target cluster;<target_table>— target table name.
-
Compare the target table columns obtained in step 3 with the source table structure — for example, with the output of the
\d <source_table>command inpsql, where<source_table>is the source table name. Columns with data types skipped by Trino are not visible in the catalog — compare them directly with the source table structure.
8. Switch services
-
Switch services to the target cluster — learn more in the Connect to PostgreSQL cluster instructions. For access to the cluster from the internet, attach a public floating IP address.
-
If you attached a public floating IP address to the target PostgreSQL cluster and no longer need it, detach it.
-
If you no longer need the Trino cluster, delete it.
-
If you no longer need the server with the source MySQL database, delete it.