1. Project Overview

A heterogeneous database migration and synchronization (data transfer) tool.

A tool for synchronizing heterogeneous database data and structures.

The dbswitch tool provides migration and synchronization from source databases to target databases inbatches,supporting both full and incremental data synchronization.

2. Open Source License

Released under the BSD open-source license.

3. Screenshots

• The WEB system workflow is as follows:

Establish a connection to the source database -> Establish a connection to the target database -> Configure tasks -> Publish tasks -> Execute tasks manually or through scheduled system jobs -> View execution records -> View data results in the data directory

• Some screenshots of the WEB system:

4. Feature Overview

Feature Description

In a nutshell, the dbswitch tool providesbatches,migration and synchronization from source databases to target databases, supporting both full and incremental data synchronization. It includes:

• Schema Migration

Supports conversion of field types, primary key information, and CREATE TABLE statements, and generates CREATE TABLE SQL statements.

Supports table name and field name mapping and conversion based on regular expressions.

• Data Synchronization。

Reads data from the source database in batches using JDBC and writes data to the target database in batches using INSERT/COPY methods.

Supports tables with primary keys for Incremental Change Synchronization (Change Data Calculate) functionality (Performance with datasets exceeding tens of millions of records still needs to be validated in a production environment).

Functional Design

5. Tech Stack

Architecture Design

• Module Structure and Functions
• 
  

└── dbswitch
    ├── dbswitch-common               // dbswitch common definitions module
    ├── dbswitch-core                 // dbswitch migration and synchronization implementation classes
    ├── dbswitch-product              // dbswitch database dialects
        ├── dbswitch-product-mysql    //  -> mysql dialect implementation
        ├── dbswitch-product-oracle   //  -> oracle dialect implementation
        ├── dbswitch-product-sqlserver//  -> sqlserver dialect implementation
        ├── dbswitch-product-postgres //  -> postgres dialect implementation
        ├── dbswitch-product-dm       //  -> dm dialect implementation
        ├── dbswitch-product-kingbase //  -> kingbase dialect implementation
        ├── dbswitch-product-oscar    //  -> oscar dialect implementation
        ├── dbswitch-product-gbase    //  -> gbase dialect implementation
        ├── dbswitch-product-mariadb  //  -> mariadb dialect implementation
        ├── dbswitch-product-openguass//  -> openguass dialect implementation
        ├── dbswitch-product-db2      //  -> db2 dialect implementation
        ├── dbswitch-product-sybase   //  -> sybase dialect implementation
        ├── dbswitch-product-hive     //  -> hive dialect implementation
        ├── dbswitch-product-sqlite   //  -> sqlite dialect implementation
        ├── dbswitch-product-mongodb  //  -> mongodb dialect implementation
    ├── dbswitch-data                 // Tool entry module that reads parameters from configuration files to execute heterogeneous migration and synchronization
    ├── dbswitch-admin                // Introduces Quartz scheduling services and APIs based on the modules above
    ├── dbswitch-admin-ui             // Vue2-based frontend WEB interface
    ├── dbswitch-dist                 // Project packaging module based on the maven-assembly-plugin

Build and Package

This tool is developed entirely in Java, and all dependencies in the code come from open-source projects.

1. Build and Package

• Environment Requirements:
• JDK:>=1.8 (JDK 1.8 is recommended)
• maven:>=3.6

Maven repositories are hosted overseas by default, which can make downloads slow in China. You can switch to the Alibaba Cloud repository instead. See the tutorial: Configuring the Alibaba Cloud Repository

For more information, please refer to the README.md documentation.

2. Installation and Deployment

(1) After the build and packaging command completes, a packaged file named dbswitch-relase-x.x.x.tar.gz will be generated in the dbswitch/target/ directory. Copy this file to a deployment machine with JRE installed and extract it.

(2) Deployment using Docker containers:

Assume the deployed MySQL database has the following connection details: IP address 192.168.31.57, port 3306, username test, and password 123456

docker run -d --name dbswitch \
 -e MYSQLDB_HOST=192.168.31.57 \
 -e MYSQLDB_PORT=3306 \
 -e MYSQLDB_USERNAME=test \
 -e MYSQLDB_PASSWORD='123456' \
 -v /tmp:/tmp \
 -p 9088:9088 \
 registry.cn-hangzhou.aliyuncs.com/inrgihc/dbswitch:latest

(3) Provides a Linux network-enabled environment based on docker-composeOne-Click Installation, see the distribution release link for installation commands

For detailed documentation, see: build-docker/install/README.md

Tool Usage

The dbswitch tool provides two ways to run and use its modules: the dbswitch-data module, which is launched using the conf/config.yml configuration file, and the WEB interface of the dbswitch-admin module, which is accessed using the conf/application.yml configuration file;

For more information, please refer to the README.md documentation.