1. Project Overview

DBSyncer is an open-source data synchronization middleware that supports synchronization scenarios involving MySQL, Oracle, SqlServer, PostgreSQL, Elasticsearch (ES), Kafka, File, SQL, and more. It supports uploading custom plugins to define synchronization and transformation logic, and provides monitoring dashboards for full and incremental data statistics, as well as application performance alerts.

Features

• Combined Drivers: Customize database-to-database synchronization combinations, including combinations between relational and non-relational databases, with flexible table synchronization and mapping relationships.
• Real-Time Monitoring: Monitor the real-time synchronization status, results, synchronization logs, and system logs of full or incremental data synchronization.
• Plugin Development: Customize data transformation and synchronization logic.

2. Open Source License

Licensed under Apache-2.0

3. Screenshots

✨ Preview

Driver management

Driver Details

Driver Table Field Mapping Configuration

Monitoring

Upload Plugin

4. Feature Overview

Use Cases

Connector

Data Source

Target Data Source

Supported Versions (including the following)

MySQL

✔

✔

5.7.19 and later

Oracle

✔

✔

10gR2 -11g

SqlServer

✔

✔

2008 and later

PostgreSQL

✔

✔

9.5.25 and later

ES

✔

✔

6.0 and later

Kafka

In Development

✔

2.10-0.9.0.0 and later

File

✔

✔

*.txt, *.unl

SQL

✔

Supports the relational databases listed above.

Future Plans

Redis

5. Tech Stack

Design

Architecture Diagram

Development Dependencies

• JDK - 1.8.0_40 (or a recommended newer version)
• Maven - 3.3.9 (or a recommended newer version)

Installation and Configuration

Steps

1. Install JDK 1.8 (detailed instructions omitted).
2. Download the installation package DBSyncer-1.0.0-RC.zip (you can also compile it manually).
3. Extract the installation package. On Windows, run bin/startup.bat. On Linux, run bin/startup.sh.
4. Open a browser and visit: [http://127.0.0.1:18686](http://127.0.0.1:18686)
5. Username and Password: admin/admin

Incremental Synchronization Configuration (Source Database)

MySQL

• Dump the Binlog binary logs. The Master synchronizes with the Slave, creates an IO thread to read data and write it to the relaylog, and captures incremental data through message subscriptions.

Modify the my.ini file and restart the service.

# Unique service ID
server_id=1
log-bin=mysql_bin
binlog-format=ROW
max_binlog_cache_size = 256M
max_binlog_size = 512M
# Databases to monitor for synchronization; separate multiple databases with English commas ","
replicate-do-db=test

Prepare an account for data synchronization (here, I created an account named ae86 for the test database and granted it the necessary permissions).

CREATE USER 'ae86'@'%' IDENTIFIED WITH mysql_native_password BY '123';
GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'ae86'@'%';
GRANT SELECT ON test.* TO 'ae86'@'%';
flush privileges;

Oracle

• CDN registration and subscription. Listen for insert, delete, and update events, obtain the rowid, and execute SQL queries based on the rowid to retrieve the changed data.

1. Grant the account permission to monitor changes. The target source table must also define a varchar field with a length of 18 to receive the rowid value and perform insert, update, and delete operations.

grant change notification to your_account

2. The account must be the OWNER of the monitored table.

SELECT OBJECT_ID, OBJECT_NAME, OWNER FROM ALL_OBJECTS WHERE OBJECT_TYPE = 'TABLE' AND OWNER='your_account';

SqlServer

• SQL Server 2008 provides a built-in Change Data Capture (CDC) feature for asynchronously tracking data modifications in user tables.

Requires version 2008 or later. Start the Agent service. The connecting account must be a member of the sysadmin fixed server role or the db_owner fixed database role. For all other users, SELECT permission on the source table is required. If an access control role has been defined for the capture instance, membership in that database role is also required.

1. Start the Agent and Database services.

1. Prepare a test account: test

1. Assign the sysadmin role.

1. Assign the specified database permissions.

PostgreSQL

• Monitor incremental events through replication stream technology and parse WAL logs using the built-in plugins pgoutput and test_decoding.

Modify the postgresql.conf file and restart the service.

wal_level=logical

File

• Detect modified files by monitoring file modification times, then read the latest data using file offsets.

File Monitoring Implementation

ES

• Retrieve incremental data at scheduled intervals.

The account has access permissions.

Logs

It is recommended to use log-based synchronization for MySQL, SqlServer, and PostgreSQL.

Scheduled

Assumed source table data format