1. Project Overview

MyDataHarbor is a middleware dedicated to solving distributed, high-extensibility, high-performance, transaction-level, and quasi-real-time data synchronization between any data source and any data source.

It can help users reliably, quickly, and stably perform quasi-real-time incremental synchronization or scheduled full synchronization of massive data, mainly positioning it for real-time trading systems, and can also be used for big data data synchronization (ETL field).

2. Open Source License

Licensed under GPL-3.0

3. Screenshots

4. Feature Overview

Background

In the background of microservices, the scattered storage of real-time trading system data has become a common practice, but sometimes we need to synchronize these data in real-time or scheduled full synchronization to another place.

For example, a company's C department system needs to use the data produced by the A and B departments, which inevitably requires full or incremental data synchronization. Again, for example, I want to synchronize the data in the database in real-time to Elasticsearch, Redis, etc. for search.

The application scenario of data synchronization is very common and very important in daily distributed system development, and once data synchronization appears to be problematic, it will cause data inconsistency and lead to other serious exceptions.

Currently, small companies practice modifying the code in the business program system to write data to the target data source, while medium-sized companies develop their own synchronization small programs in each department, without management and monitoring, which is a waste of resources and cannot ensure stability. Large companies have a data migration platform (such as Ali's Jingwei).

MyDataHarbor was born in response to this scene demand!

Features

Distributed Design

MyDataHarbor is a distributed middleware built on ZooKeeper, supporting horizontal expansion, dividing nodes into groups, and forming a sub-cluster under each group, with tasks performing load balancing within the sub-cluster isolation range to prevent single-point failures.

Plugin Design

The highly reasonable abstraction and pluginized design make MyDataHarbor have high extensibility, and any data migration requirements can be completed by developing plugins.

Transaction Support

MyDataHarbor has considered the problem of data loss from the beginning of design and introduced transaction support to ensure data is not lost!

Plugin Self-Description

After installing the plugin, the middleware will automatically identify the capabilities of this plugin and generate a user-friendly task creation interface, without requiring users to write complex JSON configurations directly.

Free combination

MyDataHarbor supports reusing various components from different plugins, forming a new pipeline pipeline, and these can be done through a visual way.

Task monitoring

Integrate with Java's jmx, each task has detailed monitoring, and real-time viewing of task execution status.

Batch support

Reserve batch interfaces for batch submission, effectively improving data migration speed, from motorcycle to car.

ForkJoin

For the case where DataSource cannot be multi-threaded and concurrent pulled (such as jdbc cursor taking data), internal introduction of forkjoin concurrent processing model to open multi-threaded processing, and flexible transaction control, let the speed be fast while ensuring the stability and no loss of data migration, from car to high-speed rail.

5. Tech Stack

Design

MyDataHarbor's only dependency is zookeeper, with two components: mydataharbor-console, mydataharbor-server

Supported plugins

Middleware/Protocol

Data source (DataSource)

Write source (Sink)

Author

Remarks

kafka

✅ All versions

✅ All versions

MyDataHarbor

redis

Not considered

✅ All versions

MyDataHarbor

Support connecting single-machine, master-slave, sentinel, cluster mode of various redis service ends

Currently supports string data format operations

If needed, please contact the author to extend: list, set, zset, hash

elasticsearch

Planned

✅5.6.x✅6.4.x✅6.0.x

✅6.8.x✅6.5.x✅6.6.x

✅6.7.x✅7.7.x✅7.13.x

MyDataHarbor

http

Not considered

✅

MyDataHarbor

jdbc

✅ All versions of MySQL

✅ All versions of Oracle

✅ All versions of Hive

PostgreSQL planned

✅ All versions of MySQL

✅ All versions of Oracle

PostgreSQL planned

MyDataHarbor

Three modes (full/incremental/first full then incremental)

❗ This way cannot synchronize deleted data

MongoDB

Planned

Planned

Waiting for position

binlog

Planned

×

Waiting for position

rabbitMQ

Planned

Planned

Waiting for position

RocketMQ

Planned

Planned

Waiting for position

Quick Start

MyDataHarbor's installation is very simple (please prepare a zookeeper cluster before startup):

Download binary package

Access to One Flying Open-source: https://code.exmay.com/ Download

  mydataharbor-console-xxx-bin.tar.gz
  mydataharbor-server-xxx-bin.tar.gz

xxx is the version number

mydataharbor-console

Unzip

Configuration

Enter the config directory, modify application.yml, mainly modify the following configurations

server:
  port: 8080 #console service startup port
zk: 127.0.0.1:2181 #zk address

Run

Run start.bat on Windows system

Run start.sh on Linux system, close stop.sh

start.sh script supports jmx, debug, status parameters, such as:

start.sh jmx Start remote jmx support

start.sh debug Open remote debug way to start

start.sh status View current program status

mydataharbor-server

Unzip

Configuration

Modify system.yml in config directory

zk: ["127.0.0.1:2181"] #zk address
port: 1299 #server service startup port
group: biz001 #node belongs to group
pluginRepository: http://127.0.0.1:8080 #plugin repository address

Run

Run start.bat on Windows system

Run start.sh on Linux system, close stop.sh

start.sh script supports jmx, debug, status parameters, such as:

start.sh jmx Start remote jmx support

start.sh debug Open remote debug way to start

start.sh status View current program status

Verification

Access: mydataharbor-console http://127.0.0.1:8080 Whether to see the node just started