1. Project Overview

A heterogeneous data source data transfer system based on Flink.

Core Features : Synchronize data across different heterogeneous data sources, and manage and maintain synchronization tasks.

2. Open Source License

Licensed under Apache-2.0

3. Screenshots

How to Use

1. Log in to the system. The default username and password are both admin. No permission controls are configured.

2. Data Source Management: Configure the data sources used for data transfer.

3. Task Management: Configure from_db and to_db to construct the job_graph. 

4. Task Cascade Configuration 

5. Task Lineage 

6. Task Scheduling 

7. Task Execution 

4. Feature Overview

Introduction to DatalinkX: A Heterogeneous Data Source Synchronization Service

Core Features : Synchronize data across different heterogeneous data sources, and manage and maintain synchronization tasks

Purpose: Any company with a relatively large scale and data collaboration between departments should have a project similar to DatalinkX. For example, data crawled by colleagues in the crawler team may need to be synchronized periodically to databases managed by the data warehouse team. The synchronization service centrally manages synchronization tasks, consolidates synchronization logs, and improves internal work efficiency.

Project characteristics

• Simple and Easy to Use: Quickly create data sources and synchronization tasks through a Web interface. Simple operations allow you to get started in just one minute.

• Scheduled Triggers: Integrates with xxl-job scheduling to trigger synchronization tasks using cron expressions.

• Configuration-Based Task Integration: Configure database information and task details through a graphical interface.

• High-Performance Synchronization: Uses the high-performance Flink streaming computing engine.

• Containerized Deployment: Supports Docker deployment.

5. Tech Stack

Project Technology Stack

Getting Started

Middleware

Run the docker compose -p datalinkx up -d command to start all components.

Manually Set Up Components (Linux):

xxl-job: xuxueli/xxl-job/archive/refs/tags/2.3.0.zip is a pure Java project. You can clone the source code, package it into a JAR file, and start it. xxl-job depends on MySQL, so you need to modify the corresponding database connection configuration. The database schema is located at /xxl-job-2.3.0/doc/db/tables_xxl_job.sql. Simply import it into MySQL.

flink: Download flink-1.10.3-bin-scala_2.12.tgz from flink/flink-1.10.3/, extract it, navigate to the bin directory, and execute ./start-cluster.sh. On the first run, there is only one task slot by default. Visit [http://localhost:8081](http://localhost:8081) to access the Flink dashboard.

DB Layer

Execute /datalinkx-server/src/main/resources/db.sql

Backend

1. Run the datalinkx-server and datalinkx-job modules

1.1 datalinkx-server communicates with front and depends on MySQL and Redis.

1.2 datalinkx-job is responsible for submitting tasks and managing their lifecycles. It depends on xxl-job and Flink.

• 1.2.1 After the service starts, Netty is enabled by default to listen for xxl-job task events on ${xxl.job.executor.port}.

• 1.2.2 Detailed task execution information is accessed through the RPC capabilities of datalinkx-client at ${client.dataserver}.

• 1.2.3 If you change the datalinkx-server port, you must also update the ${client.dataserver} configuration in datalinkx-job.

• 1.2.4 ${flinkx.path} specifies the path to the flinkx module.

1.3 ${flinkx.path} specifies the path to the flinkx module.

2. The flinkx module is an independent project.

2.1 You need to manually execute mvn clean install -U -Dmaven.test.skip=true -Dcheckstyle.skip=true to package the plugins.

2.2 After packaging, configure the Flink address (jobmanager.rpc.address:) and port (rest.port) in flinkx/flinkconf.

Frontend

yarn install && export NODE_OPTIONS=--openssl-legacy-provider && yarn run serve