1. Project Overview

DataX Web is a distributed data synchronization tool developed on top of DataX, providing a simple and easy-to-use operation interface, reducing the learning cost of users using DataX, shortening the task configuration time, avoiding errors during the configuration process. Users can create data synchronization tasks by selecting data sources on the page, supporting RDBMS, Hive, HBase, ClickHouse, MongoDB, etc. data sources, RDBMS data sources can batch-create data synchronization tasks, supporting real-time viewing of data synchronization progress and logs, and providing a terminate synchronization function, integrating and secondary developing xxl-job, synchronizing data according to time, auto-increment primary key increment, etc.

Task executor supports cluster deployment, supports task executor multi-node routing strategy selection, supports timeout control, failure retry, failure alarm, task dependency, and monitoring of executor CPU, memory, and load, etc. Subsequently, more data source support, data conversion UDF, table structure synchronization, data synchronization lineage, etc. will be provided for more complex business scenarios.

DataX integration visualization page, select data source to generate data synchronization task with one click, supports RDBMS, Hive, HBase, ClickHouse, MongoDB, etc. data sources, batch creation of RDBMS data synchronization tasks, integration of open-source scheduling systems, supports distributed, incremental data synchronization, real-time viewing of running logs, monitoring of executor resources, killing running processes, data source information encryption, etc.

2. Open Source License

Licensed under MIT

The product is open-source and free, and will continue to provide free community technical support.

3. Screenshots

1. Task executor configuration (using open-source project xxl-job)

• 1. "Scheduler Center OnLine": The online list of "Scheduler Center" is displayed on the right side. After task execution is completed, it will be notified to the "Scheduler Center" with failover mode to avoid single-point risk of callback.
• 2. "Executor List" displays the online executor list, which can be viewed by "OnLine Machine" to view the corresponding executor cluster machine;

Executor Attribute Explanation

1. AppName: (consistent with datax-executor's application.yml datax.job.executor.appname) 
   Each executor cluster's unique identifier AppName, the executor will periodically register automatically with AppName as the object. The executor can be automatically discovered and registered through this configuration, and the registered executor can be used for task scheduling;
2. Name: The name of the executor, because AppName restricts letters and numbers, etc., the readability is not strong, the name is used to improve the readability of the executor;
3. Sorting: The sorting of the executor, the system needs to read the available executor list in the places where the executor is needed, such as task addition, it will be read according to the sorting;
4. Registration Method: The way the scheduler gets the executor address;
    Automatic Registration: The executor automatically performs executor registration, and the scheduler can dynamically discover the executor machine address through the underlying registration table;
    Manual Input: The executor address information is manually input by humans, and multiple addresses are separated by commas for the scheduler to use;
5. Machine Address: Effective when "Registration Method" is "Manual Input", supporting manual maintenance of executor addresses;

2. Create Data Source

Fourth Step Using

3. Create Task Template

Fourth Step Using

4. Build JSON Script

• 1. Step One, Step Two, Select the data source created in Step Two, JSON Build currently supports the following data sources: hive, mysql, oracle, postgresql, sqlserver, hbase, mongodb, clickhouse. Other data sources' JSON build is under development, and manual writing is required for the time being;

• 2. Field Mapping

• 3. Click to Build, Generate JSON, At This Time, You Can Select Copy JSON Then Create Task, Select Datax Task, Paste JSON into Text Box. You Can Also Click to Select Template, Directly Generate Task.

5. Batch Create Task

6. Task Creation Introduction (Task creation associated with template creation is no longer introduced, please refer to 4. Build JSON Script)

Supports DataX tasks, Shell tasks, Python tasks, PowerShell tasks

• Blocking Processing Strategy: Processing strategy when the scheduler is too dense and the executor cannot handle it;
• Single Machine Serial: After the scheduling request enters the single machine executor, the scheduling request enters the FIFO queue and runs in serial;
• Discard Subsequent Scheduling: After the scheduling request enters the single machine executor, if the executor exists with running scheduling tasks, this request will be discarded and marked as failed;
• Override Previous Scheduling: After the scheduling request enters the single machine executor, if the executor exists with running scheduling tasks, the running scheduling tasks will be terminated and the queue will be cleared, and then the local scheduling task will be run;
• It is recommended to set the blocking strategy to discard subsequent scheduling or single machine serial for incremental incremental tasks;
• When setting single machine serial, note to set the reasonable retry times (the number of failed retries * the execution time of each retry < the scheduling cycle of the task), if the retry times are set too many, it will cause data duplication, for example, the task executes every 30 seconds, and the execution time of each retry is 20 seconds, set the retry times to three;
• Incremental parameter settings
• Partition parameter settings

7. Task List

8. Can be clicked to view logs, get real-time log information, and terminate the running datax process

9. Task Resource Monitoring

10. The admin can create users, edit user information

4. Feature Overview

Features

• 1. Build DataX Json through Web;
• 2. DataX Json is saved in the database, making it easy to migrate and manage tasks;
• 3. Real-time viewing of extraction logs, similar to Jenkins' log console output function;
• 4. Displaying DataX running records, can be operated on the page to stop DataX jobs;
• 5. Supports DataX scheduled tasks, supports dynamic modification of task status, start/stop tasks, and terminate running tasks, with immediate effect;
• 6. The scheduler adopts a centralized design, supporting cluster deployment;
• 7. Task distributed execution, task "executor" supports cluster deployment;
• 8. The executor will periodically automatically register tasks, and the scheduling center will automatically discover and trigger the execution of registered tasks;
• 9. Routing strategy: when deploying executor clusters, provide rich routing strategies, including: first, last, round-robin, random, consistent hash, least frequently used, least recently used, failover, busy transfer, etc.;
• 10. Blocking processing strategy: processing strategy when the scheduler is too dense and the executor cannot handle it, including: single machine serial (default), discard subsequent scheduling, override previous scheduling;
• 11. Task timeout control: supports customizing task timeout time, and when the task runs out of time, it will actively interrupt the task;
• 12. Task failure retry: supports customizing task failure retry times, and when the task fails, it will actively retry according to the preset failure retry times;
• 13. Task failure alarm; default provides email failure alarm, and reserves extension interfaces for easy extension of SMS, DingTalk, etc. alarm methods;
• 14. User management: supports online management of system users, with two roles: administrators and ordinary users;
• 15. Task dependency: supports configuring child task dependencies, when the parent task is executed and executed successfully, it will actively trigger the execution of child tasks, multiple child tasks are separated by commas;
• 16. Running report: supports real-time viewing of running data, and scheduling reports, such as scheduling date distribution graph, scheduling success distribution graph, etc.;
• 17. Specify incremental fields, configure scheduled tasks to automatically get the data interval each time, task failure retry, ensure data security;
• 18. The page can configure DataX startup JVM parameters;
• 19. After the data source is configured, add a manual test function;
• 20. Can configure task templates for commonly used tasks, after building JSON, can select associated templates to create tasks;
• 21. Add support for jdbc adding hive data sources, can select data sources on the build JSON page to generate column information and simplify configuration;
• 22. Prioritize getting DataX file directory through environment variables, no need to specify JSON and log directories when deploying clusters;
• 23. Configure hive partitions dynamically through dynamic parameters, can also be used in combination with incremental to dynamically insert partitions;
• 24. Task type is extended from DataX tasks to Shell tasks, Python tasks, PowerShell tasks;
• 25. Add support for HBase data sources, JSON build can get hbaseConfig, column through HBase data sources;
• 26. Add support for MongoDB data sources, users only need to select collectionName to complete JSON build;
• 27. Add monitoring pages for executor CPU, memory, and load;
• 28. Add 24 types of DataX JSON configuration samples
• 29, Public Fields (Creation Time, Creator, Modification Time, Modifier) are automatically filled when inserted or updated
• 30, Token verification for Swagger interfaces
• 31, Task timeout time is increased, and the datax process can be killed for tasks that have timed out, which can be used in conjunction with the retry strategy to avoid datax being stuck due to network problems
• 32, Project management module is added, and tasks can be classified and managed
• 33, Batch task creation function is added for RDBMS data sources, and tables can be selected to generate DataX synchronization tasks in batches according to templates
• 34, JSON construction supports ClickHouse data sources
• 35, Executioner CPU, memory, and load monitoring page is graphically displayed
• 36, Incremental extraction of RDBMS data sources is optimized, and the main key is increased in the self-increment way, and the page parameter configuration is optimized
• 37, MongoDB data source connection method is changed, and HBase data source JSON construction module is reconstructed
• 38, Script type tasks add stop function
• 39, RDBMS JSON construction adds postSql, and supports building multiple preSql, postSql
• 40, Data source information encryption algorithm is modified, and code optimization is performed
• 41, Log page adds DataX execution result statistics data

5. Tech Stack

Architecture diagram：

System Requirements

• Language: Java 8 (JDK version is recommended to be 1.8.201 or later)
• Python 2.7 (Python 3 needs to modify and replace the three Python files in datax/bin, and the replacement files are in doc/datax-web/datax-python3)
• Environment: MacOS, Windows,Linux
• Database: MySQL 5.7