1. Project Overview

dataCompare is a big data comparison and data probing platform.

(1) Supports data comparison between Hive tables, MySQL, and Doris. Data comparison tasks can be configured automatically, avoiding the need to repeatedly write SQL for data processing.

(2) Supports data probing with only a small amount of configuration.

2. Open Source License

Licensed under Apache-2.0

3. Screenshots

System Feature Demonstrations

System Homepage 

(1) Data Comparison Feature Demonstration

Database configuration

MySQL Configuration

Hive Configuration

Job Configuration 

Comparison Results 

(2) Data Probes

Job Configuration

Probe Results

4. Feature Overview

Data Comparison:

(1) Configure data comparison tasks through an interactive UI, generating comparison tasks quickly with low-code and minimal configuration.

(2) Supports volume comparison, consistency comparison, and automatic discovery of differential cases.

(3) Currently supports JDBC databases such as MySQL, Apache Hive, and Apache Doris.

(4) Supports automatically sending email alert reports with comparison results.

Data Probes:

(1) Complete data probing with low-code and minimal configuration.

(2) Supports primary key, enumeration value, and null value probing.

5. Tech Stack

Software Architecture

System Flowchart

Technology Stack:

Backend: Spring Boot + Mybatis

Database: MySQL

Parser: Antrl

Data Storage and Computing Engines: Hive, Spark, etc.

System Execution

System Runtime Requirements:

java jdk8

mysql 5.7.36

Runtime Configuration

(1) Execute the SQL files in the sql directory against the database to create the database and tables.

(2) In the project root directory, run mvn clean package -Dmaven.test.skip=true -Ptest, and use dataCompare.jar from the target directory.

(3) Modify the database configuration in application.yml, replacing the database connection information with that of the database configured in Step (1).

(4) Run java -jar -Dspring.config.location=application.yml dataCompare.jar (application.yml and the JAR file must be in the same directory).

(5) Visit [http://127.0.0.1/](http://127.0.0.1/) to access the application (Username: admin, Password: admin123).

Environment Installation and Configuration

(1) To configure Hive data comparison, you must first install the Hive environment (refer to the Docker-based quick Hive installation guide for instructions). For details, see the README.md documentation.

(2) After installation, select Hive when creating a new data source connection. Use the following connection URL: jdbc:hive2://ip:10000