1. Project Overview

DataVines is a one-stop open-source data observability platform, providing metadata management, data overview reports, data quality management, data distribution queries, and data trend insights, and is committed to helping users comprehensively understand and manage data, so that you can have a clear mind.

2. Open Source License

Licensed under Apache-2.0

3. Screenshots

4. Feature Overview

Features

Data catalog

• Scheduled acquisitionData source metadata, constructing data catalog
• Scheduled listeningMetadata changesSituation
• Supports metadataLabel management

Data quality monitoring

• Built-in 27 data quality check rules, ready to use
• Supports 4 data quality check rule types, single table, single column, custom SQL, cross-table accuracy, two-table value comparison
• Supports configuring scheduled tasks forScheduled checks
• Supports configuring SLA forAlerts for check results

Data overview

• Supports scheduled execution of data detection, outputData overview report
• SupportAutomatic identificationColumn types, automatic matching of suitable data overview indicators
• SupportTable row numbers, trendsMonitoring
• Supports columnData distributionSituation view

Plugin-Based Design

The platform is built around a plugin-based architecture, and all the following modules support user-defined plugins for extensions

• Data Source: Supported databases include MySQL, Impala, Starocks, Doris, Presto, Trino, ClickHouse, and PostgreSQL
• Validation Rules: Includes 27 built-in validation rules, such as null checks, non-null checks, and enumeration checks
• Job Execution Engine: Supports both Spark and Local execution engines. The Spark engine currently supports only Spark 2.4, while the Local engine is a JDBC-based local execution engine that does not require dependencies on other execution engines.
• Alert Channels: SupportedEmail
• Error Data Storage: Supports MySQL and Local Files(supported only by the Local execution engine)
• Service Registry: Supports MySQL, PostgreSQL, and ZooKeeper

Multiple operating modes

• ProvidesWeb pageConfigure check jobs, run jobs, view job execution logs, view error data and check results
• SupportOnline generationJob execution scripts, submitted through datavines-submit.sh, can be used with scheduling systems

Easy deployment & high availability

• Few dependencies, easy deployment
• Minimal dependencies on MySQL to start the project and complete data quality tasks
• Supports horizontal expansion and automatic fault tolerance
• Decentralized Architecture, with support for horizontal scaling of Server nodes to improve performance
• JobAutomatic fault toleranceGuarantees that jobs are not lost and will not be executed repeatedly

5. Tech Stack

Architecture design

Environment Dependencies

1. Java runtime environment: Jdk8
2. Datavines supports JDBC engine, if your data volume is small or you just want to do functional verification, you can use JDBC engine
3. If you want to run Datavines based on Spark, you need to ensure that your server has the conditions to run Spark applications

Installation

Using Maven 3.6.1 or later versions

$ mvn clean package -Prelease -DskipTests

Datavines Environment Setup Guide

Preconditions

Before setting up the Datavines development environment, make sure you have installed the following software

• Git: Version control system
• JDK: Backend development
• Maven: Java package management system

Clone the Repository

Download the Git repository using your Git management tool. The following example uses git-core

mkdir datavines
cd datavines
Visit Yifei Open Source: [https://code.exmay.com/](https://code.exmay.com/) to download

Compile the Source Code

• If you use a MySQL database, remember to modify pom.xml and change the scope of the mysql-connector-java dependency to compile. No changes are needed when using PostgreSQL.
• Run mvn clean install -Prelease -Dmaven.test.skip=true

Database

Datavines metadata is stored in a relational database. Currently supported relational databases include MySQL and PostgreSQL. The following example uses MySQL. Start the database and create a new database for Datavines metadata storage. Here, the database name is datavines.

After creating the new database, run the SQL file located at script/sql/datavines-mysql.sql directly in MySQL to complete database initialization.

Start the Backend

The following steps will guide you through starting the Datavines backend service.

Prerequisites

• Open the project: Open the project using a development tool. This example uses IntelliJ IDEA. After opening the project, allow some time for IntelliJ IDEA to download all dependencies.
• Required Modifications
• If you use MySQL as the metadata database, first modify datavines/pom.xml and change the scope of the mysql-connector-java dependency to compile. This is not necessary when using PostgreSQL.
• Modify the database configuration in the datavines-server/src/main/resources/application.yaml file.
• This example uses MySQL, with datavines as the database name and root/123456 as the username/password.
• spring:datasource: driver-class-name: com.mysql.cj.jdbc.Driver url: jdbc:mysql://127.0.0.1:3306/datavines?useUnicode=true&characterEncoding=UTF-8 username: root password: 123456

Start the service

Start DatavinesServer

Add the following to VM Options:

-Dspring.profiles.active=mysql, where mysql specifies the configuration file to use. -Dlogging.config=classpath:server-logback.xml

When you see the following log in the Console, it indicates that the service has started successfully: [INFO] 2022-04-10 12:29:05.447 io.datavines.server.DatavinesServer:[61] - Started DatavinesServer in 3.97 seconds (JVM running for 4.69)

Start the Frontend Service

• Prerequisites: Install node (v16.9.0) and npm (8.19.3) locally.
• Execute the following commands in sequence to start the local frontend service for development and debugging.
• 
  

cd datavines\ui
npm install
npm fund
npm run dev:prod

Quick start

Please refer to the official documentation: Quick Start Guide

Development guide

Please refer to the official documentation: Development Guide

Source: Datavines Official Documentation