1. Project Overview

Datavines

DataVines is a one-stop open-source data observability platform, providing metadata management, data overview reports, data quality management, data distribution queries, and data trend insights, and is committed to helping users comprehensively understand and manage data, so that you can have a clear mind.

2. Open Source License

Licensed under Apache-2.0

3. Screenshots

4. Feature Overview

DataVines is a user-friendly data quality service platform, supporting multiple indicators and validation rules. It provides visibility into data lineage, quality, and distribution patterns, helping organizations establish robust data governance practices. The platform helps teams identify and resolve data issues before they impact business decisions, ensuring data remains trustworthy and reliable throughout its lifecycle.

Features

Data catalog

• Scheduled acquisitionData source metadata, constructing data catalog
• Scheduled listeningMetadata changesSituation
• Supports metadataLabel management

Data quality monitoring

• Built-in 27 data quality check rules, ready to use
• Supports 4 data quality check rule types, single table, single column, custom SQL, cross-table accuracy, two-table value comparison
• Supports configuring scheduled tasks forScheduled checks
• Supports configuring SLA forAlerts for check results

Data overview

• Supports scheduled execution of data detection, outputData overview report
• SupportAutomatic identificationColumn types, automatic matching of suitable data overview indicators
• SupportTable row numbers, trendsMonitoring
• Supports columnData distributionSituation view

Multiple operating modes

• ProvidesWeb pageConfigure check jobs, run jobs, view job execution logs, view error data and check results
• SupportOnline generationJob execution scripts, submitted through datavines-submit.sh, can be used with scheduling systems

Easy deployment & high availability

• Few dependencies, easy deployment
• Minimal dependencies on MySQL to start the project and complete data quality tasks
• Supports horizontal expansion and automatic fault tolerance
• No centralized design, Server node supports horizontal expansion to improve performance
• JobAutomatic fault toleranceGuarantees that jobs are not lost and will not be executed repeatedly

5. Tech Stack

Architecture design

Installation

Using Maven 3.6.1 or later versions

$ mvn clean package -Prelease -DskipTests

Environment Dependencies

1. Java runtime environment: Jdk8
2. Datavines supports JDBC engine, if your data volume is small or you just want to do functional verification, you can use JDBC engine
3. If you want to run Datavines based on Spark, you need to ensure that your server has the conditions to run Spark applications

Quick start

Please refer to the official documentation: Quick Start Guide

Development guide

Please refer to the official documentation: Development Guide

DataVines represents a comprehensive solution for data observability and quality management. Its combination of data catalog, quality monitoring, summary analysis, and multi-table verification capabilities provides the tools needed to establish robust data governance practices for organizations.

 

The platform's plugin-based architecture ensures flexibility and extensibility, enabling it to adapt to the evolving data landscape and demands. By supporting numerous data sources, execution engines, and verification rules, DataVines can meet the needs of organizations of various scales and industries.

 

Whether you're just starting your data quality journey or looking to enhance existing practices, DataVines provides a solid foundation to ensure data accuracy, reliability, and trustworthiness throughout its entire lifecycle.