1. Project Overview

Visualization Distributed Data Analysis Platform

A data analysis platform similar to Kettle, primarily designed for data analysis while also handling some ETL tasks. Visualization uses a low-code approach to orchestrate DAG tasks, which are automatically executed by the cluster.

2. Open Source License

Author has not specified open source protocol

3. Screenshots

Partial Screenshots

4. Feature Overview

Features

• Configurable Workflows: Provides a drag-and-drop interface for configuring workflows. Data operations are configured through forms, allowing users to complete the corresponding configurations with minimal coding;

• Distributed Cluster Computing Based on Spring Cloud: Requires relatively few external dependencies and supports convenient containerized cluster deployment;

• DAG Task Scheduling: Node tasks are automatically executed by the cluster;

• Multiple Data Sources: Currently supports CSV and JDBC (MySQL and PostgreSQL drivers), with additional data source types planned for future releases;

• File Upload and Download with Chunking Support;

• Workflow Instance Execution Tracking: Supports querying task execution status and displaying it as a task timeline;

• Multi-Tenant Permission Management: Supports parent-child tenant relationships;

• Dynamic Frontend Page Routing: Supports generating frontend routes dynamically based on permissions;

Usage scenario

• Asynchronous Excel Export

• Some state-owned enterprise projects require Excel export functionality. Visual can export Excel files asynchronously, reducing development workload.

• Data analysis that can be handled by a single node.

• This tool currently does not support data sharding. For big data analysis, you need to manually partition data at task nodes, use other OLAP tools, or increase available memory;

• A JVM with a few GB of memory can generally process millions of records.

• Suitable for scenarios requiring data analysis without the need to build a complete big data cluster;

• Suitable for scenarios requiring configurable workflows, such as regularly generating analytical reports.

• Currently, an external scheduled task component is required to create workflow instances. Many scheduled task components are already available on the market and can be used together with this tool. (Integration of a built-in scheduling component is being considered for future releases.)

• Cross-Data-Source Join Operations: For example, joining one table from PolarDB or a CSV file with another table in MySQL;

• Frequently Changing Data Retrieval Logic: For example, when the method of calculating a particular metric changes. With this tool, you only need to modify the configuration to update the logic;

• Backend for Data Visualization Dashboards: Similar to the canvas tools provided by datav and mapv. This tool can be used to quickly build the backend for large-screen data visualization dashboards;

5. Tech Stack

Architecture Diagram

Deployment Prerequisites

JDK1.8+、MySQL5.7+、Redis6+、NodeJS、MinIO、InfluxDB

Note: The author's development environment is Mac. Some dependencies may differ on Windows. During development, the author encountered a Netty dependency that needed to be replaced.

Project Structure

• visual-discovery: Service registry based on Eureka

• visual-engine: Computing node. For distributed clusters, start multiple instances.

• visual-core: Basic implementation of configurable functionality

• visual-portal: Frontend project

• visual-gateway: Gateway service

• visual-auth: Permission management service

Deployment Steps

1. Start MySQL, Redis, Minio, and InfluxDB

2. Start visual-discovery (using the standard Spring Boot startup method)

3. Start visual-gateway (SpringBoot startup method)

4. Start visual-auth (SpringBoot startup method)

5. Start visual-engine (SpringBoot startup method)

6. Start visual-portal (yarn test; startup commands for other environments are not configured for now)

Task Workflow

1. Create a workflow template under Workflow Management - Workflow Definitions. Drag the "Task Node" from the left panel onto the canvas, then click the node to configure it. After completing the configuration, click Publish Workflow.

2. Click Run Instance in the list of defined workflows.

3. Wait for the task to complete, then check its execution status under Workflow Management - Workflow Instance List - View Execution Logs.

Tenant Instructions

• During registration, you can choose between a tenant user and a regular user.

• Regular users are associated with the corresponding tenant only after permissions have been granted.

• Select the current tenant space using the "Enterprise/Organization/Team" option in the upper-right corner. All users are associated with the public tenant by default.

Storage Space Instructions

• You can choose between personal storage and storage belonging to an associated tenant.

• All configuration items are stored in the selected storage space.

Files Used in Workflow Instances

There are two types of file paths used in workflows:

• If the path prefix is spaceShare, the file is located in the spaceShare folder.

• In all other cases, the path prefix is "Workflow Template ID/Workflow Instance ID/Path Configured for the File". This approach prevents files from being overwritten when multiple workflow instances are running.

For more information, please refer to the README.md document