1. Project Overview

Introduction to YunQue

YunQue, where 'Yun' represents big data, and 'Que' represents simplicity and freedom.

YunQue is a data integration tool that integrates heterogeneous data sources and helps enterprises build application architectures such as data warehouses and data lakes.

2. Open Source License

Licensed under Apache-2.0

3. Screenshots

Design philosophy

With YunQue, data can be interconnected and integrated across multiple databases, enabling data migration, desensitization, and aggregation.

4. Feature Overview

Goals and Features

1. YunQue currently supports 14 data source types and enables synchronization across heterogeneous data sources.

2. YunQue supports drag-and-drop graphical operations. Users can easily configure input and output components through the interface for simple visual operations.

3. YunQue integrates with computing engines to consolidate data from heterogeneous sources into a unified platform, supporting data lakes, data warehouses, and data integration.

4. YunQue integrates with message queues, enabling data to be read once and delivered to multiple databases multiple times for reuse. It supports massive datasets and high-speed data migration.

5. YunQue supports both standalone and clustered deployments with a simple architecture.

6. YunQue provides clear and user-friendly logs, enabling real-time monitoring of migrated data volumes and migration status. Detailed error logs make troubleshooting easier.

7. YunQue features clean, highly readable code, elegant implementation, comprehensive documentation, and complete reference materials.

8. YunQue supports containerized deployment using Docker and K8S clusters.

9. YunQue is easy to integrate, seamlessly connecting with open-source scheduling systems and integration platforms, with flexible configuration options.

10. YunQue complies with the Apache License and supports commercial deployment.

5. Tech Stack

Framework Design

The framework consists of three components: Reader, Channel, and Writer.

Reader serves as the data source. By implementing the common Reader interface, data can be extracted using SQL.

Channel serves as the queue. Available queue options include the default LinkedBlockingQueue and Kafka. Using Kafka enables a write-once, read-many approach.

Writer serves as the target data source. By implementing the common Writer interface, data can be delivered to the target using SQL.