1. Project Overview

A data transfer platform based on kettle. Leveraging kettle's flexibility and rich component ecosystem, it can flexibly implement various complex requirements. This platform can function solely as a kettle scheduling platform or as a flexible data transfer platform. Combining both capabilities provides the ideal solution.

2. Open Source License

Using GPL-2.0 open-source protocol

3. Screenshots

4. Feature Overview

All jobs created on the platform are ultimately generated as kettle jobs. If special requirements arise, they can be modified directly in kettle to meet those needs. Flexibility is the foundation of long-term sustainability.

Other unconventional jobs can be designed directly in kettle and then centrally scheduled and managed within this platform.

This platform is highly flexible, embodying the principle that true strength comes from being inclusive. Although it is already powerful on its own, it also offers exceptional extensibility and compatibility. Any excellent external solution can be integrated into it. Therefore, this platform also serves as an ideal centralized management platform for all backend tasks.

kettle is an excellent open-source ETL tool that has been widely adopted. In general, users operate and manage it through the client application. However, problems arise when it is deployed in production environments, where hundreds or even thousands of jobs may need to be managed. At that scale, managing everything through the client application becomes extremely difficult. The solutions known to the author include command-line invocation, which is extremely cumbersome to operate and difficult to maintain, making it far from practical. Another option is to develop a web-based management tool. kettle comes with its own web-based management tool, but its interface is extremely rudimentary and difficult to use, making it practically unsuitable for production environments. The author has not come across any other tools that provide a better way to manage large numbers of jobs.

Based on the circumstances described above and the author's many years of experience in kettle customization, development, and usage, the author designed and developed Data Master. Built on kettle, it moves commonly used scenarios to the web, enabling operations personnel and even ordinary customers to handle common requirements. The author's goal is to support 70% of requirements through the web interface, while the remaining specialized and less common requirements can be implemented directly in kettle by data engineers and then centrally scheduled and managed through this platform.

Platform Features

1. Supports scheduling tasks from multiple repositories within a single scheduler. It also supports executing tasks from one repository across multiple scheduling nodes, as well as managing multiple scheduling nodes simultaneously through a single management system.

2. New support for configuring jobs to execute immediately upon startup, as well as scheduled execution using cron, making scheduling configurations more flexible.

3. The platform supports creating the following types of jobs: data transfer jobs (temporarily discontinued because the design is somewhat complex and not very practical; data synchronization tasks are currently available), KM jobs, JavaScript jobs, shell jobs, and SQL jobs.

4. The platform can automatically detect abnormal job execution information, making it easier to monitor platform operations. When there are a large number of jobs, checking the execution status of each job individually becomes extremely difficult.

5. The platform provides basic execution logs that serve as a historical record of job execution, making subsequent troubleshooting easier.

6. The platform provides a pre-designed data transfer logging feature that is closer to the business layer and can be used according to user requirements.

7. The platform provides transformation management. Currently, this feature only supports viewing, deleting, and viewing transformation diagrams. An import function will be added in the future based on actual requirements.

8. Database management handles the database information configured in kettle. If the Data World platform is adopted comprehensively, all databases here will be associated with data resources managed in Data World through JNDI. This allows a single configuration to be shared across the entire platform. If databases are configured directly in kettle, their passwords can be decrypted here.

Additional Notes

1. This platform was first put into use in 2015 and has been deployed and used by our company in multiple locations. Since becoming open source, it has also attracted a large number of external users. All basic job scheduling functions are already stable. Newly added features are primarily enhancements. Minor bugs are inevitable, and feedback is welcome. Let's work together to make it better.

2. This system is designed based on database repositories. File repositories are not currently supported. Oracle and MySQL are currently supported.

3. Regarding kettle versions, the current version was developed using version 8.3, while previous versions were developed using version 5.4. Version 8.3 is currently recommended.

5. Tech Stack

Framework Overview

This system is fully parameterized and adopts a parameter organization structure similar to that of front-end frameworks, making it highly flexible and extensively configurable.

Technical Architecture

The following technologies were selected for this project:

1. Database: Currently Oracle and MySQL. Support for various other databases can be implemented quickly.

2. Backend: springboot2 + beetlsql + redis

3. Frontend: vue3 + vxe-table + element

Framework Features

1. Technology Stack: The frontend and backend adopt some of the most popular technologies and relatively recent versions: springboot2, beetlsql, fastjson, druid, vue3, vxe-table, element-plus, and various databases.

2. DAO Tool: The database layer uses beetlSql, an excellent domestically developed DAO tool. The author considers it better than hibernate and mybatis. hibernate is overly abstracted, and most people only understand its basic features even after years of use. The XML-based SQL configuration in mybatis is also rather inconvenient. beetlSql combines the advantages of both: it requires relatively little code, offers excellent code readability, and uses template syntax similar to JavaScript. SQL is organized using Markdown, making it simple and easy to learn. When problems arise, you can directly download the source code to understand and modify its internal implementation. The author has also contributed code to the official project. This is an excellent DAO tool that truly gives developers control over their own technology.

3. Multi-Data Source Support: Data sources can be configured dynamically and flexibly, allowing one configuration to be reused in multiple places. Switching data sources in code is extremely convenient. The same data source configuration can also be used directly in kettle. This system encapsulates the Db utility to simplify common database operations and make switching between database access methods convenient, significantly improving the development experience.

4. Simple Deployment: The project has very few service dependencies. After repeated consideration, the main dependencies are currently redis (optional) and a database (currently compatible with mysql and oracle), making the project simpler and allowing it to be deployed and run quickly and easily.

5. Fully Parameterized: This system is fully parameterized and adopts a parameter organization structure similar to that of front-end frameworks, making it highly flexible and extensively configurable. Complete parameter documentation is also provided.

6. Unified Interfaces: All platform interfaces support convenient integration and access from PC clients, mobile apps, and third-party applications. External integration and data sharing do not require additional code development and can be achieved through simple configuration.

7. Logging Encapsulation: Uses the logback logging framework and integrates a conversion framework to ensure compatibility with logging frameworks such as log4j and common-longging, achieving unified logging. It also implements a custom logger that supports recording all system exception logs in the database, facilitating subsequent system maintenance and early warnings. In addition, the author has implemented a lightweight logback wrapper, making logging convenient during development without affecting class names or stack trace information in log entries.

8. Object-Oriented Design: Adopts an object-oriented system architecture in which everything is treated as an object. Plans to support various data sources, including JDBC-compatible databases, files, FTP, and other data formats.

9. Unified Dictionary: Provides a unified dictionary service that consolidates all dictionaries across the platform. It can flexibly integrate data from different sources to provide dictionary services. For details, refer to the Dictionary Management section in the Development Standards.

10. File Service: Provides a unified file service across the entire platform. Supports uploading files to various storage systems, including local file systems, FTP, HDFS, and databases. All files uploaded to the platform can be easily searched and downloaded.

11. Unified Configuration: Implements extremely lightweight distributed configuration management with support for configuration inheritance and no external dependencies.

12. Unified Validation and Conversion: Implements a unified validation and conversion mechanism that allows validation and conversion rules to be configured for the parameters of each request.

13. Detailed System Logs: Records system operations with precision. In addition to conventional information such as request paths, IP addresses, and operators, this system automatically translates log parameter descriptions into Chinese. It records request parameters, return results, execution times, and other details, making it easier to identify system performance bottlenecks and perform targeted optimizations.

14. A Different Permission System: Provides a simple yet powerful access control system. Roles in this framework can have child roles, and parent roles automatically inherit the permissions of their child roles. Permissions can be granted to unauthenticated users or all authenticated users. Button-level operation permissions are also supported.

15. Low-Code Development: This system already implements generic create, read, update, and delete (CRUD) operations and provides specialized features such as generic data template downloads, bulk data uploads, data exports, inline list editing, and record duplication. Additional common methods can also be customized for different systems as needed. Most requirements can be implemented through configuration without writing development code. The system does not generate code; configurations are cached in memory, so performance is not affected. Configuration changes take effect immediately, while the system remains flexible enough to accommodate virtually any special requirement. This is the biggest difference between this framework and most low-code frameworks currently available on the market.

16. Insert and Update: Supports deduplication based on business fields. During new record creation and bulk uploads, the system checks for existing records using business fields and automatically determines whether to insert or update.

17. Internationalization Support: Integrates internationalization capabilities, making it convenient to develop multilingual applications.

18. Scheduled Tasks: Integrates xxl-job as the scheduled task management mechanism, making it easy to develop and manage scheduled tasks.

19. Intelligent Data Exchange: Provides an intelligent data exchange system that allows near-real-time synchronization between different data sources, including local files, FTP, and kafka. This makes it easy to implement complex requirements such as efficient cross-network data exchange.

20. Bean Generation Tool: Includes a built-in POJO generation tool based on beetl templates. Import doc/idea-run/pojo.run.xml into IDEA, then modify the doc/pojo.json configuration file as needed to generate entity classes.