[Open Source] A Lightweight, Easy-to-Use Distributed Task Scheduling and Computing Framework
![[Open Source] A Lightweight, Easy-to-Use Distributed Task Scheduling and Computing Framework](https://yy123.ink/api/img/1782599054196-qmiy0s.jpg)
1. Project Overview
A Distributed Task Scheduling and Distributed Computing Framework
A distributed task scheduling and computing framework that provides not only standard task scheduling features but also advanced capabilities such as pausing or canceling running tasks, resuming paused tasks, task sharding, failure retries, broadcast tasks, task dependencies, workflow tasks (DAG), separate deployment of managers and workers, and a Web administration console.
Lightweight and easy to use, particularly suitable for executing long-running tasks. Feature-rich, stable, and reliable, with proven performance in production environments.
2. Open Source License
Licensed under Apache-2.0
3. Screenshots
4. Feature Overview
Features
-
It has two roles: Manager (Supervisor) and Executor (Worker). Supervisor and Worker can be deployed separately. -
Supervisor and Worker discover each other through a registry. Supported registries include: Database, Redis, Consul, Nacos, Zookeeper, and Etcd. -
Supervisor is responsible for creating tasks and dispatching them to Workers for execution. Supported task dispatch methods include: Redis and Http. -
A Job group must be specified. Jobs are dispatched only to Workers belonging to the designated group. -
Provides task sharding capabilities. Override the JobSplitter#split method to split a task into multiple subtasks, enabling distributed task execution and parallel processing. -
Supports pausing and canceling running tasks. Paused tasks can be resumed, and failed tasks can be retried. -
Supports task execution state persistence (Savepoint), allowing manually or unexpectedly paused tasks to resume from their previous execution state. -
When a task throws PauseTaskException during execution, all tasks under the corresponding instance will be paused, including tasks dispatched to other Worker machines. -
Supports broadcast tasks, which are dispatched to all Workers within the specified group. -
Supports parent-child dependencies between Jobs. Once dependencies are configured, multiple Jobs are executed sequentially according to the specified dependency order. -
Supports DAG workflows. The jobExecutor can be configured using complex DAG expressions, such as: A->B,C,(D->E)->F,G->H -
Supports automatic resumption of running Tasks after a Worker (service) deployment is completed (see the shutdown_strategy configuration). -
Provides a Web administration console for job configuration, task monitoring, and other operations through a graphical interface.
Comparison
|
|
Quartz |
Elastic-Job |
Xxl-Job |
Disjob |
|
Trigger Types |
Cron |
Cron |
Cron, fixed interval, parent-child dependencies |
Cron, specified time, fixed interval, fixed delay, parent-child dependencies |
|
Task Orchestration |
No |
No |
No |
DAG Expressions |
|
Task Sharding |
No |
Static Sharding |
Broadcast Tasks |
Broadcast Tasks, Dynamic Sharding |
|
Stop and Resume |
No |
No |
Terminate Running Tasks |
Pause running tasks and resume paused tasks |
|
Save Execution Snapshots |
No |
No |
No |
Yes |
|
Failure Retries |
No |
Failover |
Yes |
Yes |
|
Admin Console |
No |
Yes |
Yes |
Yes |
|
Monitoring and Alerts |
No |
|
|
Email, DingTalk, WeCom, Feishu, SMS, and more |
5. Tech Stack
Architecture
-
Overall Workflow
-
Code Structure