1. Project Overview

Jumbo task scheduling platform for Meiyou big data research and development's distributed computing task scheduling system, providing Spark, Flink, etc. batch processing tasks DAG scheduling and flow processing tasks running management and status monitoring, and with Yarn application management, repeated application detection, large memory application detection, etc. functions.

2. Open Source License

Licensed under Apache-2.0

3. Screenshots

Initial configuration

• Open: http://localhost:17070

• Input account admin, password admin
• Click: Permission Management->User Management, modify the current account's email to a valid and existing email address, otherwise it will cause email sending failure
• Add Cluster Cluster Management->Cluster Management->Add New

Yarn management address is the WEB UI address of Yarn ResourceManager Package storage directory is the storage path when uploading packages to the HDFS cluster, such as /data/big-whale/storage Supports Flink task proxy user Flow processing task blacklist and batch processing task blacklist are internal customized task allocation rules, do not fill

• Add Cluster User Cluster Management->Cluster User->Add

This configuration means: the Yarn resource queue (--queue) and proxy user (--proxy-user) that platform users can use under the selected cluster

• Add proxy cluster management->Proxy Management->Add

Can add multiple instances (only supports IP addresses, can specify port numbers, default is 22), when executing the script, it will randomly select one instance to execute, in the case where an instance is unreachable, it will continue to randomly select the next instance, and when all instances are unreachable, the execution will fail. After selecting a cluster, it will be one of the proxies for submitting Spark or Flink tasks under that cluster

• Add compute framework version cluster management -> version management -> add

The submission commands of Spark or Flink tasks in the same cluster may be different, such as the submission command of Spark 1.6.0 version is spark-submit, and the submission command of Spark 2.1.0 version is spark2-submit

Usage

1. Offline scheduling

1.1 New

• Currently supports three types of batch jobs: "Shell", "Spark Batch", and "Flink Batch"
• You can add corresponding DAG nodes by dragging the corresponding batch job icons from the left toolbar

Supports time parameters${now} ${now - 1d} ${now - 1h@yyyyMMddHHmmss}At (d days, h hours, m minutes, s seconds, @yyyyMMddHHmmss as the formatted parameter), non-"Shell" type batch processing tasks should upload the corresponding program package, which in this case is the jar package of the Spark batch task. The "Resource Options" can be left blank. There are two editing modes, "Visual View" and "Code View", which can be switched between. Clicking "Test" can test whether the current node is correctly configured and can run normally. To prevent the platform thread from being occupied by a large number of threads, when submitting Spark or Flink tasks, it will be forced to execute in the "background" mode. The corresponding Spark configuration is --conf spark.yarn.submit.waitAppCompletion=false, and the Flink configuration is -d. However, based on the "background job status update task" callback, when implementing the DAG execution engine, it can ensure that the tasks submitted by the current node are completed before executing the tasks of the next node.

• DAG node supports failure retry
• By connecting nodes in a certain order, a complete DAG can be constructed

• After DAG construction is complete, clicking 'Save' completes the scheduling settings

1.2 Operations

• Open the offline scheduling list

• Clicking the left sidebar 'Scheduling Instance' can view the scheduling instance list, running status, and node startup log

• Clicking the left sidebar 'Manual Execution' can trigger scheduling execution

2. Real-time Tasks

2.1 Add

• Currently supports 'Spark Stream' and 'Flink Stream' two types of stream processing tasks

• Enable monitoring can monitor the status of tasks, including abnormal restart, batch accumulation alarm, etc.

2.2 Operations

• Open real-time task list

• Click the left sidebar 'Log' to view task startup logs
• Click the left sidebar 'Execute' to trigger task startup

3. Task Alarm

• After correct configuration of email or Dingding alarm, when the task runs abnormally, it will send the corresponding alarm email or notification in time, so as to handle it in time

<Jinyong Task Alarm>
Agent: agent1
Type: Script execution failed
User: admin
Task: Schedule Example 1 - shell_test
Time: 2021-03-05 15:18:23
<Jinyong Task Alarm>
Cluster: Cluster1
Type: Spark offline task exception (FAILED)
User: admin
Task: Schedule Example 1 - spark_test
Time: 2021-03-05 15:28:33
<Jinyong Task Alarm>
Cluster: Cluster1
Type: Spark real-time task batch accumulation, restarted
User: admin
Task: sparkstream_test
Time: 2021-03-05 15:30:41

• Besides the above-mentioned alarm information, there are other alarm information not listed here

4. Feature Overview

Features

• Based on SSH's script execution mechanism, deployment is simple and fast, requiring only a single service
• Based on Yarn Rest Api's task status synchronization mechanism, there is no version limit for Spark and Flink
• Supports failed retries
• Supports task dependencies
• Supports complex task orchestration (DAG)
• Supports real-time task execution management and monitoring
• Supports Yarn application management

5. Tech Stack

1. Architecture diagram

Deployment

1. Preparation

• Java 1.8+
• Mysql 5.1.0+
• Download the project or clone the project
• To solve the problem of GitHub README.md images not loading normally, please add the relevant domain name resolution rules to the hosts file, reference: hosts

2. Installation

• Create a database: big-whale
• Run the database script: big-whale.sql
• Configure the relevant database account and password, as well as SMTP information, according to the Spring Boot environment
• Configuration: big-whale.properties configuration item explanation ssh.user: The SSH remote login username with script execution permission (the platform will use this user as the unified script execution user) ssh.password: SSH remote login user password dingding.enabled: Whether to enable Dingding alarm dingding.watcher-token: Dingding public group robot token yarn.app-memory-threshold: Yarn application memory threshold (unit: MB), -1 disables detection yarn.app-white-list: Yarn application white list (applications in the list that apply for memory exceeding the threshold will not be alarmed)
• Modification: $FLINK_HOME/bin/flink, Reference: flink (because flink submits tasks can only read local jar packages, so need to download jar packages from hdfs and replace the jar package path parameter in the script when executing the submit command)
• Packaging: mvn clean package

3. Start

• Check if port 17070 is occupied, if occupied, close the occupied process or modify the project port number configuration and repackage
• Copy the big-whale.jar in the target directory, execute the command: java -jar big-whale.jar