1. Project Overview

Compass is a big data task diagnosis platform designed to improve users' troubleshooting efficiency and reduce the costs associated with abnormal tasks.

2. Open Source License

Licensed under Apache-2.0

3. Screenshots

4. Feature Overview

Compass is a big data task diagnosis platform designed to improve users' troubleshooting efficiency and reduce the costs associated with abnormal tasks.

Its main features are as follows:

• Non-intrusive, real-time diagnosis. Experience diagnostic results without modifying your existing scheduling platform.
• Supports multiple mainstream scheduling platforms, such as DolphinScheduler, Airflow, and self-developed platforms.
• Supports task log diagnosis and parsing for multiple Spark versions and Hadoop 2.x and 3.x.
• Supports workflow-level anomaly diagnosis, identifying various failures and abnormal baseline execution times.
• Supports engine-level anomaly diagnosis, covering 14 types of anomalies, including data skew, large table scans, and memory waste.
• Supports custom log matching rules and anomaly threshold adjustments, allowing users to optimize configurations for their actual scenarios.

Overview of diagnostic types currently supported by Compass:

Diagnostic Dimension

Diagnostic Type

Type Description

Failure Analysis

Execution Failure

Tasks that ultimately failed during execution

First-Time Failure

Tasks that succeeded after more than one retry

Persistent Failures

Tasks that failed during execution within the last 10 days

Execution Time Analysis

Abnormal Baseline Timing

Tasks that finished earlier or later than their historically normal completion times

Abnormal Baseline Duration

Tasks that ran significantly longer or shorter than their historically normal execution durations

Long Execution Duration

Tasks that ran for more than 2 hours

Error Analysis

SQL Failure

Tasks that failed due to SQL execution issues

Shuffle Failure

Tasks that failed due to shuffle execution issues

Out of Memory

Tasks that failed due to out-of-memory issues

Cost Analysis

Memory Waste

Tasks where peak memory usage accounts for a very low proportion of total available memory

CPU Waste

Tasks where driver/executor computation time accounts for a very low proportion of total CPU computation time

Efficiency Analysis

Large Table Scans

Tasks that scan too many rows due to missing partition restrictions

OOM Warning

Tasks where the cumulative memory usage of broadcast tables accounts for an excessively high proportion of either driver or executor memory

Data Skew

Tasks where the maximum amount of data processed by a task in a stage is significantly greater than the median

Abnormal Job Duration

Tasks where the proportion of job idle time to total job execution time is excessively high

Abnormal Stage Duration

Tasks where the proportion of stage idle time to total stage execution time is excessively high

Long-Tail Tasks

Tasks where the maximum task execution duration in a stage is significantly greater than the median

HDFS Slowdown

Tasks where task processing speed is excessively slow within a stage

Excessive Speculative Task Execution

Tasks where speculative task execution occurs frequently within a stage

Abnormal Global Sorting

Tasks where global sorting results in excessively long execution times

5. Tech Stack

Compass Deployment Guide

Compass depends on the scheduling platform, Hadoop, Spark, Canal, MySQL, Kafka, Redis, Zookeeper, and Elasticsearch. Make sure all required environments are prepared in advance.

Compass supports both standalone and cluster deployments, with flexible horizontal scaling for individual modules.

Project Directory

compass
├── bin
│   ├── compass_env.sh                  Environment variables and basic component configuration
│   ├── start_all.sh                    Startup script
│   └── stop_all.sh                     Shutdown script
├── conf
│   └── application-hadoop.yml          Hadoop-related configuration
├── task-application                    Associates task instances, applicationId, and hdfs_log_path
├── task-canal                          Subscribes to scheduling platform MySQL table metadata and publishes it to Kafka
├── task-canal-adapter                  Synchronizes scheduling platform MySQL table metadata to the Compass platform
├── task-detect                         Workflow-level anomaly detection
├── task-metadata                       Synchronizes Yarn and Spark task metadata to Elasticsearch
├── task-parser                         Log parsing and Spark task anomaly detection
├── task-portal                         Visualization service for abnormal tasks
└── task-syncer                         Abstraction and mapping of scheduling platform task relationship tables

How to Use

1. Code Compilation

Visit Yifei Open Source: [https://code.exmay.com/](https://code.exmay.com/) to download.
cd compass
mvn package -DskipTests

2. Configuration Changes

cd dist/compass
vim bin/compass_env.sh
# Scheduler MySQL
export SCHEDULER_MYSQL_ADDRESS="ip:port"
export SCHEDULER_MYSQL_DB="scheduler"
export SCHEDULER_DATASOURCE_USERNAME="user"
export SCHEDULER_DATASOURCE_PASSWORD="pwd"
# Compass MySQL
export COMPASS_MYSQL_ADDRESS="ip:port"
export COMPASS_MYSQL_DB="compass"
export SPRING_DATASOURCE_USERNAME="user"
export SPRING_DATASOURCE_PASSWORD="pwd"
# Kafka
export SPRING_KAFKA_BOOTSTRAPSERVERS="ip1:port,ip2:port"
# Redis
export SPRING_REDIS_CLUSTER_NODES="ip1:port,ip2:port"
# Zookeeper
export SPRING_ZOOKEEPER_NODES="ip1:port,ip2:port"
# Elasticsearch
export SPRING_ELASTICSEARCH_NODES="ip1:port,ip2:port"

3. One-Click Deployment

./bin/start_all.sh

System architecture

System architecture diagram

Architecture Overview

The overall architecture consists of 3 layers:

• Scheduling System Integration Layer: Integrates with schedulers, Yarn, Spark, HDFS, and other systems, synchronizing tasks and their log metadata to the diagnostic system;
• Service Layer: Includes data collection, metadata association&model standardization, anomaly detection, and the Portal module;
• Infrastructure Component Layer: Includes MySQL, Elasticsearch, Kafka, Redis, Zookeeper, and other components.

Workflow stages of specific modules:

• Data Collection Stage: The task-canal/adapter modules subscribe to and synchronize workflow metadata from the scheduling system, including users, DAGs, jobs, and execution records, to the diagnostic platform. The task-metadata module periodically synchronizes Yarn ResourceManager and Spark HistoryServer App metadata to the diagnostic system, associating log storage paths to prepare for subsequent data processing stages;
• Data Association and Model Standardization Stage: The task-syncer module standardizes synchronized data into User, Project, Flow, Task, and TaskInstance models. The task-application module associates workflow-level and engine-level metadata;
• Workflow-Level & Engine-Level Anomaly Detection Stage: At this point, standardized data models are available, enabling further Workflow anomaly detection based on these models. The task-detect module detects workflow-level abnormal tasks, such as execution failures and abnormal baseline durations. The task-parser module detects engine-level abnormal tasks, such as SQL failures and Shuffle failures;
• Business Views: The task-portal module provides features including user report overviews, one-click diagnosis, workflow-level task diagnosis, engine-level job Application diagnosis, diagnostic recommendations and detailed reports, and whitelists.

For more details, please refer to the Deployment Guide.