[Open Source] A simple-to-use, powerful hybrid scientific big data pipeline system.

Open SourceProcess Management

2026-06-28 05:37 2 次浏览 • 0 次下载 0KB

[Open Source] A simple-to-use, powerful hybrid scientific big data pipeline system.

1. Project Overview

πFlow is an easy-to-use and powerful big data pipeline system.

A hybrid scientific big data pipeline system featuring a rich collection of processor components and providing Shell, DSL, Web configuration interfaces, task scheduling, task monitoring, and more.

2. Open Source License

Licensed under Apache-2.0

3. Screenshots

Page Preview

Login:

Pipeline List:

Create Pipeline:

Configure Pipeline:

Run Pipeline:

Monitor Pipeline:

Pipeline Logs:

Pipeline Group List:

Configure Pipeline Group:

Monitor Pipeline Group:

Running Pipelines List:

Pipeline Template List:

Data Source:

Scheduling:

Custom Components:

4. Feature Overview

Features

Easy to Use

Visually Configure Pipelines
Monitor Pipeline
View Pipeline Logs
Checkpoint Functionality
Pipeline Scheduling

Highly Extensible:

Supports custom development of data processing components

Excellent Performance:

Developed based on the distributed computing engine Spark.

Powerful Features:

Provides over 100 data processing components.
Including Hadoop, Spark, MLlib, Hive, Solr, Redis, MemCache, ElasticSearch, JDBC, MongoDB, HTTP, FTP, XML, CSV, JSON, and more.
Integrates algorithms related to the field of microbiology.

5. Tech Stack

Architecture

Requirements

JDK 1.8
Scala-2.11.8
Apache Maven 3.1.0
Spark version 2.1.0 or later.
Hadoop-2.6.0

Start

Build πFlow:

install external package

mvn install:install-file -Dfile=/../piflow/piflow-bundle/lib/spark-xml_2.11-0.4.2.jar -DgroupId=com.databricks -DartifactId=spark-xml_2.11 -Dversion=0.4.2 -Dpackaging=jar
    mvn install:install-file -Dfile=/../piflow/piflow-bundle/lib/java_memcached-release_2.6.6.jar -DgroupId=com.memcached -DartifactId=java_memcached-release -Dversion=2.6.6 -Dpackaging=jar
    mvn install:install-file -Dfile=/../piflow/piflow-bundle/lib/ojdbc6-11.2.0.3.jar -DgroupId=oracle -DartifactId=ojdbc6 -Dversion=11.2.0.3 -Dpackaging=jar
    mvn install:install-file -Dfile=/../piflow/piflow-bundle/lib/edtftpj.jar -DgroupId=ftpClient -DartifactId=edtftp -Dversion=1.0.0 -Dpackaging=jar

mvn clean package -Dmaven.test.skip=true

[INFO] Replacing original artifact with shaded artifact.
    [INFO] Reactor Summary:
    [INFO]
    [INFO] piflow-project ..................................... SUCCESS [  4.369 s]
    [INFO] piflow-core ........................................ SUCCESS [01:23 min]
    [INFO] piflow-configure ................................... SUCCESS [ 12.418 s]
    [INFO] piflow-bundle ...................................... SUCCESS [02:15 min]
    [INFO] piflow-server ...................................... SUCCESS [02:05 min]
    [INFO] ------------------------------------------------------------------------
    [INFO] BUILD SUCCESS
    [INFO] ------------------------------------------------------------------------
    [INFO] Total time: 06:01 min
    [INFO] Finished at: 2020-05-21T15:22:58+08:00
    [INFO] Final Memory: 118M/691M
    [INFO] ------------------------------------------------------------------------

Run πflow Server:

See the README.md documentation for details

下载地址免费

资源直链https://gitee.com/opensci/piflow

去下载