开源工作流任务调度系统：Apache DolphinScheduler

Posted 2023-04-16

参考技术A

A distributed and easy-to-extend visual workflow scheduler system

dedicated to solving the complex task dependencies in data processing, making the scheduler system out of the box for data processing.

Its main objectives are as follows:

https://dolphinscheduler.apache.org/en-us/docs/latest/user_doc/guide/installation/standalone.html

https://dolphinscheduler.apache.org/en-us/docs/latest/user_doc/guide/quick-start.html

Before explaining the architecture of the scheduling system, let\'s first understand the commonly used terms of the scheduling system

System architecture diagram

Start process activity diagram

The centralized design concept is relatively simple. The nodes in the distributed cluster are divided into roles according to roles, which are roughly divided into two roles:

Problems in centralized thought design:

In the above figure, MainFlowThread waits for the end of SubFlowThread1, SubFlowThread1 waits for the end of SubFlowThread2, SubFlowThread2 waits for the end of SubFlowThread3, and SubFlowThread3 waits for a new thread in the thread pool, then the entire DAG process cannot end, so that the threads cannot be released. In this way, the state of the child-parent process loop waiting is formed. At this time, unless a new Master is started to add threads to break such a "stalemate", the scheduling cluster will no longer be used.

It seems a bit unsatisfactory to start a new Master to break the deadlock, so we proposed the following three solutions to reduce this risk:

note: The Master Scheduler thread is executed by FIFO when acquiring the Command.

So we chose the third way to solve the problem of insufficient threads.

Fault tolerance is divided into service downtime fault tolerance and task retry, and service downtime fault tolerance is divided into master fault tolerance and worker fault tolerance.

The service fault-tolerance design relies on ZooKeeper\'s Watcher mechanism, and the implementation principle is shown in the figure:

Among them, the Master monitors the directories of other Masters and Workers. If the remove event is heard, fault tolerance of the process instance or task instance will be performed according to the specific business logic.

Fault tolerance range: From the perspective of host, the fault tolerance range of Master includes: own host + node host that does not exist in the registry, and the entire process of fault tolerance will be locked;

Fault-tolerant content: Master\'s fault-tolerant content includes: fault-tolerant process instances and task instances. Before fault-tolerant, it compares the start time of the instance with the server start-up time, and skips fault-tolerance if after the server start time;

Fault-tolerant post-processing: After the fault tolerance of ZooKeeper Master is completed, it is re-scheduled by the Scheduler thread in DolphinScheduler, traverses the DAG to find the "running" and "submit successful" tasks, monitors the status of its task instances for the "running" tasks, and "commits successful" tasks It is necessary to determine whether the task queue already exists. If it exists, the status of the task instance is also monitored. If it does not exist, resubmit the task instance.

Fault tolerance range: From the perspective of process instance, each Master is only responsible for fault tolerance of its own process instance; it will lock only when handleDeadServer ;

Fault-tolerant content: When sending the remove event of the Worker node, the Master only fault-tolerant task instances. Before fault-tolerant, it compares the start time of the instance with the server start-up time, and skips fault-tolerance if after the server start time;

Fault-tolerant post-processing: Once the Master Scheduler thread finds that the task instance is in the "fault-tolerant" state, it takes over the task and resubmits it.

Note: Due to "network jitter", the node may lose its heartbeat with ZooKeeper in a short period of time, and the node\'s remove event may occur. For this situation, we use the simplest way, that is, once the node and ZooKeeper timeout connection occurs, then directly stop the Master or Worker service.

Here we must first distinguish the concepts of task failure retry, process failure recovery, and process failure rerun:

Next to the topic, we divide the task nodes in the workflow into two types.

Each business node can be configured with the number of failed retries. When the task node fails, it will automatically retry until it succeeds or exceeds the configured number of retries. Logical node Failure retry is not supported. But the tasks in the logical node support retry.

If there is a task failure in the workflow that reaches the maximum number of retries, the workflow will fail to stop, and the failed workflow can be manually rerun or process recovery operation

In the early scheduling design, if there is no priority design and the fair scheduling design is used, the task submitted first may be completed at the same time as the task submitted later, and the process or task priority cannot be set, so We have redesigned this, and our current design is as follows:

https://dolphinscheduler.apache.org/en-us/docs/latest/user_doc/architecture/design.html

开源大数据：Apache DolphinScheduler

Apache DolphinScheduler 分布式易扩展的可视化DAG工作流任务调度系统

Apache DolphinScheduler是一个分布式去中心化，易扩展的可视化DAG工作流任务调度系统。致力于解决数据处理流程中错综复杂的依赖关系，使调度系统在数据处理流程中开箱即用

Apache DolphinScheduler 架构

Apache DolphinScheduler 特性

• 以DAG图的方式将Task按照任务的依赖关系关联起来，可实时可视化监控任务的运行状态
• 支持丰富的任务类型：Shell、MR、Spark、SQL(mysql、postgresql、hive、sparksql),Python,Sub_Process、Procedure等
• 支持工作流定时调度、依赖调度、手动调度、手动暂停/停止/恢复，同时支持失败重试/告警、从指定节点恢复失败、Kill任务等操作
• 支持工作流优先级、任务优先级及任务的故障转移及任务超时告警/失败
• 支持工作流全局参数及节点自定义参数设置
• 支持资源文件的在线上传/下载，管理等，支持在线文件创建、编辑
• 支持任务日志在线查看及滚动、在线下载日志等
• 实现集群HA，通过Zookeeper实现Master集群和Worker集群去中心化
• 支持对Master/Worker cpu load，memory，cpu在线查看
• 支持工作流运行历史树形/甘特图展示、支持任务状态统计、流程状态统计
• 支持补数
• 支持多租户
• 支持国际化
• 还有更多等待伙伴们探索

Apache DolphinScheduler 启动流程活动

DolphinScheduler 界面图