A novel fault-tolerant scheduling approach for collaborative workflows in an edge-IoT environment

• Published in: Digital communications and networks 2022-12, Vol.8 (6), p.911-922
• Main Authors: Long, Tingyan, Ma, Yong, Wu, Lei, Xia, Yunni, Jiang, Ning, Li, Jianqi, Fu, Xiaodong, You, Xiangmi, Zhang, Bo
• Format: Article
• Language: English
• Subjects: DQN algorithm; Edge computing; Fault tolerance; Primary-backup model
• ISSN: 2352-8648; 2352-8648
• DOI: 10.1016/j.dcan.2022.08.010

As a newly emerging computing paradigm, edge computing shows great capability in supporting and boosting 5G and Internet-of-Things (IoT) oriented applications, e.g., scientific workflows with low-latency, elastic, and on-demand provisioning of computational resources. However, the geographically distributed IoT resources are usually interconnected with each other through unreliable communications and ever-changing contexts, which brings in strong heterogeneity, potential vulnerability, and instability of computing infrastructures at different levels. It thus remains a challenge to enforce high fault-tolerance of edge-IoT scientific computing task flows, especially when the supporting computing infrastructures are deployed in a collaborative, distributed, and dynamic environment that is prone to faults and failures. This work proposes a novel fault-tolerant scheduling approach for edge-IoT collaborative workflows. The proposed approach first conducts a dependency-based task allocation analysis, then leverages a Primary-Backup (PB) strategy for tolerating task failures that occur at edge nodes, and finally designs a deep Q-learning algorithm for identifying the near-optimal workflow task scheduling scheme. We conduct extensive simulative case studies on multiple randomly-generated workflow and real-world edge-IoT server position datasets. Results clearly suggest that our proposed method outperforms the state-of-the-art competitors in terms of task completion ratio, server active time, and resource utilization.