Bounding substreams in distributed stream processing

• Published in: Information systems (Oxford) 2023-07, Vol.117, p.102251, Article 102251
• Main Authors: Trofimov, Artem, Sokolov, Nikita, Marshalkin, Nikita, Kuralenok, Igor, Novikov, Boris
• Format: Article
• Language: English
• Subjects: Data streams; Punctuations; State management; Stream join; Substreams; Watermarks
• ISSN: 0306-4379; 1873-6076
• DOI: 10.1016/j.is.2023.102251

A common problem in distributed stream processing is to split a stream into finite chunks of messages (substreams) and to determine their boundaries: stateful streaming operators should clear outdated state; time window operators should release output after all elements within the specified time range have arrived. Most state-of-the-art SPEs use punctuations to divide a stream into bounded substreams of messages. The punctuation approach is powerful but has limitations: it does not support cyclic dataflows, is poorly scalable in some cases due to intensive use of broadcasts, and becomes inefficient when the number of chunks or cluster size becomes significant. We introduce a new substream tracking technique called trAcker that overcomes the limits of punctuations. We experimentally evaluate the properties of trAcker in both synthetic and real-world environments. Experiments show that our technique is scalable, outperforms punctuations for a large number of substreams, and efficiently handles real-world cyclic dataflows. •Punctuations can be inefficient for substreams bounding due to high network overhead.•Propagation of substream termination events via an extra agent reduces network traffic.•Network traffic reduction improves the overall performance of stream processing tasks.•An extra agent can be distributed and may scale up to the required substreams number.