A New Architecture for Real Time Data Stream Processing

Processing a data stream in real time is a crucial issue for several applications, however processing a large amount of data from different sources, such as sensor networks, web traffic, social media, video streams and other sources, represents a huge challenge. The main problem is that the big data...

Full description

Saved in:
Bibliographic Details
Published in:International journal of advanced computer science & applications 2017-01, Vol.8 (11)
Main Authors: Ounacer, Soumaya, Amine, Mohamed, Ardchir, Soufiane, Daif, Abderrahmane, Azouazi, Mohamed
Format: Article
Language:English
Subjects:
Big Data
Computer architecture
Data processing
Data transmission
Decision making
Fault tolerance
Machine learning
Network latency
Real time
Video data
Citations: Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Processing a data stream in real time is a crucial issue for several applications, however processing a large amount of data from different sources, such as sensor networks, web traffic, social media, video streams and other sources, represents a huge challenge. The main problem is that the big data system is based on Hadoop technology, especially MapReduce for processing. This latter is a high scalability and fault tolerant framework. It also processes a large amount of data in batches and provides perception blast insight of older data, but it can only process a limited set of data. MapReduce is not appropriate for real time stream processing, and is very important to process data the moment they arrive at a fast response and a good decision making. Ergo the need for a new architecture that allows real-time data processing with high speed along with low latency. The major aim of the paper at hand is to give a clear survey of the different open sources technologies that exist for real-time data stream processing including their system architectures. We shall also provide a brand new architecture which is mainly based on previous comparisons of real-time processing powered with machine learning and storm technology.
ISSN:2158-107X
2156-5570
DOI:10.14569/IJACSA.2017.081106