The MeDICi Integration Framework: A Platform for High Performance Data Streaming Applications

Building high performance analytical applications for data streams generated from sensors is a challenging software engineering problem. Such applications typically comprise a complex pipeline of processing components that capture, transform and analyze the incoming data stream. In addition, applica...

Full description

Saved in:
Bibliographic Details
Main Authors: Gorton, I., Wynne, A., Almquist, J., Chatterton, J.
Format: Conference Proceeding
Language:English
Subjects:
Application software
component model
Computer security
Data analysis
Independent component analysis
integration
Middleware
Performance analysis
Pipelines
Production
Software engineering
Throughput
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Building high performance analytical applications for data streams generated from sensors is a challenging software engineering problem. Such applications typically comprise a complex pipeline of processing components that capture, transform and analyze the incoming data stream. In addition, applications must provide high throughput, be scalable and easily modifiable so that new analytical components can be added with minimum effort. In this paper we describe the MeDICi integration framework (MIF), which is a middleware platform we have created to address these challenges. The MIF extends an open source messaging platform with a component-based API for integrating components into analytical pipelines. We describe the features and capabilities of the MIF, and show how it has been used to build a production analytical application for detecting cyber security attacks. The application was composed from multiple independently developed components using several different programming languages. The resulting application was able to process network sensor traffic in real time and provide insightful feedback to network analysts as soon as potential attacks were recognized.
DOI:10.1109/WICSA.2008.21