SCIMITAR: Scalable Stream-Processing for Sensor Information Brokering

Current sensor collection capabilities produce an incredible amount of data that needs to be processed, analyzed, and distributed in a timely and efficient manner. Information Management (IM) services supporting a publish-subscribe and query paradigm can be a powerful general purpose approach to ena...

Full description

Saved in:
Bibliographic Details
Main Authors: Rohloff, Kurt, Cleveland, Jeffrey, Loyall, Joseph, Blocher, Timothy
Format: Report
Language:English
Subjects:
CLOUD COMPUTING
Computer Systems
DETECTORS
DISTRIBUTED DATA PROCESSING
INFORMATION BROKERING
INFORMATION EXCHANGE
PUBLISH-SUBSCRIBE
STREAM COMPUTING
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Current sensor collection capabilities produce an incredible amount of data that needs to be processed, analyzed, and distributed in a timely and efficient manner. Information Management (IM) services supporting a publish-subscribe and query paradigm can be a powerful general purpose approach to enabling this information exchange between decoupled and dynamic information producers and consumers. These IM services will only be of value, however, if they can support operations in a manner that is responsive to the sheer quantity and frequency of data produced by surveillance platforms. Cloud computing is the technology of choice for providing the resources and services needed to enable and mange large-scale distributed computation. To date, there has been little work to develop highly scalable, dynamic IM processing and dissemination services in a cloud computing environment. In this paper we discuss our design, implementation and evaluation of a prototype cloud-based information broker which is a critical component of a highly scalable, distributed IM System. The brokering prototype is designed using a distributed stream processing framework and is shown to scale nearly linearly with the number of computing nodes as information load and subscription quantity increases. Military Communications Conference \201MILCOM\202, November 18-20, 2013, San Diego, CA, pp. 1856-1861.