Dynamical System Theory for the Detection of Anomalous Behavior in Computer Programs

Code injection is a common approach which is utilized to exploit applications. We introduce some of the well-established techniques and formalisms of dynamical system theory into analysis of program behavior via system calls to detect code injections into an applications execution space. We accept a...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on human-machine systems 2012-11, Vol.42 (6), p.1579-1589
Main Authors: Kanaskar, N., Seker, R., Jiang Bian, Phoha, V. V.
Format: Article
Language:English
Subjects:
Anomalous behavior
Applied sciences
approximate entropy
central tendency measure (CTM) dynamical system
Cleaning
Complexity
Complexity theory
Computer programs
Computer science
control theory
systems
Computer systems performance. Reliability
Control theory. Systems
Cybernetics
Dynamical systems
Dynamics
Entropy
Exact sciences and technology
Formalism
intrusion detection
Malware
Memory and file management (including protection and security)
Memory organisation. Data processing
Meteorology
percent determinism
percent ratio
percent recurrence
recurrence plots
Software
system call sequence
System theory
Time series analysis
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Code injection is a common approach which is utilized to exploit applications. We introduce some of the well-established techniques and formalisms of dynamical system theory into analysis of program behavior via system calls to detect code injections into an applications execution space. We accept a program as a blackbox dynamical system whose internals are not known, but whose output we can observe. The blackbox system observable in our model is the system calls the program makes. The collected system calls are treated as signals which are used to reconstruct the system's phase space. Then, by using the well-established techniques from dynamical system theory, we quantify the amount of complexity of the system's (program's) behavior. The change in the behavior of a compromised system is detected as anomalous behavior compared with the baseline established from a clean program. We test the proposed approach against DARPA-98 dataset and a real-world exploit and present code injection experiments to show the applicability of our approach.
ISSN:1094-6977
2168-2291
1558-2442
2168-2305
DOI:10.1109/TSMCC.2012.2208187