A program-based anomaly intrusion detection scheme using multiple detection engines and fuzzy inference

• Published in: Journal of network and computer applications 2009-11, Vol.32 (6), p.1219-1228
• Main Authors: Hoang, Xuan Dau, Hu, Jiankun, Bertok, Peter
• Format: Article
• Language: English
• Subjects: Anomaly intrusion detection; Applied sciences; Computer science; control theory; systems; Exact sciences and technology; Fuzzy logic; Hidden Markov model; Information systems. Data bases; Memory organisation. Data processing; Multiple detection engines; Program intrusion detection; Services and terminals of telecommunications; Software; Systems, networks and services of telecommunications; Telecommunications; Telecommunications and information theory; Telemetry. Remote supervision. Telewarning. Remote control
• ISSN: 1084-8045; 1095-8592
• DOI: 10.1016/j.jnca.2009.05.004

In this paper, a hybrid anomaly intrusion detection scheme using program system calls is proposed. In this scheme, a hidden Markov model (HMM) detection engine and a normal database detection engine have been combined to utilise their respective advantages. A fuzzy-based inference mechanism is used to infer a soft boundary between anomalous and normal behaviour, which is otherwise very difficult to determine when they overlap or are very close. To address the challenging issue of high cost in HMM training, an incremental HMM training with optimal initialization of HMM parameters is suggested. Experimental results show that the proposed fuzzy-based detection scheme can reduce false positive alarms by 48%, compared to the single normal database detection scheme. Our HMM incremental training with the optimal initialization produced a significant improvement in terms of training time and storage as well. The HMM training time was reduced by four times and the memory requirement was also reduced significantly.