Prediction of Human Activity by Discovering Temporal Sequence Patterns

• Published in: IEEE transactions on pattern analysis and machine intelligence 2014-08, Vol.36 (8), p.1644-1657
• Main Authors: Li, Kang, Fu, Yun
• Format: Article
• Language: English
• Subjects: Activity prediction; Algorithms; Applied sciences; causality; Computer science; control theory; systems; Computer systems and distributed systems. User interface; Constituents; Construction; Context; Context modeling; context-cue; Data mining; Data processing. List processing. Character string processing; Databases, Factual; Exact sciences and technology; Games; Hidden Markov models; Human; Human Activities - classification; Humans; Image Processing, Computer-Assisted - methods; Male; Markov processes; Mathematical models; Memory organisation. Data processing; Models, Statistical; Pattern analysis; predictability; Predictive models; Semantics; Sequences; Software; Spatio-Temporal Analysis; Temporal logic
• ISSN: 0162-8828; 1939-3539; 2160-9292
• DOI: 10.1109/TPAMI.2013.2297321

Early prediction of ongoing human activity has become more valuable in a large variety of time-critical applications. To build an effective representation for prediction, human activities can be characterized by a complex temporal composition of constituent simple actions and interacting objects. Different from early detection on short-duration simple actions, we propose a novel framework for long -duration complex activity prediction by discovering three key aspects of activity: Causality, Context-cue, and Predictability. The major contributions of our work include: (1) a general framework is proposed to systematically address the problem of complex activity prediction by mining temporal sequence patterns; (2) probabilistic suffix tree (PST) is introduced to model causal relationships between constituent actions, where both large and small order Markov dependencies between action units are captured; (3) the context-cue, especially interactive objects information, is modeled through sequential pattern mining (SPM), where a series of action and object co-occurrence are encoded as a complex symbolic sequence; (4) we also present a predictive accumulative function (PAF) to depict the predictability of each kind of activity. The effectiveness of our approach is evaluated on two experimental scenarios with two data sets for each: action-only prediction and context-aware prediction. Our method achieves superior performance for predicting global activity classes and local action units.