Multi-Modal Deep Learning-Based Violin Bowing Action Recognition

In this paper, a deep learning-based violin action recognition is proposed. By fusing the sensing signals from depth camera modality and inertial sensor modalities, violin bowing actions can be recognized by the proposed deep learning scheme. The actions performed by a violinist are captured by a de...

Full description

Saved in:
Bibliographic Details
Main Authors: Liu, Bao-Yun, Jen, Yi-Hsin, Sun, Shih-Wei, Su, Li, Chang, Pao-Chi
Format: Conference Proceeding
Language:English
Subjects:
Accelerometers
Cameras
Deep learning
Sensors
Skeleton
Three-dimensional displays
Wearable sensors
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In this paper, a deep learning-based violin action recognition is proposed. By fusing the sensing signals from depth camera modality and inertial sensor modalities, violin bowing actions can be recognized by the proposed deep learning scheme. The actions performed by a violinist are captured by a depth camera, and recorded by wearable sensors on the forearm of a violinist. In the proposed system, 3D convolution neural network (3D-CNN) and long short-term memory (LSTM) deep learning algorithms are adopted to generate the action models from depth camera modality and inertial sensor modalities. The features and models obtained from multi-modalities are used to classify different violin bowing actions. A fusion process from different modalities can achieve satisfactory recognition accuracy. In this paper, we generate a violin bowing actions dataset for the preliminary study and the system performance evaluation.
ISSN:2575-8284
DOI:10.1109/ICCE-Taiwan49838.2020.9257995