Video-based bird posture recognition using dual feature-rates deep fusion convolutional neural network

• Published in: Ecological indicators 2022-08, Vol.141, p.109141, Article 109141
• Main Authors: Lin, Chih-Wei, Chen, Zhongsheng, Lin, Mengxiang
• Format: Article
• Language: English
• Subjects: 00–01; 99–00; Behavior rate; Bird behavior recognition; birds; Convolutional neural network; data collection; habitats; Information fusion; medical treatment; neural networks; posture; remediation
• ISSN: 1470-160X; 1872-7034
• DOI: 10.1016/j.ecolind.2022.109141

[Display omitted] •The first study focuses on multiple behavior recognition of different birds based on the video stream.•The TNL module generates features with various rates and uses a transpose mechanism to enhance features.•DF2-Net transforms features’ rates and iteratively fuses them to obtain the relationship of behavior at various rates.•We collected the new bird postures (behaviors) dataset based on videos that contain eight types of postures. It is necessary to detect changes in birds’ behaviors promptly to realize their health and habitat status. Moreover, promptly provide appropriate medical treatment and environmental remediation. Automating bird behavior recognition can solve this problem and assist in breeding and protecting birds. This paper proposes the transposed non-local module based on a time pyramid network to establish a dual feature-rates deep fusion net (DF2-Net) for bird behavior recognition. The time pyramid network uses spatial alignment and time pooling operations to extract features with different rates from features of different depths and then fuse features containing the information. On this basis, the transposed non-local (TNL) module uses features with different rates to calculate the relationship matrix of each time slice and spatial position. Then TNL uses the transpose operation to make the original feature take advantage of the corresponding relationship when it multiplies the original feature by the matrix. The module further deeply integrates the relational information of behaviors at different rates to enhance features of different rates, respectively, which can improve the recognition effect of the model on dynamic behaviors. Our study has three contributions: (1) The TNL module takes features with different rates as inputs and uses a transpose mechanism to get two relationships in different directions that match their corresponding input. (2) Dual feature-rates deep fusion net (DF2-Net) transforms a single rate feature into two rate features and iteratively fuses them to obtain information and behavior relationships at various rates. (3) We collect the unique video dataset of birds’ behaviors to fill in the blank of the video dataset and support the study of birds’ behaviors. The experiments compare the DF2-Net with the well-known video-based recognition model on the self-collected birds’ behavior dataset, containing eight behaviors. DF2-Net has the best classification accuracy and achieves 80.87%, 81.35%, 80.70%,