New method for modeling digital twin behavior perception of cows: Cow daily behavior recognition based on multimodal data

• Published in: Computers and electronics in agriculture 2024-11, Vol.226, p.109426, Article 109426
• Main Authors: Zhang, Yi, Zhang, Yu, Jiang, Hailong, Du, Haitao, Xue, Aidi, Shen, Weizheng
• Format: Article
• Language: English
• Subjects: Behavior recognition; Dairy cow; Digital twin; Multimodal classification; Transformer
• ISSN: 0168-1699
• DOI: 10.1016/j.compag.2024.109426

•Integrates video and sensor data to improve cow behavior recognition.•Utilizes different modal data to enhance model performance.•Utilizes existing farm sensor and video data to enhance data utilization.•Enhances modeling elements in the digital twin architecture for dairy cows. The cow digital shadow reflects the behavior, health condition, and productivity of cows, playing a crucial role in ensuring animal welfare, increasing individual productivity, and improving breeding efficiency. To fully utilize the existing multimodal data on farms and build a cow digital shadow with rich behavioral information, this study proposes a multimodal data fusion algorithm for recognizing cow behaviors such as drinking, feeding, lying, standing, and walking. This algorithm leverages the strengths of different data modalities, complementing each other, and enhances the performance of the cow behavior classification model. The algorithm integrates motion sensor and video data, collected by custom-made collars with inertial measurement units (IMUs) sensors placed at the top of the cow’s neck and cameras in the barn, using EfficientNet V2 S, BiLSTM, and Transformer networks. Experimental results demonstrate recognition accuracies of 98.80 %, precision of 97.15 %, and recall rates of 96.93 %, showing significant improvements over single-modal data behavior recognition algorithms. This method maximizes the utility of existing multimodal data to generate a cow digital shadow with detailed behavioral information, enhancing the modeling and simulation element of the cow digital twin architecture and laying the foundation for developing a comprehensive cow behavior data model.