Synchronous spatio-temporal signature verification via Fusion Triplet Supervised Network

• Published in: Engineering applications of artificial intelligence 2023-08, Vol.123, p.106378, Article 106378
• Main Authors: Xie, Liyang, Wu, Zhongcheng, Zhang, Xian, Li, Yong
• Format: Article
• Language: English
• Subjects: Fusion Triplet Supervised Network; Person verification; Spatio-temporal signature data; The Electronic Handwritten Signature Acquisition System
• ISSN: 0952-1976
• DOI: 10.1016/j.engappai.2023.106378

Handwritten signature has been regarded as one of the most accepted and practical means of person verification since ancient times. Unlike existing methods to trace the signer only considering a single spatial or temporal information, in this paper, we propose to combine two forms of signature data (stroke images and sensor signals) to realize person verification using the novel Fusion Triplet Supervised Network (fuseTSN). First, the Electronic Handwritten Signature Acquisition System 1.0 (EHS-AS 1.0) connected to the Wacom tablet is developed to construct a synchronous static–dynamic Chinese signature dataset (SynCS) and to transform existing dynamic signature datasets into static–dynamic datasets. Then, the hybrid feature extractor (ResCNN-BiLSTM with attention) as the backbone is embedded into the fuseTSN model, mapping the images and time-series signal into a common hyperspace. Furthermore, the similarity of the input [Anchor, Positive, Test] is measured to produce the final output. Finally, the proposed fuseTSN is tested on three public datasets and also evaluated on our SynCS dataset comprised of 24000 unique triplet label samples. The results show that the accuracy of the fuseTSN model is at least 5.01% higher than that of a unimodal system with only stroke image or sensor data on the SynCS dataset and outperforms the current state-of-the-art multimodal methods for writer-independence (WI) verification on public datasets. In addition, we present the fuseTSN’s reliable judgment basis for spatial strokes in the heat-map and further find that the combination of time attributes (OA and OE) can significantly improve the model performance. •An Electronic Handwritten Signature Acquisition System is developed based on the Wacom tablet.•The Synchronous Static-dynamic Chinese Signature Database is created, called SynCS.•A novel Fusion Triplet Supervised Network is proposed to mine the potential relationship between spatio-temporal data.•The key features of spatio-temporal signals learned by the fusion feature extractor are visualized.