Partially-Supervised Metric Learning via Dimensionality Reduction of Text Embeddings Using Transformer Encoders and Attention Mechanisms

• Published in: IEEE access 2024, Vol.12, p.77536-77554
• Main Authors: Hodgson, Ryan, Wang, Jingyun, Cristea, Alexandra I., Graham, John
• Format: Article
• Language: English
• Subjects: Accuracy; Algorithms; attention mechanisms; Clustering; Clustering algorithms; Coders; Datasets; Dimensionality reduction; Machine learning; Measurement; metric learning; Natural language processing; Neural networks; Recurrent neural networks; Supervised learning; Task analysis; transformer networks; Transformers
• ISSN: 2169-3536; 2169-3536
• DOI: 10.1109/ACCESS.2024.3403991

Real-world applications of word embeddings to downstream clustering tasks may experience limitations to performance, due to the high degree of dimensionality of the embeddings. In particular, clustering algorithms do not scale well when applied to highly dimensional data. One method to address this is through the use of dimensionality reduction algorithms (DRA). Current state of the art algorithms for dimensionality reduction (DR) have been demonstrated to contribute to improvements in clustering accuracy and performance. However, the impact that a neural network architecture can have on the current state of the art Parametric Uniform Manifold Approximation and Projection (UMAP) algorithm is yet unexplored. This work investigates, for the first time, the effects of using attention mechanisms in neural networks for Parametric UMAP, through the application of network architectures that have had considerable effect upon the wider machine learning and natural language processing (NLP) fields - namely, the transformer-encoder, and the bidirectional recurrent neural network. We implement these architectures within a semi-supervised metric learning pipeline, with results demonstrating an improvement in the clustering accuracy, compared to conventional DRA techniques, on three out of four datasets, and comparable SoA accuracy on the fourth. To further support our analysis, we also investigate the effects of the transformer-encoder metric-learning pipeline upon the individual class accuracy of downstream clustering, for highly imbalanced datasets. Our analyses indicate that the proposed pipeline with transformer-encoder for parametric UMAP confers a significantly measurable benefit to the accuracy of underrepresented classes.