Pre-Trained Language Models and Their Applications

[Display omitted] Pre-trained language models have achieved striking success in natural language processing (NLP), leading to a paradigm shift from supervised learning to pre-training followed by fine-tuning. The NLP community has witnessed a surge of research interest in improving pre-trained model...

Full description

Saved in:
Bibliographic Details
Published in:Engineering (Beijing, China) China), 2023-06, Vol.25 (6), p.51-65
Main Authors: Wang, Haifeng, Li, Jiwei, Wu, Hua, Hovy, Eduard, Sun, Yu
Format: Article
Language:English
Subjects:
Natural language processing
Pre-trained models
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:[Display omitted] Pre-trained language models have achieved striking success in natural language processing (NLP), leading to a paradigm shift from supervised learning to pre-training followed by fine-tuning. The NLP community has witnessed a surge of research interest in improving pre-trained models. This article presents a comprehensive review of representative work and recent progress in the NLP field and introduces the taxonomy of pre-trained models. We first give a brief introduction of pre-trained models, followed by characteristic methods and frameworks. We then introduce and analyze the impact and challenges of pre-trained models and their downstream applications. Finally, we briefly conclude and address future research directions in this field.
ISSN:2095-8099
DOI:10.1016/j.eng.2022.04.024