The Landscape and Challenges of HPC Research and LLMs

Recently, language models (LMs), especially large language models (LLMs), have revolutionized the field of deep learning. Both encoder-decoder models and prompt-based techniques have shown immense potential for natural language processing and code-based tasks. Over the past several years, many resea...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2024-02
Main Authors: Chen, Le, Ahmed, Nesreen K, Dutta, Akash, Bhattacharjee, Arijit, Yu, Sixing, Quazi Ishtiaque Mahmud, Abebe, Waqwoya, Phan, Hung, Sarkar, Aishwarya, Butler, Branden, Hasabnis, Niranjan, Gal Oren, Vo, Vy A, Juan Pablo Munoz, Willke, Theodore L, Mattson, Tim, Jannesari, Ali
Format: Article
Language:English
Subjects:
Computation
Encoders-Decoders
Large language models
Natural language processing
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Recently, language models (LMs), especially large language models (LLMs), have revolutionized the field of deep learning. Both encoder-decoder models and prompt-based techniques have shown immense potential for natural language processing and code-based tasks. Over the past several years, many research labs and institutions have invested heavily in high-performance computing, approaching or breaching exascale performance levels. In this paper, we posit that adapting and utilizing such language model-based techniques for tasks in high-performance computing (HPC) would be very beneficial. This study presents our reasoning behind the aforementioned position and highlights how existing ideas can be improved and adapted for HPC tasks.
ISSN:2331-8422