PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedback

Large Language Models for Code (Code LLM) are flourishing. New and powerful models are released on a weekly basis, demonstrating remarkable performance on the code generation task. Various approaches have been proposed to boost the code generation performance of pre-trained Code LLMs, such as superv...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2023-07
Main Authors: Shen, Bo, Zhang, Jiaxin, Chen, Taihong, Daoguang Zan, Geng, Bing, Fu, An, Zeng, Muhan, Yu, Ailun, Ji, Jichuan, Zhao, Jingyang, Guo, Yuenan, Wang, Qianxiang
Format: Article
Language:English
Subjects:
Benchmarks
Feedback
Large language models
Machine learning
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Large Language Models for Code (Code LLM) are flourishing. New and powerful models are released on a weekly basis, demonstrating remarkable performance on the code generation task. Various approaches have been proposed to boost the code generation performance of pre-trained Code LLMs, such as supervised fine-tuning, instruction tuning, reinforcement learning, etc. In this paper, we propose a novel RRTF (Rank Responses to align Test&Teacher Feedback) framework, which can effectively and efficiently boost pre-trained large language models for code generation. Under this framework, we present PanGu-Coder2, which achieves 62.20% pass@1 on the OpenAI HumanEval benchmark. Furthermore, through an extensive evaluation on CoderEval and LeetCode benchmarks, we show that PanGu-Coder2 consistently outperforms all previous Code LLMs.
ISSN:2331-8422