Language Models as a Service: Overview of a New Paradigm and its Challenges

Some of the most powerful language models currently are proprietary systems, accessible only via (typically restrictive) web or software programming interfaces. This is the Language-Models-as-a-Service (LMaaS) paradigm. In contrast with scenarios where full model access is available, as in the case...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2023-11
Main Authors: Emanuele La Malfa, Petrov, Aleksandar, Frieder, Simon, Weinhuber, Christoph, Burnell, Ryan, Raza Nazar, Cohn, Anthony G, Shadbolt, Nigel, Wooldridge, Michael
Format: Article
Language:English
Subjects:
Accessibility
Language
User interfaces
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Some of the most powerful language models currently are proprietary systems, accessible only via (typically restrictive) web or software programming interfaces. This is the Language-Models-as-a-Service (LMaaS) paradigm. In contrast with scenarios where full model access is available, as in the case of open-source models, such closed-off language models present specific challenges for evaluating, benchmarking, and testing them. This paper has two goals: on the one hand, we delineate how the aforementioned challenges act as impediments to the accessibility, replicability, reliability, and trustworthiness of LMaaS. We systematically examine the issues that arise from a lack of information about language models for each of these four aspects. We conduct a detailed analysis of existing solutions and put forth a number of considered recommendations, and highlight the directions for future advancements. On the other hand, it serves as a comprehensive resource for existing knowledge on current, major LMaaS, offering a synthesized overview of the licences and capabilities their interfaces offer.
ISSN:2331-8422