CoSimLex: A Resource for Evaluating Graded Word Similarity in Context

State of the art natural language processing tools are built on context-dependent word embeddings, but no direct method for evaluating these representations currently exists. Standard tasks and datasets for intrinsic evaluation of embeddings are based on judgements of similarity, but ignore context;...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2020-10
Main Authors: Carlos Santos Armendariz, Purver, Matthew, Ulčar, Matej, Pollak, Senja, Ljubešić, Nikola, Robnik-Šikonja, Marko, Granroth-Wilding, Mark, Vaik, Kristiina
Format: Article
Language:English
Subjects:
Datasets
Evaluation
Natural language processing
Similarity
Word sense disambiguation
Words (language)
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:State of the art natural language processing tools are built on context-dependent word embeddings, but no direct method for evaluating these representations currently exists. Standard tasks and datasets for intrinsic evaluation of embeddings are based on judgements of similarity, but ignore context; standard tasks for word sense disambiguation take account of context but do not provide continuous measures of meaning similarity. This paper describes an effort to build a new dataset, CoSimLex, intended to fill this gap. Building on the standard pairwise similarity task of SimLex-999, it provides context-dependent similarity measures; covers not only discrete differences in word sense but more subtle, graded changes in meaning; and covers not only a well-resourced language (English) but a number of less-resourced languages. We define the task and evaluation metrics, outline the dataset collection methodology, and describe the status of the dataset so far.
ISSN:2331-8422