On Robustness and Sensitivity of a Neural Language Model: A Case Study on Italian L1 Learner Errors

In this paper, we propose a comprehensive linguistic study aimed at assessing the implicit behavior of one of the most prominent Neural Language Models (NLM) based on Transformer architectures, BERT Devlin et al., when dealing with a particular source of noisy data, namely essays written by L1 Itali...

Full description

Saved in:
Bibliographic Details
Published in:IEEE/ACM transactions on audio, speech, and language processing speech, and language processing, 2023, Vol.31, p.426-438
Main Authors: Miaschi, Alessio, Brunato, Dominique, Dell'Orletta, Felice, Venturi, Giulia
Format: Article
Language:English
Subjects:
Case studies
Evaluation
Grammar lexicon relationship
Hierarchies
interpretability
Italian language
Language modeling
learner errors
Linguistics
Morphosyntax
NLP
Noise measurement
Orthography
Probes
Representations
Robustness
Semantic processing
Semantics
Sensitivity
Sensitivity analysis
Sentences
Similarity
Task analysis
transformers
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In this paper, we propose a comprehensive linguistic study aimed at assessing the implicit behavior of one of the most prominent Neural Language Models (NLM) based on Transformer architectures, BERT Devlin et al., when dealing with a particular source of noisy data, namely essays written by L1 Italian learners containing a variety of errors targeting grammar, orthography and lexicon. Differently from previous works, we focus on the pre-training stage and we devise two complementary evaluation tasks aimed at assessing the impact of errors on sentence-level inner representations in terms of semantic robustness and linguistic sensitivity. While the first evaluation perspective is meant to probe the model's ability to encode the semantic similarity between sentences also in the presence of errors, the second type of probing task evaluates the influence of errors on BERT's implicit knowledge of a set of raw and morpho-syntactic properties of a sentence. Our experiments show that BERT's ability to compute sentence similarity and to correctly encode multi-leveled linguistic information of a sentence are differently modulated by the category of errors and that the error hierarchies in terms of robustness and sensitivity change across layer-wise representations.
ISSN:2329-9290
2329-9304
DOI:10.1109/TASLP.2022.3226333