Social media text normalization for Turkish

Text normalization is an indispensable stage in processing noncanonical language from natural sources, such as speech, social media or short text messages. Research in this field is very recent and mostly on English. As is known from different areas of natural language processing, morphologically ri...

Full description

Saved in:
Bibliographic Details
Published in:Natural language engineering 2017-11, Vol.23 (6), p.835-875
Main Authors: ERYİǦİT, GÜLŞEN, TORUNOǦLU-SELAMET, DİLARA
Format: Article
Language:English
Subjects:
Artificial intelligence
Computer engineering
Data mining
Datasets
Digital media
English language
Error analysis
Messages
Natural language processing
Natural morphology
Short message service
Social media
Social networks
Statistical models
Text messaging
Turkish language
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Text normalization is an indispensable stage in processing noncanonical language from natural sources, such as speech, social media or short text messages. Research in this field is very recent and mostly on English. As is known from different areas of natural language processing, morphologically rich languages (MRLs) pose many different challenges when compared to English. Turkish is a strong representative of MRLs and has particular normalization problems that may not be easily solved by a single-stage pure statistical model. This article introduces the first work on the social media text normalization of an MRL and presents the first complete social media text normalization system for Turkish. The article conducts an in-depth analysis of the error types encountered in Web 2.0 Turkish texts, categorizes them into seven groups and provides solutions for each of them by dividing the candidate generation task into separate modules working in a cascaded architecture. For the first time in the literature, two manually normalized Web 2.0 datasets are introduced for Turkish normalization studies. The exact match scores of the overall system on the provided datasets are 70.40 per cent and 67.37 per cent (77.07 per cent with a case insensitive evaluation).
ISSN:1351-3249
1469-8110
DOI:10.1017/S1351324917000134