CALAM: model-based compilation and linguistic statistical analysis of Urdu corpus

In this paper, we introduce an efficient framework for the compilation of an Urdu corpus along with ground truth and transcription in Unicode format. A novel scheme of the annotation based on four-level XML has been incorporated for the corpus CALAM. In addition to compilation and benchmarking test,...

Full description

Saved in:
Bibliographic Details
Published in:Sadhana (Bangalore) 2020-12, Vol.45 (1), Article 20
Main Authors: Choudhary, Prakash, Nain, Neeta
Format: Article
Language:English
Subjects:
Annotations
Engineering
Evaluation
Frequency distribution
Ground truth
Linguistics
Quantitative analysis
Statistical analysis
Viability
Words (language)
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In this paper, we introduce an efficient framework for the compilation of an Urdu corpus along with ground truth and transcription in Unicode format. A novel scheme of the annotation based on four-level XML has been incorporated for the corpus CALAM. In addition to compilation and benchmarking test, the framework generates the word frequency distribution according to category sapient useful for linguistic evaluation. This paper presents the statistical analysis with corpus data based on transcript text and frequency of occurrences. The observation of statistical analysis is conducted using vital statistics like rank of words, the frequency of words, ligatures length (number of ligatures with combination of two to seven characters), entropy and perplexity of the corpus. Besides rudimental statistics coverage, some additional statistical features are also evaluated like Zipf’s linguistic rule and measurement of dispersion in corpus information. The experimental results obtained from statistical observation are presented for asserting viability and usability of the corpus data as a standard platform for linguistic research on the Urdu language.
ISSN:0256-2499
0973-7677
DOI:10.1007/s12046-019-1237-3