CALAM: model-based compilation and linguistic statistical analysis of Urdu corpus

• Published in: Sadhana (Bangalore) 2020-12, Vol.45 (1), Article 20
• Main Authors: Choudhary, Prakash, Nain, Neeta
• Format: Article
• Language: English
• Subjects: Annotations; Engineering; Evaluation; Frequency distribution; Ground truth; Linguistics; Quantitative analysis; Statistical analysis; Viability; Words (language)
• ISSN: 0256-2499; 0973-7677
• DOI: 10.1007/s12046-019-1237-3

In this paper, we introduce an efficient framework for the compilation of an Urdu corpus along with ground truth and transcription in Unicode format. A novel scheme of the annotation based on four-level XML has been incorporated for the corpus CALAM. In addition to compilation and benchmarking test, the framework generates the word frequency distribution according to category sapient useful for linguistic evaluation. This paper presents the statistical analysis with corpus data based on transcript text and frequency of occurrences. The observation of statistical analysis is conducted using vital statistics like rank of words, the frequency of words, ligatures length (number of ligatures with combination of two to seven characters), entropy and perplexity of the corpus. Besides rudimental statistics coverage, some additional statistical features are also evaluated like Zipf’s linguistic rule and measurement of dispersion in corpus information. The experimental results obtained from statistical observation are presented for asserting viability and usability of the corpus data as a standard platform for linguistic research on the Urdu language.