An evaluation dataset for the toponym resolution task

• Published in: Computers, environment and urban systems environment and urban systems, 2006-07, Vol.30 (4), p.400-417
• Main Author: Leidner, Jochen L.
• Format: Article
• Language: English
• Subjects: Applied sciences; Buildings. Public works; Documentation. Terminology; Evaluation; Exact sciences and technology; Extensional referent disambiguation; Geo-coding; Natural language processing; Place names; Spatial grounding; Toponym resolution
• ISSN: 0198-9715; 1873-7587
• DOI: 10.1016/j.compenvurbsys.2005.07.003

Toponym resolution is the task of linking place name instances in a text with spatial footprints, given the context in which they occur. Whereas a lot of work on the evaluation of temporal resolution is ongoing (e.g. [Setzer, A., & Gaizauskas, R. (2000). On the importance of annotating temporal event–event relations in text. In LREC 2000 Workshop on annotation standards for temporal information in natural language, Vol. 3 (pp. 1281–1286). Athens, Greece]), to date no reference resource is available to evaluate competing algorithms for toponym resolution. It is thus argued that a shareable, reusable evaluation resource is necessary. To this end, a new proposal for the markup of toponyms in text corpora with their referents and an associated tool data methodology are presented: the Toponym Resolution Markup Language (TRML) is an XML-based markup language, and TAME, the toponym annotation markup editor, is a tool that implements it. A novel evaluation resource is described which comprises a large-scale reference gazetteer server and a human-annotated news corpus in which toponyms are associated with latitude/longitude coordinates of the location they refer to. The reliability of the annotation task is established by determining inter-annotator agreement of the human annotators.