Converting heterogeneous statistical tables on the web to searchable databases

• Published in: International journal on document analysis and recognition 2016-06, Vol.19 (2), p.119-138
• Main Authors: Embley, David W., Krishnamoorthy, Mukkai S., Nagy, George, Seth, Sharad
• Format: Article
• Language: English
• Subjects: Computer Science; Image Processing and Computer Vision; Original Paper; Pattern Recognition
• ISSN: 1433-2833; 1433-2825
• DOI: 10.1007/s10032-016-0259-1

Much of the world’s quantitative data reside in scattered web tables. For a meaningful role in Big Data analytics, the facts reported in these tables must be brought into a uniform framework. Based on a formalization of header-indexed tables, we proffer an algorithmic solution to end-to-end table processing for a large class of human-readable tables. The proposed algorithms transform header-indexed tables to a category table format that maps easily to a variety of industry-standard data stores for query processing. The algorithms segment table regions based on the unique indexing of the data region by header paths, classify table cells, and factor header category structures of two-dimensional as well as the less common multidimensional tables. Experimental evaluations substantiate the algorithmic approach to processing heterogeneous tables. As demonstrable results, the algorithms generate queryable relational database tables and semantic-web triple stores. Application of our algorithms to 400 web tables randomly selected from diverse sources shows that the algorithmic solution automates end-to-end table processing.