Comparing and evaluating the effectiveness of mobile Web adequacy evaluation tools

• Published in: Universal access in the information society 2017-06, Vol.16 (2), p.411-424
• Main Authors: Kaur, Arvinder, Dani, Diksha
• Format: Article
• Language: English
• Subjects: Accessibility; Adequacy; Automation; Best practice; Completeness; Computer Communication Networks; Computer Science; Computers and Society; Electronic commerce; Electronic devices; Evaluation; Guidelines; Hand tools; Information Storage and Retrieval; Information Systems Applications (incl.Internet); IT in Business; Long Paper; Page layout; Statistical analysis; User Interfaces and Human Computer Interaction; Websites
• ISSN: 1615-5289; 1615-5297
• DOI: 10.1007/s10209-016-0466-z

The number of Web sites is growing exponentially and so are the people who are accessing them on mobile devices including people with special abilities. Mobile-friendly Web sites further increase traffic to the Web site, customer participation, and boost e-Commerce. In such a scenario, it is important to ensure that the Web sites are accessible on mobile devices for all users. Automatic tools provide an easy and a less error-prone mechanism for evaluating the mobile adequacy of these Web sites. The present study focuses primarily on the comparison between four tools of mobile Web adequacy evaluation, namely mobileOK checker, taw, EvalAcess Mobile, and mobiReady. The comparison is made based on three parameters, i.e., correctness, completeness, coverage with respect to the conformance of Mobile Web Best Practices (MWBP), and four groups of mobile Web best practice guidelines, i.e., Navigability and Links, Page Layout and Content, Page Definition, and User Input. Intrareliability of the tools and inter-reliability between the tools are also investigated. The improvement required in MWBP and the role of various mobile platforms in mobile adequacy today are also discussed. The study was conducted using automatic tools, manual evaluation, and statistical tools. The results of the study indicate that the coverage of the automated tools was found to lie between 45 and 68 %. The completeness ranged from 14 % (by MobileOK) to 59 % (by EvalAccess mobile). The correctness of the tools ranged from 42 to 51 %. Statistical analysis highlighted that the tools showed no significant differences in reporting false positives, false negatives and true positives, except taw and EvalAccess which reported some difference in reporting true positives. The intrareliability evaluation showed that the tools were highly stable. Intrareliability evaluation done by Krippendorff’s alpha revealed that taw and mobileOK show the similar results while evaluating the MWBP guidelines in most cases. However, for more accurate evaluations manual evaluation along with automatic evaluation must be performed. The practical significance of such a study can lead to improving the guidelines, modification of the tools, proactive use of the tools, and an overall awareness about building the Web sites accessible for all.