Can Crowdworkers be used to Rate the Quality of Ultrasound Images?

The present study examined whether inexperienced individuals can evaluate ultrasonography (US) images using a new rating scale and visual aids. A US image assessment tool and visual aids in the form of alignable comparison standards (one good and one poor example) were developed to rate images of th...

Full description

Saved in:
Bibliographic Details
Published in:Proceedings of the Human Factors and Ergonomics Society Annual Meeting 2018-09, Vol.62 (1), p.595-599
Main Authors: Ashdown, Amanda, Scerbo, Mark W., Knapp, Barry J., Toreno, Felicia, Goodmurphy, Craig, Byars, Don
Format: Article
Language:English
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The present study examined whether inexperienced individuals can evaluate ultrasonography (US) images using a new rating scale and visual aids. A US image assessment tool and visual aids in the form of alignable comparison standards (one good and one poor example) were developed to rate images of the hepatorenal interface along eight characteristics. Twenty novice participants from Amazon.com’s Mechanical Turk service used the visual aids to evaluate 42 US images in good and poor categories. The results revealed that crowdworkers could distinguish the good from poor US images. Their ratings were highly and positively correlated with those of experts. However, the crowdworkers’ ratings were more severe than those of the experts. These results suggest that inexperienced individuals can use a rating scale and visual aids to reliably distinguish good from poor US images but may require a more extensive set of visual cues or additional training to match ratings provide by experts.
ISSN:1071-1813
2169-5067
DOI:10.1177/1541931218621136