Crowdsourcing as a tool in the clinical assessment of intelligibility in dysarthria: How to deal with excessive variation

• Published in: Journal of communication disorders 2021-09, Vol.93, p.106135-106135, Article 106135
• Main Authors: Ziegler, Wolfram, Lehner, Katharina, Klonowski, Madleen, Geißler, Nadine, Ammer, Franziska, Kurfeß, Christina, Grötzbach, Holger, Mandl, Alexander, Knorr, Felicitas, Strecker, Katrin, Schölderle, Theresa, Matern, Sina, Weck, Christiane, Gröne, Berthold, Brühl, Stefanie, Kirchner, Christiane, Kleiter, Ingo, Sühn, Ursula, von Eichmann, Joachim, Möhrle, Christina, Spencer, Pete Guy, Ilg, Rüdiger, Klintwort, Doris, Lubecki, Daniel, Marinho, Steffy, Hogrefe, Katharina
• Format: Article
• Language: English
• Subjects: Crowdsourcing; Dysarthria; Intelligibility; Quality control; Validation
• ISSN: 0021-9924; 1873-7994
• DOI: 10.1016/j.jcomdis.2021.106135

•Involvement of laypersons in clinical intelligibility assessment is needed.•Crowdsourcing is a way to involve laypersons in clinical dysarthria assessment.•Excessive variability of crowd scores is constrained by weighted aggregation.•Cost-benefit considerations suggest panels of 9 listeners.•The proposed method immunizes crowd-based intelligibility scores against spamming. Independent laypersons are essential in the assessment of intelligibility in persons with dysarthria (PWD), as they reflect intelligibility limitations in the most ecologically valid way, without being influenced by familiarity with the speaker. The present work investigated online crowdsourcing as a convenient method to involve lay people as listeners, with the objective of exploring how to constrain the expected variability of crowd-based judgements to make them applicable in clinical diagnostics. Intelligibility was assessed using a word transcription task administered via crowdsourcing. In study 1, speech samples of 23 PWD were transcribed by 18 crowdworkers each. Four methods of aggregating the intelligibility scores of randomly sampled panels of 4 to 14 listeners were compared for accuracy, i.e. the stability of the resulting intelligibility estimates across different panels, and their validity, i.e. the degree to which they matched data obtained under controlled laboratory conditions (“gold standard”). In addition, we determined an economically acceptable number of crowdworkers per speaker which is needed to obtain accurate and valid intelligibility estimates. Study 2 examined the robustness of the chosen aggregation method against downward outliers due to spamming in a larger sample of 100 PWD. In study 1, an interworker aggregation method based on negative exponential weightings of the scores as a function of their distance from the “best” listener's score (exponentially weighted mean) outperformed three other methods (median value, arithmetic mean, maximum). Under cost-benefit considerations, an optimum panel size of 9 crowd listeners per examination was determined. Study 2 demonstrated the robustness of this aggregation method against spamming crowd listeners. Though intelligibility data collected through online crowdsourcing are noisy, accurate and valid intelligibility estimates can be obtained by appropriate aggregation of the raw data. This makes crowdsourcing a suitable method for incorporating real-world perspectives into clinical dysarthria assessment.