Interactive sonification of images in serious games as an education aid for visually impaired children

• Published in: British journal of educational technology 2020-03, Vol.51 (2), p.473-497
• Main Authors: Radecki, Andrzej, Bujacz, Michał, Skulimowski, Piotr, Strumiłło, Paweł
• Format: Article
• Language: English
• Subjects: Acoustics; Algorithms; Assistive Technology; Audio Equipment; Blindness; Braille; Children; Color; Computer & video games; Data collection; Education; Educational Games; Educational software; Electronic devices; Elementary Education; Elementary School Students; Elementary Schools; Geometric Concepts; Geometry; Handheld Devices; Interaction; Learning; Learning Processes; Machine learning; Mathematical analysis; Mathematics; Mobile operating systems; Nonverbal Communication; Shape recognition; Software; Sound; Sound generation; Speech sounds; Students with Disabilities; Synthesis; Teaching Methods; Touch; Tracks (paths); Usability; Visual Impairments
• ISSN: 0007-1013; 1467-8535
• DOI: 10.1111/bjet.12852

The paper presents an application for interactive sonification of images intended for use on mobile devices in education of blind children at elementary school level. The paper proposes novel sonification algorithms for converting colour and grayscale images into sound. The blind user can interactively explore image content through multi‐touch gestures and select image sub‐regions for sonification with real‐time control of the synthesized sound parameters such as amplitude, frequency and timbre. Additionally, images may contain text fields read by text‐to‐speech synthesizer. The usability of one of the proposed sonification schemes is tested by collecting data on the tracking accuracy and recognition speed of basic shapes, such as lines, curves, as well as figures and simple functions. In order to facilitate the learning process for blind children, a number of games were proposed that use the devised sonification schemes. The first game—“Hear the Invisible”—is intended for two players—one child draws a shape and the task of the other one is to guess the displayed shape by means of the available sonification methods. The second proposed game is “Follow the Rabbit” and is intended for a single player who tracks a colourful “Rabbit” that runs along a path representing a given geometric shape. The obtained results show that the proposed sonification methods allow to reach previously unattainable levels of flexibility in exploration of shapes and colours. The main application of the described interactive tool is to teach geometry and mathematics in schools for blind children. The developed games are meant to enhance the learning process and motivate the children. Practitioner Notes What is known about the topic Sonification is a generic term for data‐based non‐speech sound generation. Interactive sonification is a relatively fresh area of study, where the listener heavily influences the process of sound synthesis. Earlier studies on sonification and educational computer games have shown that it is possible to demonstrate shapes such as plots and geometry using only non‐verbal sounds. What the paper adds The paper proposes an interactive sonification algorithm that can be used on greyscale or colour images using additive synthesis and the HSV colour representation. The presented software also allows to record and analyse user interaction with the images, making it a good research tool for current and future sonification algorithms. The paper presents a pilot stu