Accessibility of mobile apps for visually impaired users: problems encountered by user evaluation, inspections, and automated tools

Abstract

Mobile apps have grown considerably, supporting many everyday activities such as news reading, social media, and online banking. Mobile apps should be usable and accessible to everyone, including visually impaired people. Accessibility evaluation techniques are essential to identify accessibility problems and help to avoid problems that hinder access by people with disabilities. Different accessibility evaluation methods may be used in different stages of software development. These methods include approaches such as employing automated tools to detect specifc issues in the source code and more comprehensive methods, such as user evaluation and expert inspections. Expert inspections and automated tools can help detect problems earlier in the development process, though they cannot cover all the problems visually impaired users encounter. However, knowing the types of accessibility problems encountered by each method is vital for developers, testers, and designers to allocate the different methods appropriately in different stages of the development cycle. This project aimed to compare the outcomes of automated evaluations, manual inspection and user evaluation methods applied to mobile apps focusing on visually impaired users. The study compared the results from different methods by comparing results from a systematic mapping of the literature and the analysis of user evaluations performed in a previous study with two mobile apps. The study compared the results from the usability evaluations of Saraiva and Receita Federal by nine users with visual impairment and 189 instances of problems in two applications. These results were compared to the automated evaluation of the two apps by the tools MATE (Mobile Accessibility Testing) and Accessibility Scanner. Next, the apps were inspected by two groups of professionals: 17 experts in different areas of software development (full-stack developers, testers and front-end developers) and ten specialists in Human-Computer Interaction (HCI) with previous experience with accessibility. The results showed a difference between the different accessibility evaluation methods. Inspections by HCI specialists and tests with users with disabilities found a greater diversity of types of problems. Automated tools showed limited performance in the detection of types of problems. In accessibility inspections, There is a difference between specialists in software development and specialists in HCI. The results showed a difference in the number of violations that specialists in software development fnd compared to HCI specialists. This study contributes to the understanding of how different methods can contribute to the accessibility evaluation of mobile apps, helping different stakeholders make decisions about when and how to apply different methods.