Usability evaluation with screen reader users: a video presentation of the PCTA’s experimental setting and rules

Introduction Usability evaluation is essential to make sure that software products newly released are easy to use, efficient, and effective to reach goals, and satisfactory to users. For example, when a software company wants to develop and sell a new product, the company needs to evaluate usability of the new product before launching it at a market to avoid the possibility that the new product may contain usability problems, which span from cosmetic problems to severe functional problems. Three widely used methods for usability evaluation are Think Aloud (TA), Heuristic Evaluation (HE) and Cognitive Walkthrough (CW). TA method is commonly employed with a lab-based user testing, while there are variants of TA methods, including thinking out aloud at user's workplace instead of at labs. What we discuss here is the TA method that is combined with a lab-based user testing, in which test users use products while simultaneously and continuously thinking out aloud, and experimenters record users' behaviors and verbal protocols in the laboratory. HE is a usability inspection method, in which a small number of evaluators find usability problems in a user interface design by examining an interface and judging its compliance with well-known usability principles, called heuristics. CW is a theory-based method, in which evaluators evaluate every step necessary to perform a scenario-based task, and look for usability problems that would interfere with learning by exploration. These three methods have their own advantages and disadvantages. For instance, TA method provides good qualitative data from a small number of test users, but laboratory environment may influence test user's behaviors. HE is a cheap, fast and easy-to-use method, while it often finds too specific and low-priority usability problems, including even not real problems. CW helps find mismatches between users' and designers' conceptualization of a task, but it needs extensive knowledge of cognitive psychology and technical details to apply. However, even though these advantages and disadvantages show overall characteristics of three major usability evaluation methods, we cannot compare them quantitatively and see their efficiency clearly. Because one of reasons why so-called discounted methods, such as HE and CW, were developed is to save costs of usability evaluation, cost-related criteria for comparing usability evaluation are meaningful to usability practitioners as well as usability researchers. One of the most disputable issues related to cost of usability evaluation is sample size. That is, how many users or evaluators are needed to achieve a targeted usability evaluation performance, for example, 80% of overall discovery rate? The sample size of usability evaluation is known to depend on an estimate of problem discovery rate across participants. The overall discovery rate is a common quantitative measure that is used to show the effectiveness of a specific usability evaluation method in most of usability evaluation studies. It is also called overall detection rate or thoroughness measure, which is the ratio of 'the sum of unique usability problems detected by all experiment participants' against 'the number of usability problems that exist in the evaluated systems', ranging between 0 and 1. The overall discovery rates were reported more than any other criterion measure in the usability evaluation experiments and also a key component for projecting required sample size for usability evaluation study. Thus, how many test users or evaluators participate in the usability evaluation is a critical issue, considering its cost-effectiveness.

Determination of the effectiveness of two methods for usability evaluation using a CPOE medication ordering system

BackgroundUsability is a key factor affecting the acceptance of mobile health applications (mHealth apps) for elderly individuals, but traditional usability evaluation methods may not be suitable for use in this population because of aging barriers. The objectives of this study were to identify, explore, and summarize the current state of the literature on the usability evaluation of mHealth apps for older adults and to incorporate these methods into the appropriate evaluation stage.MethodsElectronic searches were conducted in 10 databases. Inclusion criteria were articles focused on the usability evaluation of mHealth apps designed for older adults. The included studies were classified according to the mHealth app usability evaluation framework, and the suitability of evaluation methods for use among the elderly was analyzed.ResultsNinety-six articles met the inclusion criteria. Research activity increased steeply after 2013 (n = 92). Satisfaction (n = 74) and learnability (n = 60) were the most frequently evaluated critical measures, while memorability (n = 13) was the least evaluated. The ratios of satisfaction, learnability, operability, and understandability measures were significantly related to the different stages of evaluation (P < 0.05). The methods used for usability evaluation were questionnaire (n = 68), interview (n = 36), concurrent thinking aloud (n = 25), performance metrics (n = 25), behavioral observation log (n = 14), screen recording (n = 3), eye tracking (n = 1), retrospective thinking aloud (n = 1), and feedback log (n = 1). Thirty-two studies developed their own evaluation tool to assess unique design features for elderly individuals.ConclusionIn the past five years, the number of studies in the field of usability evaluation of mHealth apps for the elderly has increased rapidly. The mHealth apps are often used as an auxiliary means of self-management to help the elderly manage their wellness and disease. According to the three stages of the mHealth app usability evaluation framework, the critical measures and evaluation methods are inconsistent. Future research should focus on selecting specific critical measures relevant to aging characteristics and adapting usability evaluation methods to elderly individuals by improving traditional tools, introducing automated evaluation tools and optimizing evaluation processes.

Participatory research methods are being used internationally to gather data on complex social, cultural, and political concerns that effect the use of technology [4]. Researchers have found it difficult to include people with disabilities in these studies [5, 6, 7]. The Accessible Learning Through Text-to-Speech Project will utilize online focus groups as a method of integrating people with disabilities into a participatory research project. The Alt-Learning Project will have three primary target populations; users of screen readers with vision, users of screen readers who are blind, and professionals responsible for the delivery of assistive technology. The online focus groups will allow the observation and collection of data as a participant would normally utilize their screen reader applications at home, school, or workplace.

This paper describes the interactive software system development process focused on blind people making use of the user experience evaluation methodology focused on blind people (MEUXuc). What led us to carry out this work is that based on the related works there are no ISSs that specify how they were evaluated and whether or not they are usable for the target user, in cases where the blind user is involved (one user). The results are not optimal due to the type of evaluation used, in turn, it is observed that there are few systems that focus on learning mathematics and that at the same time are inclusive and adapt to blind users. 15 blind users participated in this work to carry out the tests, between 18 and 52 years old, of both sexes, all with minimal knowledge in the use of a mobile device. The tests that were performed were: low-level prototype analysis, semi-closed Card Sorting, Thinking Aloud, HED/UT Scale and PANAS. As a result, it is observed that by making use of a methodology focused on blind users, most of the factors referred to by the user experience are met, such as: usability, accessibility, utility, among others. In this way, the investigation and realization of this interactive software systems allows to conclude that blind people are able to perform various tests as long as the methods are adapted to their needs to be able to evaluate systems that are focused on them, thus making the systems more accessible, useful, usable, valuable, among others.

Aim: To measure the effect of thinking aloud on perceptual accuracy and visual search behavior during chest radiograph interpretation for pulmonary nodules. Background: Thinking Aloud (TA) is an empirical research method used by researchers in cognitive psychology and behavioural analysis. In this pilot study we wanted to examine whether TA had an effect on the perceptual accuracy and search patterns of subjects looking for pulmonary nodules on adult posterioranterior chest radiographs (PA CxR). Method: Seven academics within Medical Radiation Sciences at The University of Sydney participated in two reading sessions with and without TA. Their task was to localize pulmonary nodules on 30 PA CxR using mouse clicks and rank their confidence levels of nodule presence. Eye-tracking recordings were collected during both viewing sessions. Time to first fixation, duration of first fixation, number of fixations, cumulative time of fixation and total viewing time were analysed. In addition, ROC analysis was conducted on collected outcome using DBM methodology. Results: Time to first nodule fixation was significantly longer (p=0.001) and duration of first fixation was significantly shorter (p=0.043). No significant difference was observed in ROC AUC scores between control and TA conditions. Conclusion: Our results confirm that TA has little effect on perceptual ability or performance, except for prolonging the task. However, there were significant differences in visual search behavior. Future researchers in radio-diagnosis could use the think aloud condition rather than silence so as to more closely replicate the clinical scenario.

Think Aloud (TA) studies provide rich verbal data about reasoning during a problem solving task. Using TA and protocol analysis, investigators can identify the information that is concentrated on during problem solving and how that information is used to facilitate problem resolution. From this, inferences can be made about the reasoning processes that were used during the problem-solving task. In the past, the validity of data obtained from TA studies has been suspect because of inconsistencies in data collection and the inability to verify findings obtained from the slow, laborious process of protocol analysis. This article describes a means of obtaining more accurate verbal data and analyzing it in a standardized step-by-step manner.

The COVID-19 pandemic exacerbated the lack of accessible tech skills training for screen reader users. To address this gap, we developed remote tools and techniques to support screen reader users in learning web development. In this paper, we describe our design and implementation of a free, remote web development workshop, designed to introduce 12 screen reader users to HTML, CSS, and JavaScript. We taught the workshop remotely using synchronous lectures followed by 1-on-1 time with teaching assistants (TAs), and included a resource-rich website, tactile diagrams, and discussion forum. We evaluated the impact of the workshop, as well as techniques and tools using pre- and post-workshop surveys, observational field notes, exit interviews, and unit quizzes. At the end of the workshop, all students demonstrated their knowledge of web development basics by creating and publishing their own websites; showed an increase in self-efficacy; and maintained a high level of interest in the subject. Participating in this workshop benefited our TAs who reported increased confidence in understanding accessibility concepts, increased interest in pursuing work related to accessibility, and plans to apply what they learned. Based on these findings, we recommend techniques and tools to support remote screen reader users learning web development; and suggestions for engaging with post-secondary institutions to pair service learning with tech skills training. We close with recommendations for implementing and adapting the workshop using our open-educational materials.

Little is known about the navigation tactics employed by screen reader users when they face problematic situations on the Web. Understanding how these tactics are operationalised and knowing the situations that bring about such tactics paves the way towards modeling navigation behaviour. Modeling the navigation of users is of utmost importance as it allows not only to predict interactive behaviour, but also to assess the appropriateness of the content in a link, the information architecture of a site and the design of a web page. Current navigation models do not consider the extreme adaptations, namely coping tactics, that screen reader users undergo on the Web. Consequently, their prediction power is lessened and coping tactics are mistakenly considered outlying behaviours. We draw from existing navigation models for sighted users to suggest the incorporation of emerging behaviours in navigation models for screen reader users. To do so, we identify the navigation coping tactics screen reader users exhibit on the Web, including deliberately clicking on low scented links, escaping from useless or inaccessible content and backtracking to a shelter. Our findings suggest that, especially in problematic situations, navigation is not driven by information scent or utility, but by the need of increasing autonomy and the need of escaping from the current web patch.

It is not surprising that rich Internet content, such as Flash and DHTML, is some of the most pervasive content because of its visual attractiveness to the sighted majority. Such visually rich content has been causing severe accessibility problems, especially for people with visual disabilities. For Flash content, the kinds of accessibility information necessary for screen readers is not usually provided in the existing content. A typical example of such missing data is the lack of alternative text for buttons, hypertext links, widget roles, and so on. One of the major reasons is that the current accessibility framework of Flash content imposes a burden on content authors to make their content accessible. As a result, adding support for accessibility tends to be neglected, and screen reader users are left out of the richer Internet experiences.Therefore, we decided to develop an automatic accessibility transcoding system for Flash content to allow users to access a wider range of existing content, and to reduce the workload for content authors by using an automatic repair algorithm. It works as a client-side transcoding system based on the internal object model inside the Flash content. It adds and repairs accessibility information for existing Flash content, so screen readers can present more accessible information to users. Our experiment using the pilot system showed that 55% of the missing alternative texts for buttons in the tested websites could be added automatically.

Usability evaluation methods have become critical in the Web domain to ensure the success of Web applications. Aim: Since a large number of proposals have been presented during the last few years, a question arises: Which usability evaluation methods have proven to be the most effective in the Web domain? Method: This paper presents a systematic review that was motivated by previous results obtained from a systematic mapping study in the Web usability evaluation field. Results: A total of 18 studies were selected from an initial set of 206 in order to extract, code, and synthesize empirical data concerning the effectiveness of usability evaluation methods for the Web. Conclusions: We detected a need of more empirical studies and more standardized effectiveness measures for comparing usability evaluation methods. Our results suggest several evaluation methods which may be useful in allowing researchers and practitioners to perform effective Web usability evaluations.

Navigating back and forth from a list of links (index) to its target pages is common on the web, but tethers screen-reader users to unnecessary cognitive and mechanical steps. This problem worsens when indexes lack information scent: cues that enable users to select a link with confidence during fact-finding. This paper investigates how blind users who navigate the web with screen-readers can bypass a scentless index with guided tours: a much simpler browsing pattern that linearly concatenates items of a collection. In a controlled study (N=11) at the Indiana School for the Blind and Visually Impaired (ISBVI), guided tours lowered user's cognitive effort and significantly decreased time-on-task and number of pages visited when compared to an index with poor information scent. Our findings suggest that designers can supplement indexes with guided tours to benefit screen-reader users in a variety of web navigation contexts.

A review of multiple online courses at one institution was conducted by a skilled screen reader user for the purpose of assessing the extent to which the courses were navigable and understandable to online students using assistive technologies. This paper identifies features of online courses that may present problems for screen reader users and recommends solutions to address those problems. The following two overarching recommendations are suggested and elaborated: 1) Just as a roadmap assists drivers in navigating unfamiliar terrain, principles of clarity, consistency, and organization should be applied to the design of online courses to orient students to the virtual learning environment. 2) Web pages and course documents should make effective use of metadata, i.e., machine-understandable information about computer-based content, in order for course material to be accurately understood by students using a screen reader.

Visual content is strongly present in online social media. To ensure the accessibility of these contents to screen reader users, it is necessary to provide alternatives. In this paper, we report an initial investigation about the practices of visual content description in Brazilian online social media by examining the interfaces of the four most used in the country (YouTube, Facebook, WhatsApp, and Instagram) and surveying the description habits adopted by sighted users and the difficulties faced by screen readers users in understanding visual content. For this purpose, we conducted online surveys with sighted and visually impaired social media users. The results showed, among other aspects, that most sighted users do not describe visual content and even those that describe it, have difficulties to create the alternatives, mainly because they don't know what should be included or how to elaborate them. Moreover, for screen reader users, the greatest difficulty is the lack of alternatives for visual content provided by people. From the results, were identified actions and improvements to be adopted in the interfaces of online social media, in order to make them more inclusive to the Brazilian visually impaired.

Data visualization has become an increasingly important means of effective data communication and has played a vital role in broadcasting the progression of COVID-19. Accessible data representations, however, have lagged behind, leaving areas of information out of reach for many blind and visually impaired (BVI) users. In this work, we sought to understand (1) the accessibility of current implementations of visualizations on the web; (2) BVI users’ preferences and current experiences when accessing data-driven media; (3) how accessible data representations on the web address these users’ access needs and help them navigate, interpret, and gain insights from the data; and (4) the practical challenges that limit BVI users’ access and use of data representations. To answer these questions, we conducted a mixed-methods study consisting of an accessibility audit of 87 data visualizations on the web to identify accessibility issues, an online survey of 127 screen reader users to understand lived experiences and preferences, and a remote contextual inquiry with 12 of the survey respondents to observe how they navigate, interpret, and gain insights from accessible data representations. Our observations during this critical period of time provide an understanding of the widespread accessibility issues encountered across online data visualizations, the impact that data accessibility inequities have on the BVI community, the ways screen reader users sought access to data-driven information and made use of online visualizations to form insights, and the pressing need to make larger strides towards improving data literacy, building confidence, and enriching methods of access. Based on our findings, we provide recommendations for researchers and practitioners to broaden data accessibility on the web.