A systematic review of software usability studies

Introduction Usability evaluation is essential to make sure that software products newly released are easy to use, efficient, and effective to reach goals, and satisfactory to users. For example, when a software company wants to develop and sell a new product, the company needs to evaluate usability of the new product before launching it at a market to avoid the possibility that the new product may contain usability problems, which span from cosmetic problems to severe functional problems. Three widely used methods for usability evaluation are Think Aloud (TA), Heuristic Evaluation (HE) and Cognitive Walkthrough (CW). TA method is commonly employed with a lab-based user testing, while there are variants of TA methods, including thinking out aloud at user's workplace instead of at labs. What we discuss here is the TA method that is combined with a lab-based user testing, in which test users use products while simultaneously and continuously thinking out aloud, and experimenters record users' behaviors and verbal protocols in the laboratory. HE is a usability inspection method, in which a small number of evaluators find usability problems in a user interface design by examining an interface and judging its compliance with well-known usability principles, called heuristics. CW is a theory-based method, in which evaluators evaluate every step necessary to perform a scenario-based task, and look for usability problems that would interfere with learning by exploration. These three methods have their own advantages and disadvantages. For instance, TA method provides good qualitative data from a small number of test users, but laboratory environment may influence test user's behaviors. HE is a cheap, fast and easy-to-use method, while it often finds too specific and low-priority usability problems, including even not real problems. CW helps find mismatches between users' and designers' conceptualization of a task, but it needs extensive knowledge of cognitive psychology and technical details to apply. However, even though these advantages and disadvantages show overall characteristics of three major usability evaluation methods, we cannot compare them quantitatively and see their efficiency clearly. Because one of reasons why so-called discounted methods, such as HE and CW, were developed is to save costs of usability evaluation, cost-related criteria for comparing usability evaluation are meaningful to usability practitioners as well as usability researchers. One of the most disputable issues related to cost of usability evaluation is sample size. That is, how many users or evaluators are needed to achieve a targeted usability evaluation performance, for example, 80% of overall discovery rate? The sample size of usability evaluation is known to depend on an estimate of problem discovery rate across participants. The overall discovery rate is a common quantitative measure that is used to show the effectiveness of a specific usability evaluation method in most of usability evaluation studies. It is also called overall detection rate or thoroughness measure, which is the ratio of 'the sum of unique usability problems detected by all experiment participants' against 'the number of usability problems that exist in the evaluated systems', ranging between 0 and 1. The overall discovery rates were reported more than any other criterion measure in the usability evaluation experiments and also a key component for projecting required sample size for usability evaluation study. Thus, how many test users or evaluators participate in the usability evaluation is a critical issue, considering its cost-effectiveness.

Usability in a medical technology context assessment of methods for usability evaluation of medical equipment

Different usability evaluation methods (UEMs) are often combined to assess the compliance of interactive systems with usability standards and guidelines. In practice, combinations of UEMs have been applied in a rather ad hoc manner resulting in inadequate analyses of rich usability data sets. While the potential offered by recent advances in technologies such as eye tracking for usability is recognized, and the challenges of evaluating complex web systems such as portals which involve different interaction styles are highlighted, the issue of effectively integrating multiple UEMs to make sense of observations has emerged. This paper presents a framework developed for effectively integrating UEMs in usability evaluations by matching the methods' capabilities and limitations with a classification of usability problems. The framework is applied to evaluating the usability of the Mawhiba web portal with target user populations which demonstrates how better coverage of usability issues can be achieved.

Combined Expert and User-Driven Usability Assessment of Trauma Decision Support Systems Improves User-Centered Design

The complexity of an electronic medication administration record (eMAR) has been underestimated by most designers in the past. Usability issues, such as poorly designed user application flow in eMAR, are therefore of vital importance, since they can have a negative impact on nursing activities and result in poor outcomes. The purpose of this study was to evaluate the usability of an eMAR application during its development. A usability evaluation was conducted during the development of the eMAR application. Two usability methods were used: a heuristic evaluation complemented by usability testing. Each eMAR application version provided by the vendor was evaluated by 2 hospital pharmacists, who applied the heuristic method. They reviewed the eMAR tasks, detected usability problems and their heuristic violations, and rated the severity of the usability problems. Usability testing was used to assess the final application version by observing how 3 nurses interacted with the application. Thirty-four versions were assessed before the eMAR application was considered usable. During the heuristic evaluation, the usability problems decreased from 46 unique usability problems in version 1 (V1) to 9 in version 34 (V34). In V1, usability problems were categorized into 154 heuristic violations, which decreased to 27 in V34. The average severity rating also decreased from major usability problem (2.96) to no problem (0.23). During usability testing, the 3 nurses did not encounter new usability problems. A thorough heuristic evaluation is a good method for obtaining a usable eMAR application. This evaluation points key areas for improvement and decreases usability problems and their severity.

The lack of standard assessment criteria for reliably comparing usability evaluation methods (UEMs) is an important gap in HCI knowledge. Recently, metrics for assessing thoroughness, validity, and effectiveness of UEMs, based on user data, have been proposed to bridge this gap. This paper reports our findings of applying these proposed metrics in a study that compared heuristic evaluation (HE) to HE-Plus (an extended version of HE). Our experiment showed better overlap among the HE-Plus evaluators than the HE evaluators, demonstrating greater reliability of the method. When evaluation data, from testing the usability of the same website, was used in calculating the UEM performance metrics, HE-Plus was found to be a superior method to HE in all assessment criteria with a 17%, 39%, and 67% improvement in the aspects of thoroughness, validity, and effectiveness, respectively. The paper concludes with a discussion concerning the limitations of the effectiveness of the UEM from which the real users' data was obtained.

E-Government aims to deliver benefits to government and citizens by improving transparency, efficiency, trust, and citizen participation. However, e-government initiatives face several barriers. One of them is poor usability. To advance quality of usability, literatures indicate that usability evaluation is a key success factor. There are many approaches to conduct usability evaluation. Each of it has advantages and challenges. On the other hand, there are several aspects that must be considered related to usability evaluation in e-government context. It includes large stakeholders and their diversity, extra needs for ethical practices, as well as high privacy. Therefore, it is crucial to investigate the right usability evaluation method in e-government. In order to address this issue, a study using Systematic Literature Review (SLR) was conducted to identify the suitable usability evaluation methods. There are 519 literatures that have been selected in the initial stage. It was then followed by an extraction process, which produced 22 selected references. Each method was grouped into usability testing, inspection, and inquiry. These results can guide academics and practitioners to carry out usability evaluation in e-government.

Purpose This study aims to systematically review the published literature on the topic of academic library website usability evaluation. It determines the most significant subattributes in the usability evaluation of academic library websites. The study also looks at the metrics that have been used to evaluate the usability of academic library websites. Then, it identifies the most significant usability evaluation method (UEM) that has been discussed in the literature. Lastly, it identifies the research gaps in the literature regarding the proposed approaches to usability evaluation of academic library websites. Design/methodology/approach In this systematic literature review (SLR), the research conducted in the area of usability evaluation of academic library websites from 1995 to 2023 was reviewed. Findings The results of this study disclose that “effectiveness” is the most significant usability subattribute in the context of usability evaluation of academic library websites. Furthermore, “user testing” is the most significant UEM that has been applied in most of the primary studies. Some of the metrics used for the measurement of effectiveness were user preference, error rate and number of correct answers. Similarly, some of the metrics used, for efficiency, were response time, click count, time on task, logical information architecture and efficient information retrieval and so on. The common research gap in the reviewed literature is a lack of a generalized framework applicable to any academic library website for its usability evaluation. Practical implications This study is useful for library website developers to better understand the required features of a library website. Originality/value This is the first SLR on the topic of academic libraries’ website usability evaluation.

The rapid proliferation of novel interaction paradigms—including Extended Reality (XR), tangible and natural user interfaces, and haptic systems—together with evolving user experience attributes such as emotional, social, and collaborative interactions, has fundamentally transformed human–computer interaction. These interaction modalities, increasingly integrated with artificial intelligence, expose the limitations of traditional usability and UX evaluation methods developed for conventional graphical user interfaces. Human–AI interaction introduces distinct design challenges related to transparency, explainability, user trust, and shared control, requiring evaluation approaches that shift from technology-centered assessment toward genuinely human-centered analysis.This research proposes a metric-based, AI-powered evaluation platform designed to systematically test and benchmark contemporary user interfaces and interaction modalities. The platform employs enhanced event logging and behavioral coding mechanisms to capture comprehensive multimodal interaction data, including task performance metrics, behavioral patterns, temporal action sequences, gaze trajectories, gesture dynamics, and physiological responses. Machine learning models analyze this multidimensional data to predict standard usability attributes—efficiency, effectiveness, and satisfaction as defined by ISO 9241-11—alongside emerging UX dimensions such as cognitive load, emotional engagement, and social presence.The platform architecture integrates high-performance XR-capable computing hardware with professional-grade graphics processing, multi-monitor visualization, eye-tracking sensors, haptic feedback devices, motion-capture systems, and biometric sensing infrastructure. Data collection is supported by curated training datasets derived from prior usability studies, expert heuristic evaluations, standardized task baselines, and cross-cultural UX assessment data. A custom-engineered, shock-resistant, transportable case with integrated power management and modular connectivity enables rapid deployment across laboratory, field, and organizational contexts, addressing ecological validity limitations inherent in traditional lab-based usability testing.Automated usability evaluation is driven by a hybrid AI framework combining three complementary models. Random Forest ensemble classifiers provide robust prediction across heterogeneous interaction data while offering interpretable feature importance measures to identify key usability determinants. Long Short-Term Memory (LSTM) networks model temporally ordered interaction sequences, enabling detection of behavioral signatures associated with confusion, flow states, hesitation, and error recovery. Support Vector Machines with radial basis function kernels capture complex non-linear relationships in high-dimensional usability data and perform effectively under limited expert-labeled training conditions.The central hypothesis posits that AI-driven automated usability evaluation can achieve accuracy and reliability comparable to expert-conducted heuristic evaluations and cognitive walkthroughs, validated through controlled A/B experiments comparing automated predictions with expert assessments across diverse interface types and user populations.By democratizing access to rigorous UX evaluation through automation, multimodal sensing, and portable deployment, the proposed platform aims to accelerate iterative design cycles for emerging interactive and human–AI systems while maintaining methodological rigor comparable to expert evaluation.

Usability evaluation methods have become critical in the Web domain to ensure the success of Web applications. Aim: Since a large number of proposals have been presented during the last few years, a question arises: Which usability evaluation methods have proven to be the most effective in the Web domain? Method: This paper presents a systematic review that was motivated by previous results obtained from a systematic mapping study in the Web usability evaluation field. Results: A total of 18 studies were selected from an initial set of 206 in order to extract, code, and synthesize empirical data concerning the effectiveness of usability evaluation methods for the Web. Conclusions: We detected a need of more empirical studies and more standardized effectiveness measures for comparing usability evaluation methods. Our results suggest several evaluation methods which may be useful in allowing researchers and practitioners to perform effective Web usability evaluations.

A usability model is a hierarchical structure encompassing the key elements such as users, user interface and interaction between them. It is a generic template which is independent of usability evaluation methods and provides flexibility for adaptation in different contexts and domains. In this paper, a usability model for medical user interfaces, especially for ventilator systems in Intensive Care Unit (ICU), is proposed based on Norman’s action-oriented seven-step model to capture a related medical context. A ventilator system is a therapeutic device, which provides a respiratory support to critically-ill patients. Currently, a usability of user interfaces of ventilator systems is evaluated by typical usability evaluation methods from software industry. These evaluation methods miss out important elements in medical context. Therefore, a need for a specialized usability model for medical user interfaces is fulfilled with a proposed usability model encompassing vital elements such as medical user, user interface, ICU environment and time required. This usability model is validated first, through a human work analysis using videos of selected tasks with medical user interfaces and then, with an overview of critical factors affecting medical user interfaces in ICU. In future, a proposed usability model can be integrated with a suitable usability evaluation method for evaluating medical user interfaces to identify related medical usability problems more effectively.

Heuristic evaluation (HE) is an inspection-based usability evaluation method in which a number of evaluators, typically 3–5, assess the usability of a system based on a set of usability guidelines. HE was first introduced by Nielsen and Molich and then revised by Nielsen. Since its introduction, HE has gained wide popularity among human-computer interaction (HCI) and user experience (UX) practitioners and is now one of the most common usability evaluation methods. A few years after the introduction of HE, some researchers realized that novice evaluators perform poorly on complex systems. This issue, known as the expertise effect, is one of the major challenges of HE; i.e., the more experienced the evaluator is, the better the results of the evaluation. Consequently, some researchers argue that the results of HE are the product of the evaluator’s experience rather than the method itself. To address this issue, we interviewed 15 usability experts, all of whom had at least four years of experience and who came from both academia and industry. We analyzed their responses and developed a step-by-step protocol.

Research problem: No study has explored how incorporating personas into heuristic evaluation of products, namely websites, affects the kinds of findings reported and the recommendations presented by usability evaluators. Research questions: (1) Do findings resulting from heuristic evaluations of a website without the use of personas differ from findings resulting from heuristic evaluations of the same website with the use of personas? (2) Do findings from persona-based heuristic evaluations in which evaluators develop their own personas differ from findings from persona-based heuristic evaluations in which evaluators are given personas? (3) If findings and recommendations are different, how do they differ? (4) How does the use of personas affect the evaluators' confidence in the findings of a heuristic evaluation? Literature review: First, previous research of heuristic evaluation has concluded that although heuristic evaluation is inexpensive and does not require advance planning, it has several shortcomings, including its too-intense focus on minor issues and its inability to capture all usability issues. Second, data-driven personas, which have long been a resource in user-centered design, have been suggested as a way to improve or enhance heuristic evaluation, and several studies suggest that usability professionals are indeed using personas in their evaluations. However, no empirical study has assessed heuristic evaluations that include personas. Methodology: In this exploratory study involving three sections of an advanced technical writing course, groups of evaluators conducted a heuristic evaluation of a website. Each section was randomly assigned a different condition with which they would conduct the heuristic evaluation: (a) a traditional heuristic evaluation, (b) a persona-led heuristic evaluation in which the personas were given to the evaluators, or (c) a persona-led heuristic evaluation in which the evaluators themselves created their own personas. Each group wrote a report identifying the major problems with the website and provided recommendations to solve the identified problems. The evaluators completed pretesting demographic surveys and posttesting confidence surveys. Results and discussion: This exploratory study found few detectable differences in the findings reported by groups that used personas in heuristic evaluation and groups that did not use personas. The groups that used personas were more likely to report findings related to navigation than the groups that did not use personas, while the groups that did not use personas were more likely to report findings related to design than the groups that used personas. The groups that created their own personas were more likely than the other groups to include complex issues in their reports and include language that directly references users and user needs. All groups were confident in their findings.

This article describes an experiment comparing three Usability Evaluation Methods: User Testing (UT), Document-based Inspection (DI), and Expert Inspection (EI) for evaluating Virtual Environments (VEs). Twenty-nine individuals (10 end-users and 19 junior usability experts) participated during 1 hr each in the evaluation of two VEs (a training VE and a 3D map). Quantitative results of the comparison show that the effectiveness of UT and DI is significantly better than the effectiveness of EI. For each method, results show their problem coverage: DI- and UT-based diagnoses lead to more problem diversity than EI. The overlap of identified problems amounts to 22% between UT and DI, 20% between DI and EI, and 12% between EI and UT for both virtual environments. The identification impact of the whole set of usability problems is 60% for DI, 57% for UT, and only 36% for EI for both virtual environments. Also reliability of UT and DI is significantly better than reliability of EI. In addition, a qualitative analysis identified 35 classes describing the profile of usability problems found with each method. It shows that UT seems particularly efficient for the diagnosis of problems that require a particular state of interaction to be detectable. On the other hand, DI supports the identification of problems directly observable, often related to learnability and basic usability. This study shows that DI could be viewed as a “4-wheel drive SUV evaluation type” (less powerful under certain conditions but able to go everywhere, with any driver), whereas UT could be viewed as a “Formula 1 car evaluation type” (more powerful but requiring adequate road and a very skilled driver). EI is found (considering all metrics) to be not efficient enough to evaluate usability of VEs.

BackgroundUsability is a key factor affecting the acceptance of mobile health applications (mHealth apps) for elderly individuals, but traditional usability evaluation methods may not be suitable for use in this population because of aging barriers. The objectives of this study were to identify, explore, and summarize the current state of the literature on the usability evaluation of mHealth apps for older adults and to incorporate these methods into the appropriate evaluation stage.MethodsElectronic searches were conducted in 10 databases. Inclusion criteria were articles focused on the usability evaluation of mHealth apps designed for older adults. The included studies were classified according to the mHealth app usability evaluation framework, and the suitability of evaluation methods for use among the elderly was analyzed.ResultsNinety-six articles met the inclusion criteria. Research activity increased steeply after 2013 (n = 92). Satisfaction (n = 74) and learnability (n = 60) were the most frequently evaluated critical measures, while memorability (n = 13) was the least evaluated. The ratios of satisfaction, learnability, operability, and understandability measures were significantly related to the different stages of evaluation (P < 0.05). The methods used for usability evaluation were questionnaire (n = 68), interview (n = 36), concurrent thinking aloud (n = 25), performance metrics (n = 25), behavioral observation log (n = 14), screen recording (n = 3), eye tracking (n = 1), retrospective thinking aloud (n = 1), and feedback log (n = 1). Thirty-two studies developed their own evaluation tool to assess unique design features for elderly individuals.ConclusionIn the past five years, the number of studies in the field of usability evaluation of mHealth apps for the elderly has increased rapidly. The mHealth apps are often used as an auxiliary means of self-management to help the elderly manage their wellness and disease. According to the three stages of the mHealth app usability evaluation framework, the critical measures and evaluation methods are inconsistent. Future research should focus on selecting specific critical measures relevant to aging characteristics and adapting usability evaluation methods to elderly individuals by improving traditional tools, introducing automated evaluation tools and optimizing evaluation processes.