A framework for integrating usability evaluations methods: The Mawhiba web portal case study

Usability evaluation methods have become critical in the Web domain to ensure the success of Web applications. Aim: Since a large number of proposals have been presented during the last few years, a question arises: Which usability evaluation methods have proven to be the most effective in the Web domain? Method: This paper presents a systematic review that was motivated by previous results obtained from a systematic mapping study in the Web usability evaluation field. Results: A total of 18 studies were selected from an initial set of 206 in order to extract, code, and synthesize empirical data concerning the effectiveness of usability evaluation methods for the Web. Conclusions: We detected a need of more empirical studies and more standardized effectiveness measures for comparing usability evaluation methods. Our results suggest several evaluation methods which may be useful in allowing researchers and practitioners to perform effective Web usability evaluations.

The aim of this review is to summarize, analyze various research studies and identify different research gaps regarding usability standards and models, usability evaluation methods, usability metric, usability at different phases of software development life cycle and application domains of usability. This systematic review of usability studies between 1990 and 2016 has been conducted and 150 studies are identified. We conclude that researchers have not reached at consensus w.r.t. software usability models. We identify that Efficiency, Effectiveness, Satisfaction, and Learnability are commonly addressed attributes in various existing software usability models and standards. Further, developers do not have sufficient knowledge to decide the appropriate usability evaluation method to use in given domain. On the contrary, Usability Testing, Heuristic Evaluation and Questionnaire are identified frequently used methods for usability evaluation. Our findings investigate different metrics and measurement approaches used for usability estimation. But, current methods for usability measurement in practice do not include all ISO and ANSI defined aspects of usability into a single metric. Although, we identify studies concerning the integration of usability and software engineering into a single framework with generalizable results, their practical implementation is still missing and significantly needed. Conversely, this study highlights the fact that around 71% of studies address usability related issues during Design-Phase of software development life cycle. At present, usability issues have been identified in various domains but around 33.82% of studies identify that usability evaluation approach is widely used in Web-Domain.

Introduction Usability evaluation is essential to make sure that software products newly released are easy to use, efficient, and effective to reach goals, and satisfactory to users. For example, when a software company wants to develop and sell a new product, the company needs to evaluate usability of the new product before launching it at a market to avoid the possibility that the new product may contain usability problems, which span from cosmetic problems to severe functional problems. Three widely used methods for usability evaluation are Think Aloud (TA), Heuristic Evaluation (HE) and Cognitive Walkthrough (CW). TA method is commonly employed with a lab-based user testing, while there are variants of TA methods, including thinking out aloud at user's workplace instead of at labs. What we discuss here is the TA method that is combined with a lab-based user testing, in which test users use products while simultaneously and continuously thinking out aloud, and experimenters record users' behaviors and verbal protocols in the laboratory. HE is a usability inspection method, in which a small number of evaluators find usability problems in a user interface design by examining an interface and judging its compliance with well-known usability principles, called heuristics. CW is a theory-based method, in which evaluators evaluate every step necessary to perform a scenario-based task, and look for usability problems that would interfere with learning by exploration. These three methods have their own advantages and disadvantages. For instance, TA method provides good qualitative data from a small number of test users, but laboratory environment may influence test user's behaviors. HE is a cheap, fast and easy-to-use method, while it often finds too specific and low-priority usability problems, including even not real problems. CW helps find mismatches between users' and designers' conceptualization of a task, but it needs extensive knowledge of cognitive psychology and technical details to apply. However, even though these advantages and disadvantages show overall characteristics of three major usability evaluation methods, we cannot compare them quantitatively and see their efficiency clearly. Because one of reasons why so-called discounted methods, such as HE and CW, were developed is to save costs of usability evaluation, cost-related criteria for comparing usability evaluation are meaningful to usability practitioners as well as usability researchers. One of the most disputable issues related to cost of usability evaluation is sample size. That is, how many users or evaluators are needed to achieve a targeted usability evaluation performance, for example, 80% of overall discovery rate? The sample size of usability evaluation is known to depend on an estimate of problem discovery rate across participants. The overall discovery rate is a common quantitative measure that is used to show the effectiveness of a specific usability evaluation method in most of usability evaluation studies. It is also called overall detection rate or thoroughness measure, which is the ratio of 'the sum of unique usability problems detected by all experiment participants' against 'the number of usability problems that exist in the evaluated systems', ranging between 0 and 1. The overall discovery rates were reported more than any other criterion measure in the usability evaluation experiments and also a key component for projecting required sample size for usability evaluation study. Thus, how many test users or evaluators participate in the usability evaluation is a critical issue, considering its cost-effectiveness.

Determination of the effectiveness of two methods for usability evaluation using a CPOE medication ordering system

Usability in a medical technology context assessment of methods for usability evaluation of medical equipment

Usability evaluation methods have gained a substantial attention in networks particularly in Intrusion Detection System (IDS) as these evaluation methods are envisioned to achieve usability and define usability defects for a large number of practical software’s. Despite a good number of available survey and methods on usability evaluation, we feel that there is a gap in existing literature in terms of usability evaluation methods, IDS interfaces and following usability guidelines in IDS development. This paper reviews the state of the art for improving usability of networks that illustrates the issues and challenges in the context of design matters. Further, we propose the taxonomy of key issues in evaluation methods and usability problems. We also define design heuristics for IDS users and interfaces that improves detection of usability defects and interface usability compared to conventional evaluation heuristics. The similarities and differences of usability evaluation methods and usability problems are summarized on the basis of usability factors, current evaluation methods and interfaces loopholes

E-Government aims to deliver benefits to government and citizens by improving transparency, efficiency, trust, and citizen participation. However, e-government initiatives face several barriers. One of them is poor usability. To advance quality of usability, literatures indicate that usability evaluation is a key success factor. There are many approaches to conduct usability evaluation. Each of it has advantages and challenges. On the other hand, there are several aspects that must be considered related to usability evaluation in e-government context. It includes large stakeholders and their diversity, extra needs for ethical practices, as well as high privacy. Therefore, it is crucial to investigate the right usability evaluation method in e-government. In order to address this issue, a study using Systematic Literature Review (SLR) was conducted to identify the suitable usability evaluation methods. There are 519 literatures that have been selected in the initial stage. It was then followed by an extraction process, which produced 22 selected references. Each method was grouped into usability testing, inspection, and inquiry. These results can guide academics and practitioners to carry out usability evaluation in e-government.

BackgroundUsability is a key factor affecting the acceptance of mobile health applications (mHealth apps) for elderly individuals, but traditional usability evaluation methods may not be suitable for use in this population because of aging barriers. The objectives of this study were to identify, explore, and summarize the current state of the literature on the usability evaluation of mHealth apps for older adults and to incorporate these methods into the appropriate evaluation stage.MethodsElectronic searches were conducted in 10 databases. Inclusion criteria were articles focused on the usability evaluation of mHealth apps designed for older adults. The included studies were classified according to the mHealth app usability evaluation framework, and the suitability of evaluation methods for use among the elderly was analyzed.ResultsNinety-six articles met the inclusion criteria. Research activity increased steeply after 2013 (n = 92). Satisfaction (n = 74) and learnability (n = 60) were the most frequently evaluated critical measures, while memorability (n = 13) was the least evaluated. The ratios of satisfaction, learnability, operability, and understandability measures were significantly related to the different stages of evaluation (P < 0.05). The methods used for usability evaluation were questionnaire (n = 68), interview (n = 36), concurrent thinking aloud (n = 25), performance metrics (n = 25), behavioral observation log (n = 14), screen recording (n = 3), eye tracking (n = 1), retrospective thinking aloud (n = 1), and feedback log (n = 1). Thirty-two studies developed their own evaluation tool to assess unique design features for elderly individuals.ConclusionIn the past five years, the number of studies in the field of usability evaluation of mHealth apps for the elderly has increased rapidly. The mHealth apps are often used as an auxiliary means of self-management to help the elderly manage their wellness and disease. According to the three stages of the mHealth app usability evaluation framework, the critical measures and evaluation methods are inconsistent. Future research should focus on selecting specific critical measures relevant to aging characteristics and adapting usability evaluation methods to elderly individuals by improving traditional tools, introducing automated evaluation tools and optimizing evaluation processes.

Purpose This study aims to systematically review the published literature on the topic of academic library website usability evaluation. It determines the most significant subattributes in the usability evaluation of academic library websites. The study also looks at the metrics that have been used to evaluate the usability of academic library websites. Then, it identifies the most significant usability evaluation method (UEM) that has been discussed in the literature. Lastly, it identifies the research gaps in the literature regarding the proposed approaches to usability evaluation of academic library websites. Design/methodology/approach In this systematic literature review (SLR), the research conducted in the area of usability evaluation of academic library websites from 1995 to 2023 was reviewed. Findings The results of this study disclose that “effectiveness” is the most significant usability subattribute in the context of usability evaluation of academic library websites. Furthermore, “user testing” is the most significant UEM that has been applied in most of the primary studies. Some of the metrics used for the measurement of effectiveness were user preference, error rate and number of correct answers. Similarly, some of the metrics used, for efficiency, were response time, click count, time on task, logical information architecture and efficient information retrieval and so on. The common research gap in the reviewed literature is a lack of a generalized framework applicable to any academic library website for its usability evaluation. Practical implications This study is useful for library website developers to better understand the required features of a library website. Originality/value This is the first SLR on the topic of academic libraries’ website usability evaluation.

The lack of standard assessment criteria for reliably comparing usability evaluation methods (UEMs) is an important gap in HCI knowledge. Recently, metrics for assessing thoroughness, validity, and effectiveness of UEMs, based on user data, have been proposed to bridge this gap. This paper reports our findings of applying these proposed metrics in a study that compared heuristic evaluation (HE) to HE-Plus (an extended version of HE). Our experiment showed better overlap among the HE-Plus evaluators than the HE evaluators, demonstrating greater reliability of the method. When evaluation data, from testing the usability of the same website, was used in calculating the UEM performance metrics, HE-Plus was found to be a superior method to HE in all assessment criteria with a 17%, 39%, and 67% improvement in the aspects of thoroughness, validity, and effectiveness, respectively. The paper concludes with a discussion concerning the limitations of the effectiveness of the UEM from which the real users' data was obtained.

AI-augmented usability evaluation framework for software requirements specification in cyber physical human systems

Multimodal video games designed for increasing cognition of people who are blind should be friendly and pleasant to use, instead of adding complexity to the interaction, leading people to acquire cognitive skills while interacting. There are specific issues that make multimodal usability evaluation different from the evaluation of traditional user interfaces in the context of improving cognition of people who are blind. In this context, identifying how well the Usability Evaluation Methods (UEM) meet the evaluation criteria to assess multimodal games for people who are blind is necessary. In this paper, we conducted an expert opinion survey to analyze how usability evaluation has been done by researchers and practitioners in this field. As a result, we propose the PrincipLes for Evaluating Usability of Multimodal Video Games for People who are Blind (PLUMB), a set of evaluation good practices that should be observed while planning the evaluation. This paper builds on the literature about how multimodal features affect people who are blind interaction with multimodal interfaces by focusing on their practical evaluation.

A usability model is a hierarchical structure encompassing the key elements such as users, user interface and interaction between them. It is a generic template which is independent of usability evaluation methods and provides flexibility for adaptation in different contexts and domains. In this paper, a usability model for medical user interfaces, especially for ventilator systems in Intensive Care Unit (ICU), is proposed based on Norman’s action-oriented seven-step model to capture a related medical context. A ventilator system is a therapeutic device, which provides a respiratory support to critically-ill patients. Currently, a usability of user interfaces of ventilator systems is evaluated by typical usability evaluation methods from software industry. These evaluation methods miss out important elements in medical context. Therefore, a need for a specialized usability model for medical user interfaces is fulfilled with a proposed usability model encompassing vital elements such as medical user, user interface, ICU environment and time required. This usability model is validated first, through a human work analysis using videos of selected tasks with medical user interfaces and then, with an overview of critical factors affecting medical user interfaces in ICU. In future, a proposed usability model can be integrated with a suitable usability evaluation method for evaluating medical user interfaces to identify related medical usability problems more effectively.

The rapid proliferation of novel interaction paradigms—including Extended Reality (XR), tangible and natural user interfaces, and haptic systems—together with evolving user experience attributes such as emotional, social, and collaborative interactions, has fundamentally transformed human–computer interaction. These interaction modalities, increasingly integrated with artificial intelligence, expose the limitations of traditional usability and UX evaluation methods developed for conventional graphical user interfaces. Human–AI interaction introduces distinct design challenges related to transparency, explainability, user trust, and shared control, requiring evaluation approaches that shift from technology-centered assessment toward genuinely human-centered analysis.This research proposes a metric-based, AI-powered evaluation platform designed to systematically test and benchmark contemporary user interfaces and interaction modalities. The platform employs enhanced event logging and behavioral coding mechanisms to capture comprehensive multimodal interaction data, including task performance metrics, behavioral patterns, temporal action sequences, gaze trajectories, gesture dynamics, and physiological responses. Machine learning models analyze this multidimensional data to predict standard usability attributes—efficiency, effectiveness, and satisfaction as defined by ISO 9241-11—alongside emerging UX dimensions such as cognitive load, emotional engagement, and social presence.The platform architecture integrates high-performance XR-capable computing hardware with professional-grade graphics processing, multi-monitor visualization, eye-tracking sensors, haptic feedback devices, motion-capture systems, and biometric sensing infrastructure. Data collection is supported by curated training datasets derived from prior usability studies, expert heuristic evaluations, standardized task baselines, and cross-cultural UX assessment data. A custom-engineered, shock-resistant, transportable case with integrated power management and modular connectivity enables rapid deployment across laboratory, field, and organizational contexts, addressing ecological validity limitations inherent in traditional lab-based usability testing.Automated usability evaluation is driven by a hybrid AI framework combining three complementary models. Random Forest ensemble classifiers provide robust prediction across heterogeneous interaction data while offering interpretable feature importance measures to identify key usability determinants. Long Short-Term Memory (LSTM) networks model temporally ordered interaction sequences, enabling detection of behavioral signatures associated with confusion, flow states, hesitation, and error recovery. Support Vector Machines with radial basis function kernels capture complex non-linear relationships in high-dimensional usability data and perform effectively under limited expert-labeled training conditions.The central hypothesis posits that AI-driven automated usability evaluation can achieve accuracy and reliability comparable to expert-conducted heuristic evaluations and cognitive walkthroughs, validated through controlled A/B experiments comparing automated predictions with expert assessments across diverse interface types and user populations.By democratizing access to rigorous UX evaluation through automation, multimodal sensing, and portable deployment, the proposed platform aims to accelerate iterative design cycles for emerging interactive and human–AI systems while maintaining methodological rigor comparable to expert evaluation.

Mobile devices are used more and more frequently by society. In recent years has increased the sale of those devices and therefore in the applications provided to the users. The educational apps are one of the most important, because they offer more significant opportunities for development and growth to society, developing in various areas. In this context, usability becomes a key factor for its adoption.The objective of this paper is to identify which are the usability methods that are used in mobile educational applications by subject area. Therefore, a systematic mapping review was conducted. A total of 1503 studies were identified, and only 174 were selected for this review. The results show that the user test is the most widely used usability evaluation method in mobile educational applications, and the preferred learning topics are foreign languages and information and communication technologies. These results can guide experts and developers to construct usable mobile educational apps by selecting the usability evaluation methods according to the learning topic.