E-Government Usability Evaluation

Purpose This study aims to systematically review the published literature on the topic of academic library website usability evaluation. It determines the most significant subattributes in the usability evaluation of academic library websites. The study also looks at the metrics that have been used to evaluate the usability of academic library websites. Then, it identifies the most significant usability evaluation method (UEM) that has been discussed in the literature. Lastly, it identifies the research gaps in the literature regarding the proposed approaches to usability evaluation of academic library websites. Design/methodology/approach In this systematic literature review (SLR), the research conducted in the area of usability evaluation of academic library websites from 1995 to 2023 was reviewed. Findings The results of this study disclose that “effectiveness” is the most significant usability subattribute in the context of usability evaluation of academic library websites. Furthermore, “user testing” is the most significant UEM that has been applied in most of the primary studies. Some of the metrics used for the measurement of effectiveness were user preference, error rate and number of correct answers. Similarly, some of the metrics used, for efficiency, were response time, click count, time on task, logical information architecture and efficient information retrieval and so on. The common research gap in the reviewed literature is a lack of a generalized framework applicable to any academic library website for its usability evaluation. Practical implications This study is useful for library website developers to better understand the required features of a library website. Originality/value This is the first SLR on the topic of academic libraries’ website usability evaluation.

Usability in a medical technology context assessment of methods for usability evaluation of medical equipment

Introduction Usability evaluation is essential to make sure that software products newly released are easy to use, efficient, and effective to reach goals, and satisfactory to users. For example, when a software company wants to develop and sell a new product, the company needs to evaluate usability of the new product before launching it at a market to avoid the possibility that the new product may contain usability problems, which span from cosmetic problems to severe functional problems. Three widely used methods for usability evaluation are Think Aloud (TA), Heuristic Evaluation (HE) and Cognitive Walkthrough (CW). TA method is commonly employed with a lab-based user testing, while there are variants of TA methods, including thinking out aloud at user's workplace instead of at labs. What we discuss here is the TA method that is combined with a lab-based user testing, in which test users use products while simultaneously and continuously thinking out aloud, and experimenters record users' behaviors and verbal protocols in the laboratory. HE is a usability inspection method, in which a small number of evaluators find usability problems in a user interface design by examining an interface and judging its compliance with well-known usability principles, called heuristics. CW is a theory-based method, in which evaluators evaluate every step necessary to perform a scenario-based task, and look for usability problems that would interfere with learning by exploration. These three methods have their own advantages and disadvantages. For instance, TA method provides good qualitative data from a small number of test users, but laboratory environment may influence test user's behaviors. HE is a cheap, fast and easy-to-use method, while it often finds too specific and low-priority usability problems, including even not real problems. CW helps find mismatches between users' and designers' conceptualization of a task, but it needs extensive knowledge of cognitive psychology and technical details to apply. However, even though these advantages and disadvantages show overall characteristics of three major usability evaluation methods, we cannot compare them quantitatively and see their efficiency clearly. Because one of reasons why so-called discounted methods, such as HE and CW, were developed is to save costs of usability evaluation, cost-related criteria for comparing usability evaluation are meaningful to usability practitioners as well as usability researchers. One of the most disputable issues related to cost of usability evaluation is sample size. That is, how many users or evaluators are needed to achieve a targeted usability evaluation performance, for example, 80% of overall discovery rate? The sample size of usability evaluation is known to depend on an estimate of problem discovery rate across participants. The overall discovery rate is a common quantitative measure that is used to show the effectiveness of a specific usability evaluation method in most of usability evaluation studies. It is also called overall detection rate or thoroughness measure, which is the ratio of 'the sum of unique usability problems detected by all experiment participants' against 'the number of usability problems that exist in the evaluated systems', ranging between 0 and 1. The overall discovery rates were reported more than any other criterion measure in the usability evaluation experiments and also a key component for projecting required sample size for usability evaluation study. Thus, how many test users or evaluators participate in the usability evaluation is a critical issue, considering its cost-effectiveness.

Usability evaluation methods have become critical in the Web domain to ensure the success of Web applications. Aim: Since a large number of proposals have been presented during the last few years, a question arises: Which usability evaluation methods have proven to be the most effective in the Web domain? Method: This paper presents a systematic review that was motivated by previous results obtained from a systematic mapping study in the Web usability evaluation field. Results: A total of 18 studies were selected from an initial set of 206 in order to extract, code, and synthesize empirical data concerning the effectiveness of usability evaluation methods for the Web. Conclusions: We detected a need of more empirical studies and more standardized effectiveness measures for comparing usability evaluation methods. Our results suggest several evaluation methods which may be useful in allowing researchers and practitioners to perform effective Web usability evaluations.

The evaluation of interactive adaptive systems is considered as a difficult endeavour due to the complex nature of these systems. Issues arise in the need to identify, for a given evaluation situation the set of suitable usability evaluation methods to be used by the evaluators. This paper describes a framework for supporting the choice of appropriate methods for the usability evaluation of interactive adaptive systems. It is based on the comparison of usability evaluation methods using relevant criteria. This framework is then validated through a case study; it concerns an adaptive hypermedia system in transportation field.

The aim of this review is to summarize, analyze various research studies and identify different research gaps regarding usability standards and models, usability evaluation methods, usability metric, usability at different phases of software development life cycle and application domains of usability. This systematic review of usability studies between 1990 and 2016 has been conducted and 150 studies are identified. We conclude that researchers have not reached at consensus w.r.t. software usability models. We identify that Efficiency, Effectiveness, Satisfaction, and Learnability are commonly addressed attributes in various existing software usability models and standards. Further, developers do not have sufficient knowledge to decide the appropriate usability evaluation method to use in given domain. On the contrary, Usability Testing, Heuristic Evaluation and Questionnaire are identified frequently used methods for usability evaluation. Our findings investigate different metrics and measurement approaches used for usability estimation. But, current methods for usability measurement in practice do not include all ISO and ANSI defined aspects of usability into a single metric. Although, we identify studies concerning the integration of usability and software engineering into a single framework with generalizable results, their practical implementation is still missing and significantly needed. Conversely, this study highlights the fact that around 71% of studies address usability related issues during Design-Phase of software development life cycle. At present, usability issues have been identified in various domains but around 33.82% of studies identify that usability evaluation approach is widely used in Web-Domain.

BackgroundUsability is a key factor affecting the acceptance of mobile health applications (mHealth apps) for elderly individuals, but traditional usability evaluation methods may not be suitable for use in this population because of aging barriers. The objectives of this study were to identify, explore, and summarize the current state of the literature on the usability evaluation of mHealth apps for older adults and to incorporate these methods into the appropriate evaluation stage.MethodsElectronic searches were conducted in 10 databases. Inclusion criteria were articles focused on the usability evaluation of mHealth apps designed for older adults. The included studies were classified according to the mHealth app usability evaluation framework, and the suitability of evaluation methods for use among the elderly was analyzed.ResultsNinety-six articles met the inclusion criteria. Research activity increased steeply after 2013 (n = 92). Satisfaction (n = 74) and learnability (n = 60) were the most frequently evaluated critical measures, while memorability (n = 13) was the least evaluated. The ratios of satisfaction, learnability, operability, and understandability measures were significantly related to the different stages of evaluation (P < 0.05). The methods used for usability evaluation were questionnaire (n = 68), interview (n = 36), concurrent thinking aloud (n = 25), performance metrics (n = 25), behavioral observation log (n = 14), screen recording (n = 3), eye tracking (n = 1), retrospective thinking aloud (n = 1), and feedback log (n = 1). Thirty-two studies developed their own evaluation tool to assess unique design features for elderly individuals.ConclusionIn the past five years, the number of studies in the field of usability evaluation of mHealth apps for the elderly has increased rapidly. The mHealth apps are often used as an auxiliary means of self-management to help the elderly manage their wellness and disease. According to the three stages of the mHealth app usability evaluation framework, the critical measures and evaluation methods are inconsistent. Future research should focus on selecting specific critical measures relevant to aging characteristics and adapting usability evaluation methods to elderly individuals by improving traditional tools, introducing automated evaluation tools and optimizing evaluation processes.

A usability model is a hierarchical structure encompassing the key elements such as users, user interface and interaction between them. It is a generic template which is independent of usability evaluation methods and provides flexibility for adaptation in different contexts and domains. In this paper, a usability model for medical user interfaces, especially for ventilator systems in Intensive Care Unit (ICU), is proposed based on Norman’s action-oriented seven-step model to capture a related medical context. A ventilator system is a therapeutic device, which provides a respiratory support to critically-ill patients. Currently, a usability of user interfaces of ventilator systems is evaluated by typical usability evaluation methods from software industry. These evaluation methods miss out important elements in medical context. Therefore, a need for a specialized usability model for medical user interfaces is fulfilled with a proposed usability model encompassing vital elements such as medical user, user interface, ICU environment and time required. This usability model is validated first, through a human work analysis using videos of selected tasks with medical user interfaces and then, with an overview of critical factors affecting medical user interfaces in ICU. In future, a proposed usability model can be integrated with a suitable usability evaluation method for evaluating medical user interfaces to identify related medical usability problems more effectively.

Different usability evaluation methods (UEMs) are often combined to assess the compliance of interactive systems with usability standards and guidelines. In practice, combinations of UEMs have been applied in a rather ad hoc manner resulting in inadequate analyses of rich usability data sets. While the potential offered by recent advances in technologies such as eye tracking for usability is recognized, and the challenges of evaluating complex web systems such as portals which involve different interaction styles are highlighted, the issue of effectively integrating multiple UEMs to make sense of observations has emerged. This paper presents a framework developed for effectively integrating UEMs in usability evaluations by matching the methods' capabilities and limitations with a classification of usability problems. The framework is applied to evaluating the usability of the Mawhiba web portal with target user populations which demonstrates how better coverage of usability issues can be achieved.

Determination of the effectiveness of two methods for usability evaluation using a CPOE medication ordering system

Combined Expert and User-Driven Usability Assessment of Trauma Decision Support Systems Improves User-Centered Design

The rapid proliferation of novel interaction paradigms—including Extended Reality (XR), tangible and natural user interfaces, and haptic systems—together with evolving user experience attributes such as emotional, social, and collaborative interactions, has fundamentally transformed human–computer interaction. These interaction modalities, increasingly integrated with artificial intelligence, expose the limitations of traditional usability and UX evaluation methods developed for conventional graphical user interfaces. Human–AI interaction introduces distinct design challenges related to transparency, explainability, user trust, and shared control, requiring evaluation approaches that shift from technology-centered assessment toward genuinely human-centered analysis.This research proposes a metric-based, AI-powered evaluation platform designed to systematically test and benchmark contemporary user interfaces and interaction modalities. The platform employs enhanced event logging and behavioral coding mechanisms to capture comprehensive multimodal interaction data, including task performance metrics, behavioral patterns, temporal action sequences, gaze trajectories, gesture dynamics, and physiological responses. Machine learning models analyze this multidimensional data to predict standard usability attributes—efficiency, effectiveness, and satisfaction as defined by ISO 9241-11—alongside emerging UX dimensions such as cognitive load, emotional engagement, and social presence.The platform architecture integrates high-performance XR-capable computing hardware with professional-grade graphics processing, multi-monitor visualization, eye-tracking sensors, haptic feedback devices, motion-capture systems, and biometric sensing infrastructure. Data collection is supported by curated training datasets derived from prior usability studies, expert heuristic evaluations, standardized task baselines, and cross-cultural UX assessment data. A custom-engineered, shock-resistant, transportable case with integrated power management and modular connectivity enables rapid deployment across laboratory, field, and organizational contexts, addressing ecological validity limitations inherent in traditional lab-based usability testing.Automated usability evaluation is driven by a hybrid AI framework combining three complementary models. Random Forest ensemble classifiers provide robust prediction across heterogeneous interaction data while offering interpretable feature importance measures to identify key usability determinants. Long Short-Term Memory (LSTM) networks model temporally ordered interaction sequences, enabling detection of behavioral signatures associated with confusion, flow states, hesitation, and error recovery. Support Vector Machines with radial basis function kernels capture complex non-linear relationships in high-dimensional usability data and perform effectively under limited expert-labeled training conditions.The central hypothesis posits that AI-driven automated usability evaluation can achieve accuracy and reliability comparable to expert-conducted heuristic evaluations and cognitive walkthroughs, validated through controlled A/B experiments comparing automated predictions with expert assessments across diverse interface types and user populations.By democratizing access to rigorous UX evaluation through automation, multimodal sensing, and portable deployment, the proposed platform aims to accelerate iterative design cycles for emerging interactive and human–AI systems while maintaining methodological rigor comparable to expert evaluation.

Previous studies have found out that educational applications using the Augmented Reality (AR) technology have provided immersion, motivation, fun and high level of engagement. However, designing and developing of the AR courseware with these characteristics while ensuring the effectiveness in teaching and learning process can be quite challenging. The increasingly use of AR has called for the creation of more usable products of high quality, which in turn, makes the Usability Evaluation testing an essential process. Previous researchers stressed that until very recently, where the Usability Evaluation of AR interfaces had not been systematically examined. Furthermore, nowadays usability is a fundamental factor to consider when developing educational courseware. Therefore, a good design for the usability evaluation methodology is needed which is best suited for Down Syndrome (DS) learners in early reading. This paper describes the review of usability evaluation in learning courseware and presenting the usability evaluation that has been successfully applied to previous AR applications. Next, we will present several existing usability evaluation methods as a guideline in conducting the usability evaluation for AR BACA SindD. Lastly, this paper proposed a Design of Usability Evaluation Methodology Framework for AR BACA SindD, especially for the DS children in this case study. Hopefully, the methods that were used for the evaluation as well as the suitable usability evaluation methodology proposed, can be deployed to evaluate the effectiveness of the courseware to disabled children as general and primarily to the DS children.

Recently, mobile educational games became a trend for primary school students, because it makes children learn in an entertaining way, and since nowadays they spend more time with their mobile devices, especially smartphones, the usage of this games is widespread. This kind of games must consider different aspects so these can cover all the purpose that they want to provide, such as usability, playability, learnability, effectiveness, simplicity, and so on. That is why the usability evaluation plays an important role in it. However, despite the fact that a lot of usability evaluation methods exist, most of them are focused on traditional computer usage and those are not 100% compatible with mobile phone usage. Therefore, a systematic literature review was conducted in order to identify usability evaluation guidelines for mobile educational games, which are concerning primary school students as users. This work is the first step toward making a set of usability guidelines for the evaluation of mobile educational games for Primary school students.

Mobile devices are used more and more frequently by society. In recent years has increased the sale of those devices and therefore in the applications provided to the users. The educational apps are one of the most important, because they offer more significant opportunities for development and growth to society, developing in various areas. In this context, usability becomes a key factor for its adoption.The objective of this paper is to identify which are the usability methods that are used in mobile educational applications by subject area. Therefore, a systematic mapping review was conducted. A total of 1503 studies were identified, and only 174 were selected for this review. The results show that the user test is the most widely used usability evaluation method in mobile educational applications, and the preferred learning topics are foreign languages and information and communication technologies. These results can guide experts and developers to construct usable mobile educational apps by selecting the usability evaluation methods according to the learning topic.