Technical Review: Current Issues of Usability Testing

Evaluation of Premium Ultrasound Systems With Sonographers: How Comparative Usability Testing Can Reveal Human Factors Issues Prior to Errors in the Exam Room1

Usability testing is a recommended main testing method to evaluate the usability or ease of use of a software. However, this approach can induce bias. Usability testing with survey method using questionnaire instruments and interview has a risk of generating lack of objectivity from the participants because the facilitator can influence the evaluation results. The evaluation process is not optimal and the biases that arise from usability testing can affect the success or failure of a software in the market. A method for validating usability testing results needs to be studied to ensure whether the test results obtained are valid and unbiased. Electroencephalogram (EEG) is one of the tool that can be used to validate usability testing. This study aims to conduct usability testing of the three most popular e-marketplace applications in Indonesia. From the results it can be seen that usability testing using user-based testing gives a fairly good result in term of accuracy. Usability testing using EEG gives lower results than user-based testing. Nevertheless, we cannot rule out EEG as a usability testing method in the future.

Usability is one factor that influences software design success during the software testing process. Usability testing is used to evaluate whether the software meets user needs or not. Usability testing is usually carried out using a certain questionnaire method, but the distribution and measurement of the results are carried out separately. This makes the researchers spend more time testing the application. Therefore, this study provides a solution by designing and building web-based applications to measure the usability of a software design or product with the SUS, UMUX, and UMUX-lite questionnaires. The development method used in this study is Rapid Application Development (RAD) with the stages of the process starting with problem identification, literature study, needs planning, RAD design workshops, implementation, and report writing. The application was built using the Laravel framework and the database using MySQL. The main features of the online questionnaire application were the creation of a questionnaire and the analysis of the results of the questionnaire. The functionality of the application was tested by using the Black Box testing method. Then the developed application was also utilized in usability testing using the SUS, UMUX, and UMUX-lite questionnaires. The results of application usability testing using the SUS, UMUX, and UMUX-Lite methods obtained a final score of 88; 91,28, and 82,05 where all these values mean the application can be accepted and used. Keywords: Usability Testing, Rapid Application Development, SUS, UMUX, UMUX-Lite

Information System usability level depends on acceptance and system convenience to be run by users. One of the methods to measure usability level is by conducting usability testing. This article elaborates usability testing for Crop and Farmer Activity Information System. This system is one of the agriculture information systems that is developed to record system activities for each farm field. This system is considered as one of the important role of Information and Communication Technology (ICT) for agriculture. This system has been developing since 2017 and needs to be assessed and tested. To assess the system, usability testing was conducted by taking sampling from two regions in Central Java, Temanggung and Gombong. The respondents are system administrators, farmers, and general users with each of respondents has different criteria. There are 58 respondents participated in this research: 49 farmers, 3 system administrators, and 6 general users. Usability testing was carried out by giving respondents several test tasks based on the system. Each respondent had different kind of test task in accordance with the system functionality for each user. The result of the test found that system administrator user interface assessment value gained average percentage of 69%, while the farmers gained 76% and general users gained 79%. From the test, it also bring some recommendations for system refinement. Those recommendations were taken from user inputs and user test results. The recommendations have been made to bring better system environment.

Usability has become an important consideration for product development, and as a result, there is a growing need for systems and tools that can support usability test projects. However, few studies so far have developed such systems and tools. During a usability test project, many participants take up different roles, such as project managers, usability testers, and subjects. We implement a web-based usability test system with which a project manager can manage and control all participants and documents throughout the entire usability test process, from the design of a usability test project to the analysis of the test results. A usability test generates many documents, such as subject agreement forms and before/after questionnaires. Since many different subjects can participate in a usability test, consistency during testing with different participants and efficient document management are the keys to success for a usability test. Since all users that participate in usability test projects can access web-based usability test systems through a web browser, regardless of the place where they are, the reliability of the testing results can improve since the tests are conducted in the locations where the target products are meant to be used. In particular, our system is useful for disabled individuals who cannot move.

Stimulated by regulations and standards as well as commercial imperatives, medical device manufacturers are striving to make medical devices safer by decreasing the potential for harmful use errors. Accordingly, manufacturers are observing and interviewing intended users about their interactions with devices en route to developing user interface requirements; applying human factors engineering (HFE) principles when designing user interfaces; and conducting formative and summative usability tests to improve and validate their devices’ interactive quality. This has represented significant work for manufacturers, particularly the majority who started with little HFE knowledge and experience. Today, the “awakening” to HFE in medical device development is essentially over—an established manufacturer who is just now discovering HFE has been out of touch. After all, the U.S. Food and Drug Administration (FDA) and AAMI conducted their first joint conference on HFE in 1995 when they previewed HFE expectations related to recent changes in quality system regulation (QSR). So, it has been 17 years since HFE entered the medical industry’s zeitgeist. HFE specialists working in the medical industry used to focus on making devices more usable (i.e., user friendly). Since the QSR change, use safety has become the primary focus, resulting in user interfaces that are less vulnerable to harmful use errors. It is rather obvious how HFE affects the safety of some medical devices. In the design of a surgical instrument, for example, HFE’s aim would be to reduce the chance of cutting anything other than the intended target. This goal could be accomplished by including a blade guard and a mechanical interlock that requires more than one deliberate action to enable cutting.

The user experience of human-robot interaction (HRI) has become more and more important, and the HRI evaluation of service robots has become an indispensable research topic. This paper proposes that besides the commonly used usability testing, emotional factors should be considered. The emotional response of users when using robot products should be measured. The paper also describes that, based on the results of usability testing and emotion measurement, the Kano model can be referred to analyze the requirements to determine the requirements which users concern most, which attract users, and which users think the product must have. Finally, we conduct a usability testing and requirements analysis of a test service robot Alpha 2 as an example.

The only way to judge a product's acceptance in the workplace is through its use. However, before a product is released into the marketplace, its developers would like to predict its acceptability in the target market. One predictor of acceptability is usability test results. Typically, usability testing takes place outside of the user's natural environment in a usability test lab, an artificial environment. This article suggest ways in which ethnographic principles, historically used to describe a culture from the point of view of someone within that culture, can be used along with traditional usability testing to predict a product's acceptability in the marketplace.

The growth of the Internet and related technologies has enabled the development of a new breed of dynamic websites that is growing rapidly in use and that has had a huge impact on many businesses. One type of websites that have been widely spread and are being widely adopted is the educational websites. There are many forms of educational websites, such as free online websites and Web-based server software. This creates challenges regarding their continuing evaluation and monitoring in order to measure their efficiency and effectiveness, to assess user satisfaction and, ultimately, to improve their quality. The lack of an adaptive usability checklist for improvement of the usability assessment process for educational systems represents a missing piece in ‘usability testing’. This paper presents an adaptive Domain-Specific Inspection (DSI) checklist as a tool for evaluating the usability of educational systems. The results show that the adaptive educational usability checklist helped evaluators to facilitate the evaluation process. It also provides an opportunity for website owners to choose the usability area(s) that they think need to be evaluated. Moreover, this method was more efficient and effective than user testing (UT) and heuristics evaluation (HE) methods.

Usability testing is a method of evaluating the user experience of a product or service by testing it with real users. Usability testing is an essential step in the product development process, as it helps to ensure that the product is easy to use and meets the needs of its intended users. Websites are an important communication and interaction channel for many businesses, and it is important to ensure that they are easy to use and provide a good user experience. It provides various information technology facilities that can support companies in improving and optimizing their business processes. However, some website still has deficiencies in terms of website interactivity and a responsive display when the website changes to a User Interface display on several gadget devices. Therefore, the main objective of this testing process is to ensure that each scenario in the application has run as expected and meets the standard requirements written in three main aspects: System Aspect, User Aspect, and Interaction Aspect. This study uses the UAT (User Acceptance Testing) method to determine the responses of respondents (users) to the system that has been built. Based on the data from the questionnaire results, three aspects are assessed: the system aspect, which gets an average score of 3.243, and the user aspect, which is around 3.186. and interaction aspects with an average value of 3.22. In short, the research that has been carried out using usability testing analysis in our case study has generated positive ratings that can satisfy users when accessing the website, although some respondents still give pretty bad ratings.

The term document testing refers broadly to a cluster of techniques used to assess the accuracy, appeal, and appropriateness of a document. Appropriateness includes characteristics such as accessibility, effectiveness, and usability. Document testing is widely acknowledged as a critical aspect of developing and producing usable texts--from operator's manuals to on-line documentation, from form letters to in-house guides for service representatives (Ramey, 1989; Dumas & Redish, 1993). Researchers urge a broad view of testing--one that encourages testing throughout the design and development processes, incorporates a balanced representation of assessment strategies, and considers the context in which a document is used (Dumas, 1989; Jeffries & Desurvire, 1992; Sullivan, 1989). Researchers confirm that a well-designed, on-going testing program has numerous tangible and intangible benefits that include saving time and frustration in document design and development; saving money in service calls, maintenance, training, and revisions; and improving customer service as well as the organization's image (Guillemette, 1989; Potosnak & Koffler, 1986; Redish, 1994). CATEGORIES OF DOCUMENT TESTING Three categories of document testing--text-based, expert-based, and user-based (Schriver, 1989)--are distinguished by the way in which the information is collected and the strengths of each method for providing information about document quality and usability (Sullivan, 1989). Text-based testing concentrates on the words and sentences of a document, focusing on local-level features of a document and then drawing conclusions about characteristics such as reading level. Text-based testing includes checklists as well as readability tests and computer programs to assess structural and stylistic features. Expert-based testing gathers information from professionals who provide formal and informal peer reviews, technical reviews, editorial reviews, and document design reviews. Such testing is useful for assessing the technical accuracy of a document as well as the selection of supporting evidence and level of detail for the intended audience. Reader-based or usability testing gets information about a document's usability directly from the users in two ways: after they have finished reading a document (which is called retrospective testing) and as they read and use a document (which is called concurrent testing). Retrospective testing includes such methods as questionnaires, interviews, focus groups, comprehension tests, and reader-feedback cards. Retrospective user-based testing often provides useful information from actual users. However, the feedback should be used cautiously because reader's memories are not necessarily accurate, and the information they provide is often vague. Several kinds of concurrent testing can be especially useful. * Observations: A test administrator watches a reader read a document, usually in order to perform a task. For example, the tester can observe the amount of time and apparent ease (or difficulty) a reader or user has in locating information or performing individual tasks or sequences of tasks. * Think-aloud protocols: A reader is asked to think aloud--that is, to say aloud all comments, reactions, and opinions, which are tape-recorded and often transcribed. * Reading protocols: A reader is asked to read aloud as well as comment on the text. In a cued-reading protocol, a reader is asked to stop at marks (cues) in the text and comment--for example, provide a gist of content and/or predict what's to come. For all types of protocols, tape recordings or transcripts can be examined to identify patterns of problems. * Co-discovery: Two (or occasionally more) users are asked to work together to discuss and solve a problem or complete a task, such as installing computer software. …

Usability testing has gained a rather stable status as a method for usability evaluation even though it has both low reliability and validity. The sources of result variance are well acknowledged among researchers and practitioners. However, the validity problem has not been explicated or exemplified although it is frequently discussed in the literature how the results of usability tests should be interpreted and to what extent results are generalizable. We employ Activity Theory and a case example to argue that the validity problem is mainly caused by the fact that what we are testing are artifacts and what people are using in their real life activities are tools and these two entities are qualitatively different. Basing on our analysis, the effects of the reliability and validity problems on the application of usability testing and its role as one of the tools in the design process are discussed.

Determining fidelity of mixed prototypes: Effect of media and physical interaction

Pressure from customers, the trade press, and the competition is causing companies to focus more on delivering usable products. But even when the commitment is there, companies still find it difficult to get their developers to believe in the results of usability tests and to make improvements based on them. The authors report on what Lotus has done to bring developers closer to the process, describing tools, techniques, and concepts to optimize user interfaces. They have found four approaches particularly valuable in getting developers and other members of the product team involved in usability testing: informal setups, sit-in sessions, usability nights, and videotapes. Their experience shows that these methods are more effective than conventional practices, in which developers merely receive reports of key problems from the test sessions. >

Recently, demand of a suits for disabled in wheelchair who is accessible to economic activities has been increased. It is necessary to design the online suits customizing system considering customizability, accessibility, and economics, in order to solve the problem related to the suits of the disabled in wheelchair. Development of a web usability testing tool specialized in the disabled in wheelchair is crucial to design the online suits customizing system. The present study is intended to develop a web usability testing tool to analyze the usability of online men’s suits customizing system for the disabled in wheelchair systematically. A three-step approach was applied in the study: (1) literature review of ten studies related to various website usability testing, (2) classification of factors for web usability testing tool into website components and evaluation criterion through factor elimination/integration and frequency analysis of usage, and (3) selection of evaluation items through analysis of relationship between website components and evaluation criterion. As a result, 26 website components and 6 evaluation criterion were established in the web usability testing tool. Website components were consisted of 7 style (e.g., color, layout), 8 information (e.g., terminology, title), 8 function (e.g., link, navigation), and 3 custom suits (e.g., suit fabrics, body size). Evaluation criterion were consisted of interactivity, attractiveness, accessibility, error prevention, informativeness, and consistency. A total of 36 evaluation items were selected through analysis of relationship between 26 website components and 6 evaluation criterion. The usability testing tool developed in the present study can be used for identifying the usability of online customizing system specialized in the disabled in wheelchair effectively. key words Disabled in Wheelchair, Men’s Custom Suits, Online Customizing System