Making Usability Test Data Actionable! A Quantitative Test-Driven Prototyping Approach

Design thinking involves divergent and convergent thinking towards a satisficing solution with issues such as rationalization, creativity and how it functions during the process with design activities. The goal of this research is to understand design activities and cognitive processes during design phases through comparative analysis of novice and expert visual communication designers under theoretical framework. Research method for the study consists of three stages; 1) literature review, 2) experiment, and 3) analysis. After reviewing design thinking, visual communication design, and design activities and cognitive processes in design phases, a visual communication design task is performed by novice and expert designers in an experimental setting. Collected data includes concurrent self-reporting, writings, sketches, computer iterations, retrospective interview and researcher`s observational notes. Data assembled in one-minute time frames were examined by two coders for verbal and visual analyses. For the design process, all expert designers went through the phases of problem structuring, concept generation, concept visualization, design refinement, and detailing whereas two of the novice designers jumped into visualization followed by refinement and detailing. Experts worked on detailed design in between phases instead of at the end which indicates the ability to segment and conquer design task according to confronted circumstances. In reflection to design activities, novice designers produced significantly larger number of sketches compared to expert designers. Expert designers produced small number of sketches but they made substantial progress during the process. Novice designers used graphic software as a confirmatory tool whereas expert designers started working with computer from preliminary design phase for exploratory purposes. As for cognitive processes, expert designers exhibited more divergent thinking compared to novice designers. Expert designers went through divergent to convergent thinking for concept generation, and repeated the process one more time during visualization until meeting a satisficing design solution. Novices who overlooked concept generation went through divergent to convergent thinking during sketching and visualization using computer. The goal of this study was to investigate the thinking processes of visual communication designers during the entire design phases through comparative analysis of novice and expert designers. This study contributes to the comprehensive exploration of the design phases as most of the prior researches concentrated on the conceptual stage of design process. This sets example for examining design disciplines according to the design objectives, functions, artifacts, and how designers operate while designing. For the future research, specific issues found from the study need to be further investigated with larger number of participants for finding patterns of visual communication designers` thinking processes.

The manual metric measurement is time-consuming due to the large number of users and test criteria. Additionally, not all tester is able to complete all metrics measurement correctly in time. Software testers can process and analyze test data much more easily when using information systems in conducting usability testing. The purposes of this study is to develop a web-based information system of ISO metrics measurement and conducting usability tests on the Shopee application with 10 users. The level of usability of the Shopee application is highly dependent on measuring user happiness with the application. Usability testing matrics measurements are based on ISO measures that include effectiveness, efficiency, and satisfaction. The system is feasible to use and runs normally, according to the calculation results interpreted based on the Likert Scale, which came to 66.924%. However, there are still many users who do not understand how to use the features. It’s expected that this research will help sotware tester to improve the quality in conducting usability testing as well as provide feedbacks from the user in order to elevate the ease of use of the application.
 

Hallway testing, Remote Usability testing, Expert review, Automated expert review and A/B testing are the methods commonly used for Usability testing. However, there is no reliable system that integrates Brain Computer Interface (BCI) into the testing process with focus given towards user emotion analysis using electroencephalography (EEG) signals. This paper proposes a system that would be able to identify user emotions while they are conducting usability tests and the results would be able to increase the accuracy of the usability test. In the proposed system the results of the usability test would be displayed in real time on a dashboard and a summary report can be generated for distribution.

Data searching is an essential process in design, and research within the design field indicates a distinction in how expert designers and novice designers approach design problems. Expert designers, possessing a wealth of specialized knowledge, tend to search for additional information less frequently during the design process. In contrast, novice designers, due to their lack of professional knowledge, often need to search for, redefine, or organize information. (Ho, 2001; Cross et al., 1994). Consequently, novice designers tend to spend a significant amount of time repeatedly searching for information, leading not only to time wastage but also to designs that lack comprehensive consideration. Therefore, this study posits that the data collection process at the onset of design significantly affects the quality of design outcomes, necessitating further exploration of the relationship between various search strategies and design quality.The methodology involves think-aloud protocols, with the selection of coding systems based on search strategies divided into 0 to 3 dimensions as proposed by Gero and McNeill (1998) within the "problem domain" abstraction. Dimension 0 defines the system from the product's usage and user needs; 1 pertains to the product's interaction, styling, and imagery; 2 involves subsystems from product specifications, functions, behaviors, principles, and pain points; 3 considers details, integrating product principles with design elements and local details for effective optimization within the design concept. Additionally, regarding the quality of design outcomes, this study draws from the basic criteria for product design concepts proposed by Li, Feng-Qiang et al. (2016), encompassing ten aspects: functionality (F), usability (U), aesthetics and form (A), innovation (I), sustainability (ST), possibility (P), safety and regulation (SA), and marketability (M). These ten design criteria serve as a standard for assessing the quality of product design. Through the "problem domain" and "ten design criteria," this study aims to explore the differences in data search processes between novice and expert designers and the resulting impact on design quality.The findings reveal that novice designers mainly focus on the shallow, basic search dimensions of 1 and 2, lacking in-depth understanding of product details, which often leads to insufficient information for design execution. Experts iterate across dimensions 0 to 3, gaining a comprehensive understanding of the product, which better supports the subsequent design process. In terms of design criteria, novices focus on aesthetics/form (A) and innovation (I), with functionality also considered to some extent, while other aspects appear to be unconsciously neglected. Moreover, interviews and data organization revealed that novices tend to search from their personal experiences, whereas experts start from user needs, leading to more effective and accurate searches. Hence, in design education, teaching students to conduct comprehensive data collection could significantly enhance their design quality.

Information System usability level depends on acceptance and system convenience to be run by users. One of the methods to measure usability level is by conducting usability testing. This article elaborates usability testing for Crop and Farmer Activity Information System. This system is one of the agriculture information systems that is developed to record system activities for each farm field. This system is considered as one of the important role of Information and Communication Technology (ICT) for agriculture. This system has been developing since 2017 and needs to be assessed and tested. To assess the system, usability testing was conducted by taking sampling from two regions in Central Java, Temanggung and Gombong. The respondents are system administrators, farmers, and general users with each of respondents has different criteria. There are 58 respondents participated in this research: 49 farmers, 3 system administrators, and 6 general users. Usability testing was carried out by giving respondents several test tasks based on the system. Each respondent had different kind of test task in accordance with the system functionality for each user. The result of the test found that system administrator user interface assessment value gained average percentage of 69%, while the farmers gained 76% and general users gained 79%. From the test, it also bring some recommendations for system refinement. Those recommendations were taken from user inputs and user test results. The recommendations have been made to bring better system environment.

This research project focused upon the design strategies adopted by expert and novice designers. It was based upon a desire to compare the design problem solving strategies of novices, in this case key stage three pupils studying technolgy within the United Kingdom National Curriculum, with designers who could be considered to have developed expertise. The findings helped to provide insights into potential teaching strategies to suit novice designers. Verbal protocols were made as samples of expert and novice designers solved a design problem and talked aloud as they worked. The verbalisations were recorded on video tape. The protocols were transcribed and segmented, with each segment being assigned to a predetermined coding system which represented a model of design problem solving. The results of the encoding were analysed and consideration was also given to the general design strategy and heuristics used by the expert and novice designers. The drawings and models produced during the generation of the protocols were also analysed and considered. A number of significant differences between the problem solving strategies adopted by the expert and novice designers were identified. First of all, differences were observed in the way expert and novice designers used the problem statement and solution validation during the process. Differences were also identified in the way holistic solutions were generated near the start of the process, and also in the cycles of exploration and the processes of integration. The way design and technological knowledge was used provided further insights into the differences between experts and novices, as did the role of drawing and modelling during the process. In more general terms, differences were identified in the heuristics and overall design strategies adopted by the expert and novice designers. The above findings provided a basis for discussing teaching strategies appropriate for novice designers. Finally, opportunities for future research were discussed.

Usability engineering and usability testing are concepts that continue to evolve. Interesting research studies and new ideas come up every now and then. This paper tests the hypothesis of using an EDA-based physiological measurements as a usability testing tool by considering three measures; which are observers‟ opinions, self-reported data and EDA-based physiological sensor data. These data were analyzed comparatively and statistically. It concludes by discussing the findings that has been obtained from those subjective and objective measures, which partially supports the hypothesis.

Recently, demand of a suits for disabled in wheelchair who is accessible to economic activities has been increased. It is necessary to design the online suits customizing system considering customizability, accessibility, and economics, in order to solve the problem related to the suits of the disabled in wheelchair. Development of a web usability testing tool specialized in the disabled in wheelchair is crucial to design the online suits customizing system. The present study is intended to develop a web usability testing tool to analyze the usability of online men’s suits customizing system for the disabled in wheelchair systematically. A three-step approach was applied in the study: (1) literature review of ten studies related to various website usability testing, (2) classification of factors for web usability testing tool into website components and evaluation criterion through factor elimination/integration and frequency analysis of usage, and (3) selection of evaluation items through analysis of relationship between website components and evaluation criterion. As a result, 26 website components and 6 evaluation criterion were established in the web usability testing tool. Website components were consisted of 7 style (e.g., color, layout), 8 information (e.g., terminology, title), 8 function (e.g., link, navigation), and 3 custom suits (e.g., suit fabrics, body size). Evaluation criterion were consisted of interactivity, attractiveness, accessibility, error prevention, informativeness, and consistency. A total of 36 evaluation items were selected through analysis of relationship between 26 website components and 6 evaluation criterion. The usability testing tool developed in the present study can be used for identifying the usability of online customizing system specialized in the disabled in wheelchair effectively. key words Disabled in Wheelchair, Men’s Custom Suits, Online Customizing System

Usability engineering and usability testing are concepts that continue to evolve. Interesting research studies and new ideas come up every now and then. This paper tests the hypothesis of using an EDA based physiological measurements as a usability testing tool by considering three measures which are observers opinions, self reported data and EDA based physiological sensor data. These data were analyzed comparatively and statistically. It concludes by discussing the findings that has been obtained from those subjective and objective measures, which partially supports the hypothesis.

BackgroundLynch syndrome is an underdiagnosed hereditary condition carrying an increased lifetime risk for colorectal and endometrial cancer and affecting nearly 1 million people in the United States. Cascade screening, systematic screening through family members of affected patients, could improve identification of Lynch syndrome, but this strategy is underused due to multi-level barriers including low knowledge about Lynch syndrome, low access to genetics services, and challenging family dynamics.MethodsWe used intervention mapping, a 6-step methodology to create stakeholder-driven interventions that meet the needs of a target population, to develop an intervention to improve cascade screening for Lynch syndrome. The intervention development process was guided by input from key stakeholders in Lynch syndrome care and patients. We conducted usability testing on the intervention with Lynch syndrome patients using qualitative semi-structured interviewing and rapid qualitative analysis.ResultsWe developed a workbook intervention named Let’s Talk that addresses gaps in knowledge, skills, self-efficacy, outcome expectancy and other perceived barriers to cascade screening for Lynch syndrome. Let’s Talk contained educational content, goal setting activities, communication planning prompts and supplemental resources for patients to plan family communication. Evidence-based methods used in the workbook included information chunking, guided practice, goal setting and gain-framing. We conducted usability testing focused on the complexity and relative advantage of the intervention through 45-min virtual interviews with 10 adult patients with Lynch syndrome recruited from a national advocacy organization in the United States. Usability testing results suggested the intervention was acceptable in terms of complexity and relative advantage to other available resources, but additional information for communication with young or distant family members and a web-based platform could enhance the intervention’s usability.ConclusionsIntervention mapping provided a framework for intervention development that addressed the unique needs of Lynch syndrome patients in overcoming barriers to cascade screening. Future work is needed to transform Let’s Talk into a web-based tool and evaluate the effectiveness of the intervention in clinical practice with patients and genetic counselors. Intervention mapping can be useful to researchers as an evidence-based technique to develop stakeholder-centered interventions for addressing the needs of other unique populations.

Introduction: Cardiac rehabilitation improves outcomes for patients with ischemic heart disease or heart failure, but is underused. New strategies to improve delivery of cardiac rehabilitation are needed. We developed a mobile application for technology-facilitated home cardiac rehabilitation and sought to determine its usability. Methods: We recruited patients eligible for cardiac rehabilitation who had access to a smartphone, tablet, or computer with internet access to participate in usability testing of the mobile application. The mobile application includes physical activity goal setting, logs for physical activity and health measures, health education, reminders, and feedback (Figure). Participants were introduced to the mobile application and then observed while completing pre-specified tasks with the mobile application. Participants completed the System Usability Scale (0-100), rated likelihood to use the mobile application (0-100), and participated in a semi-structured interview. Based on participant feedback, we made iterative revisions to the mobile application. Results: We conducted usability testing in 13 participants. The first version of the mobile application was used by the first 5 participants, and revised versions were used by the final 8 participants. From the first version to revised versions, task completion success rate improved from 44% to 78% (p=0.05), System Usability Scale improved from 54% to 76% (p=0.04), and rated likelihood of using the mobile application remained high at 76% and 87% (p = 0.30). Interview responses revealed a need for introductory training ( “Initially, training with a technical person, instead of me relying on myself”) and on-demand help (“ If I had problems I’d try to find out how to fix it on this or call you” ). Additionally, many participants were interested in sharing data with providers (“ I can show my doctor what I’ve been working on ”). Conclusions: With participant feedback and iterative revisions, we significantly improved the usability of a mobile application for cardiac rehabilitation. Patient expectations for using a mobile application for cardiac rehabilitation include introductory training, on-demand help, and sharing data with providers. Iterative mixed-method evaluation may be useful for improving the usability of health technology.

There is a lack of research to guide collaborative treatment decision-making for children who have complex behavioral problems, despite the extensive use of mental health services in this population. We developed and pilot-tested a one-page Option Grid™ patient decision aid to facilitate shared decision-making for these situations. An editorial team of parents, child psychiatrists, researchers, and other stakeholders developed the scope and structure of the decision aid. Researchers included information about a carefully chosen number of psychosocial and pharmacological treatment options, using descriptions based on the best available evidence. Using semi-structured qualitative interviews (n = 18), we conducted user testing with four parents and four clinical prescribers and field testing with four parents, four clinical prescribers, and two clinic administrators. The researchers coded and synthesized the interview responses using mixed inductive and deductive methods. Parents, clinicians, and administrators felt the Option Grid had significant value, although they reported that additional training and other support would be required in order to successfully implement the Option Grid and achieve shared decision-making in clinical practice.

Usability testing is a recommended main testing method to evaluate the usability or ease of use of a software. However, this approach can induce bias. Usability testing with survey method using questionnaire instruments and interview has a risk of generating lack of objectivity from the participants because the facilitator can influence the evaluation results. The evaluation process is not optimal and the biases that arise from usability testing can affect the success or failure of a software in the market. A method for validating usability testing results needs to be studied to ensure whether the test results obtained are valid and unbiased. Electroencephalogram (EEG) is one of the tool that can be used to validate usability testing. This study aims to conduct usability testing of the three most popular e-marketplace applications in Indonesia. From the results it can be seen that usability testing using user-based testing gives a fairly good result in term of accuracy. Usability testing using EEG gives lower results than user-based testing. Nevertheless, we cannot rule out EEG as a usability testing method in the future.

In teaching online technical communication courses, shaping an electronic interface requires extensive consideration of the user experience, both for students and for faculty members who design and teach the courses. Technical communication faculty members should provide strong examples of effective user experiences and should be leaders in making the interfaces of online learning management systems as usable as possible.Principles of usability designed for general web sites may or may not apply to learning management systems designed for educational purposes. In order to create effective online technical communication courses, one needs to consider both usability concerns and pedagogical concerns.To assess the usability and pedagogical effectiveness of online courses, faculty members may use indirect means such as heuristic analyses. In addition, they may use direct means such as usability testing, student feedback, and analytic tools. Each approach has advantages as well as limitations. Faculty members will gain the richest information through using multiple approaches.In assessing usability and pedagogical effectiveness, faculty members also need to consider the situational constraints and resources in their unique contexts. Understanding and adapting their approaches to use resources well and to work within constraints will benefit their abilities to enhance their student users' experiences with online courses.

It is all too common for products, such as consumer appliances, information systems, mobile apps, and websites, to cause trouble and frustration. For example, products are often difficult or dull to use, make tasks less flexible or more tedious, shift attention away from important or gratifying activities, and simply fail to deliver expected benefits or experiences. By identifying such trouble and frustration in the lab prior to widespread use, usability tests have proven a valuable method for informing redesign efforts. A usability test consists of having test users exercise a product and think aloud about their experience using it, while an evaluator observes the users and listens in on their thoughts. On this basis, the evaluator identifies usability problems and assesses the user experience. This book describes how to conduct usability tests. After providing context about concepts and testing, the main chapters of the book cover the steps involved in preparing for a usability test, executing the test sessions, and analyzing the test data. Throughout the chapters, concrete guidance is balanced against more complex issues with an impact on the robustness, validity, completeness, impact, and cost of a usability test. The book concludes with an outlook to variations of usability testing and alternatives to it.