Assessing the Utility of the System Usability Scale for Evaluating Voice-based User Interfaces

BackgroundUnderstanding the impact of socio-economic inequality on health outcomes is arguably more relevant than ever before given the global repercussions of Covid-19. With limited resources, innovative methods to track disease, population needs, and current health and social service provision are essential. To best make use of currently available data, there is an increasing reliance on technology. One approach of interest is the implementation and integration of mapping software. This research aimed to determine the usability and acceptability of a methodology for mapping public health data using GIS technology.MethodsPrototype multi-layered interactive maps were created demonstrating relationships between socio-economic and health data (vaccination and admission rates). A semi-structured interview schedule was developed, including a validated tool known as the System Usability Scale (SUS), which assessed the usability of the mapping model with five stakeholder (SH) groups. Fifteen interviews were conducted across the 5 SH and analysed using content analysis. A Kruskal-Wallis H test was performed to determine any statistically significant difference for the SUS scores across SH. The acceptability of the model was not affected by the individual use of smart technology among SHs.ResultsThe mean score from the SUS for the prototype mapping models was 83.17 out of 100, indicating good usability. There was no statistically significant difference in the usability of the maps among SH (p = 0.094). Three major themes emerged with respective sub-themes from the interviews including: (1) Barriers to current use of data (2) Design strengths and improvements (3) Multiple benefits and usability of the mapping model.ConclusionIrrespective of variations in demographics or use of smart technology amongst interviewees, there was no significant difference in the usability of the model across the stakeholder groups. The average SUS score for a new system is 68. A score of 83.17 was calculated, indicative of a “good” system, as falling within the top 10% of scores. This study has provided a potential digital model for mapping public health data. Furthermore, it demonstrated the need for such a digital solution, as well as its usability and future utilisation avenues among SH.

Background and importanceAn online personalised health record (PHR) is a valid tool to reduce medication discrepancies (MDs), defined as unexplained differences among medication regimens. The success of a PHR depends...

BackgroundWearable devices have the potential to promote a healthy lifestyle because of their real-time data monitoring capabilities. However, device usability is a critical factor that determines whether they will be adopted on a large scale. Usability studies on wearable devices are still scarce.ObjectiveThis study aims to compare the functions and attributes of seven mainstream wearable devices and to evaluate their usability.MethodsThe wearable devices selected were the Apple Watch, Samsung Gear S, Fitbit Surge, Jawbone Up3, Mi Band, Huawei Honor B2, and Misfit Shine. A mixed method of feature comparison and a System Usability Scale (SUS) evaluation based on 388 participants was applied; the higher the SUS score, the better the usability of the product.ResultsFor features, all devices had step counting, an activity timer, and distance recording functions. The Samsung Gear S had a unique sports track recording feature and the Huawei Honor B2 had a unique wireless earphone. The Apple Watch, Samsung Gear S, Jawbone Up3, and Fitbit Surge could measure heart rate. All the devices were able to monitor sleep, except the Apple Watch. For product characteristics, including attributes such as weight, battery life, price, and 22 functions such as step counting, activity time, activity type identification, sleep monitoring, and expandable new features, we found a very weak negative correlation between the SUS scores and price (r=−.10, P=.03) and devices that support expandable new features (r=−.11, P=.02), and a very weak positive correlation between the SUS scores and devices that support the activity type identification function (r=.11, P=.02). The Huawei Honor B2 received the highest score of mean 67.6 (SD 16.1); the lowest Apple Watch score was only 61.4 (SD 14.7). No significant difference was observed among brands. The SUS score had a moderate positive correlation with the user’s experience (length of time the device was used) (r=.32, P<.001); participants in the medical and health care industries gave a significantly higher score (mean 61.1, SD 17.9 vs mean 68.7, SD 14.5, P=.03).ConclusionsThe functions of wearable devices tend to be homogeneous and usability is similar across various brands. Overall, Mi Band had the lowest price and the lightest weight. Misfit Shine had the longest battery life and most functions, and participants in the medical and health care industries had the best evaluation of wearable devices. The perceived usability of mainstream wearable devices is unsatisfactory and customer loyalty is not high. A consumer’s SUS rating for a wearable device is related to their personal situation instead of the device brand. Device manufacturers should put more effort into developing innovative functions and improving the usability of their products by integrating more cognitive behavior change techniques.

BackgroundRepeated applications of short-term dietary assessment instruments are recommended for estimating usual dietary intake. For this purpose, NutriDiary, a smartphone app for collecting weighed dietary records (WDRs) in the German population, was developed.ObjectiveWe aim to describe NutriDiary and evaluate its usability and acceptability.MethodsNutriDiary was developed as a WDR, allowing users to enter food items via text search, barcode scanning, or free text entry. The sample for the evaluation study included 74 participants (n=51, 69% female, aged 18‐64 years), including 27 (37.5%) experts and 47 (63.5%) laypersons (including n=22, 30%, nutrition students). Participants completed a 1-day WDR and entered a predefined sample meal (n=17 foods) the following day by using NutriDiary. An evaluation questionnaire was answered from which the system usability scale (SUS) score (0‐100) was calculated. A backward selection procedure (PROC REG in SAS; SAS Institute) was used to identify potential predictors for the SUS score (age, sex, status [expert or laypersons], and operating system [iOS or Android]).ResultsThe median SUS score of 75 (IQR 63‐88) indicated good usability. Age was the only characteristic identified as a potential predictor for a lower SUS score (P<.001). The median completion time for an individual WDR was 35 (IQR 19‐52) minutes. Older participants took longer to enter the data than younger ones (18‐30 y: median 1.5, IQR 1.1‐2.0 min/item vs 45‐64 y: median 1.8, IQR 1.3‐2.3 min/item). Most participants expressed a preference for NutriDiary over the traditional paper-based method.ConclusionsGood usability and acceptability make NutriDiary promising for use in epidemiological studies.

Gamification is the integration of game mechanics in non-gaming contexts, such as education, to enhance motivation and engagement. This study aimed to evaluate the usability of a gamification-based Moodle learning management system (LMS) by having 48 computer science students aged between 15 and 24 years perform tasks and provide feedback on the System Usability Scale (SUS). 29 participants provided feedback, with 18 having prior LMS experience and 11 reporting no prior experience. An Independent Samples T-Test determined that there was no significant difference in System Usability Scale scores between male and female groups or between experienced and non-experienced groups. This suggests that gender and prior experience with an LMS do not affect the perceived usability of a gamification-based Moodle LMS.

A pair of commercially available smart glasses, Epson BT-200, was evaluated with different information display modes for assisting machine operators. Two information display modes were selected, displaying animated images only, and displaying animated images with text illustrations. Direction prompts were added in the system. Whether the prompts are helpful or not was also discussed in this study. Task completion time and error rate were collected during the experiment. System Usability Scale (SUS) and NASA-Task Load Index (NASA-TLX) were used to collect subjective information after each experiment. Twenty participants (10 males and 10 females) were recruited. The results showed that the display mode of having smart glasses displaying animated images versus displaying animated images with text illustrations showed no significant difference in task completion time and error rate. But the significant difference was found in the result of SUS scores. Participants preferred the display mode of animate images with text illustrations according to the SUS scores. Moreover, the results indicated that the use of direction prompts had significant influences upon all the measures. Participants completed the tasks faster and had lower error rate by using the smart glasses with direction prompts. And the results of subjective ratings also showed higher SUS score and lower NASA-TLX score were associated with using smart glasses with direction prompts. Thus, the implementation of using smart glasses to guide machine operations should be considered the design of adding direction prompts to increase efficiency and effectiveness of the operations.KeywordsAugmented realityInformation displaySmart glasses

The System Usability Scale (SuS) survey is a widely respected tool for measuring usability. Generally, a SuS score is administered directly after a usability test to assess the usability and user experience of digital products. However, some researchers have used SuS as a survey as part of longitudinal ‘in the wild’ trials where SuS is often completed some period after the trial. The aim of this research was to determine if a participant’s memory of a product’s usability would change if a SuS survey was administered at different times after a test. Hence, we sought to understand if recalling the usability of a digital technology was affected by temporal bias or memory decay. This paper includes results and findings from two studies, study 1 involved evaluating a web application and study 2 involved evaluating a virtual learning environment. Collectively the two studies had 212 participants (n = 212). The findings conclude that there is no significant change of the user’s recollection of the usability of digital product as evidenced by an analysis of users who completed multiple SuS surveys over a short term period of 3 weeks or over an extended period of time of 6 months. RESEARCH HIGHLIGHTS The system usability scale is used to test for memory decay and temporal bias in judging the user experience of technologies at different time points. 212 participants took part in two studies ranging from 3 weeks to 6 months. There is no evidence that there is a temporal bias or memory decay when users complete a SuS survey at the two different time points of 3 weeks and 6 months.

BackgroundTransparency is increasingly called for in health care, especially, when it comes to patients’ access to their electronic health records. In Sweden, the e-service Journalen is a national patient accessible electronic health record (PAEHR), accessible online via the national patient portal. User characteristics and perceived benefits of using a PAEHR influence behavioral intention for use and adoption, but poor usability that increases the effort expectancy can have a negative impact. It is, therefore, of interest to explore how users of the PAEHR Journalen perceive its usability and usefulness.ObjectiveThe aim of this study was to explore how the users of the Swedish PAEHR experience the usability of the system and to identify differences in these experiences based on the level of transparency of the region.MethodsA survey study was conducted to elicit opinions and experiences of patients using Journalen. The data were collected from June to October 2016. The questionnaire included questions regarding the usability of the system from the System Usability Scale (SUS). The SUS analysis was the focus of this paper. Analysis was performed on different levels: nationally looking at the whole data set and breaking it down by focusing on 2 different regions to explore differences in experienced usability based on the level of transparency.ResultsDuring the survey period, 423,141 users logged into Journalen, of which 2587 unique users completed the survey (response rate 0.61%). The total mean score for all respondents to the SUS items was 79.81 (SD 14.25), which corresponds to a system with good usability. To further explore whether the level of transparency in a region would affect the user’s experience of the usability of the system, we analyzed the 2 regions with the most respondents: Region Uppsala (the first to launch, with a high level of transparency), and Region Skåne (an early implementer, with a low level of transparency at the time of the survey). Of the participants who responded to at least 1 SUS statement, 520 stated that they had received care in Region Skåne, whereas 331 participants had received care in Region Uppsala. Uppsala’s mean SUS score was 80.71 (SD 13.41), compared with Skåne’s mean of 79.37 (SD 13.78).ConclusionsThe Swedish national PAEHR Journalen has a reasonably good usability (mean SUS score 79.81, SD 14.25); however, further research into more specific usability areas are needed to ensure usefulness and ease of use in the future. A somewhat higher SUS score for the region with high transparency compared with the region with low transparency could indicate a relationship between the perceived usability of a PAEHR and the level of transparency offered, but further research on the relationship between transparency and usability is required.

Home healthcare devices have gained attention recently as more and more care is being delivered in the patients’ home rather than in a hospital setting. This trend is accelerating as healthcare systems work to restrain costs and provide patients with the tangible benefits of receiving care in their own home. Because the user population of these devices is moving away from trained professionals, the usability of home healthcare devices takes on greater importance. Poor usability of these devices could result in negative consequences, ranging from incorrect diagnoses to patient injury or death. In this study, the usability of four common home healthcare devices (a manual blood pressure monitor, a temporal artery thermometer, a pulse oximeter, and a spirometer) was evaluated by 74 participants. The objective metrics of effectiveness (success), efficiency (time on task) and the subjective metric of satisfaction were collected for each device, and correlated with the users’ subjective assessment of the usability of each device, as measured with the System Usability Scale (SUS). Results show that there is a positive correlation between effectiveness and SUS scores, no correlation between efficiency and SUS, and a positive correlation between satisfaction and SUS.

Adx International is one of the international shipping services that has collaborated with various marketplaces. Along with the development of technology, Adx International chose a website as a promotional material as well as a system for the transaction process. This research will discuss problems with the UI/UX design of the website by distributing questionnaires to users. After that, several problems were found such as problems with navigation design (search filters), colors that are not aligned, excessive use of icons and there is no guidebook for using the Adx International website. Therefore, further user interface analysis is required. To evaluate the sustainability of this platform, a combination of Heuristic and System Usability Scale (SUS) methods are used in this research because the two methods have different but complementary focuses. Heuristic provides perspectives from experts or experts, while SUS provides perspectives from users. This research uses 10 heuristic aspects and 10 SUS questionnaire questions. The result of this study is to provide recommendations for improvement from both perspectives and identify aspects that need to be improved. The SUS score in this study for the Adx International website is 55. In addition, the results of this study present recommendations for design improvements from various aspects, therefore it can be concluded that this research is successful.

Rationale: Asthma attacks result in significant morbidity to children, caregivers and increase in emergency visits. Asthma attacks can be fatal. Early recognition and management of worsening symptoms at home can potentially prevent an attack. However, existing home monitoring tools rely on subjective recognition and reporting of symptoms. Tests such as peak-flow and lung function are limited by poor usability and carer burden. Additionally, emerging tools such as wearable devices lack long term usability data (Shin: 2019). Albus Home is a passive contactless device that detects changes in cough and respiratory rate providing a potential therapeutic window to prevent attacks in children. We describe the usability of this tool for long term daily use as reported by children and caregivers. Methods: Children with asthma aged 6-16 years recruited into the Childhood Home Asthma Monitoring (CHAMP) study and their caregivers completed the System Usability Scale (SUS) (Bangor:2008). Participants in the study had nightly monitoring of the physiological parameters by the Albus Home contactless device for up to 12 months. SUS is a widely used and validated instrument to assess the usability of a system (Lewis:2018). It comprises 10 questions on a Likert five-point scale, with 5 questions asked positively and negatively respectively. The SUS score is computed across 10 questions and ranges from 0 to 100.), A score over 80 demonstrates excellent usability. Results: Of 96 participants who started Albus Home monitoring, SUS responses were obtained from 74 (77%) families (mean child age 11, SD 4; 43M:31F). SUS responses were completed by 70 caregivers, and by 25 children with asthma (mean age 13, SD 2), after they had used the Albus Home for median of 10 months. The median caregiver and child SUS scores were 86 and 80 respectively (Figure 1). Ninety one percent of caregivers reported that the “Albus Home was easy to use” and “felt very confident using Albus Home”. Conclusion: The usability assessment of the Albus Home contactless monitoring system yielded highly positive results with the majority of caregivers and children reporting favorable usability experience. Feedback from children and caregivers was useful in developing the second generation Albus Home solution. These insights highlight the potential of integrating Albus Home for long term daily use in asthma management in children.

Usability engineering analyzes the interaction between the intended users and a device. Its implementation is mandatory for manufacturers to obtain regulatory approval for the European market. The aim of this evaluation was assessing the role of usability testing in the development process. For this purpose, a continuous glucose monitoring (CGM) device under development was investigated to determine whether it could be used safely and effectively by the intended users. Conduct of the usability testing was based on the international standard IEC 62366-1. Medical device use of CGM-experienced and non-experienced users (n = 15 each) was observed without initial training in use scenarios containing 18 tasks. The success rate of task completion was determined and the System Usability Scale (SUS) score was calculated from a questionnaire. A prototype of the FiberSense CGM System (EyeSense GmbH, Großostheim, Germany), comprising of a single-use sensor and a reusable detector, was investigated. Most use errors made by both user groups were related to ease of handling of the reusable detectors. The SUS scores achieved in this study were below the pre-defined SUS score acceptance criterion of ≥68. The most frequently mentioned reason for use errors was an incomprehensible and non-chronological instructions for use (IFU). The evaluation provides valuable insights on how to improve usability of the prototype device and demonstrates the value of conducting structured usability testing prior to product finalization. The results reflected areas for improvement of the user interface, mainly by restructuring the IFU, provision of an additional leaflet, and device training prior to use.

This study evaluates the KBGAN iHealth© and KBGAN iFeed© mobile apps designed for buffalo health and feeding management, particularly for agricultural extension professionals (AEPs) in selected Philippine municipalities. These apps aim to address challenges in buffalo management, such as limited access to veterinary expertise, personalized recommendations, organized data, communication channels, and difficulties in calculating ideal feed compositions and meeting the distinct needs of smallholder farmers and AEPs. Despite System Usability Scale (SUS) scores indicating marginal acceptability for both apps, weighted mean scores by AEPs for statements assessed on a 5-point Likert scale (1 as strongly disagree and 5 as strongly agree), demonstrate that AEPs reported high confidence in the accuracy of buffalo health diagnostics (Mean of 4.20) and health management recommendations (Mean of 4.17) provided by KBGAN iHealth©. Similarly, KBGAN iFeed© received favorable ratings, with AEPs expressing agreement on the accuracy of feeding recommendations (Mean of 3.89) and the facilitation of feeding ration computations (Mean of 4.00). These positive perceived performance outcomes, coupled with increased confidence (Mean of 4.03) and motivation (Mean of 4.00) among AEPs, suggest the potential for frequent and consistent app usage despite usability concerns. Chi-square tests examining the relationship between AEP characteristics and SUS scores for the apps reveal significant associations between AEPs' education and experience levels and SUS scores for KBGAN iFeed©. The choice of IT device also influences KBGAN iHealth© SUS scores. Proposed enhancements by AEPs include refining algorithms, improving the user interface for navigation, speed, and efficiency, and incorporating features such as photo uploads and geotagging.

The Nintendo Switch is a popular hybrid portable gaming console that employs a novel modular design, enabling it to be played in multiple controller configurations: Handheld mode, left and right JoyCon Sticks, and the JoyCon Grip. However, there are notes of user complaints about the complexity of performing basic tasks as a result of the unintuitiveness of its user interface. This can result to poor gaming experience and may potentially affect customer buying decisions. To assess the affordance of the user interface design of the Nintendo Switch, a total of 16 Filipino participants, classified as either novice or expert users, based on familiarity with the Switch or other gaming consoles and their functionalities, were made to perform a predetermined set of tasks on navigating through the Nintendo Switch’s basic menus, and playing a racing game across different controller orientations. Affordance, overall usability, and gaming experience satisfaction were measured through 5-point Likert scales (5 being the highest) for affordance surveys, system usability scale (SUS) surveys, and in-task and post-task surveys, respectively. Separate sets of surveys were done for general navigation and gaming tasks. Additionally, quantitative measures of effectiveness (task completion success rates) and efficiency (time to successfully complete tasks) were taken for general navigation tasks, and compared between novice and expert users. Task efficiency rates were also compared to baseline values obtained from experienced Nintendo Switch users. Results show a significant difference between expert and novice users’ effectiveness rates, with expert users all successfully performing 73% of navigation tasks, compared to novice rates of 18%. Efficiency rates for both expert and novice participant groups consistently fell significantly behind baseline task completion times. Affordance survey results showed consistent ratings across different controller configurations for expert users, and high preference for JoyCon Grip and Handheld mode configurations for novice participants. Of all SUS scores, only Game Navigation for expert users scored above the 68-point threshold. Recommendations to improve the Nintendo Switch user interface affordance and overall user satisfaction were made.

Rational: Lymphangioleiomyomatosis (LAM) is a rare, progressive interstitial lung disease (ILD) predominantly affecting women of reproductive age. Improving access to structured exercise through home-based programs is attractive in LAM and other ILDs, but monitoring exercise pulse oximetry and outcomes can be challenging for patients far from specialized pulmonology centres. A preliminary study of a remotely-monitored exercise program demonstrated improvement in 6-minute walk test (6MWT) distance without safety incident, indicating mobile technology with remote monitoring may provide a feasible solution for broader access. For use during LAMFit, a fully-remote, structured exercise program for LAM, we aimed to develop and test a mobile health application to enable remote assessment of 6MWT and home monitoring of pulse oximetry during exercise. Methods: Application development involved multi-stakeholder engagement and collaboration between LAMFit investigators, patient representatives, LAM care providers, and software developers. Feature design was informed by 2002 American Thoracic Society (ATS) guidelines for the 6MWT. The application was deployed in a 28-week LAMFit pilot in 25 women with LAM (mean±SD, age 52.6±14 years). Weekly exercise sessions were remotely monitored via the application, and remote 6MWT and validated patient questionnaires were completed at weeks 2, 14, and 27. System Usability Scale (SUS) scores, a standardized metric to assess user perceptions and ease of system use, were collected at weeks 2 and 28.Results: Collaborative co-design facilitated the creation of a mobile application for the LAMFit program. Iterative design, testing and refinement, guided by steering team feedback, incorporated adjustments in language, font size, icons, and audio prompts, and enabled 6MWT to continue during brief connection-loss with the Bluetooth-enabled pulse oximeter. All 25 patients successfully used the application to complete surveys and multiple 6MWT. System usability (SUS) at week 2 was 50.625 (SD 3.227), and 52.62 (SD 4.292) at week 28. On exit survey, 88% of participants found it feasible to conduct a remote 6MWT at home using the mobile health app. Conclusion: A collaborative co-design approach enabled alignment of the application with users’ needs, enabling successful implementation of the technology. SUS scores increased during the study, suggesting improvements in perceptions of usability with increasing familiarity. However, both scores are below the industry average SUS of 68, indicating potential for further improvement in application usability. Examination of application usability during LAMFit's pilot study provides insights to inform the refinement of the application prior to scaling up the program, optimizing a digital-health solution that supports patients with LAM exercise routines