System Usability Scale: A Meta-Analysis of How SUS Relates to Workload, Task Time, and Error Rate

A pair of commercially available smart glasses, Epson BT-200, was evaluated with different information display modes for assisting machine operators. Two information display modes were selected, displaying animated images only, and displaying animated images with text illustrations. Direction prompts were added in the system. Whether the prompts are helpful or not was also discussed in this study. Task completion time and error rate were collected during the experiment. System Usability Scale (SUS) and NASA-Task Load Index (NASA-TLX) were used to collect subjective information after each experiment. Twenty participants (10 males and 10 females) were recruited. The results showed that the display mode of having smart glasses displaying animated images versus displaying animated images with text illustrations showed no significant difference in task completion time and error rate. But the significant difference was found in the result of SUS scores. Participants preferred the display mode of animate images with text illustrations according to the SUS scores. Moreover, the results indicated that the use of direction prompts had significant influences upon all the measures. Participants completed the tasks faster and had lower error rate by using the smart glasses with direction prompts. And the results of subjective ratings also showed higher SUS score and lower NASA-TLX score were associated with using smart glasses with direction prompts. Thus, the implementation of using smart glasses to guide machine operations should be considered the design of adding direction prompts to increase efficiency and effectiveness of the operations.KeywordsAugmented realityInformation displaySmart glasses

This article describes the psychometric properties of the Emotional Metric Outcomes (EMO) questionnaire and the System Usability Scale (SUS) using data collected as part of a large-sample unmoderated usability study (n = 471). The EMO is a concise multifactor standardized questionnaire that provides an assessment of transaction-driven personal and relationship emotional outcomes, both positive and negative. The SUS is a well-known standardized usability questionnaire designed to assess perceived usability. In previous research, psychometric evaluation using data from a series of online surveys showed that the EMO and its component scales had high reliability and concurrent validity with loyalty and overall experience metrics but did not find the expected four-factor structure. Previous structural analyses of the SUS have had mixed results. Analysis of the EMO data from the usability study revealed the expected four-factor structure. The factor structure of the SUS appeared to be driven by item tone. The estimated reliability of the SUS (.90) was consistent with previous estimates. The EMO and its subscales were also quite reliable, with the estimates of reliability for the various EMO scales ranging from .86 to .96. Regression analysis using SUS, EMO, and Effort as predictors revealed different key drivers for the outcome metrics of Satisfaction and Likelihood-to-Recommend. The key recommendations are to include the EMO as part of the battery of poststudy standardized questionnaires, along with the SUS (or similar questionnaire), but to be cautious in reporting SUS subscales such as Usable and Learnable.

Voice-based User Interfaces (VUIs) challenge our existing conceptions of usability since the standardized evaluation tools we use were typically developed for interfaces with visual feedback, whereas VUI's have predominantly eyes-free interactions. We experimented with the use of a well-validated tool, the System Usability Scale (SUS), to evaluate two existing, commercially available VUIs. We administered the SUS to 12 participants after they completed a set of scenario tasks on Amazon's Alexa and Apple's Siri. The results were consistent with previous studies comparing subjective rating scores and SUS, suggesting that the SUS is a valid evaluation tool for VUIs. Additionally, despite large, significant differences in adjective scale ratings and SUS scores, both systems performed similarly on the learnability items. We conclude with recommendations on the use of the SUS for evaluating VUIs.

Background and importanceAn online personalised health record (PHR) is a valid tool to reduce medication discrepancies (MDs), defined as unexplained differences among medication regimens. The success of a PHR depends...

The System Usability Scale (SuS) survey is a widely respected tool for measuring usability. Generally, a SuS score is administered directly after a usability test to assess the usability and user experience of digital products. However, some researchers have used SuS as a survey as part of longitudinal ‘in the wild’ trials where SuS is often completed some period after the trial. The aim of this research was to determine if a participant’s memory of a product’s usability would change if a SuS survey was administered at different times after a test. Hence, we sought to understand if recalling the usability of a digital technology was affected by temporal bias or memory decay. This paper includes results and findings from two studies, study 1 involved evaluating a web application and study 2 involved evaluating a virtual learning environment. Collectively the two studies had 212 participants (n = 212). The findings conclude that there is no significant change of the user’s recollection of the usability of digital product as evidenced by an analysis of users who completed multiple SuS surveys over a short term period of 3 weeks or over an extended period of time of 6 months. RESEARCH HIGHLIGHTS The system usability scale is used to test for memory decay and temporal bias in judging the user experience of technologies at different time points. 212 participants took part in two studies ranging from 3 weeks to 6 months. There is no evidence that there is a temporal bias or memory decay when users complete a SuS survey at the two different time points of 3 weeks and 6 months.

BackgroundTransparency is increasingly called for in health care, especially, when it comes to patients’ access to their electronic health records. In Sweden, the e-service Journalen is a national patient accessible electronic health record (PAEHR), accessible online via the national patient portal. User characteristics and perceived benefits of using a PAEHR influence behavioral intention for use and adoption, but poor usability that increases the effort expectancy can have a negative impact. It is, therefore, of interest to explore how users of the PAEHR Journalen perceive its usability and usefulness.ObjectiveThe aim of this study was to explore how the users of the Swedish PAEHR experience the usability of the system and to identify differences in these experiences based on the level of transparency of the region.MethodsA survey study was conducted to elicit opinions and experiences of patients using Journalen. The data were collected from June to October 2016. The questionnaire included questions regarding the usability of the system from the System Usability Scale (SUS). The SUS analysis was the focus of this paper. Analysis was performed on different levels: nationally looking at the whole data set and breaking it down by focusing on 2 different regions to explore differences in experienced usability based on the level of transparency.ResultsDuring the survey period, 423,141 users logged into Journalen, of which 2587 unique users completed the survey (response rate 0.61%). The total mean score for all respondents to the SUS items was 79.81 (SD 14.25), which corresponds to a system with good usability. To further explore whether the level of transparency in a region would affect the user’s experience of the usability of the system, we analyzed the 2 regions with the most respondents: Region Uppsala (the first to launch, with a high level of transparency), and Region Skåne (an early implementer, with a low level of transparency at the time of the survey). Of the participants who responded to at least 1 SUS statement, 520 stated that they had received care in Region Skåne, whereas 331 participants had received care in Region Uppsala. Uppsala’s mean SUS score was 80.71 (SD 13.41), compared with Skåne’s mean of 79.37 (SD 13.78).ConclusionsThe Swedish national PAEHR Journalen has a reasonably good usability (mean SUS score 79.81, SD 14.25); however, further research into more specific usability areas are needed to ensure usefulness and ease of use in the future. A somewhat higher SUS score for the region with high transparency compared with the region with low transparency could indicate a relationship between the perceived usability of a PAEHR and the level of transparency offered, but further research on the relationship between transparency and usability is required.

Home healthcare devices have gained attention recently as more and more care is being delivered in the patients’ home rather than in a hospital setting. This trend is accelerating as healthcare systems work to restrain costs and provide patients with the tangible benefits of receiving care in their own home. Because the user population of these devices is moving away from trained professionals, the usability of home healthcare devices takes on greater importance. Poor usability of these devices could result in negative consequences, ranging from incorrect diagnoses to patient injury or death. In this study, the usability of four common home healthcare devices (a manual blood pressure monitor, a temporal artery thermometer, a pulse oximeter, and a spirometer) was evaluated by 74 participants. The objective metrics of effectiveness (success), efficiency (time on task) and the subjective metric of satisfaction were collected for each device, and correlated with the users’ subjective assessment of the usability of each device, as measured with the System Usability Scale (SUS). Results show that there is a positive correlation between effectiveness and SUS scores, no correlation between efficiency and SUS, and a positive correlation between satisfaction and SUS.

Adx International is one of the international shipping services that has collaborated with various marketplaces. Along with the development of technology, Adx International chose a website as a promotional material as well as a system for the transaction process. This research will discuss problems with the UI/UX design of the website by distributing questionnaires to users. After that, several problems were found such as problems with navigation design (search filters), colors that are not aligned, excessive use of icons and there is no guidebook for using the Adx International website. Therefore, further user interface analysis is required. To evaluate the sustainability of this platform, a combination of Heuristic and System Usability Scale (SUS) methods are used in this research because the two methods have different but complementary focuses. Heuristic provides perspectives from experts or experts, while SUS provides perspectives from users. This research uses 10 heuristic aspects and 10 SUS questionnaire questions. The result of this study is to provide recommendations for improvement from both perspectives and identify aspects that need to be improved. The SUS score in this study for the Adx International website is 55. In addition, the results of this study present recommendations for design improvements from various aspects, therefore it can be concluded that this research is successful.

Rationale: Asthma attacks result in significant morbidity to children, caregivers and increase in emergency visits. Asthma attacks can be fatal. Early recognition and management of worsening symptoms at home can potentially prevent an attack. However, existing home monitoring tools rely on subjective recognition and reporting of symptoms. Tests such as peak-flow and lung function are limited by poor usability and carer burden. Additionally, emerging tools such as wearable devices lack long term usability data (Shin: 2019). Albus Home is a passive contactless device that detects changes in cough and respiratory rate providing a potential therapeutic window to prevent attacks in children. We describe the usability of this tool for long term daily use as reported by children and caregivers. Methods: Children with asthma aged 6-16 years recruited into the Childhood Home Asthma Monitoring (CHAMP) study and their caregivers completed the System Usability Scale (SUS) (Bangor:2008). Participants in the study had nightly monitoring of the physiological parameters by the Albus Home contactless device for up to 12 months. SUS is a widely used and validated instrument to assess the usability of a system (Lewis:2018). It comprises 10 questions on a Likert five-point scale, with 5 questions asked positively and negatively respectively. The SUS score is computed across 10 questions and ranges from 0 to 100.), A score over 80 demonstrates excellent usability. Results: Of 96 participants who started Albus Home monitoring, SUS responses were obtained from 74 (77%) families (mean child age 11, SD 4; 43M:31F). SUS responses were completed by 70 caregivers, and by 25 children with asthma (mean age 13, SD 2), after they had used the Albus Home for median of 10 months. The median caregiver and child SUS scores were 86 and 80 respectively (Figure 1). Ninety one percent of caregivers reported that the “Albus Home was easy to use” and “felt very confident using Albus Home”. Conclusion: The usability assessment of the Albus Home contactless monitoring system yielded highly positive results with the majority of caregivers and children reporting favorable usability experience. Feedback from children and caregivers was useful in developing the second generation Albus Home solution. These insights highlight the potential of integrating Albus Home for long term daily use in asthma management in children.

The System Usability Scale (SUS), developed by Brooke (Usability evaluation in industry, Taylor & Francis, London, pp 189-194, 1996), had a great success among usability practitioners since it is a quick and easy to use measure for collecting users' usability evaluation of a system. Recently, Lewis and Sauro (Proceedings of the human computer interaction international conference (HCII 2009), San Diego CA, USA, 2009) have proposed a two-factor structure-Usability (8 items) and Learnability (2 items)-suggesting that practitioners might take advantage of these new factors to extract additional information from SUS data. In order to verify the dimensionality in the SUS' two-component structure, we estimated the parameters and tested with a structural equation model the SUS structure on a sample of 196 university users. Our data indicated that both the unidimensional model and the two-factor model with uncorrelated factors proposed by Lewis and Sauro (Proceedings of the human computer interaction international conference (HCII 2009), San Diego CA, USA, 2009) had a not satisfactory fit to the data. We thus released the hypothesis that Usability and Learnability are independent components of SUS ratings and tested a less restrictive model with correlated factors. This model not only yielded a good fit to the data, but it was also significantly more appropriate to represent the structure of SUS ratings.

Rational: Lymphangioleiomyomatosis (LAM) is a rare, progressive interstitial lung disease (ILD) predominantly affecting women of reproductive age. Improving access to structured exercise through home-based programs is attractive in LAM and other ILDs, but monitoring exercise pulse oximetry and outcomes can be challenging for patients far from specialized pulmonology centres. A preliminary study of a remotely-monitored exercise program demonstrated improvement in 6-minute walk test (6MWT) distance without safety incident, indicating mobile technology with remote monitoring may provide a feasible solution for broader access. For use during LAMFit, a fully-remote, structured exercise program for LAM, we aimed to develop and test a mobile health application to enable remote assessment of 6MWT and home monitoring of pulse oximetry during exercise. Methods: Application development involved multi-stakeholder engagement and collaboration between LAMFit investigators, patient representatives, LAM care providers, and software developers. Feature design was informed by 2002 American Thoracic Society (ATS) guidelines for the 6MWT. The application was deployed in a 28-week LAMFit pilot in 25 women with LAM (mean±SD, age 52.6±14 years). Weekly exercise sessions were remotely monitored via the application, and remote 6MWT and validated patient questionnaires were completed at weeks 2, 14, and 27. System Usability Scale (SUS) scores, a standardized metric to assess user perceptions and ease of system use, were collected at weeks 2 and 28.Results: Collaborative co-design facilitated the creation of a mobile application for the LAMFit program. Iterative design, testing and refinement, guided by steering team feedback, incorporated adjustments in language, font size, icons, and audio prompts, and enabled 6MWT to continue during brief connection-loss with the Bluetooth-enabled pulse oximeter. All 25 patients successfully used the application to complete surveys and multiple 6MWT. System usability (SUS) at week 2 was 50.625 (SD 3.227), and 52.62 (SD 4.292) at week 28. On exit survey, 88% of participants found it feasible to conduct a remote 6MWT at home using the mobile health app. Conclusion: A collaborative co-design approach enabled alignment of the application with users’ needs, enabling successful implementation of the technology. SUS scores increased during the study, suggesting improvements in perceptions of usability with increasing familiarity. However, both scores are below the industry average SUS of 68, indicating potential for further improvement in application usability. Examination of application usability during LAMFit's pilot study provides insights to inform the refinement of the application prior to scaling up the program, optimizing a digital-health solution that supports patients with LAM exercise routines

The increasing reliance on digital platforms in higher education necessitates the evaluation of system usability to ensure effective user interaction and satisfaction. This study aims to evaluate the usability of the Academic Information System (AIS) at [University Name] using the System Usability Scale (SUS) method. SUS is a reliable, standardized tool for measuring the usability of interactive systems, providing a quick and quantitative assessment. Data were collected from a sample of 100 students and academic staff who frequently use the AIS for various academic activities, including course registration, grade checking, and academic planning. The results of the SUS analysis yielded an average score of 72.5, indicating that the system falls within the “Good” usability category. However, several usability issues were identified, such as navigation complexity and visual layout inconsistencies, which slightly reduced user satisfaction. These findings highlight the importance of continuous usability testing and user-centered design in the development of academic systems. The study recommends specific design improvements to enhance user experience and system performance. Overall, the SUS method proved effective in identifying usability strengths and weaknesses, offering valuable insights for future system optimization.

Background. Education has undergone a significant transformation due to digital technology, which has increased its flexibility, accessibility, and engagement. With gamifikasi strategies that increase user engagement, the Duolingo app is a highly preferred language learning resource, especially for learning English. Purpose. The System Usability Scale (SUS) and End User Computing Satisfaction (EUCS) methods will be used in this study to evaluate the usability and user satisfaction of the Duolingo program. Method. 404 respondents were obtained for this study using a quantitative approach using the Slovin formula sampling technique. Ten SUS items and five components of EUCS, content, accuracy, format, ease of use, and timeliness were included in the online survey used to collect the data. Descriptive statistics were used in the analysis to assess SUS and EUCS scores. Result. The app has a good usability level, with an average SUS score of 78.86 (grade B+). The five components, content (4.4), accuracy (4.2), format (4.5), ease of use (4.5), and timeliness (4.4), produced excellent results for EUCS. Most respondents were between the ages of 18 and 24, and they spent an average of less than 30 minutes a day. Conclusion. Particularly when it comes to usability and interface design, Duolingo offers a positive user experience and high satisfaction. Its gamifikasi strategy increases the effectiveness and engagement of English learning, especially for younger users.

Background: Machine learning models have been reported to assist in the proper management of cancer through accurate prognostication. Integrating such models as a web-based prognostic tool or calculator may help to improve cancer care and assist clinicians in making oral cancer management-related decisions. However, none of these models have been recommended in daily practices of oral cancer due to concerns related to machine learning methodologies and clinical implementation challenges. An instance of the concerns inherent to the science of machine learning is explainability. Objectives: This study measures the usability and explainability of a machine learning-based web prognostic tool that was designed for prediction of oral tongue cancer. We used the System Usability Scale (SUS) and System Causability Scale (SCS) to evaluate the explainability of the prognostic tool. In addition, we propose a framework for the evaluation of post hoc explainability of web-based prognostic tools. Methods: A SUS- and SCS-based questionnaire was administered amongst pathologists, radiologists, cancer and machine learning researchers and surgeons (n = 11) to evaluate the quality of explanations offered by the machine learning-based web prognostic tool to address the concern of explainability and usability of these models for cancer management. The examined web-based tool was developed by our group and is freely available online. Results: In terms of the usability of the web-based tool using the SUS, 81.9% (45.5% strongly agreed; 36.4% agreed) agreed that neither the support of a technical assistant nor a need to learn many things were required to use the web-based tool. Furthermore, 81.8% agreed that the evaluated web-based tool was not cumbersome to use (usability). The average score for the SCS (explainability) was 0.74. A total of 91.0% of the participants strongly agreed that the web-based tool can assist in clinical decision-making. These scores indicated that the examined web-based tool offers a significant level of usability and explanations about the outcome of interest. Conclusions: Integrating the trained and internally and externally validated model as a web-based tool or calculator is poised to offer an effective and easy approach towards the usage and acceptance of these models in the future daily practice. This approach has received significant attention in recent years. Thus, it is important that the usability and explainability of these models are measured to achieve such touted benefits. A usable and well-explained web-based tool further brings the use of these web-based tools closer to everyday clinical practices. Thus, the concept of more personalized and precision oncology can be achieved.

The development of Virtual Reality (VR) technology for competency assessment with multiple users is a breakthrough in the assessment of skills. The Multi-User Virtual Environment for Professional Certification Assessment (MUVE PCA) is a VR-based application that provides a VR experience to users. This study aimed to obtain information on the usefulness of the application and whether it can be accepted by users using the Virtual Reality System Usability Questionnaire (VRSUQ) and the System Usability Scale (SUS). Data collection was carried out by piloting the MUVE PCA application installed on the Oculus Head Mounted Device (HMD) to vocational school students, vocational school teachers, students, and lecturers in the field of Information Technology (IT) in computer network environments. The results showed a SUS score of 70.41, slightly above the global SUS standard, indicating that the system is considered acceptable. The average VRSUQ score of 3.52 is slightly below the standards, indicating that the system needs to be improved in some areas, especially in terms of convenience.