A novel graphical user interface for ultrasound-guided shoulder arthroscopic surgery

Camera positioning and orientation is important to applications in domains such as transportation since the objects to be imaged vary greatly in shape and size. In a typical transportation application that requires capturing still images, inductive loops buried in the ground or laser trigger sensors are used when a vehicle reaches the image capture zone to trigger the image capture system. The camera in such a system is in a fixed position pointed at the roadway and at a fixed orientation. Thus the problem is to determine the optimal location and orientation of the camera when capturing images from a wide variety of vehicles. Methods from Design for Six Sigma, including identifying important parameters and noise sources and performing systematically designed experiments (DOE) can be used to determine an effective set of parameter settings for the camera position and orientation under these conditions. In the transportation application of high occupancy vehicle lane enforcement, the number of passengers in the vehicle is to be counted. Past work has described front seat vehicle occupant counting using a camera mounted on an overhead gantry looking through the front windshield in order to capture images of vehicle occupants. However, viewing rear seat passengers is more problematic due to obstructions including the vehicle body frame structures and seats. One approach is to view the rear seats through the side window. In this situation the problem of optimally positioning and orienting the camera to adequately capture the rear seats through the side window can be addressed through a designed experiment. In any automated traffic enforcement system it is necessary for humans to be able to review any automatically captured digital imagery in order to verify detected infractions. Thus for defining an output to be optimized for the designed experiment, a human defined standard image reference (SIR) was used to quantify the quality of the line-of-sight to the rear seats of the vehicle. The DOE-SIR method was exercised for determining the optimal camera position and orientation for viewing vehicle rear seats over a variety of vehicle types. The resulting camera geometry was used on public roadway image capture resulting in over 95% acceptable rear seat images for human viewing.

Mobility for graphical user interfaces (GUIs) is a challenging problem, as different GUIs need to be constructed for different device capabilities and changing context, preferences and users’ locations. GUI developers frequently create multiple user interface versions for different devices. The solution lies in using a single, abstract, user interface description that is used later to automatically generate user interfaces for different devices. Various techniques are proposed to adapt GUIs from an abstract specification to a concrete interface. Design-time techniques have the possibility of creating better performing GUIs but, in contrast to run-time techniques, lack flexibility and mobility. Run-time techniques’ mobility and autonomy can be significantly improved by using mobile agent technology and an indirect GUI generation paradigm. Using indirect generation enables analysis of computer-human interaction and application of artificial intelligence techniques to be made at run-time, increasing GUIs’ performance and usability.

AI (Artificial Intelligence) and GUI (Graphical User Interface) are the two most cutting-edge technologies. These technologies individually can make a comprehensive change in industries and alter economic and social interactions and connections. In general, Now-a-days AI is integrated with human technology and used in some wide range of sectors like healthcare, finance to education, banking, manufacturing, transportation, etc. GUI is used in gaming, designing, AR (Augmented reality) and VR (Virtual reality), media, smart home devices, etc., Integration of Artificial Intelligence and Graphical User Interface is beneficial for both users and app developers. The convergence of AI and GUI gives dynamic applications like Data Visualization, Insights, Smart assistants, Adaptions, Navigations, and Predictive analytics. This integration will increase the ease of access for new information to users. Now we are facing some general problems based on these technologies like Adaptive User Interfaces, Automated user interface testing, AI-Enhanced data visualization, Intelligent GUI Design Assistance, Real-Time Emotion Recognition for GUI Adaptation, etc., User interfaces are often designed with a one-size-fits-all approach, which does not account for individual user preferences and abilities. Designing perceptive and attractive GUIs requires significant space and time. This chapter focus on how can we develop a UI (User Interface) with base of AI which can be capable of solving the above obstacles and gives the simple objectives and technical solutions. By this integration we can execute some future projects like AI-Enhanced Customer Support Portals, Adaptive Gaming Interfaces, Interactive Financial Management Systems, and more.

Mobile applications use Accessibility Services to assist users with disabilities (UWDs) in using Graphical User Interface (GUI)-based apps. However, such mobile assistive technologies are not fully secure as these can obtain data from the GUI objects, enhance this data for UWDs and transmit it to a server, thereby potentially exposing sensitive data to the external world. SEAPHISH (SEcuring Accessibility using PHISHing) is a platform aimed towards protecting against such an attack by providing defense by deception. This platform generates a phishing app i.e. an app similar to the original app installed on the user’s smartphone by extracting GUI elements and properties from the original app. A simulation for SEAPHISH can help determine the situations when an attack against a particular app can be performed with a high degree of probability. But performing effective simulations requires a fundamental understanding of the properties of GUI layouts of apps at large. This thesis aims at providing a framework that analyzes GUI layouts and their transitions using a large base of approximately three million Android apps. Various state-of-the-art tools that use different strategies in traversing layouts are explored. We created a framework where the tool Backstage performs static analysis and another tool AndroidRipper performs dynamic analysis of the layouts to help build a GUI model of the app. Using these models, we investigate the layouts of Android apps by collecting statistics on various GUI elements and screens. This investigation enhances SEAPHISH with statistically significant real-world constraints thereby providing defense against malwares and reducing the security vulnerabilities faced by UWDs.

Implementation of a Chatbot System using AI and NLP

User interfaces represent a major part in software products. Generally, it is difficult to document the content and the structure of the user interface. However, there is a need to save and preserve the overall state and structure of the user interface for later communication and modifications in requirements, design and implementation. The Graphical User Interface (GUI) storage format should be universal and can be transferred from one application to another (such as XML). GUI states are usually defined as the overall combination of properties of all the components or widgets of the GUI This paper suggests an alternative back end representation of user interfaces and their state from their original format within the applications. The user interface model is transformed to a schema saved in an XML file. The schema structure represents the user interface state. This state considers only the structure of the user interface and ignores controls' specific properties. XML format is widely used and accepted by many applications. It is the infrastructure language for web pages and database management systems. User interface documentation is useful for future evaluation and comparison, useful for stakeholders' communication. It is also for some GUI testing activities such as regression testing where we trigger testing if there is a change in the user interface.

Visual and aural are two most important channels of information processing. While most of the interaction with computers have been designed around the visual channel, there are circumstances where voice based man-machine interaction becomes preferable, and in some cases, necessary, given that voice based interaction comes naturally to humans and can be used by illiterate people easily.

The contemporary theory of metaphor defines the metaphor as a major and indispensable part of our ordinary, conventional way of conceptualizing the world and it asserts that the metaphors are used to define our relationship with the physical environment and to create a context of communication. The metaphors are the core idioms of the contemporary graphical user interface (GUI) and have an important role in helping users interact with computer systems. The desktop metaphor began with the first GUI Xerox Star and was popularized by the Lisa of Apple Macintosh. Realizing the physical-office metaphor and reflecting the familiar world of the potential users by using icons of file, folder and trashcan, the metaphor-based GUIs made it relatively easy for all to use the computer productively. Nowadays almost all of the software industry is based on the desktop metaphor paradigm. However, besides the researchers that favor the use of metaphors in the development of the system image by accentuating their assistance to learning, there are also other researchers, who reject the metaphor-based design approach in GUIs by referring to their limitations. Considering these discussions and various approaches, this study aims to make an outline of the historical development of metaphor-based GUIs; to determine the current state of metaphor within GUI design by brief examples and finally to identify future prospects and research themes. The study claims that metaphor still remains to be an inevitable part of the GUI to communicate it with the users and despite its potential limitations -rather than rejecting the metaphor totally- it seems better-to try compatible and supplementary solutions like visual formalism.

Background and Objective: In the last decade, the field of medicine has ingressed into a new era of technological advancements, driven by an ever increasing demand to reduce patient costs and risks, improve patient safety, efficiency, and surgical outcomes. The need for alternative ways of training and surgery is stronger than ever. Virtual reality based training and surgery systems hold significant promise in this direction. However, development of realistic virtual surgery systems for invasive orthopaedic surgical procedures remains one of the most challenging problems in the field of virtual reality based surgery and training because of the involvement of complex musculoskeletal structures and surgical tools. In recent years, some advances have been made in this area but they have limited practical scope because of their support for small range of procedures and training scenarios. The tools developed so far are either restricted in their ability or follow non patient-centric approaches and hence, cannot be considered viable alternatives to the conventional techniques for invasive orthopaedic surgery and training. In this paper, we discuss the challenges and complexities associated with the development of a virtual reality based system for orthopaedic training and surgery, and present our image guidance based navigation system, developed as part of our ongoing research initiative to build a comprehensive tool for realistic virtual orthopedic surgery and training.Methods: Our image guidance based interactive navigation system provides a common interface for the assembly of different components crucial for a realistic virtual reality based training and surgery application. Presently, the system incorporates various features including rigid body registration, patient-specific three-dimensional model generation, two-dimensional and three-dimensional interactive visualizations, and real time intraoperative surgical guidance. In this paper, we outline the details of our present system along with its key features.Results: A preliminary version of our proposed virtual reality based orthopaedic training and surgery navigation system is presented. To demonstrate the applicability of our system, a sample application showing the anatomically detailed three -dimensional representations of a patient's knee, derived from the pre-operative image scans, along with the corresponding two-dimensional image details is presented. To the best of our knowledge, this is the first attempt that constructs and integrates patient-specific, anatomically correct, and comprehensive three-dimensional models, with all possible soft tissue details, to provide patient-specific visualization and training capabilities. Preliminary feedback by the orthopaedic surgeons on the prototype of our system is very encouraging and pin points some additional features that can further strengthen the efficacy of our tool and its clinical adoption.Conclusion: A comprehensive virtual reality based navigation system for orthopaedic training and surgery is presented. The system utilizes patient-specific two-dimensional image modalities and provides corresponding two-dimensional and three-dimensional, interactive visualization capabilities along with realtime tracking of surgical instruments. The present system can be used as an effective tool for anatomy education, surgical planning, diagnosis, and real-time intra-operative surgical navigation. Additional components such as haptics and real-time tissue deformations are currently under development and will soon be integrated with this platform.

Cross-modal Transfer Learning Based on an Improved CycleGAN Model for Accurate Kidney Segmentation in Ultrasound Images

Graphical User Interfaces (GUIs) are amongst the most common user interfaces, enabling interactions with applications through mouse movements and key presses. Tools for automated testing of programs through their GUI exist, however they usually rely on operating system or framework specific knowledge to interact with an application. Due to frequent operating system updates, which can remove required information, and a large variety of different GUI frameworks using unique underlying data structures, such tools rapidly become obsolete, Consequently, for an automated GUI test generation tool, supporting many frameworks and operating systems is impractical. We propose a technique for improving GUI testing by automatically identifying GUI widgets in screen shots using machine learning techniques. As training data, we generate randomized GUIs to automatically extract widget information. The resulting model provides guidance to GUI testing tools in environments not currently supported by deriving GUI widget information from screen shots only. In our experiments, we found that identifying GUI widgets in screen shots and using this information to guide random testing achieved a significantly higher branch coverage in 18 of 20 applications, with an average increase of 42.5% when compared to conventional random testing.

This paper presents a distributed, multi-user virtual reality testbed for a mobile robot. The main components of the testbed include user interface, networking, simulation and robot control. The user interface is implemented as a combination of a virtual environment and a standard graphical user interface (GUI). The virtual environment is developed using the Virtual Reality Modeling Language (VRML) and it corresponds to the laboratory environment where the robot is placed and tested. The standard GUI is implemented using Java Programming language. The World Wide Web (WWW) browser is used to integrate the virtual environment (embedded VRML) and the standard GUI (Java applet) in a single user interface. Users can access a dedicated WWW server and download the user interface. Networking component (communication among users and robots) is implemented using the Distributed Interactive Simulation (DIS) protocol. The existing robot development system has been utilized to provide simulation capabilities. A separate stand-alone Java application has been developed to communicate and control the robot using a wireless Ethernet link. In addition to user's direct control, control algorithms (e.g. fuzzy logic based) can be used to replace the user. A Nomad200 mobile robot has been used for initial measurements and evaluations of the testbed.

Intraoperative patient registration and tracking are key enabling components of Computer-Assisted Orthopaedic Surgery (CAOS). Optical tracking systems that detect rigidly-attached reference frames with reflective markers are the standard in current CAOS systems, due to their large field of view and high accuracy [1], [2]. Recent improvements in depth sensing technology have led to the widespread availability of RGBD stereo- cameras. Their adoption in CAOS is appealing, as it would allow to reconstruct a patient’s anatomy with a simple ‘snapshot’ of the scene [3], [4]. However, inte- grating such cameras into a CAOS system would not be straightforward, as would require changes to the layout traditionally used in current systems. Indeed, while the camera is normally positioned laterally to the patient, a markerless system would need to look directly into the surgical incision, at a much closer distance so as to maximise the density of points in the Region-of-Interest (RoI). While these constraints present some challenges, the light weight and small form factor of many state-of- the-art RGBD cameras allow to envision solutions that would not be possible with bulkier systems. In this abstract we present a preliminary study for an integrated solution composed of a lightweight RGBD camera connected to the robot-assisted burring tool of a commercial CAOS system. We present preliminary results on the tracking accuracy of the system, and discuss the advantages and disadvantages of such a solution.

Analyses some of the components of graphical user interfaces (GUI), applied to CD-ROM databases (windows, dialogue boxes, menus, commands, buttons, check boxes and icons) and remarks on the degree to which these are based on Windows software. Notes the constraints to GUIs that still remain even after databases have appeared in the marketplace: particularly mouse based prob lems. Discusses ways of evaluating GUIs and their specific components and reports a study of databases in which these types of interfaces are used. Notes the significant variability between products in the way that they apply GUIs and discusses whether database users would benefit from a certain degree of standardization in the basic interface structure using GUIs, or whether users would prefer greater control over the design of the local user interface. Concludes that the introduction of Windows based CD-ROM databases with graphical user interfaces will increasingly revolutionize the design of such data bases. Makes 3 recommendations that would aim to enhance the user friend liness of CD-ROM interfaces: research to develop criteria for evaluating interfaces in the GUI environment; standardization of interface design between some products, particularly for novice users; and facilities to allow experienced users to develop and tailor their own interface design.

Background: Studies have demonstrated the development of an osseous reaction at the drill sites of anchors after arthroscopic shoulder surgery. Purpose: To investigate the drill-hole size at 18 years after arthroscopic Bankart repair using either fast polygluconate acid (PGA) or slow polylevolactic acid (PLLA) absorbable tacks and to compare the functional outcomes and development of osteoarthritis. Study design: Randomized controlled trial; Level of evidence, 2. Methods: 40 patients with unidirectional anterior shoulder instability, treated with arthroscopic Bankart repair, were randomized into the PGA group (n = 20) or the PLLA group (n = 20). Plain radiographs of both shoulders, as well as computed tomography (CT) images of the operated shoulder, were used to evaluate the drill-hole size, volume, and degenerative changes. Functional outcomes were assessed by use of the Rowe score, Constant score, and Western Ontario Shoulder Instability (WOSI) index. Results: Of the 40 patients, 32 patients returned for the follow-up (15 PGA and 17 PLLA). No significant differences were found in the population characteristics between the study groups. The mean follow-up time was 18 years for both groups. No significant differences were seen in range of motion, strength in abduction, or Constant, Rowe, and WOSI scores between the groups. Recurrence rate was 33% in the PGA group and 6% in the PLLA group during the follow-up period (P = .07). The drill-hole appearance on plain radiographs (invisible/hardly visible/visible/cystic) was 11/2/2/0 and 6/5/5/1 for the PGA and PLLA groups, respectively (P = .036). The mean ± SD drill-hole volume as estimated on CT images was 89 ± 94 and 184 ± 158 mm3 in the PGA and PLLA groups, respectively (P = .051). Degenerative changes (normal/minor/moderate/severe) on plain radiographs were 7/4/4/0 and 3/8/5/1 for the PGA and PLLA groups, respectively (P = .21), and on CT images were 5/7/3/0 and 2/6/6/3 for the PGA and PLLA groups, respectively (P = .030). Conclusion: This long-term follow-up study demonstrated that the PLLA group had significantly more visible drill holes than the PGA group on plain radiographs. However, this difference was not evident on CT imaging, with both groups having several visible cystic drill holes and a substantial drill-hole volume defect. No significant differences were found between the study groups in terms of clinical outcomes.