VisuoSpatial Foresight for Multi-Step, Multi-Task Fabric Manipulation

Surgical movements have an important stylistic quality that individuals without formal surgical training can use to identify expertise. In our prior work, we sought to characterize quantitative metrics associated with surgical style and developed a near-real-time detection framework for stylistic deficiencies using a commercial haptic device. In this paper, we implement bimanual stylistic detection on the da Vinci Research Kit (dVRK) and focus on one stylistic deficiency, "Anxious", which may describe movements under stressful conditions. Our goal is to potentially correct these "Anxious" movements by exploring the effects of three different types of haptic cues (time-variant spring, damper, and spring-damper feedback) on performance during a basic surgical training task using the da Vinci Research Kit (dVRK). Eight subjects were recruited to complete peg transfer tasks using a randomized order of haptic cues and with baseline trials between each task. Overall, all cues lead to a significant improvement over baseline economy of volume and time-variant spring haptic cues lead to significant improvements in reducing the classified "Anxious" movements and also corresponded with significantly lower path length and economy of volume for the non-dominant hand. This work is the first step in evaluating our stylistic detection model on a surgical robot and could lay the groundwork for future methods to actively and adaptively reduce the negative effect of stress in the operating room.

Research in surgical robotics and automation has made remarkable advancements in recent years thanks to new methods in computer vision, control, and deep learning. Autonomous end-effector manipulation is a challenging task in surgical robotics, and cutting with scissor tools is largely unexplored. A concurrent work explored path and trajectory generation for cutting deformable materials using the da Vinci Research Kit (dVRK) [1]. However, an efficient and realistic simulation is necessary for methods such as reinforcement learning (RL) or learned trajectory planning. Our previous work built a simulation for the dVRK in Unity for training RL algorithms on rigid body tasks [2]. To our knowledge, there is no dVRK simulation available that includes the cutting of deformable materials. This paper introduces a cutting simulation of a deformable mesh, which can represent a tissue layer, built onto our Unity dVRK simulation.

Although substantial progresses have been made in robot-assisted laparoscopic surgery, the graspers for existing surgical systems generally remain nonsensorized forceps design with limited functions. This article presents the design, development, and preliminary evaluation of the MUSHA Hand II, a multifunctional hand with force sensors for robot-assisted laparoscopic surgery. The proposed hand has three snake-like underactuated fingers that can be folded into a $\phi$ 12 mm cylindrical form. Each finger has a three-axis force sensor, to provide force information. After been deployed into an abdominal cavity, the hand can be configured to either grasper mode, retractor mode, or palpation mode for different tasks. Underactuated finger design enhances the adaptivity in grasping and the compliance in interaction with the environment. In addition, fingertip force sensors can be utilized for palpation to obtain a real-time stiffness map of organs. Using the da Vinci Research Kit as a robotic testbed, the functionality of the hand has been demonstrated and experiments have been conducted, including robotic palpation and organ manipulation. The results suggest that the hand can effectively enhance the functionality of a robotic surgical system and overcome the limits on force sensing introduced by the use of robots in laparoscopic surgery.

In a teleoperated system, misalignment between the master and slave manipulators can result from clutching, errors in the kinematic model, and/or sensor errors. This study examines the effects of type and magnitude of misalignment on the performance of the teleoperator. We first characterized the magnitude and direction of orientation misalignment created when clutching and unclutching during use of two surgical robots: the Raven II and the da Vinci Research Kit. We then purposely generated typical misalignments in order to measure the impact of such misalignment on user performance of a peg transfer task with the Raven II. Users were able to compensate for misalignment angles up to approximately 20 degrees in both tool orientation and camera viewpoint misalignment. These results can be used to guide the design and control of teleoperated systems for a variety of applications.

With progressive advancements in modern technologies, image-guided automation, which plans and manipulates particular tasks by utilizing visual cues, is becoming an emerging sector in medical robotics. Its prevalence nowadays requires a reliable hand-to-eye calibration owing to various configurations of surgical robots, especially for those with remote center of motion (RCM) constraints, e.g., da Vinci Research Kit (dVRK). In this article, we proposed a novel unified calibration method, which leverages the structure characteristics and establishes an explicit geometric model to solve the hand-to-eye relationship only with a monocular camera's feedback. By freely moving the patient side manipulator in dVRK, our method can first recover the 3-D poses of its shaft at various configurations using Plücker coordinates. To obtain the hand-to-eye translation relationship, these recovered poses are further converged by softly intersecting them within a minimal spatial sphere while preserving their coordinates using an aggregating sphere loss. Meanwhile, the orientation mapping can be recovered by leveraging the corresponding longitudinal-axis information. To improve the calibration accuracy, a local optimization strategy that utilizes the bundle adjustment to refine the transformation using the previous outcomes is also introduced. With such a hierarchical calibration scheme, the hand-to-eye relationship of any RCM constrained robot can be successfully achieved. Extensive simulations and experiments are conducted based on dVRK platform. By comparing with the traditional method, the validation results prove the feasibility and superiority of our unified algorithm in calibration accuracy and robustness.

The paper examines a particular class of robotic applications, i.e. “domestic robots,” in order to stress that such robots will likely affect current legal frameworks of privacy and data protection. Since most of these machines act, new responsibilities of humans for the behaviour of others should be expected in the legal field. More particularly, focus is on the protection of people’s “opaqueness” and the transparency with which domestic robots should collect, process, and make use of personal data. Whilst the aim of the law to govern the process of technological innovation concerns here the regulation of producers and designers of robots through specific sets of norms, or the regulation of users behaviour through the design of their robots, three issues are fated to remain open. They concern: (i) a new expectation of privacy; (ii) the realignment of the traditional distinction between data processors and data controllers; and, (iii) a novel set of challenges to the principle of privacy by design. Although the claim and goal of lawmakers will probably revolve around the protection of individuals against every harm, e.g. psychological problems related to the interaction with domestic robots and the processing of third parties’ information, the intent to embed normative constraints into the internal control architecture of such artificial agents entails a major risk. If there is no need to humanize our robotic applications, we should not robotize human life either.

Robotic assisted surgery is becoming widely adopted by surgeons for a number of reasons, which include improved instrumentation control and dexterity as well as faster patient recovery times and cosmetic advantages. Robotic assisted surgery is currently one of the fastest growing applications in robotics. Although the traditional robotic actuators which are currently used have advanced performance which can, in some aspects, surpass that of humans, they simply do not have the capabilities and diversity required to meet the demand for new applications in robotic surgery. Novel transducers which have advanced capabilities and which allow safe operation in delicate environments are needed. Ionic polymer–metal composites (IPMCs) have extensive desirable characteristics when compared with traditional actuators and as their transduction mechanisms can mimic biological muscle they have much potential for future advanced biomedical and surgical robotics. In this research, a complete two degree-of-freedom (2DOF) surgical robotic instrument has been developed, which with the attachment of surgical tools (scalpel, etc.) has the ability to undertake surgical procedures. The system integrates an IPMC sensor and actuator at each joint. A gain scheduled (GS) controller, which is tuned with an iterative feedback tuning (IFT) algorithm, has been developed to ensure an accurate and adaptive response. The main advantages of this device over traditional devices are the improved safety through a natural compliance of the joints as well as the mechanical simplicity which ensures ease of miniaturisation for minimally invasive surgery (MIS). The components of the system have been tested and shown to have the capabilities required to operate the device for certain surgical procedures, specifically a device work envelope of 1600 mm2, compliance of 0.0668 m/N while still maintaining enough force to cut tissue, IPMC sensor accuracy between 3–22% and a control system which has shown to guarantee zero steady state error.

Automation of surgical processes (SPs) is an utterly complex, yet highly demanded feature by medical experts. Currently, surgical tools with advanced sensory and diagnostic capabilities are only available. A major criticism towards the newly developed instruments that they are not fitting into the existing medical workflow often creating more annoyance than benefit for the surgeon. The first step in achieving streamlined integration of computer technologies is gaining a better understanding of the SP. Surgical ontologies provide a generic platform for describing elements of the surgical procedures. Surgical Process Models (SPMs) built on top of these ontologies have the potential to accurately represent the surgical workflow. SPMs provide the opportunity to use ontological terms as the basis of automation, allowing the developed algorithm to easily integrate into the surgical workflow, and to apply the automated SPMs wherever the linked ontological term appears in the workflow. In this work, as an example to this concept, the subtask level ontological term “blunt dissection” was targeted for automation. We implemented a computer vision-driven approach to demonstrate that automation on this task level is feasible. The algorithm was tested on an experimental silicone phantom as well as in several ex vivo environments. The implementation used the da Vinci surgical robot, controlled via the Da Vinci Research Kit (DVRK), relying on a shared code-base among the DVRK institutions. It is believed that developing and linking further building blocks of lower level surgical subtasks could lead to the introduction of automated soft tissue surgery. In the future, the building blocks could be individually unit tested, leading to incremental automation of the domain. This framework could potentially standardize surgical performance, eventually improving patient outcomes.

Master manipulators represent a key component of Robot-Assisted Minimally Invasive Surgery (RAMIS). In this paper, an Analytic Hierarchy Process (AHP) method is used to construct an effective workspace analysis framework, which can assist the configuration selection and design evaluation of a portable master manipulator for surgical robot control and training. The proposed framework is designed based on three criteria: 1) compactness, 2) workspace quality, and 3) mapping efficiency. A hardware prototype, called the Hamlyn Compact Robotic Master (Hamlyn CRM), is constructed following the proposed framework. Experimental verification of the platform is conducted on the da Vinci Research Kit (dVRK) with which a da Vinci robot is controlled as a slave. The proposed Hamlyn CRM is compared with Phantom Omni, a commercial portable master device, with results demonstrating the relative merits of the new platform in terms of task completion time, average control speed and number of clutching.

The introduction of the laparoscope led to the progress of surgery to a new era, where surgeries that were deemed major are now being performed through keyhole incisions with comparable outcomes to open surgery. However, with this new technique rose several problems like inaccurate depth perception, diminished tactile feedback, need for experienced assistance, and reduction in degrees of motion of the surgeons hands all of which inspired surgeons and engineers to look for mechanical tools to help in reducing these problems. Henceforth; came the application of robotics in surgery. A PubMed and Medline search was performed on cardiac robotic surgery and its applications in mitral valve repair and coronary artery surgery. A total of twenty one articles were picked that allude to the subject. A history of robotic surgery was outlined followed by applications of robotic manipulation in cardiac surgery was narrated. A quick overview of this technology in telemedicine was then outlined followed by future prospects of this technology in surgery was contemplated. The experience of the group from St. Mary's Hospital, London in this field was outlined. During the period of 4 years a total of 102 cases of robotic cardiac surgery were performed. The mean length of hospital stay was 3.1 days with a standard deviation of 1.4 days and the morbidity of the series explained. There was no mortality. Early studies have shown that minimally invasive cardiac surgery is feasible and yields results similar to conventional cardiac surgery, yet it is more technically demanding on the surgeon. As advantageous as this new modality is, further multicenter studies are needed to prove its efficacy.

Neural-based multi-task learning has been successfully used in many real-world large-scale applications such as recommendation systems. For example, in movie recommendations, beyond providing users movies which they tend to purchase and watch, the system might also optimize for users liking the movies afterwards. With multi-task learning, we aim to build a single model that learns these multiple goals and tasks simultaneously. However, the prediction quality of commonly used multi-task models is often sensitive to the relationships between tasks. It is therefore important to study the modeling tradeoffs between task-specific objectives and inter-task relationships. In this work, we propose a novel multi-task learning approach, Multi-gate Mixture-of-Experts (MMoE), which explicitly learns to model task relationships from data. We adapt the Mixture-of-Experts (MoE) structure to multi-task learning by sharing the expert submodels across all tasks, while also having a gating network trained to optimize each task. To validate our approach on data with different levels of task relatedness, we first apply it to a synthetic dataset where we control the task relatedness. We show that the proposed approach performs better than baseline methods when the tasks are less related. We also show that the MMoE structure results in an additional trainability benefit, depending on different levels of randomness in the training data and model initialization. Furthermore, we demonstrate the performance improvements by MMoE on real tasks including a binary classification benchmark, and a large-scale content recommendation system at Google.

The field of spine surgery has changed significantly over the past few decades as once technological fantasy has become reality. The advent of stereotaxis, intra-operative navigation, endoscopy, and percutaneous instrumentation have altered the landscape of spine surgery. The concept of minimally invasive spine (MIS) surgery has blossomed over the past ten years and now robot-assisted spine surgery is being championed by some as another potential paradigm altering technological advancement. The application of robotics in other surgical specialties has been shown to be a safe and feasible alternative to the traditional, open approach. In 2004 the Mazor Spine Assist robot was approved by FDA to assist with placement of pedicle screws and since then, more advanced robots with promising clinical outcomes have been introduced. Currently, robotic platforms are limited to pedicle screw placement. However, there are centers investigating the role of robotics in decompression, dural closure, and pre-planned osteotomies. Robot-assisted spine surgery has been shown to increase the accuracy of pedicle screw placement and decrease radiation exposure to surgeons. However, modern robotic technology also has certain disadvantages including a high introductory cost, steep learning curve, and inherent technological glitches. Currently, robotic spine surgery is in its infancy and most of the objective evidence available regarding its benefits draws from the use of robots in a shared-control model to assist with the placement of pedicle screws. As artificial intelligence software and feedback sensor design become more sophisticated, robots could facilitate other, more complex surgical tasks such as bony decompression or dural closure. The accuracy and precision afforded by the current robots available for use in spinal surgery potentially allow for even less tissue destructive and more meticulous MIS surgery. This article aims to provide a contemporary review of the use of robotics in MIS surgery.

One of the challenges in Natural Language Processing (NLP) is the unstructured nature of texts, in which useful information is not easily identifiable. Information Extraction (IE) aims to alleviate it by enabling automatic extraction of structured information from such text sources. The resulting structured information will facilitate easier querying, organizing, and analyzing of data from texts. In this thesis, we are interested in two IE related tasks: (i) named entity classification and (ii) template filling. Specifically, this thesis examines the problem of learning classifiers of text spans and explore its application for extracting named entities and template slot-fillers. In general, our goal is to construct a method to learn classifiers that: (i) require less supervision, (ii) work well with high-dimensional sparse feature spaces and (iii) are able to classify unseen items (i.e. named entities/slot-fillers not observed in training data). The key idea of our contribution is the utilization of unseen conjunctive features. A conjunctive feature is a combination of features from different feature sets. For example, to classify a phrase, one might have one feature set for the context and another set for the phrase itself. When learning a classifier, only a factor of these conjunctive features will be observed in the training set, leaving the rest (i.e. unseen features) unusable for predicting items in test time. We hypothesize that utilizing such unseen conjunctions is useful to address all of the aspects of the goal. We develop a general regularization framework specifically designed for sparse conjunctive feature spaces. Our strategy is based on employing tensors to represent the conjunctive feature space, and forcing the model to induce low-dimensional embeddings of the feature vectors via low-rank regularization on the tensor parameters. Such compressed representation will help prediction by generalizing to novel examples where most of the conjunctions will be unseen in the training set. We conduct experiments on learning named entity classifiers and template filling, focusing on extracting unseen items. We show that when learning classifiers under minimal supervision, our approach is more effective in controlling model capacity than standard techniques for linear classification. Uno de los retos en Procesamiento del Lenguaje Natural (NLP, del inglés Natural Language Processing) es la naturaleza no estructurada del texto, que hace que la información útil y relevante no sea fácilmente identificable. Los métodos de Extracción de Información (IE, del inglés Information Extraction) afrontan este problema mediante la extracción automática de información estructurada de dichos textos. La estructura resultante facilita la búsqueda, la organización y el análisis datos textuales. Esta tesis se centra en dos tareas relacionadas dentro de IE: (i) clasificación de entidades nombradas (NEC, del inglés Named Entity Classification), y (ii) rellenado de plantillas (en inglés, template filling). Concretamente, esta tesis estudia el problema de aprender clasificadores de secuencias textuales y explora su aplicación a la extracción de entidades nombradas y de valores para campos de plantillas. El objetivo general es desarrollar un método para aprender clasificadores que: (i) requieran poca supervisión; (ii) funcionen bien en espacios de características de alta dimensión y dispersión; y (iii) sean capaces de clasificar elementos nunca vistos (por ejemplo entidades o valores de campos que no hayan sido vistos en fase de entrenamiento). La idea principal de nuestra contribución es la utilización de características conjuntivas que no aparecen en el conjunto de entrenamiento. Una característica conjuntiva es una conjunción de características elementales. Por ejemplo, para clasificar la mención de una entidad en una oración, se utilizan características de la mención, del contexto de ésta, y a su vez conjunciones de los dos grupos de características. Cuando se aprende un clasificador en un conjunto de entrenamiento concreto, sólo se observará una fracción de estas características conjuntivas, dejando el resto (es decir, características no vistas) sin ser utilizado para predecir elementos en fase de evaluación y explotación del modelo. Nuestra hipótesis es que la utilización de estas conjunciones nunca vistas pueden ser potencialmente muy útiles, especialmente para reconocer entidades nuevas. Desarrollamos un marco de regularización general específicamente diseñado para espacios de características conjuntivas dispersas. Nuestra estrategia se basa en utilizar tensores para representar el espacio de características conjuntivas y obligar al modelo a inducir "embeddings" de baja dimensión de los vectores de características vía regularización de bajo rango en los parámetros de tensor. Dicha representación comprimida ayudará a la predicción, generalizando a nuevos ejemplos donde la mayoría de las conjunciones no han sido vistas durante la fase de entrenamiento. Presentamos experimentos sobre el aprendizaje de clasificadores de entidades nombradas, y clasificadores de valores en campos de plantillas, centrándonos en la extracción de elementos no vistos. Demostramos que al aprender los clasificadores bajo mínima supervisión, nuestro enfoque es más efectivo en el control de la capacidad del modelo que las técnicas estándar para la clasificación lineal

Anchor-guided online meta adaptation for fast one-Shot instrument segmentation from robotic surgical videos.

Few-shot learning under the <inline-formula><tex-math notation="LaTeX">$N$</tex-math></inline-formula> -way <inline-formula><tex-math notation="LaTeX">$K$</tex-math></inline-formula> -shot setting (i.e., <inline-formula><tex-math notation="LaTeX">$K$</tex-math></inline-formula> annotated samples for each of <inline-formula><tex-math notation="LaTeX">$N$</tex-math></inline-formula> classes) has been widely studied in relation extraction (e.g., FewRel) and image classification (e.g., Mini-ImageNet). Named entity recognition (NER) is typically framed as a sequence labeling problem where the entity classes are inherently entangled together because the entity number and classes in a sentence are not known in advance, leaving the <inline-formula><tex-math notation="LaTeX">$N$</tex-math></inline-formula> -way <inline-formula><tex-math notation="LaTeX">$K$</tex-math></inline-formula> -shot NER problem so far unexplored. In this paper, we first formally define a more suitable <inline-formula><tex-math notation="LaTeX">$N$</tex-math></inline-formula> -way <inline-formula><tex-math notation="LaTeX">$K$</tex-math></inline-formula> -shot setting for NER. Then we propose <small>FewNER</small> , a novel meta-learning approach for few-shot NER. <small>FewNER</small> separates the entire network into a task-independent part and a task-specific part. During training in <small>FewNER</small> , the task-independent part is meta-learned across multiple tasks and the task-specific part is learned for each individual task in a low-dimensional space. At test time, <small>FewNER</small> keeps the task-independent part fixed and adapts to a new task via gradient descent by updating only the task-specific part, resulting in it being less prone to overfitting and more computationally efficient. Compared with pre-trained language models (e.g., BERT and ELMo) which obtain the transferability in an implicit manner (i.e., relying on large-scale corpora), <small>FewNER</small> explicitly optimizes the capability of “learning to adapt quickly” through meta-learning. The results demonstrate that <small>FewNER</small> achieves state-of-the-art performance against nine baseline methods by significant margins on three adaptation experiments (i.e., intra-domain cross-type, cross-domain intra-type and cross-domain cross-type).