An Integrated System for User-Adaptive Robotic Grasping

Evaluation of alternate multimedia for web-based asynchronous learning

In recent years, natural language machine interfaces have become increasingly common. These interfaces allow for more intuitive communication with machines, reducing the complexity of interacting with these systems and enabling their use by non-expert users. Most of these natural language interfaces rely on speech, including such well-known devices as the iPhone's Siri application, Cortana, Amazon's Alexa and Echo devices, and others. Given their intuitive functionality, natural language interfaces have also been investigated as a method for controlling unmanned aerial vehicles (UAVs), allowing non-subject matter experts to use these tools in their scientific pursuits. This paper examines a speech-based natural language interface for defining UAV trajectories. To determine the efficacy of this interface, a user study is also presented that examines how users perform with this interface compared to a traditional mouse-based interface. The results of the user study are described in order to show how accurately users were able to define trajectories as well as user preference for using the speech-based system both before and after participating in the user study. Additional data are presented on whether users had previous experience with speech-based interfaces and how long they spent training with the interface before participating in the study. The user study demonstrates the potential of speech- based interfaces for UAV trajectory generation and suggests methods for future improvement and incorporation of natural language interfaces for UAV pilots.

Spoken dialog systems today do not vary the prosody of their utterances, although prosody is known to have many useful expressive functions. In a corpus of memory quizzes, we identify eleven dimensions of prosodic variation, each with its own expressive function. We identified the situations in which each was used, and developed rules for detecting these situations from the dialog context and the prosody of the interlocutor’s previous utterance. We implemented the resulting rules and had 21 users interact with two versions of the system. Overall they preferred the version in which the prosodic forms of the acknowledgments were chosen to be suitable for each specific context. This suggests that simple adjustments to system prosody based on local context can have value to users. Index Terms: Confidence, Emotion, User Modeling, Sensitive, Tutorial, Quiz, Social Interaction, Pace

When encountering unfamiliar APIs, developers tend to seek help from API tutorials and Stack Overflow (SO). API tutorials help developers understand the API knowledge in a general context, while SO often explains the API knowledge in a specific programming task. Thus, tutorials and SO posts together can provide more API knowledge. However, it is non-trivial to retrieve API knowledge from both API tutorials and SO posts based on natural language queries. Two major problems are irrelevant API knowledge in two different resources and the lexical gap between the queries and documents. In this article, we regard a fragment in tutorials and a Question and Answering (Q&A) pair in SO as a knowledge item (KI). We generate ⟨ API, FRA ⟩ pairs (FRA stands for fragment) from tutorial fragments and APIs and build ⟨ API, QA ⟩ pairs based on heuristic rules of SO posts. We fuse ⟨ API, FRA ⟩ pairs and ⟨ API, QA ⟩ pairs to generate API knowledge (AK for short) datasets, where each data item is an ⟨ API, KI ⟩ pair. We propose a novel approach, called PLAN, to automatically retrieve API knowledge from both API tutorials and SO posts based on natural language queries. PLAN contains three main stages: (1) API knowledge modeling, (2) query mapping, and (3) API knowledge retrieving. It first utilizes a deep-transfer-metric-learning-based relevance identification (DTML) model to effectively find relevant ⟨ API, KI ⟩ pairs containing two different knowledge items (⟨ API, QA ⟩ pairs and ⟨ API, FRA ⟩ pairs) simultaneously. Then, PLAN generates several potential APIs as a way to reduce the lexical gap between the query and ⟨ API, KI ⟩ pairs. According to potential APIs, we can select relevant ⟨ API, KI ⟩ pairs to generate potential results. Finally, PLAN returns a list of ranked ⟨ API, KI ⟩ pairs that are related to the query. We evaluate the effectiveness of PLAN with 270 queries on Java and Android AK datasets containing 10,072 ⟨ API, KI ⟩ pairs. Our experimental results show that PLAN is effective and outperforms the state-of-the-art approaches. Our user study further confirms the effectiveness of PLAN in locating useful API knowledge.

Brain-computer interfaces (BCIs) translate brain activity into digital commands for interaction with the physical world. The technology has great potential in several applied areas, ranging from medical applications to entertainment industry, and creates new conditions for basic research in cognitive neuroscience. The BCIs of today, however, offer only crude online classification of the user's current state of mind, and more sophisticated decoding of mental states depends on time-consuming offline data analysis. The present paper addresses this limitation directly by leveraging a set of improvements to the analytical pipeline to pave the way for the next generation of online BCIs. Specifically, we introduce an open-source research framework that features a modular and customizable hardware-independent design. This framework facilitates human-in-the-loop (HIL) model training and retraining, real-time stimulus control, and enables transfer learning and cloud computing for the online classification of electroencephalography (EEG) data. Stimuli for the subject and diagnostics for the researcher are shown on separate displays using web browser technologies. Messages are sent using the Lab Streaming Layer standard and websockets. Real-time signal processing and classification, as well as training of machine learning models, is facilitated by the open-source Python package Timeflux. The framework runs on Linux, MacOS, and Windows. While online analysis is the main target of the BCI-HIL framework, offline analysis of the EEG data can be performed with Python, MATLAB, and Julia through packages like MNE, EEGLAB, or FieldTrip. The paper describes and discusses desirable properties of a human-in-the-loop BCI research platform. The BCI-HIL framework is released under MIT license with examples at: bci.lu.se/bci-hil (or at: github.com/bci-hil/bci-hil).

GINKO, an integrated learning and planning system that has been applied to an autonomous mobile robot domain, is described. The goal of GINKO's learning system is to partition the robot's configuration space into regions in which actions exhibit a uniform qualitative behavior. This partitioning is performed by an inductive learning algorithm that classifies regions of the configuration space with regard to the effects of the robot's actions when executed in those regions. GINKO's learning is driven by its attempts to perform tasks. Thus, the learned effects of actions are directly applicable to normal system performance. >

Algorithmic data fusion for multi-sensory system becomes extremely challenging, particularly where the elemental sensory units do vary in type, size and characteristics. Although traditional theories on sensory data fusion fit quite satisfactorily in searching a pre-defined object with a tentative dimension and depth perception, they fail to do justice in cases where profile of the object do vary from a point-mass to a finite spatial dimension. The present paper dwells on modeling, algorithm and experimental analysis of two novel fusion rule-bases, which are implemented in a small-sized tactile array sensor to be used in robot gripper. A new proposition has also been developed for assessing the decision threshold, signaling the presence of object inside the grasp-zone of the gripper. Besides, the developed model evaluates the approximate planar area of the grasped object alongwith its shape in real-time. The model also provides estimate for the gripping force required to sustain a stable grasp of the object vis-a-vis slippage characteristics, if any.

A key challenge facing natural language interfaces is enabling users to understand the capabilities of the underlying system. We propose a novel approach for generating explanations of a natural language interface based on semantic parsing. We focus on counterfactual explanations, which are post-hoc explanations that describe to the user how they could have minimally modified their utterance to achieve their desired goal. In particular, the user provides an utterance along with a demonstration of their desired goal; then, our algorithm synthesizes a paraphrase of their utterance that is guaranteed to achieve their goal. In two user studies, we demonstrate that our approach substantially improves user performance, and that it generates explanations that more closely match the user's intent compared to two ablations.

Bidirectional Human-Robot Learning: Imitation and Skill Improvement

This work presents the development of a wireless, low cost, wearable sensor jacket for the purpose of controlling a robot arm by mimicking the motion and behaviour of a humans arm. The intended use of our system is to provide remote daily nursing care by using our system from a distant place such as a nursing home or hospital, to control a stationed robotic arm placed in an elderly or patient's home. The final system is comprised of a wearable jacket which is embedded with IMU and flex sensors to detect and track the wearers arm movements and behaviour. The system is capable of detecting up to 5 degrees of freedom of the human arm and replicate these motions using a 6 DOF robot arm. The usability, accuracy and precision of our jacket system is evaluated through a user study and the results demonstrated that our system was more accurate and easier to use for operators than a conventional robotic arm joystick controller. In a water bottle transfer task our developed wearable jacket system demonstrated an average error distance of 29.36mm from the target point, while the results using the conventional joystick demonstrated an average error distance of 37.48mm. Furthermore subjects using our system were able to complete the transfer task in an average time of 44.1s per trial which was more efficient than the joystick method in which subjects averaged 55.55s per trial. Finally, we report a feasibility study with the jacket and a subject to demonstrate the capability of this system of giving a patient water to drink. The feasibility experiment showed an 86.66% success rate in giving a patient water via video stream teleoperation control.

The integration of people with disabilities into the working world is an important, yet challenging field of research. While different inclusion efforts exist, people with disabilities are still under-represented in the open labor market. This paper investigates the approach of using a collaborative robot arm to support people with disabilities with their reintegration into the workplace. However, there is currently little literature about the acceptance of an industrial robot by people with disabilities and in cases where a robot leads to stress, fear, or any other form of discomfort, this approach is not feasible. For this reason, a first user study was performed in a sheltered workshop to investigate the acceptance of a robot arm by workers with disabilities. As a first step in this underdeveloped field, two main aspects were covered. Firstly, the reaction and familiarization to the robot arm within a study situation was closely examined in order to separate any effects that were not caused by the moving robot. Secondly, the reaction toward the robot arm during collaboration was investigated. In doing so, five different distances between the robot arm and the participants were considered to make collaboration in the workplace as pleasant as possible. The results revealed that it took the participants about 20 min to get used to the situation, while the robot was immediately accepted very well and did not cause fear or discomfort at any time. Surprisingly, in some cases, short distances were accepted even better than the larger distances. For these reasons, the presented approach showed to promise for future investigations.

Learning systems can utilize many practice exercises, ranging from simple multiple-choice questions to complex problem-solving activities. In this article, we propose a classification framework for such exercises. The framework classifies exercises in three main aspects: 1) the primary type of interaction; 2) the presentation mode; and 3) the integration in the learning system. For each of these aspects, we provide a systematic mapping of available choices and pointers to relevant research. For developers of learning systems, the framework facilitates the design and implementation of exercises. For researchers, the framework provides support for the design, description, and discussion of experiments dealing with student modeling techniques and algorithms for adaptive learning. One of the aims of the framework is to facilitate replicability and portability of research results in adaptive learning.

In this article we describe a general-purpose robotic grasping system for use in unstructured environments. Using computer vision and a compact set of heuristics, the system automatically generates the robot arm and hand motions reqrsired for grasping an unmodeled object. The utility of such a system is most evident in environ ments where the robot will have to grasp and manipulate a variety of unknown objects, but many of these manipu lation tasks may be relatively simple. Examples of such domains are planetary exploration and astronaut assis tance, undersea salvage and rescue, and nuclear waste site clean-up. This work implements a two-stage model of grasping: stage one is an orientation of the hand and wrist and a ballistic reach toward the object; stage two is hand preshaping and adjustment. Visual features are first extracted from the unmodeled object. These features and their relations are used by an expert system to generate a set of valid reach/grasps for the object. These grasps are then used in driving the robot hand and arm to bring the fingers into contact with the object in the desired configu ration. Experimental results are presented to illustrate the functioning of the system.

Robot grasping under the instruction of natural language has attracted increasing attention in various applications for its advantages in enabling natural and smooth human-robot interaction. At present, mainstream algorithms mainly solve problems of utilizing simple natural language instructions to guide the robot arm to perform some specific grasping. However, for two natural language instructions with different temporal logic and the same semantics, it is usually difficult for the robot to achieve semantic disambiguation, which further leads to the failure of the grasping task. In order to address this problem, we propose a new natural language instruction disambiguation method for robot grasping by combining sentence vector similarity calculation model and sentence temporal logic model. Firstly, the word vector is obtained through the Skip-gram model in Word2vec and a sentence vector is constructed. The semantic similarity of the sentence is then calculated by using the proposed cost function. Based on the semantic similarity of the sentence, the correct temporal logic form of the sentence is then extracted according to the temporal adverbial priority to further guide the grabbing process of the robot arm. The experimental results show that our method can successfully realize the semantic disambiguation for natural language instructions with different temporal logics and the same semantics, and further guide the robot arm to complete more complicated tasks than previous tasks.

In this paper, we propose a Learning from Demonstration approach for robotic grasping with compliant arms. The compliance in the robot arm for safety often causes a problem in grasping. In our approach, we construct a recurrent neural network, given the estimation of the target object position and random initial joint angles of the robot arm, that can produce the whole trajectories for grasping the target object. In order to generate smooth and stable trajectories and to decrease the number of human demonstrations, we propose a data augmentation method to increase the training data and utilize the trajectory planning technique using cubic splines for smooth and stable trajectories. Specifically, the two arms of the robot are trained respectively, and a support vector machine is used to decide which arm needs to be used for grasping the target object. The evaluation results show that our recurrent model not only has a good prediction for the final joint configurations, but also generates smooth and stable trajectory. Moreover, the model is robust to the changes in the initial joint state which means that even though the initial joint configuration is affected by disturbances, the model can still generate trajectories leading to the final joint configurations for grasping the object. Finally, we tested the proposed learning method on the Pepper robot which can successfully grasp randomly placed object on the workbench. Compared to traditional methods which need to avoid singular configurations as well as to secure accurate localization, our method turns out to be robust and efficient and can be applied to cluttered environment.