Abstract This paper presents Diver Interest via Pointing in Three Dimensions (DIP-3D), a method to indicate an object of interest from a diver to an autonomous underwater vehicle (AUV) by pointing that includes three-dimensional distance information to discriminate between multiple objects in the AUV’s field of view. Traditional dense stereo vision for distance estimation underwater is challenging because of the relative lack of saliency of scene features and degraded lighting conditions. Yet in many applications, including distance information is necessary for robotic perception of diver pointing when multiple objects appear within the robot’s image view. We subvert the challenges of underwater distance estimation by using sparse reconstruction of specific keypoints in both the left and right images from the robot’s stereo camera to perform pose estimation. Triangulated pose keypoints, along with any object detection method, enable DIP-3D to infer the location of an object of interest when multiple objects are in the AUV’s field of view. By allowing the scuba diver to point at an arbitrary object of interest and enabling the AUV to autonomously decide which object the diver is pointing to, this method permits more natural interaction between AUVs and humans in underwater-human robot collaborative tasks.

An important class of robotic applications involves multiple agents cooperating to provide state observations to plan joint actions. We study planning under uncertainty when more than one participant must proactively plan perception and/or communication acts, and decide whether the cost to obtain a state estimate is justified by the benefits accrued by the information thus obtained. The approach we introduce is suitable for settings where observations are of high quality and they—either alone or along with communication—recover the system’s joint state, but the costs incurred mean this happens only infrequently. We formulate the problem as a type of Markov decision process (mdp) to be solved over macro-actions, sidestepping the construction of the full joint belief space, a well-known source of intractability. We then give a suitable Bellman-like recurrence that immediately suggests a means of solution. In their most general form, policies for these problems simultaneously describe (1) low-level actions to be taken, (2) stages when system-wide state is recovered, and (3) commitments to future rescheduling acts. The formulation expresses multi-agency in a variety of distinct practical forms, including: one party assisting by providing observations of, or reference points for, another; several agents communicating sensor information to fuse data and recover joint state; multiple agents coordinating activities to arrive at states that make joint state simultaneously observable to all individuals. Though solved in centralized form over joint states, the mdp is structured to allow decentralized execution, under some assumptions of synchrony in activities. After providing small-scale simulation studies of the general formulation, we discuss a specific scenario motivated by underwater gliders. We report on a physical robot implementation mocked-up to respect these same constraints, showing that joint plans are found and executed effectively by individual robots after appropriate projection. On the basis of our experience with hardware, we examine enhancements to the model that address nonidealities we have identified in practice, including the assumptions regarding synchrony.

Abstract This paper proposes a novel system to conduct visual servoing of a mobile robot using multiple uncalibrated, wall-mounted cameras. Specifically, we utilise a constellation of such sensors to cover a wide area by allowing robot control to be passed between cameras in regions where their fields of view overlap. This method, in conjunction with the fact that all computing is also executed offboard, allows for simpler and cheaper robots to be deployed in controlled and finite spaces. Our method simplifies the natural installation cycle of a newly deployed camera network, eliminating the need for explicit camera positioning or orientation, both globally (relative to a building plan) and locally (among viewpoints). Our system memorises pixel-wise topological connections between viewpoints by leveraging natural human exploration of the environment. We detect graph edges through simultaneous detections of the same person across different cameras, allowing us to automatically construct an evolving graph that represents overlapping fields of view within the camera network. In combination with a hybrid-A*-based planner, our approach allows efficient planning and control of robots across a wide area by traversing cameras between areas of overlap. We validate our approach through autonomous traversals in a productive office environment, using a network of six cameras, and compare our performance against both human teleoperation and a traditional Simultaneous Localisation and Mapping (SLAM) approach.

Abstract In collective motion , perceptually-limited individuals move in an ordered manner, without centralized control. The perception of each individual is highly localized, as is its ability to interact with others. While natural collective motion is robust, most artificial swarms are brittle . This particularly occurs when vision is used as the sensing modality, due to ambiguities and information-loss inherent in visual perception. This paper presents mechanisms for robust collective motion inspired by studies of locusts. First, we develop a robust distance estimation method that combines visually perceived horizontal and vertical sizes of neighbors. Second, we introduce intermittent locomotion as a mechanism that allows robots to reliably detect peers that fail to keep up, and disrupt the motion of the swarm. We show how such faulty robots can be avoided in a manner that is robust to errors in classifying them as faulty. Through extensive physics-based simulation experiments, we show dramatic improvements to swarm resilience when using these techniques. We show these are relevant to both distance-based Avoid–Attract models, as well as to models relying on Alignment , in a wide range of experiment settings.

We propose a learning-based method to extract symbolic representations of the belief state and its dynamics in order to solve planning problems in a continuous-state partially observable Markov decision processes (POMDP) problem. While existing approaches typically parameterize the continuous-state POMDP into a finite-dimensional Markovian model, they are unable to preserve fidelity of the abstracted model. To improve accuracy of the abstracted representation, we introduce a memory-dependent abstraction approach to mitigate the modeling error. The first major contribution of this paper is we propose a Neural Network based method to learn the non-Markovian transition model based on the Mori-Zwanzig (M-Z) formalism. Different from existing work in applying M-Z formalism to autonomous time-invariant systems, our approach is the first work generalizing the M-Z formalism to robotics, by addressing the non-Markovian modeling of the belief dynamics that is dependent on historical observations and actions. The second major contribution is we theoretically show that modeling the non-Markovian memory effect in the abstracted belief dynamics improves the modeling accuracy, which is the key benefit of the proposed algorithm. Simulation experiment of a belief space planning problem is provided to validate the performance of the proposed belief abstraction algorithms.