DFVS: Deep Flow Guided Scene Agnostic Image Based Visual Servoing

We investigated whether a deep neural network (DNN)-based source enhancement function can be self-optimized by reinforcement learning (RL). The use of a DNN is a powerful approach to describing the relationship between two sets of variables and can be useful for source enhancement function design. By training the DNN using a huge amount of training data, sound quality of output signals are improved. However, collecting a huge amount of training data is often difficult in practice. To use limited training data efficiently, we focus on the "self-optimization" of DNN-based source enhancement function in which RL is commonly utilized in the development of game playing computers. As a reward for RL, quantitative metrics that reflect a human's perceptual score (perceptual score), e.g., perceptual evaluation methods for audio source separation (PEASS), are utilized. To investigate whether the sound quality is improved by RL-based source enhancement, subjective tests were conducted. It was confirmed that the output sound quality of the RL-based source enhancement function improved as the number of iterations was increased and finally outperformed the conventional method.

Ball balancers are nonlinear, electromechanical, multivariable, open-loop unstable systems widely used in research laboratories, aerospace, military, and automotive industries to evaluate control mechanism effectiveness. The inherent difficulty in precisely managing ball position, combined with actuator saturation and system sensitivity to disturbances, makes trajectory tracking a persistent challenge. Conventional controllers often exhibit oscillatory responses with steady-state errors exceeding acceptable limits. Sliding mode control (SMC) offers robustness against model uncertainties; however, chattering finite-frequency, finite-amplitude oscillations near the sliding surface caused by switching imperfections, time delays, and actuator dynamics remain a significant limitation. This study addresses chattering through explicit vibration model compensation integrated into the SMC design for a 2-DOF ball balancer system using a visual servoing approach. A double-loop control architecture is implemented, where the inner loop handles servo angular position control and the outer loop manages ball position tracking through visual servoing feedback. The sliding mode controller is designed with a power rate reaching law, synthesizing two control laws: one with explicit vibration model compensation incorporating damping and stiffness terms, and one without. Experimental validation confirmed that SMC with compensation achieved significantly reduced steady-state error (0.034 mm vs. 0.386 mm) and lower overshoot (3.95% vs. 13.81%) compared to the uncompensated variant, with chattering amplitude reduced by approximately 72%.

Deep Neural Networks (DNNs) with sparse input features have been widely used in recommender systems in industry. These models have large memory requirements and need a huge amount of training data. The large model size usually entails a cost, in the range of millions of dollars, for storage and communication with the inference services. In this paper, we propose a hybrid hashing method to combine frequency hashing and double hashing techniques for model size reduction, without compromising performance. We evaluate the proposed models on two product surfaces. In both cases, experiment results demonstrated that we can reduce the model size by around 90 while keeping the performance on par with the original baselines.

This paper proposes a real-time, robust and efficient 3D model-based tracking algorithm for visual servoing. A virtual visual servoing approach is used for monocular 3D tracking. This method is similar to more classical nonlinear pose computation techniques. A concise method for derivation of efficient distance-to-contour interaction matrices is described. An oriented edge detector is used in order to provide real-time tracking of points normal to the object contours. Robustness is obtained by integrating a M-estimator into the virtual visual control law via an iteratively reweighted least squares implementation. The method presented in this paper has been validated on several 2D 1/2 visual servoing experiments considering various objects. Results show the method to be robust to occlusion, changes in illumination and miss-tracking.

Self-supervised rigid object 3-D motion estimation from monocular video

This work presents different proposals to deal with common problems in robot visual servoing based on the application of discontinuous control methods. The feasibility and effectiveness of the proposed approaches are substantiated by simulation results and real experiments using a 6R industrial manipulator. The main contributions are:\n\n- Geometric invariance using sliding mode control (Chapter 3): the defined higher-order invariance is used by the proposed approaches to tackle problems in visual servoing. Proofs of invariance condition are presented.\n\n- Fulfillment of constraints in visual servoing (Chapter 4): the proposal uses sliding mode methods to satisfy mechanical and visual constraints in visual servoing, while a secondary task is considered to properly track the target object. The main advantages of the proposed approach are: low computational cost, robustness and fully utilization of the allowed space for the constraints.\n\n- Robust auto tool change for industrial robots using visual servoing (Chapter 4): visual servoing and the proposed method for constraints fulfillment are applied to an automated solution for tool changing in industrial robots. The robustness of the proposed method is due to the control law of the visual servoing, which uses the information acquired by the vision system to close a feedback control loop. Furthermore, sliding mode control is simultaneously used in a prioritized level to satisfy the aforementioned constraints. Thus, the global control accurately places the tool in the warehouse, but satisfying the robot constraints.\n\n- Sliding mode controller for reference tracking (Chapter 5): an approach based on sliding mode control is proposed for reference tracking in robot visual servoing using industrial robot manipulators. The novelty of the proposal is the introduction of a sliding mode controller that uses a high-order discontinuous control signal, i.e., joint accelerations or joint jerks, in order to obtain a smoother behavior and ensure the robot system stability, which is demonstrated with a theoretical proof.\n\n- PWM and PFM for visual servoing in fully decoupled approaches (Chapter 6): discontinuous control based on pulse width and pulse frequency modulation is proposed for fully decoupled position based visual servoing approaches, in order to get the same convergence time for camera translation and rotation.\n\nMoreover, other results obtained in visual servoing applications are also described.

<p>Visual servoing (VS) is a technique which utilizes information acquired from a visual sensor to control the motion of a robot. Despite the advantages of the classic VS techniques in controlling of the robots- specifically in unstructured environments- they suffer from some system’s shortcomings such as the lack of an effective mechanism for handling the existing constraints within the control procedure. To alleviate these issues, among the several approaches proposed in the literature, the receding horizon or model predictive control (MPC) technique is an appropriate candidate to be employed by the system. Any combination of the model predictive control and visual servoing is called visual predictive control (VPC) which is categorized as the classic and hybrid methods. The classic VPC schemes are developed based on the classic VS approaches and the hybrid ones employ the hybrid VS schemes to be integrated by MPC. On the other hand, to guarantee the convergence of the system in presence of the noises and uncertainties, the proposed VPC schemes are embedded in internal model control (IMC) framework.</p> <p>This work contributes by utilizing the available VPC scheme in control of different robots, such as 3 DOF catheter as a continuum robot and a 6 DOF manipulator from Denso robotics as a robot arm, and by developing two hybrid predictive controllers, conjugated VPC and depth-based VPC, with higher constraint handling capabilities. By utilizing the proposed VPC methods, some of the available limitations could be resolved in the single-robot control level. In addition, to enable the system to perform more sophisticated tasks under difficult conditions such as unexpected occlusions, a multi-agent system controlled by the mentioned VPC methods is employed. Under these conditions, by utilizing the data sharing ability among neighboring agents inside the network, the obstructed one could be provided with enough information to complete the task. At each stage, the performance of the proposed algorithm is verified through numerous simulations and experiments to show their capability in handling existing constraints while accounting for system uncertainties.</p>

Video hyperlinking represents a classical example of multimodal problems. Common approaches to such problems are early fusion of the initial modalities and crossmodal translation from one modality to the other. Recently, deep neural networks, especially deep autoencoders, have proven promising both for crossmodal translation and for early fusion via multimodal embedding. A particular architecture, bidirectional symmetrical deep neural networks, have been proven to yield improved multimodal embeddings over classical autoencoders, while also being able to perform crossmodal translation. In this work, we focus firstly at evaluating good single-modal continuous representations both for textual and for visual information. Word2Vec and paragraph vectors are evaluated for representing collections of words, such as parts of automatic transcripts and multiple visual concepts, while different deep convolutional neural networks are evaluated for directly embedding visual information, avoiding the creation of visual concepts. Secondly, we evaluate methods for multimodal fusion and crossmodal translation, with different single-modal pairs, in the task of video hyperlinking. Bidirectional (symmetrical) deep neural networks were shown to successfully tackle downsides of multimodal autoencoders and yield a superior multimodal representation. In this work, we extensively tests them in different settings, with different single-modal representations, within the context of video-hyperlinking. Our novel bidirectional symmetrical deep neural networks are compared to classical autoencoders and are shown to yield significantly improved multimodal embeddings that significantly (alpha=0.0001) outperform multimodal embeddings obtained by deep autoencoders with an absolute improvement in precision at 10 of 14.1% when embedding visual concepts and automatic transcripts and an absolute improvement of 4.3% when embedding automatic transcripts with features obtained with very deep convolutional neural networks, yielding 80% of precision at 10.

This paper aims the estimation of the robots motion angles through a linear fusion of two visuals information: Horizon Line position and orientation, and Image Optical Flow. The need of integration of movement estimation techniques is due to the fact that the angles estimation using Horizon information is used with efficiency in several works about aerial robots odometry, but this approach does not enable yaw angle estimation. Optical Flow approach enables the estimation of all of the angles and can be used without the presence of Horizon on the vision field, but can fail in situations which its constraints are not complied. Preliminary experiments made with aerial robots proved the efficiency of the presented method. Despite the difference on the degrees of freedom of other types of robots in comparison with aerial robots, this odometry method can be used on all the types of land robots to estimate its movements. Obviously, in particular cases, the land robot movement can be described with less of the three movement angles because the minor degree of freedom of the robot.

This study proposes an optimized hybrid visual servoing approach to overcome the imperfections of classical two-dimensional, three-dimensional and hybrid visual servoing methods. These imperfections are mostly convergence issues, non-optimized trajectories, expensive calculations and singularities. The proposed method provides more efficient optimized trajectories with shorter camera path for the robot than image-based and classical hybrid visual servoing methods. Moreover, it is less likely to lose the object from the camera field of view, and it is more robust to camera calibration than the classical position-based and hybrid visual servoing methods. The drawbacks in two-dimensional visual servoing are mostly related to the camera retreat and rotational motions. To tackle these drawbacks, rotations and translations in Z-axis have been separately controlled from three-dimensional estimation of the visual features. The pseudo-inverse of the proposed interaction matrix is approximated by a neuro-fuzzy neural network called local linear model tree. Using local linear model tree, the controller avoids the singularities and ill-conditioning of the proposed interaction matrix and makes it robust to image noises and camera parameters. The proposed method has been compared with classical image-based, position-based and hybrid visual servoing methods, both in simulation and in the real world using a 7-degree-of-freedom arm robot.

This paper showcases the integration of the Interfacing Toolbox for Robotic Arms (ITRA) with our newly developed hybrid Visual Servoing (VS) methods to automate the disassembly of electric vehicle batteries, thereby advancing sustainability and fostering a circular economy. ITRA enhances collaboration between industrial robotic arms, server computers, sensors, and actuators, meeting the intricate demands of robotic disassembly, including the essential real-time tracking of components and robotic arms. We demonstrate the effectiveness of our hybrid VS approach, combined with ITRA, in the context of Electric Vehicle (EV) battery disassembly across two robotic testbeds. The first employs a KUKA KR10 robot for precision tasks, while the second utilizes a KUKA KR500 for operations needing higher payload capacity. Conducted in T1 (Manual Reduced Velocity) mode, our experiments underscore a swift communication protocol that links low-level and high-level control systems, thus enabling rapid object detection and tracking. This allows for the efficient completion of disassembly tasks, such as removing the EV battery’s top case in 27 s and disassembling a stack of modules in 32 s. The demonstrated success of our framework highlights its extensive applicability in robotic manufacturing sectors that demand precision and adaptability, including medical robotics, extreme environments, aerospace, and construction.

Visual servoing systems are designed to solve pose alignment problems by providing the necessary linear and angular velocities using data extracted from images. Among the difficulties encountered by the traditional visual servoing approaches, there are feature detection and tracking, camera calibration, scene complexity, and robotic system constraints. Part of these problems can be solved if Convolutional Neural Networks (CNNs) are added to a visual servoing architecture. The main advantage of CNNs is the capability of understanding both the overall structure and specific details of the images corresponding to the current and desired layouts. To take a step further the state-of-the-art architectures, in this paper, we show how extra input data can improve the visual servoing behaviour. The extra data result from maps of regions induced by the feature points' positions, without the necessity of employing tracking. The results obtained on relevant data sets show that the proposed input fusion-based CNN provides an improved approximation of the linear and angular visual servoing velocities.

This paper has two purposes: investigating a featureless visual servoing approach based on mutation analysis and proposing a visual servo control method for nanomanipulations. For the first purpose, the featureless visual servoing method is needed because traditional visual servoing relies heavily on robust feature extraction and tracking, which are very difficult in natural environment. The mutation analysis based approach in this paper considers the image as a set, and designs a controller to make the distance between the initial and goal image sets converge to zero, thereby steering the initial image to the goal image. For the second purpose, atomic force microscopic (AFM) based nanomanipulations with subnanometer accuracy are very difficult because the position sensor cannot provide valuable feedback due to large noises at this precision level. We propose to use the images obtained by AFM and perform a visual servo control. This method, independent of external sensors, can directly perform control on the AFM end tip's position. The featureless controller is successfully validated on AFM images and the results suggest a potential precision enhancement for nanomanipulations.

HighlightsA deep learning architecture, which relies on multi-modal imagery from the RGB and thermal spectra, is proposed to detect PPM nests via aerial drones.A GPS-based computation method is implemented to geo-tag the position of the detected nests in the drone’s perimeter.A Kalman filter is utilized as a nest tracker to avoid reporting back the position of the same nest several times.The system is tested in a pine forest and exhibits accurate detection and localization, as evidenced by reporting back the positions of the detected PPM nests.Abstract.The pine processionary moth (PPM) is considered the main defoliator of pine trees and is a menacing threat to various other perennial species including oak and cedar. Given their negative secondary effects, spraying of pesticides has been banned as a means for the eradication of PPM; instead, an individualized approach is to be adopted, in which each nest is localized and destroyed. Detection of PPM nests using optical sensing is challenging because of the changing outdoor lighting conditions and the camouflaged appearance of moth in the underlying foliage. This article proposes a promising solution for nest detection by fusing sensory data from an RGB camera on one hand, and a thermal camera on the other, both mounted on an aerial drone. The proposed detection system is built on a two-channeled deep convolutional neural network, one for each spectrum of the collected sensory data. Experiments performed in a pine forest demonstrate successful detection rates with an average accuracy of 97% in various experiments and settings. Geo-localization is performed to report the position of the detected nests within a scanned forest map by means of an estimation scheme that is particularly designed for this purpose achieving centimeter accuracy (<20 cm). Keywords: Aerial robotics, Agriculture, Detection, Machine learning, Multi-stream CNN, Pest control.

In the last decade, research on vision-based robot control has been concentrated on two main issues: the narrow field of view of the conventional camera and the model dependency of the standard visual servoing approaches. In this paper, we propose a simple and elegant solution to these issues. To enlarge the field of view of the cameras, we use omnidirectional cameras. And to overcome the model dependency problem, we propose a new visual servoing method for omnidirectional cameras that does not need any measure of the 3D structure of the observed target with respect to which the visual servoing is performed. Only visual information measured from the reference and the current images are needed in order to compute a task function isomorphic to the camera pose and to compute the control law to be applied to the robot. We provide the theoretical proof of the existence of the isomorphism and the theoretical proof of the local stability of the control law