Deep Learning-based Algorithm for Mobile Robot Control in Textureless Environment

U ovom preglednom radu su prikazane metode vizuelnog upravljanja robotskih sistema, sa primarnim fokusom na mobilne robote sa diferencijalnim pogonom. Analizirane su standardne metode vizuelnog upravljanja bazirane na (i) greškama u parametrima slike (engl. Image-Based Visual Servoing - IBVS) i (ii) izdvojenim karakteristikama sa slike neophodnim za estimaciju položaja izabranog objekta (engl. Position-Based Visual Servoing - PBVS) i poređene sa novom metodom direktnog vizuelnog upravljanja (engl. Direct Visual Servoing - DVS). U poređenju sa IBVS i PBVS metodama, DVS metod se odlikuje višom tačnošću, ali i manjim domenom konvergencije. Zbog ovog razloga je DVS metod upravljanja pogodan za integraciju u hibridne sisteme vizuelnog upravljanja. Takođe, predstavljeni su radovi koji unapređuju sistem vizuelnog upravljanja korišćenjem stereo sistema (sistem sa dve kamere). Stereo sistem, u poređenju sa alternativnim metodama, omogućava tačniju ocenu dubine karakterističnih objekata sa slike, koja je neophodna za zadatke vizuelnog upravljanja. Predmet analize su i radovi koji integrišu tehnike veštačke inteligencije u sistem vizuelnog upravljanja. Ovim tehnikama sistemi vizuelnog upravljanja dobijaju mogućnost da uče, čime se njihov domen primene znatno proširuje. Na kraju, napominje se i mogućnost integracije vizuelne odometrije u sisteme vizuelnog upravljanja, što prouzrokuje povećanje robusnosti čitavog robotskog sistema.

Feature point detection is an important pre-processing step for quantitative evaluation of facial paralysis. Since the conventional methods such as active shape model (ASM) or active appearance model (AAM) are trained by using normal face and they are not possible to detect the feature points accurately for the face with paralysis. In this paper, we propose an automatic and accurate feature point detection method for quantitative evaluation of facial paralysis using deep convolutional neural networks (DCNN). The proposed method consists of two steps. We first use AAM for initial feature point detection. In the second step, a patch with the detected point at the center is used as an input of DCNN for refinement. Experiments demonstrated that the proposed method can significantly improve the detection accuracy of the conventional AAM.

The detection and matching of point features play an important role in most of the computer vision algorithms, such as; for 3-D reconstruction and robotics navigation, localization and mapping. Over the last years various detectors and descriptors have been proposed and successfully applied to the different applications. However, the developed detectors are based on computationally intensive algorithms, so it is desirable to implement them in hardware platform based on high performance reconfigurable systems. Nowadays, due to the flexible structure of Field Programmable Gate Array (FPGA) device, the latter can be involved in such complex algorithms of image processing. In this paper, some related works concerning implementation of features detection and matching algorithms on FPGA will be discussed in the state of the art, then we will illustrate a general framework for implementing two common features extraction detectors and two features matching algorithms on FPGA-Nios II system.

A significant amount of work has been reported in the area of visual servoing during the last decade. However, most of the contributions are applied in cases of holonomic robots. More recently, the use of visual feedback for control of nonholonomic vehicles has been reported. Some of the examples are docking and parallel parking maneuvers of cars or vision-based stabilization of a mobile manipulator to a desired pose with respect to a target of interest. Still, many of the approaches are mostly interested in the control part of visual servoing loop considering very simple vision algorithms based on artificial markers. In this paper, we present an approach for nonholonomic visual servoing based on epipolar geometry. The method facilitates a classical teach-by-showing approach where a reference image is used to define the desired pose (position and orientation) of the robot. The major contribution of the paper is the design of the control law that considers nonholonomic constraints of the robot as well as the robust feature detection and matching process based on scale and rotation invariant image features. An extensive experimental evaluation has been performed in a realistic indoor setting and the results are summarized in the paper

Converting tower-mounted videos from perspective to orthographic view is beneficial for their integration with maps and remote sensing images and can provide a clearer and more real-time data source for earth observation. This paper addresses the issue of low geometric accuracy in orthographic video generation by proposing a method that incorporates 3D GIS view matching. Firstly, a geometric alignment model between video frames and 3D GIS views is established through camera parameter mapping. Then, feature point detection and matching algorithms are employed to associate image coordinates with corresponding 3D spatial coordinates. Finally, an orthographic video map is generated based on the color point cloud. The results show that (1) for tower-based video, a 3D GIS constructed from publicly available DEMs and high-resolution remote sensing imagery can meet the spatialization needs of large-scale tower-mounted video data. (2) The feature point matching algorithm based on deep learning effectively achieves accurate matching between video frames and 3D GIS views. (3) Compared with the traditional method, such as the camera parameters method, the orthographic video map generated by this method has advantages in terms of geometric mapping accuracy and visualization effect. In the mountainous area, the RMSE of the control points is reduced from 137.70 m to 7.72 m. In the flat area, it is reduced from 13.52 m to 8.10 m. The proposed method can provide a near-real-time orthographic video map for smart cities, natural resource monitoring, emergency rescue, and other fields.

Visual servoing systems are designed to solve pose alignment problems by providing the necessary linear and angular velocities using data extracted from images. Among the difficulties encountered by the traditional visual servoing approaches, there are feature detection and tracking, camera calibration, scene complexity, and robotic system constraints. Part of these problems can be solved if Convolutional Neural Networks (CNNs) are added to a visual servoing architecture. The main advantage of CNNs is the capability of understanding both the overall structure and specific details of the images corresponding to the current and desired layouts. To take a step further the state-of-the-art architectures, in this paper, we show how extra input data can improve the visual servoing behaviour. The extra data result from maps of regions induced by the feature points' positions, without the necessity of employing tracking. The results obtained on relevant data sets show that the proposed input fusion-based CNN provides an improved approximation of the linear and angular visual servoing velocities.

Image stitching refers to the combination of multiple images with a certain over lappingarea rate acquired by the image acquisition device into one image, which has a full picture of each image. Image stitching technology is an important part in the field of digital image processing. With the increasing requirements for image quality and the limited resolution of ordinary cameras, it is impossible to obtain large angle of view and clear image at the same time. In this paper, the key technologies involved in pavement crack image stitching are intensively studied, including image feature point detection and matching, internal and external camera parameter estimation and optimization, and image fusion. The work done in this paper and the main innovations include: (1)Two main feature detection and description algorithms, SIFT algorithm and ORBalgorithm, are studied. In view of the fact that the SIFT algorithm has redundancy when the Gaussian pyramid is established, this paper appropriately reduces the number of groups and maintains the number and stability of feature point detection. Combined with the advantages and disadvantages of the improved SIFT algorithm and ORB algorithm, and the performance of the two algorithms in the detection of pavement cracks, the improvedSIF'T algorithm is used as the feature point detection algorithm, and rBRlEF algorithm for describing the feature points in the ORB algorithm is used as the feature point description algorithm. The description algorithm is called the BSIF'T algorithm. Experiments show that the robustness of BSIFT algorithm is similar to that of SIFT algorithm, which is better than ORB algorithm and faster than SIF'T algorithm. (2)Image fiusion algorithms are studied, including best seam-line search and image fusion. Aiming at the ghost phenomenon that may occur in overlapping regions, this paper proposes a best seam-line search algorithm based on distance transform.

SummaryWith the improvement of informatization and the development of computer technology, identity recognition with fingerprint has been the growing trend, but the stained fingerprint will make recognition difficult. To solve the above problem, this paper briefly introduced the traditional point matching fingerprint recognition algorithm and the damaged fingerprint recognition method based on the convolution neural network (CNN). Then in MATLAB software, the damaged fingerprint recognition method based on CNN was simulated and compared with the traditional point matching recognition method and traditional CNN recognition method. The results showed that the improved CNN recognition method iterated fewer times and achieved smaller training error during the training; the improved CNN method had lower false recognition rate and rejection rate in identifying the unknown fingerprint; the traditional point matching method spent the most time in identifying the unknown fingerprint, followed by the traditional CNN method and the improved CNN recognition method. This study used CNN for fingerprint recognition and improved CNN to improve its recognition accuracy for the damaged fingerprint. The improved CNN can effectively enhance the recognition accuracy of the damaged fingerprint, which provides a useful reference for the improvement of the accuracy and applicability of the fingerprint recognition system.

For soft continuum arms, visual servoing is a popular control strategy that relies on visual feedback to close the control loop. However, robust visual servoing is challenging as it requires reliable feature extraction from the image, accurate control models and sensors to perceive the shape of the arm, both of which can be hard to implement in a soft robot. This letter circumvents these challenges by presenting a deep neural network-based method to perform smooth and robust 3D positioning tasks on a soft arm by visual servoing using a camera mounted at the distal end of the arm. A convolutional neural network is trained to predict the actuations required to achieve the desired pose in a structured environment. Integrated and modular approaches for estimating the actuations from the image are proposed and are experimentally compared. A proportional control law is implemented to reduce the error between the desired and current image as seen by the camera. The model together with the proportional feedback control makes the described approach robust to several variations such as new targets, lighting, loads, and diminution of the soft arm. Furthermore, the model lends itself to be transferred to a new environment with minimal effort.

This paper develops a wearable sport activity classification system and its associated deep learning-based sport activity classification algorithm for accurately recognizing sport activities. The proposed wearable system used two wearable inertial sensing modules worn on athletes’ wrist and ankle to collect sport motion signals and utilized a deep convolutional neural network (CNN) to extract the inherent features from the spectrograms of the short-term Fourier transform (STFT) of the sport motion signals. The wearable inertial sensing module is composed of a microcontroller, a triaxial accelerometer, a triaxial gyroscope, an RF wireless transmission module, and a power supply circuit. All ten participants wore the two wearable inertial sensing modules on their wrist and ankle to collect motion signals generated by sport activities. Subsequently, we developed a deep learning-based sport activity classification algorithm composed of sport motion signal collection, signal preprocessing, sport motion segmentation, signal normalization, spectrogram generation, image mergence/resizing, and CNN-based classification to recognize ten types of sport activities. The CNN classifier consisting of two convolutional layers, two pooling layers, a fully-connected layer, and a softmax layer can be used to divide the sport activities into table tennis, tennis, badminton, golf, batting baseball, shooting basketball, volleyball, dribbling basketball, running, and bicycling, respectively. Finally, the experimental results show that the proposed wearable sport activity classification system and its deep learning-based sport activity classification algorithm can recognize 10 sport activities with the classification rate of 99.30%.

HighlightsWhat are the main findings?Proposed an innovative two-channel hybrid model by integrating convolutional neural networks (CNNs) with a text-based classifier.Leveraged an extended dataset derived from multiple object recognition models, increasing input data diversity and achieving a text-based classifier accuracy of 73.30%. Achieved a significant improvement of 8.33% in accuracy compared to CNN-only models, with the hybrid model attaining an accuracy of 90.46%.What is the implication of the main finding?Efficient and Scalable Methodology: Utilized EfficientNet for CNN-based feature extraction and Bag-of-Words for text representation, ensuring computational efficiency and scalability.Application Potential: Addressed challenges in indoor scene recognition, such as complex backgrounds and object diversity, demonstrating significant potential for applications in robotics, intelligent surveillance, and assistive systems.Indoor scene recognition is a computer vision task that identifies various indoor environments, such as offices, libraries, kitchens, and restaurants. This research area is particularly significant for applications in robotics, security, and assistance for individuals with disabilities, as it enables the categorization of spaces and the provision of contextual information. Convolutional Neural Networks (CNNs) are commonly employed in this field. While CNNs perform well in outdoor scene recognition by focusing on global features such as mountains and skies, they often struggle with indoor scenes, where local features like furniture and objects are more critical. In this study, the “MIT 67 Indoor Scene” dataset is used to extract and combine features from both a CNN and a text-based model utilizing object recognition outputs, resulting in a two-channel hybrid model. The experimental results demonstrate that this hybrid approach, which integrates natural language processing and image processing techniques, improves the test accuracy of the image processing model by 8.3%, achieving a notable success rate. Furthermore, this study offers contributions to new application areas in remote sensing, particularly in indoor scene understanding and indoor mapping.

Intra-operative imaging is widely used to provide visual feedback to a clinician when he/she performs a procedure. In visual servoing, surgical instruments and parts of tissue/body are tracked by processing the acquired images. This information is then used within a control loop to manoeuvre a robotic manipulator during a procedure. A comprehensive search of electronic databases was completed for the period 2000-2013 to provide a survey of the visual servoing applications in medical robotics. The focus is on medical applications where image-based tracking is used for closed-loop control of a robotic system. Detailed classification and comparative study of various contributions in visual servoing using endoscopic or direct visual images are presented and summarized in tables and diagrams. The main challenges in using visual servoing for medical robotic applications are identified and potential future directions are suggested. 'Supervised automation of medical robotics' is found to be a major trend in this field.

We present a deep neural network-based method to perform high-precision, robust and real-time 6 DOF positioning tasks by visual servoing. A convolutional neural network is fine-tuned to estimate the relative pose between the current and desired images and a pose-based visual servoing control law is considered to reach the desired pose. The paper describes how to efficiently and automatically create a dataset used to train the network. We show that this enables the robust handling of various perturbations (occlusions and lighting variations). We then propose the training of a scene-agnostic network by feeding in both the desired and current images into a deep network. The method is validated on a 6 DOF robot.

Craters are among the most abundant features on the surface of many planets with great importance for planetary scientists. They reveal chronology information about planets and may be used for autonomous spacecraft navigation and landing. Although numerous research efforts have been carried out in the field of crater detection, existing crater detection algorithms (CDAs) are only helpful in a limited number of applications. A promising crater detection approach involves two main steps: 1) hypothesis generation (HG) and 2) hypothesis verification (HV). During HG, potential crater locations are detected. The validity of the hypothesized crater locations is then tested in a HV step. In this context, we discuss some commonly used algorithms for HG such as highlight-shadow region detection and Hough transform as well as our novel and enhanced algorithms based on interest point detection and convex grouping. A key objective of this paper is to analyze their performance while paying special attention to how they affect the accuracy of the verification step. To deal with different size craters, we focus on multiscale HG. For HV, we have chosen convolutional neural networks which have recently achieved state-of-the-art performance in many computer vision applications. Due to the variation of test sets in the literature, it is often challenging to compare the performance of different CDAs in a fair way. In this paper, we present a comprehensive performance evaluation and comparison of CDAs. Each algorithm has been trained/tested using common data sets generated by a systematic approach.

With recent advancements in machine learning, especially in deep learning, the prediction of eye diseases based on fundus photography using deep convolutional neural networks (DCNNs) has attracted great attention. However, studies focusing on identifying the right disease among several candidates, which is a better approximation of clinical diagnosis in practice comparing with the case that aims to distinguish one particular eye disease from normal controls, are limited. The performance of existing algorithms for multi-class classification of fundus images is at most mediocre. Moreover, in many studies consisting of different eye diseases, labeled images are quite limited mainly due to privacy concern of patients. In this case, it is infeasible to train huge DCNNs, which usually have millions of parameters. To address these challenges, we propose to utilize a lightweight deep learning architecture called MobileNetV2 and transfer learning to distinguish four common eye diseases, including Glaucoma, Maculopathy, Pathological Myopia, and Retinitis Pigmentosa, from normal controls using a small training data. We also apply a visualization approach to highlight the loci that are most related to the disease labels to make the model more explainable. The highlighted area chosen by the algorithm itself may give some hints for further fundus image studies. Our experimental results show that our system achieves an average accuracy of 96.2%, sensitivity of 90.4%, and specificity of 97.6% on the test data via five independent runs, and outperforms two other deep learning-based algorithms both in terms of accuracy and efficiency.