Object Recognition and Simultaneous Indoor Localization Algorithm with Stereo Camera

Converting tower-mounted videos from perspective to orthographic view is beneficial for their integration with maps and remote sensing images and can provide a clearer and more real-time data source for earth observation. This paper addresses the issue of low geometric accuracy in orthographic video generation by proposing a method that incorporates 3D GIS view matching. Firstly, a geometric alignment model between video frames and 3D GIS views is established through camera parameter mapping. Then, feature point detection and matching algorithms are employed to associate image coordinates with corresponding 3D spatial coordinates. Finally, an orthographic video map is generated based on the color point cloud. The results show that (1) for tower-based video, a 3D GIS constructed from publicly available DEMs and high-resolution remote sensing imagery can meet the spatialization needs of large-scale tower-mounted video data. (2) The feature point matching algorithm based on deep learning effectively achieves accurate matching between video frames and 3D GIS views. (3) Compared with the traditional method, such as the camera parameters method, the orthographic video map generated by this method has advantages in terms of geometric mapping accuracy and visualization effect. In the mountainous area, the RMSE of the control points is reduced from 137.70 m to 7.72 m. In the flat area, it is reduced from 13.52 m to 8.10 m. The proposed method can provide a near-real-time orthographic video map for smart cities, natural resource monitoring, emergency rescue, and other fields.

To improve the navigation safety of inland river ships and enrich the methods of environmental perception, this paper studies the recognition and depth estimation of inland river ships based on binocular stereo vision (BSV). In the stage of ship recognition, considering the computational pressure brought by the huge network parameters of the classic YOLOv4 model, the MobileNetV1 network was proposed as the feature extraction module of the YOLOv4 model. The results indicate that the mAP value of the MobileNetV1-YOLOv4 model reaches 89.25%, the weight size of the backbone network was only 47.6 M, which greatly reduced the amount of computation while ensuring the recognition accuracy. In the stage of depth estimation, this paper proposes a feature point detection and matching algorithm based on the ORB algorithm at sub-pixel level, that is, firstly, the FSRCNN algorithm was used to perform super-resolution reconstruction of the original image, to further increase the density of image feature points and detection accuracy, which was more conducive to the calculation of the image parallax value. The ships’ depth estimation results indicate that when the distance to the target is about 300 m, the depth estimation error is less than 3%, which meets the depth estimation needs of inland ships. The ship target recognition and depth estimation technology based on BSV proposed in this paper makes up for the shortcomings of the existing environmental perception methods, improves the navigation safety of ships to a certain extent, and greatly promotes the development of intelligent ships in the future.

When an earthquake strikes an area, unless the local infrastructure has a structural health monitoring system, it is difficult to obtain its health condition due to a lack of data. A smartphone could be competently used for many monitoring tasks to gather large amounts of data at a low cost. In this study, a smartphone is used as a sensor to obtain the interstory drift of buildings by recording a video of the ceiling with the front camera using a feature point matching algorithm. In addition, static and dynamic tests were carried out. In the static tests, the precision and error were investigated and compared with the reference sensor. In the dynamic tests, the displacement under a seismic wave was acquired by a smartphone and compared with the reference sensor, and the illumination condition was varied to test the robustness of the algorithm. The results indicated that feature point matching was suitable for calculating the displacement, and the smartphone was competent in monitoring the interstory drift during an earthquake.

In order to reduce the proportion of outliers in feature point matching, we propose a feature point matching optimization algorithm based on a wasserstein distance and apply the algorithm to the grid cell model. The wasserstein distance is introduced to calculate the similarity of the distribution of feature points in the grids around each pair of matching points, and the outer points are eliminated by the similarity. We compare the algorithm with the robust algorithm in OpenCV, and the experimental results show that the precision of our algorithm is higher than robust, and the recall is smaller. We open source our implementations at https://github.com/liuzhenboo/Optimization-of-Feature-Point-Matching-Algorithm-Based-on-Wasserstein-Distance.

Optical Camera Communication (OCC) is an emerging technology that has attracted research interest in recent decades. Unlike previous communication technologies, OCC uses visible light as the medium to transmit data from receivers and cameras to receive the data. OCC has several advantages that can be capitalized in several implementations. However, the Internet of Things (IoT) has emerged as a technology with immense potential. Numerous research endeavors support the IoT’s prospective technology that can be implemented in various sectors, including the healthcare system. This study introduces a novel implementation of the Internet of Medical Things (IoMT) system, using OCC for real-time health monitoring and indoor location tracking. The innovative system uses standard closed-circuit television CCTV setups, integrating deep learning-based OCC to monitor multiple patients simultaneously, each represented by an LED matrix. The effectiveness of the system was demonstrated through two scenarios: the first involves dual transmitters and a single camera, highlighting real-time monitoring of vital health data; the second features a transmitter with dual cameras, focusing patient movement tracking across different camera fields of view. To accurately locate and track the position of LED arrays in the camera, the system used YOLO (You Only Look Once). Data are securely transmitted to an edge server and stored using the REST API, with a web interface providing real-time patient updates. This study highlights the potential of OCC in IoMT for advanced patient care and proposes future exploration in larger healthcare systems and other IoT domains.

Segmentation, recognition and 3D reconstruction of objects have been cutting-edge research topics, which have many applications ranging from environmental and medical to geographical applications as well as intelligent transportation. In this dissertation, I focus on the study of segmentation, recognition and 3D reconstruction of objects using LiDAR data/MRI. Three main works are that (I). Feature extraction algorithm based on sparse LiDAR data. A novel method has been proposed for feature extraction from sparse LiDAR data. The algorithm and the related principles have been described. Also, I have tested and discussed the choices and roles of parameters. By using correlation of neighboring points directly, statistic distribution of normal vectors at each point has been effectively used to determine the category of the selected point. (II). Segmentation and 3D reconstruction of objects based on LiDAR/MRI. The proposed method includes that the 3D LiDAR data are layered, that different categories are segmented, and that 3D canopy surfaces of individual tree crowns and clusters of trees are reconstructed from LiDAR point data based on a region active contour model. The proposed method allows for delineations of 3D forest canopy naturally from the contours of raw LiDAR point clouds. The proposed model is suitable not only for a series of ideal cone shapes, but also for other kinds of 3D shapes as well as other kinds dataset such as MRI. (III). Novel algorithms for recognition of objects based on LiDAR/MRI. Aimed to the sparse LiDAR data, the feature extraction algorithm has been proposed and applied to classify the building and trees. More importantly, the novel algorithms based on level set methods have been provided and employed to recognize not only the buildings and trees, the different trees (e.g. Oak trees and Douglas firs), but also the subthalamus nuclei (STNs). By using the novel algorithms based on level set method, a 3D model of the subthalamus nuclei (STNs) in the brain has been successfully reconstructed based on the statistical data of previous investigations of an anatomy atlas as reference. The 3D rendering of the subthalamic nuclei and the skull directly from MR imaging is also utilized to determine the 3D coordinates of the STNs in the brain. In summary, the novel methods and algorithms of segmentation, recognition and 3D reconstruction of objects have been proposed. The related experiments have been done to test and confirm the validation of the proposed methods. The experimental results also demonstrate the accuracy, efficiency and effectiveness of the proposed methods. A framework for segmentation, recognition and 3D reconstruction of objects has been established, which has been applied to many research areas.

In this paper, an object recognition method and a pose estimation approach using stereo vision is presented. The proposed approach was used for position based visual servoing of a 6 DoF manipulator. The object detection and recognition method was designed with the purpose of increasing robustness. A RGB color-based object descriptor and an online correction method is proposed for object detection and recognition. Pose was estimated by using the depth information derived from stereo vision camera and an SVD based method. Transformation between the desired pose and object pose was calculated and later used for position based visual servoing. Experiments were carried out to verify the proposed approach for object recognition. The stereo camera was also tested to see whether the depth accuracy is adequate. The proposed object recognition method is invariant to scale, orientation and lighting condition which increases the level of robustness. The accuracy of stereo vision camera can reach 1 mm. The accuracy is adequate for tasks such as grasping and manipulation.

In this paper, we propose a new algorithm combined with the existing object recognition algorithm. Multi-object recognition algorithms are now various, with their respective advantages and disadvantages according to their uses. However, these algorithms can only detect and recognize objects without being able to know the location of the object relative to the sensor. The ability to know the location of the object is needed so that the autonomous car can make the right decisions without harming the driver. Since it requires fast and precise object detection and recognition capabilities, the algorithm used in object recognition is YOLOv4 with CSPDarknet-53. And because object recognition uses a neural network, the algorithm in determining the location of the object needs to be made as efficient as possible without affecting the performance of the object recognition algorithm, so that the mapping algorithm is used. The YOLOv4 model used has a precision value of 57.23 percent with a detection capability of 0.03785 seconds without a mapping algorithm, and if it is added with a mapping algorithm, the detection time becomes 0.03792 seconds. Since it has fast detection time, thus it can be applied to a real-time application.

In this paper, we propose a hybrid approach for accurate depth acquisition by using a structured light-based method with a stereo matching. By projecting additional light patterns onto a scene, a structured light-based method works well on a textureless region where a stereo matching shows poor performance. In contrast, the patterns projected onto a rich textured region obstruct in estimating reliable depth information in the structured light-based method, while a stereo matching excels. We exploit these complementary characteristics by combining the results from both methods that outperform the one from either alone. In our fusion framework, a hybrid stereo matching is introduced, in which the disparity search range for each pixel is limited based on the initial depth map obtained by the structured light-based method. In addition, we introduce a confidence-based fusion method which combines the depth maps, while incorporating the advantages of each method. The experimental results show that the proposed method achieves to estimate accurate depth information, while other methods fail.

Abstract. This paper presents a novel multi-stereo camera system for robust indoor localization, leveraging point cloud data and temporal fusion techniques. The system integrates three synchronized stereo cameras to capture point clouds from multiple angles, enhancing coverage and improving point cloud density in complex indoor environments. By combining data from different perspectives and accumulating point clouds over time, the method mitigates the limitations in the short range of point clouds derived from stereo cameras, ensuring broader coverage for effective localization. To manage the computational complexity of large-scale point clouds and reduce noise in accumulated data, voxelization is applied to downsample the point clouds while preserving key geometric features. The localization process is driven by a predictive point cloud odometry method, refined through the Iterative Closest Point (ICP) algorithm. Experimental results demonstrate the system’s ability to achieve accurate localization within a pre-built LiDAR map. This study highlights the feasibility of using low-cost stereo camera systems as an alternative to LiDAR-based solutions for indoor localization.

For the implementation of stereo image-based visual servoing algorithm in the eye-in-hand robotics applications, one of the main concerns is the accurate point feature detection and matching algorithm. Since the visual servoing is carried out in the textureless environment, the feature detection process is even more challenging. To fulfill the requirement of a robust and reliable point feature detection process, in this paper we present the novel deep learning-based algorithm. The approach based on convolutional neural networks and algorithm for detection of manufacturing entities is proposed and detected regions of interest are utilized for the improvement of the point feature detection algorithm. The proposed algorithm is experimentally evaluated in real-world settings by using wheeled nonholonomic mobile robot RAICO equipped with stereo vision system. The experimental results show the improvement of 58% in the accuracy of matched point features in the images obtained during the visual servoing process. Moreover, with the implementation of the proposed deep learning-based approach, the number of successful experimental runs has increased by 80%.

Results of using an RGB-D camera (Kinect sensor) and a stereo camera, separately, in order to determine the 3D real position of characteristic points of a predetermined object in a scene are presented. KAZE algorithm was used to make the recognition, that algorithm exploits the nonlinear scale space through nonlinear diffusion filtering; 3D coordinates of the centroid of a predetermined object were calculated employing the camera calibration information and the depth parameter provided by a Kinect sensor and a stereo camera. Experimental results show it is possible to get the required coordinates with both cameras in order to locate a robot, although a balance in the distance where the sensor is placed must be guaranteed: no fewer than 0.8 m from the object to guarantee the real depth information, it is due to Kinect operating range; 0.5 m to stereo camera, but it must not be 1 m away to have a suitable rate of object recognition, besides, Kinect sensor has more precision with distance measures regarding a stereo camera.

Binocular stereo vision is an important branch of the research area in computer vision. Stereo matching is the most important process in binocular vision. In this paper, a new stereo matching scheme using shape-based matching (SBM) is presented to improve the depth reconstruction method of binocular stereo vision systems. The method works in two steps. First, an operator registers the pattern including the key features of an object to be measured. Then during the operation stage, the stereo camera snaps stereo images and finds the patterns in right and left images separately by means of the SBM. The 3D positions of the object are calculated by using the corresponding points of the stereo images and the projection matrices of the stereo camera. Since we apply robust image processing algorithms, such as the SBM, the proposed method becomes more reliable than the conventional stereo vision systems.

This paper proposes making a stereo camera using two fisheye cameras. The measurement range of the stereo camera is increased by using fisheye cameras. Fisheye images are deformed by using an equirectangular projection to simplify the stereo correspondence search in the images. Perspective transformation is applied to images so that feature points of far objects coincide in the images of the two cameras in order to remove mismatches due to the errors of intrinsic and extrinsic parameters. The corresponding points are calculated by using stereo matching in the deformed image, and the distance is measured from the disparity. The accuracy of the constructed stereo camera's distance measuring is evaluated by experiments. The performance of the constructed stereo camera is evaluated by indoor and outdoor experiments.

This paper proposes a stereo camera using two fisheye cameras. The measurement range of the stereo camera is increased by using fisheye cameras. Fisheye images are deformed by using an equirectangular projection to simplify the stereo correspondence search in the images. Stereo images are rectified to correct the influence of external parameters between a left camera and a right camera. Rectification parameters are estimated from feature points obtained from an image of an arbitrary environment. Errors due to deviation of the position and rotation of the left and right cameras are removed by the stereo rectification. The corresponding points are calculated by using stereo matching in the deformed image, and the distance is measured from the disparity. The performance of the fisheye stereo camera using equirectangular images is evaluated by simulations and experiments.