Decoupled Active Surface for Volumetric Image Segmentation

Matching 3D objects by their similarity is a fundamental problem in computer vision, multimedia databases, molecular biology, computer graphics and a variety of other fields. A challenging aspect of this problem is to find a suitable shape signature/descriptor that can be constructed and compared quickly, while still discriminating between similar and dissimilar shapes. We find that the major problems in comparing 3D mesh objects lie in the non-uniform vertex sampling and level of detail distribution, in the non-uniform polygon topology and in mesh-representation anomalies, so the primary motivation behind the work presented in this paper is the introduction of mesh-parameterization which brings meshes into a form having uniform vertex sampling, uniform polygon topology and filtered anomalies, by spherically mapping the mesh surface. Further, we present two approaches in inferring shape-descriptors from the spherically mapped objects and the results from the conducted experiments.

Pose estimation of a three-dimensional (3D) rigid object based on monocular vision is a fundamental problem in computer vision. Existing methods usually presume that there is a certain feature projection correspondence available a priori between the input image and object's 3D model, which substantially reduces the mathematical complexity of the problem. However, in actual applications this presumption rarely holds. How to estimate the pose of a general 3D object with no feature projection correspondence available a priori is still not well resolved. In this study, the authors propose a new 3D rigid object pose estimation method based on object's contour and non-Euclidean multi-feature distance map, which solves both the pose estimation problem and the feature projection correspondence problem simultaneously and iteratively. Experiment results show that our method also has good performance in convergence speed, convergence radius and noise robustness.

Accurate detection of 3D objects is a fundamental problem in computer vision and has an enormous impact on autonomous cars, augmented/virtual reality and many applications in robotics. In this work we present a novel fusion of neural network based state-of-the-art 3D detector and visual semantic segmentation in the context of autonomous driving. Additionally, we introduce Scale-Rotation-Translation score (SRTs), a fast and highly parameterizable evaluation metric for comparison of object detections, which speeds up our inference time up to 20% and halves training time. On top, we apply state-of-the-art online multi target feature tracking on the object measurements to further increase accuracy and robustness utilizing temporal information. Our experiments on KITTI show that we achieve same results as state-of-the-art in all related categories, while maintaining the performance and accuracy trade-off and still run in real-time. Furthermore, our model is the first one that fuses visual semantic with 3D object detection.

Proceedings of the 2011 joint ACM workshop on Human gesture and behavior understanding

In the areas of computer vision and computer graphics, analyzing information about three-dimensional objects from two-dimensional images is essential. Shape from shading is a very fundamental problem in computer vision, which uses features and patterns in images to infer the shape of the object in our viewing direction. The influence of solving this problem is significant. For example, a typical application of shape from shading is in astronomy, in which we can obtain landscape features of a planet by analyzing the information from the image we get.

Multi-view stereopsis (MVS) is a core problem in computer vision, which takes a set of scene views together with known camera poses, then produces a geometric representation of the underlying 3D model Using 3D reconstruction one can determine any object's 3D profile, as well as knowing the 3D coordinate of any point on the profile. The 3D reconstruction of objects is a generally scientific problem and core technology of a wide variety of fields, such as Computer Aided Geometric Design (CAGD), computer graphics, computer animation, computer vision, medical imaging, computational science, virtual reality, digital media, etc. However, though MVS problems have been studied for decades, many challenges still exist in current state-of-the-art algorithms, for example, many algorithms still lack accuracy and completeness when tested on city-scale large datasets, most MVS algorithms available require a large amount of execution time and/or specialized hardware and software, which results in high cost, and etc... This dissertation work tries to address all the challenges we mentioned, and proposed multiple solutions. More specifically, this dissertation work proposed multiple novel MVS algorithms to automatically and accurately reconstruct the underlying 3D scenes. By proposing a novel volumetric voxel-based method, one of our algorithms achieved near real-time runtime speed, which does not require any special hardware or software, and can be deployed onto power-constrained embedded systems. By developing a new camera clustering module and a novel weighted voting-based surface likelihood estimation module, our algorithm is generalized to process di erent datasets, and achieved the best performance in terms of accuracy and completeness when compared with existing algorithms. This dissertation work also performs the very first quantitative evaluation in terms of precision, recall, and F-score using real-world LiDAR groundtruth data. Last but not least, this dissertation work proposes an automatic workflow, which can stitch multiple point cloud models with limited overlapping areas into one larger 3D model for better geographical coverage. All the results presented in this dissertation work have been evaluated in our wide area motion imagery (WAMI) dataset, and improved the state-of-the-art performances by a large margin.The generated results from this dissertation work have been successfully used in many aspects, including: city digitization, improving detection and tracking performances, real time dynamic shadow detection, 3D change detection, visibility map generating, VR environment, and visualization combined with other information, such as building footprint and roads.

High-quality 3D objects play a crucial role in digital twins, while synthetic data generated from these objects have become essential in deep learning-based computer vision applications. The task of collecting and labeling real defects on industrial object surfaces has many challenges and efforts, while synthetic data generation feasibly replicates huge amounts of labeled data. However, synthetic datasets lack realism in their rendered images. To overcome this issue, this paper introduces a single framework for 3D industrial object reconstruction and synthetic defect generation for digital twin smart factory applications. In detail, NeRF is applied to reconstruct our custom industrial 3D objects through videos collected by a smartphone camera. Several NeRF-based models (i.e., Instant-NGP, Nerfacto, Volinga, and Tensorf) are compared to choose the best outcome for the next step of defect generation on the 3D object surface. To be fairly evaluated, we train four models using the Nerfstudio framework with our three custom datasets of two objects. From the experiment’s results, Instant-NGP and Nerfacto achieve the best outcomes, outperforming all other methods significantly. The exported meshes of 3D objects are refined using Blender before loading into NVIDIA Omniverse Code to generate defects on the surface with the Replicator. To evaluate the object detection performance and to verify the benefits of synthetic defect data, we conducted experiments with YOLO-based models on our synthetic and real-plus-synthetic defects. From the experiment’s results, the synthetic defect data contribute to improving YOLO models’ generalization capability with the highest and lowest accuracy mAP@0.5 enhancement of 18.8 and 1.5 percent on YOLOv6n and YOLOv8s, respectively.

Three-dimensional reconstruction technologies are fundamental problems in computer vision. Photometric stereo recovers the surface normals of a 3D object from varying shading cues, prevailing in its capability for generating fine surface normal. In recent years, deep learning-based photometric stereo methods are capable of improving the surface-normal estimation under general non-Lambertian surfaces, due to its powerful fitting ability on the non-Lambertian surface. These state-of-the-art methods however usually regress the surface normal directly from the high-dimensional features, without exploring the embedded structural information. This results in the underutilization of the information available in the features. Therefore, in this paper, we propose an efficient manifold-based framework for learning-based photometric stereo, which can better map combined high-dimensional feature spaces to low-dimensional manifolds. Extensive experiments show that our method, learning with the low-dimensional manifolds, achieves more accurate surface-normal estimation, outperforming other state-of-the-art methods on the challenging DiLiGenT benchmark dataset.

Recently, shape-based matching and retrieval of 3D polygonal models has become one of the most fundamental problems in computer vision. Dealing with families of objects instead of a single one may impose further challenges on regular geometric algorithms. In this paper we focus on the classification of 3D objects based on their geodesic distance & path calculated on a mesh using an iterative algorithm for solving the Eikonal equation. For the classification process, we use both Multiclass Support Vector Machine (M-SVM) classifier,K-Nearest Neighbors (KNN), Decision Tree (DT) and Artificial Neural Networks (ANN) to better evaluate our descriptors. We illustrate the potential of extracted characteristics by two 3D benchmarks. The recognition rates achieved in all experiments show that a small number of curve between 9 and 12 can correctly categorize a family of 3D objects.

6D object pose estimation from a single RGB-D image is a fundamental problem in computer vision and robot manipulation. Despite recent advancements, existing methods still suffer several limitations. First of all, the object shape representation extracted from the depth map is often less expressive because the object point cloud parsed from the depth map is highly incomplete due to the object self-occlusion and noisy due to the sensor artifacts. This shape representation issue further intensifies when lacking sufficient labeled data for model training, which unfortunately is another typical problem for object pose estimation considering the heavy annotation cost for real-world pose labeling. In this study, we propose to tackle the above issues in a unified way. First, we enhance the object shape representation from the partial point cloud with a novel canonical shape reconstruction module, in which an implicit canonical frame is established by incorporating the SE(3) equivariance, achieving implicit feature alignment of the partial point cloud inputs, leading to robust shape recovery. Second, based on the enhanced object representation, we further utilize the de-canonicalized and pose-dependent completed object shape as the training signal, and develop a novel weakly-supervised learning framework to leverage both labeled synthetic data and unlabeled real data to train the pose estimation model in a label-efficient way. Extensive experiments on three widely used benchmarks demonstrate the effectiveness, and superiority of our framework over state-of-the-art methods.

Image classification is a fundamental problem in computer vision, which deserves much attention in the past decade. To provide a comprehensive recall for the image classification task based on the deep learning algorithms, this paper first provides a brief history of computer vision and convolutional neural networks. Then, the current state of research and the direction of development of deep learning-based convolutional neural network models for image classification are examined. In addition, the basic model structure, convolution feature extraction, and pooling operations of standard and convolutional neural networks are also introduced. This paper summarizes the development of convolutional neural network models in recent years. In summary, based on the review of the development of image classification algorithms at home and abroad, the current mainstream image classification algorithms and frontier progress are summarized and analyzed, and the existing problems and future development directions of image classification are summarized and prospected. It can be concluded that deep learning-based methods has a great potential in this case.

This paper introduces a deep architecture for segmenting 3D objects into their labeled semantic parts. Our architecture combines image-based Fully Convolutional Networks (FCNs) and surface-based Conditional Random Fields (CRFs) to yield coherent segmentations of 3D shapes. The image-based FCNs are used for efficient view-based reasoning about 3D object parts. Through a special projection layer, FCN outputs are effectively aggregated across multiple views and scales, then are projected onto the 3D object surfaces. Finally, a surface-based CRF combines the projected outputs with geometric consistency cues to yield coherent segmentations. The whole architecture (multi-view FCNs and CRF) is trained end-to-end. Our approach significantly outperforms the existing state-of-the-art methods in the currently largest segmentation benchmark (ShapeNet). Finally, we demonstrate promising segmentation results on noisy 3D shapes acquired from consumer-grade depth cameras.

Deep neural network model is the core technology for sensors of the autonomous driving platform to perceive the external environment. Recent research have shown that it has a certain vulnerability. The artificial designed adversarial examples can make the DNN model output the wrong results. These adversarial examples not only exist in the digital world, but also in the physical world. At present, research on autonomous driving platform mainly focus on attacking a single sensor. In this paper, we introduce Camdar-adv, a method for generating image adversarial examples on three-dimensional (3D) objects, which could potentially lunch a multisensor attack toward the autonomous driving platforms. Specifically, with objects that can attack LiDAR sensors, a geometric transformation can be used to project their shape onto the two-dimensional plane. Adversarial perturbations against optical image sensor could be added to the surface of the adversarial 3D objects precisely without changing its geometry. Test results on the open-source autonomous driving data set KITTI show that Camdar-adv can generate adversarial samples for the state of the art object detection model. From a fixed viewpoint, our method can achieve an attack success rate over 99%.

Graph matching is an essential problem in computer vision that has been successfully applied to 2D and 3D feature matching and object recognition. Despite its importance, little has been published on learning the parameters that control graph matching, even though learning has been shown to be vital for improving the matching rate. In this paper we show how to perform parameter learning in an unsupervised fashion, that is when no correct correspondences between graphs are given during training. Our experiments reveal that unsupervised learning compares favorably to the supervised case, both in terms of efficiency and quality, while avoiding the tedious manual labeling of ground truth correspondences. We verify experimentally that our learning method can improve the performance of several state-of-the art graph matching algorithms. We also show that a similar method can be successfully applied to parameter learning for graphical models and demonstrate its effectiveness empirically.

Light field imaging is an emerging technology in computational photography areas. Based on innovative designs of the imaging model and the optical path, light field cameras not only record the spatial intensity of threedimensional (3D) objects, but also capture the angular information of the physical world, which provides new ways to address various problems in computer vision, such as 3D reconstruction, saliency detection, and object recognition. In this paper, three key aspects of light field cameras, i.e., model, calibration, and reconstruction, are reviewed extensively. Furthermore, light field based applications on informatics, physics, medicine, and biology are exhibited. Finally, open issues in light field imaging and long-term application prospects in other natural sciences are discussed.