Multi-scale Information Extraction Residual Network for 3D Point Clouds Classification

Nowadays, point clouds are frequently gathered by 3D scanners such as Lidar and Kinect, which produces thousands of point cloud models. Point cloud processing is vital to 3D vision, especially 3D object recognition, positioning, and navigation technology. Addressing uneven data density caused by coordinate frame transformations and the inherent problem of insufficient context connection in point clouds, the DiM-PCNet (Multi-scale and Multi-level Point Clouds Classification Network, and the Di is a prefix to represent double M in Multi-scale and Multi-level) is proposed in this paper. DiM-PCNet is provided for object classification with multi-scale and multi-level features. We encode the point cloud in multi-scale and fully fuse the features with the raw point cloud for keeping the context relationship. In DiM-PCNet, we sample the point clouds from eight parts for multi-scales feature extraction. The multi-scales features are fully fused by multi-level pyramid models. The multi-scale and multi-level strategies are applied in DiM-PCNet, in which the abundant and important features of point clouds are extracted and utilized in the 3D object classification. It is worth noting that the DiM-PCNet feature block can be embedded into the segmentation net, where the accuracy achieved is 87.1%. We conducted experiments on ShapeNet and ModelNet40 and the experimental results show that DiM-PCNet achieves state-of-the-art performance in 3D object classification. The experiment shows competitive performance on robustness and segmentation tasks.

Due to the favorable denoising performance of the discriminative learning model, a large number of discriminative learning models have been designed to remove noise. However, most discriminative learning models can only deal with the noise that already exists in the training data. In this paper, a multi-scale trainable deep residual convolutional neural network (DCMSNet) based on dilated convolution is proposed. DCMSNet consists of a chain of dilated convolution layers, convolution layers, normalized multi-scale convolution blocks (BNMCBlock), multi-scale convolution blocks (MCBlock) and multi-scale dropout convolution blocks (MCDBlock). The use of dilated convolution can avoid over-parameterization problems caused by too deep networks. In order to capture more feature information during feature extraction and image reconstruction, we designed novel BNMCBlock and MCBlock. In order to reduce the degree of coupling between image features and improve the generalization ability of the network, we also designed a MCDBlock. Meanwhile, residual learning, batch normalization and dropout are utilized to speed up the training process and boost the denoising performance. Unlike existing denoising models, DCMSNet is able to remove different degrees of noise. Compared with state-of-the-art image denoising methods, DCMSNet has achieved relatively competitive results.

In recent years, due to the significant advancements in hardware sensors and software technologies, 3D environmental point cloud modeling has gradually been applied in the automation industry, autonomous vehicles, and construction engineering. With the high-precision measurements of 3D LiDAR, its point clouds can clearly reflect the geometric structure and features of the environment, thus enabling the creation of high-density 3D environmental point cloud models. However, due to the enormous quantity of high-density 3D point clouds, storing and processing these 3D data requires a considerable amount of memory and computing time. In light of this, this paper proposes a real-time 3D point cloud environmental contour modeling technique. The study uses the point cloud distribution from the 3D LiDAR body frame point cloud to establish structured edge features, thereby creating a 3D environmental contour point cloud map. Additionally, unstable objects such as vehicles will appear during the mapping process; these specific objects will be regarded as not part of the stable environmental model in this study. To address this issue, the study will further remove these objects from the 3D point cloud through image recognition and LiDAR heterogeneous matching, resulting in a higher quality 3D environmental contour point cloud map. This 3D environmental contour point cloud not only retains the recognizability of the environmental structure but also solves the problems of massive data storage and processing. Moreover, the method proposed in this study can achieve real-time realization without requiring the 3D point cloud to be organized in a structured order, making it applicable to unorganized 3D point cloud LiDAR sensors. Finally, the feasibility of the proposed method in practical applications is also verified through actual experimental data.

The vibration signal is easily interfered by noise due to the influence of environment and other factors, which can lead to the poor adaptability, low accuracy of remaining useful life (RUL) prediction, and other problems. To solve this problem, this paper proposes a novel RUL prediction method, which is based on multiscale stacking deep residual shrinkage network (MSDRSN). MSDRSN combines the ability of stacking in improving prediction accuracy and the advantages of deep residual shrinkage network (DRSN) in denoising. First, cumulative sum (CUSUM) from statistics is used to divide the full life cycle of the rolling bearings and discover the points of failure. Second, stacking is used for feature learning on the raw data, multiple convolutional kernels of different scales are selected as base‐learners, and fully connected neural networks are selected as meta‐learners for feature fusion and learning. Then, DRSN is used to do prediction, and the acquired results are fitted with Savitzky–Golay (SG) smoothing. Finally, the effectiveness of the proposed method is proved by the IEEE PHM 2012 data challenge dataset. Compared with the multiscale convolutional neural network with fully connected layer (MSCNN‐FC) and the bidirectional long short‐term memory (BiLSTM) for RUL prediction under the noise. Using the proposed method, the mean absolute error (MSE) of the best result is 0.002 and the mean square error (MSE) is 0.014; meanwhile, the coefficient of determination (R2) of the best prediction result can reach 97.6%. It is also compared with other machine learning methods, and all the results prove the accuracy and effectiveness of the proposed method for RUL prediction applications.

Data-driven rolling-bearing fault diagnosis methods are mostly based on deep-learning models, and their multilayer nonlinear mapping capability can improve the accuracy of intelligent fault diagnosis. However, problems such as gradient disappearance occur as the number of network layers increases. Moreover, directly taking the raw vibration signals of rolling bearings as the network input results in incomplete feature extraction. In order to efficiently represent the state characteristics of vibration signals in image form and improve the feature learning capability of the network, this paper proposes fault diagnosis model MTF-ResNet based on a Markov transition field and deep residual network. First, the data of raw vibration signals are augmented by using a sliding window. Then, vibration signal samples are converted into two-dimensional images by MTF, which retains the time dependence and frequency structure of time-series signals, and a deep residual neural network is established to perform feature extraction, and identify the severity and location of the bearing faults through image classification. Lastly, experiments were conducted on a bearing dataset to verify the effectiveness and superiority of the MTF-ResNet model. Features learned by the model are visualized by t-SNE, and experimental results indicate that MTF-ResNet showed better average accuracy compared with several widely used diagnostic methods.

High quality medical images are the foundation of clinical diagnosis and treatment, but their quality may decrease due to imaging noise, artifacts, and uneven lighting. To address this issue, this paper proposes a novel Multi-scale Deep Residual Shrinkage Generative Adversarial Network (MDRSGAN) for non paired medical image enhancement. Its core innovations include: (1) Adopting a customized generator with learnable channel shared soft threshold (DRSN-CS), which can achieve hierarchical feature extraction and adaptive noise suppression; (2) Combining dual core discriminator to ensure global statistical consistency and high fidelity of local structure; (3) Introducing content perception loss and lighting loss to optimize overall details and image features. The performance of MDRSGAN is validated on fundus retina, endoscope, and self built NIR-II mouse dataset, and its performance is superior to five mainstream methods, such as a 5% increase in SNR for NIR-II mouse image enhancement. Downstream retinal vessel segmentation experiments show that IoU and DSC achieved 13% and 8% improvement, respectively, demonstrating the clinical applicability and performance advantages of this method.

Converting three-dimensional (3D) point cloud data to two-dimensional (2D) image based on virtual structured-light system is a popular method in 3D data compression because the image format is easier to process by the exiting storing and transmitting methods. When this method is used in the 3D point cloud compression, the quantization error is introduced during the coordinate mapping. To solve this problem, a virtual structured-light 3D point cloud compression algorithm based on geometric reshaping is presented. And a least-square system parameter optimization method is proposed to further improve the data decoding accuracy. In the proposed method, the 3D spatial coordinates are reshaped to a 2D matrix first, and then the 2D matrix is stored as a “Holoimage” by using the parameter optimized virtual structured-light system. The quantization error introduced by coordinate mapping between X and Y coordinates of the 3D point cloud and the 2D image pixel coordinates is suppressed, so the decoding accuracy is improved. In addition, the geometric information of the 3D point cloud is hidden synchronously when the 3D point cloud is compressed, which is of great significance in the copyright protection of the point cloud data. Experiments verify the effectiveness of the proposed algorithm. The decoding root mean square error (RMSE) of the proposed method is decreased by 79.86% on average compared with the traditional one under the same compression ratio, and the compression ratio of the proposed method is 1.96 times bigger than the traditional one under the similar decoding accuracy.

Deep learning on point clouds has attracted increasing attention in the fields of 3D computer vision and robotics. In particular, graph-based point-cloud deep neural networks (DNNs) have demonstrated promising performance in 3D object classification and scene segmentation tasks. However, the scattered and irregular graph-structured data in a graph-based point-cloud DNN cannot be computed efficiently by existing SIMD architectures and accelerators. We present Point-X, an energy-efficient accelerator architecture that extracts and exploits the spatial locality in point cloud data for efficient processing. Point-X uses a clustering method to extract fine-grained and coarse-grained spatial locality from the input point cloud. The clustering maps the point cloud into distributed compute tiles to maximize intra-tile computational parallelism and minimize inter-tile data movement. Point-X employs a chain network-on-chip (NoC) to further reduce the NoC traffic and achieve up to 3.2 × speedup over a traditional mesh NoC. Point-X’s multi-mode dataflow can support all common operations in a graph-based point-cloud DNN, i.e., edge convolution, shared multi-layer perceptron, and fully-connected layers. Point-X is synthesized in a 28nm technology and it demonstrates a throughput of 1307.1 inference/s and an energy efficiency of 604.5 inference/J on the DGCNN workload. Compared to the Nvidia GTX-1080Ti GPU, Point-X shows 4.5 × and 342.9 × improvement in throughput and efficiency, respectively.

The image captured by the camera in the rainy day contains a large amount of rain mark information. The existence of such a large amount of rain mark information will cause the background information of the image to be highly blurred, which will affect the visual effect of people's observation or the processing effect of subsequent algorithms. The deep residual network (ResNet) solves the problem that the accuracy of the network reaches saturation and then degenerates sharply when the number of network layers increases. This method is now widely used in various fields of computer vision. The use of deep residual networks to remove rain marks in images is a hot topic of research. Existing deep learning methods usually require a large number of positive and negative samples to train the network.How to train the rain well with fewer samples is a problem that needs to be solved in the field.By improving the rensnet network and densenet network, we propose a rain removal method based on densely connected deep residual network images. First, use a guided filter to extract the detailed information of the image to generate a detailed image, and then train the detailed image through the constructed multi-scale deep residual network to extract Raindrop information.The extracted rainfall marker information is formed into a negative residual map. We can add the synthesized image with rain to the negative residual map to achieve the effect of rain removal.

Multi-scale learnable key-channel attention network for point cloud classification and segmentation

Fourier ptychographic microscopy (FPM) is a newly developed microscopic technique for large field of view, high-resolution and quantitative phase imaging by combining the techniques from ptychographic imaging, aperture synthesizing and phase retrieval. In FPM, an LED array is utilized to illuminate the specimen from different angles and the corresponding intensity images are synthesized to reconstruct a high-resolution complex field. As a flexible and low-cost approach to achieve high-resolution, wide-field and quantitative phase imaging, FPM is of enormous potential in biomedical applications such as hematology and pathology. Conventionally, the FPM reconstruction problem is solved by using a phase retrieval method, termed Alternate Projection. By iteratively updating the Fourier spectrum with low-resolution-intensity images, the result converges to a high-resolution complex field. Here we propose a new FPM reconstruction framework with deep learning methods and design a multiscale, deep residual neural network for FPM reconstruction. We employ the widely used open-source deep learning library PyTorch to train and test our model and carefully choose the hyperparameters of our model. To train and analyze our model, we build a large-scale simulation dataset with an FPM imaging model and an actual dataset captured with an FPM system. The simulation dataset and actual dataset are separated as training datasets and test datasets, respectively. Our model is trained with the simulation training dataset and fine tuned with the fine-tune dataset, which contains actual training data. Both our model and the conventional method are tested on the simulation test dataset and the actual test dataset to evaluate the performances. We also show the reconstruction result of another neural network-based method for comparison. The experiments demonstrate that our model achieves better reconstruction results and consumes much less time than conventional methods. The results also point out that our model is more robust under system aberrations such as noise and blurring (fewer intensity images) compared with conventional methods.

ABSTRACTVisual–Linguistic large models (V‐LLMs) have demonstrated strong semantic understanding capabilities in understanding visual open‐world scenes. However, there is limited research on how to effectively integrate these models with traditional deep learning networks for classify and segment 3D point clouds. This paper explores the joint training mechanisms of V‐VLLMs and Convolutional Neural Networks (CNNs), constructing a pluggable module that enhances 3D object classification and component segmentation using 2D images, 3D point clouds, and language descriptions. The idea of this pluggable module is to employ contrastive learning between the generalized features extracted by large models from text and image features extracted by the CNN to enhance the robustness and understanding accuracy of the original CNN‐based point cloud network. For experiments, we validated the effectiveness and accuracy of the proposed pluggable module using multiple point cloud classification frameworks, based on datasets such as ModelNet40, ScanObjectNN, and ShapeNetCore. The experimental results demonstrate that the multimodal contrastive learning pluggable module significantly enhances performance across various deep learning frameworks, resulting in significant enhancements in classification and segmentation precision. Additionally, for the multiscale feature PointNet++ framework, the proposed multiscale contrastive learning mechanism provides further performance gains. The Code is available at https://github.com/winbigqian/MCL‐3D‐VCNN.

Unlike 2-dimensional (2D) images, direct 3-dimensional (3D) point cloud processing using deep neural network architectures is challenging, mainly due to the lack of explicit neighbor relationships. Many researchers attempt to remedy this by performing an additional voxelization preprocessing step. However, this adds additional computational overhead and introduces quantization error issues, limiting an accurate estimate of the underlying structure of objects that appear in the scene. To this end, in this article, we propose a deep network that can directly consume raw unstructured point clouds to perform object classification and part segmentation. In particular, a Deep Feature Transformation Network (DFT-Net) has been proposed, consisting of a cascading combination of edge convolutions and a feature transformation layer that captures the local geometric features by preserving neighborhood relationships among the points. The proposed network builds a graph in which the edges are dynamically and independently calculated on each layer. To achieve object classification and part segmentation, we ensure point order invariance while conducting network training simultaneously—the evaluation of the proposed network has been carried out on two standard benchmark datasets for object classification and part segmentation. The results were comparable to or better than existing state-of-the-art methodologies. The overall score obtained using the proposed DFT-Net is significantly improved compared to the state-of-the-art methods with the ModelNet40 dataset for object categorization.

Skin cancer is the irregular growth of skin cells, which is most often termed as cancer, developed by exposure of ultraviolet rays from sun. In this research paper, deep learning enabled hybrid optimization is followed for skin cancer detection and lesion segmentation. Two optimization algorithms are followed for skin lesion segmentation and cancer detection. Here, pre-processing is done by anisotropic diffusion followed by skin lesion segmentation. Here, Multi-Scale Residual Fusion Network (MSRFNet) is utilized for skin lesion segmentation, which is trained by proposed Average Subtraction Student Psychology Based Optimization (ASSPBO). After skin lesion segmentation, necessary features are extracted, followed by skin cancer detection. Skin cancer is detected by Deep Residual Network (DRN) trained by proposed Fractional ASSPBO (FrASSPBO). Moreover, performance of proposed FrASSPBO-DRN is analysed by three performance metrics like testing accuracy, True Positive Rate (TPR), and False Positive Rate (FPR) with values of 93.4%, 94%, and 8.2%.

Aiming at the problem that the difference in fault data distribution caused by the variation of gear and bearing conditions during the operation of rotating machinery affects the accuracy of fault diagnosis, a multi-condition fault diagnosis approach based on whale optimization algorithm (WOA), variational mode decomposition (VMD), and deep residual shrinkage network (DRSN) is proposed. Firstly, WOA is employed to optimize the parameters of the decomposition level and the penalty factor value in VMD, enhancing its noise resistance. Subsequently, VMD processing yields several modal components, each characterized by a permutation entropy feature vector, ensuring optimal feature extraction. DRSN is then utilized, employing an attention mechanism to adaptively set soft threshold functions, thereby improving gear fault diagnosis accuracy under multiple operating conditions. Finally, the proposed method’s performance was validated through the collection of gear running signals under various conditions, and the fault diagnosis accuracy was used as the model evaluation index to conduct comparative experiments and ablation experiments. The experimental results indicate that the WOA-VMD feature extraction and DRSN fault diagnosis techniques outperform traditional methods in multi-condition gear fault diagnosis. This study provides a new effective approach for multi-condition diagnosis of rotating machinery.