Maize stem–leaf segmentation framework based on deformable point clouds

In this study, we propose a segmentation model based on convolutional neural networks (CNNs) to address image segmentation challenges in computer vision. Prior to designing the model, the activation function and other modules of the convolutional neural network were optimized to meet specific requirements. The segmentation task was transformed into binary classification problem to simplify network calculations and improve efficiency. Additionally, the model utilized a mask map obtained from the semantic segmentation model to aid in instance segmentation. Class activation technology was introduced to extract feature mapping maps. The corresponding thermal maps were obtained to achieve target instance segmentation. To further validate the effectiveness of the segmentation model, simulation experiments were conducted on semantic segmentation and instance segmentation respectively. The results show that the accuracy of the basic semantic segmentation model reached 87.58%, while the average accuracy of the entire class of the optimized instance segmentation model reached 97.9%. Therefore, the research and design of image segmentation models demonstrate high accuracy and good robustness.

In recent years, due to the significant advancements in hardware sensors and software technologies, 3D environmental point cloud modeling has gradually been applied in the automation industry, autonomous vehicles, and construction engineering. With the high-precision measurements of 3D LiDAR, its point clouds can clearly reflect the geometric structure and features of the environment, thus enabling the creation of high-density 3D environmental point cloud models. However, due to the enormous quantity of high-density 3D point clouds, storing and processing these 3D data requires a considerable amount of memory and computing time. In light of this, this paper proposes a real-time 3D point cloud environmental contour modeling technique. The study uses the point cloud distribution from the 3D LiDAR body frame point cloud to establish structured edge features, thereby creating a 3D environmental contour point cloud map. Additionally, unstable objects such as vehicles will appear during the mapping process; these specific objects will be regarded as not part of the stable environmental model in this study. To address this issue, the study will further remove these objects from the 3D point cloud through image recognition and LiDAR heterogeneous matching, resulting in a higher quality 3D environmental contour point cloud map. This 3D environmental contour point cloud not only retains the recognizability of the environmental structure but also solves the problems of massive data storage and processing. Moreover, the method proposed in this study can achieve real-time realization without requiring the 3D point cloud to be organized in a structured order, making it applicable to unorganized 3D point cloud LiDAR sensors. Finally, the feasibility of the proposed method in practical applications is also verified through actual experimental data.

Accurate organ-level segmentation is essential for achieving high-throughput, non-destructive, and automated plant phenotyping. To address the challenge of intelligent acquisition of phenotypic parameters in tomato plants, we propose TSINet, an end-to-end dual-task segmentation network designed for effective and precise semantic labeling and instance recognition of tomato point clouds, based on the Pheno4D dataset. TSINet adopts an encoder–decoder architecture, where a shared encoder incorporates four Geometry-Aware Adaptive Feature Extraction Blocks (GAFEBs) to effectively capture local structures and geometric relationships in raw point clouds. Two parallel decoder branches are employed to independently decode shared high-level features for the respective segmentation tasks. Additionally, a Dual Attention-Based Feature Enhancement Module (DAFEM) is introduced to further enrich feature representations. The experimental results demonstrate that TSINet achieves superior performance in both semantic and instance segmentation, particularly excelling in challenging categories such as stems and large-scale instances. Specifically, TSINet achieves 97.00% mean precision, 96.17% recall, 96.57% F1-score, and 93.43% IoU in semantic segmentation and 81.54% mPrec, 81.69% mRec, 81.60% mCov, and 86.40% mWCov in instance segmentation. Compared with state-of-the-art methods, TSINet achieves balanced improvements across all metrics, significantly reducing false positives and false negatives while enhancing spatial completeness and segmentation accuracy. Furthermore, we conducted ablation studies and generalization tests to systematically validate the effectiveness of each TSINet component and the overall robustness of the model. This study provides an effective technological approach for high-throughput automated phenotyping of tomato plants, contributing to the advancement of intelligent agricultural management.

Surface defect identification is a crucial task in many manufacturing systems, including automotive, aircraft, steel rolling, and precast concrete. Although image-based surface defect identification methods have been proposed, these methods usually have two limitations: images may lose partial information, such as depths of surface defects, and their precision is vulnerable to many factors, such as the inspection angle, light, color, noise, etc. Given that a three-dimensional (3D) point cloud can precisely represent the multidimensional structure of surface defects, we aim to detect and classify surface defects using a 3D point cloud. This has two major challenges: (i) the defects are often sparsely distributed over the surface, which makes their features prone to be hidden by the normal surface and (ii) different permutations and transformations of 3D point cloud may represent the same surface, so the proposed model needs to be permutation and transformation invariant. In this paper, a two-step surface defect identification approach is developed to investigate the defects’ patterns in 3D point cloud data. The proposed approach consists of an unsupervised method for defect detection and a multi-view deep learning model for defect classification, which can keep track of the features from both defective and non-defective regions. We prove that the proposed approach is invariant to different permutations and transformations. Two case studies are conducted for defect identification on the surfaces of synthetic aircraft fuselage and the real precast concrete specimen, respectively. The results show that our approach receives the best defect detection and classification accuracy compared with other benchmark methods.

Traditionally, 3D segmentation tasks have operated in silos, focusing separately on semantic and instance segmentation. However, this disjointed approach lacks interoperability and fails to fully unleash the potential of a more integrated, multitask solution. To overcome this limitation, we introduce TUS-Net, an innovative transformer-based architecture meticulously crafted for both semantic and instance segmentation of point clouds. Our model introduces two pivotal advancements: First, it employs a superpoint-based pre-processing step that minimizes computational overhead without compromising on precision. Second, we leverage a dual-branch design within the transformer architecture, allowing it to adapt to the nuances of both segmentation tasks dynamically. Through extensive experimentation on the ScanNet dataset, our findings demonstrate that TUS-Net surpasses prevailing specialized models by a substantial margin and maintains remarkable computational efficiency. Notably, we achieve a 5.7% enhancement in mean Average Precision (mAP), for instance, segmentation, while striking an optimal balance between accuracy and runtime for semantic segmentation. These outcomes underscore the versatility, efficiency and high-performance attributes of TUS-Net, positioning it as an indispensable framework for robust 3D point cloud segmentation.

Converting three-dimensional (3D) point cloud data to two-dimensional (2D) image based on virtual structured-light system is a popular method in 3D data compression because the image format is easier to process by the exiting storing and transmitting methods. When this method is used in the 3D point cloud compression, the quantization error is introduced during the coordinate mapping. To solve this problem, a virtual structured-light 3D point cloud compression algorithm based on geometric reshaping is presented. And a least-square system parameter optimization method is proposed to further improve the data decoding accuracy. In the proposed method, the 3D spatial coordinates are reshaped to a 2D matrix first, and then the 2D matrix is stored as a “Holoimage” by using the parameter optimized virtual structured-light system. The quantization error introduced by coordinate mapping between X and Y coordinates of the 3D point cloud and the 2D image pixel coordinates is suppressed, so the decoding accuracy is improved. In addition, the geometric information of the 3D point cloud is hidden synchronously when the 3D point cloud is compressed, which is of great significance in the copyright protection of the point cloud data. Experiments verify the effectiveness of the proposed algorithm. The decoding root mean square error (RMSE) of the proposed method is decreased by 79.86% on average compared with the traditional one under the same compression ratio, and the compression ratio of the proposed method is 1.96 times bigger than the traditional one under the similar decoding accuracy.

Instance segmentation based on 3D point cloud is a key step in scene understanding. It is widely used in indoor robot navigation, outdoor autonomous driving, and other fields. But research in this area is still in its infancy. Instance segmentation not only needs to predict the semantic label of each point but also the instance label of each point. Therefore, semantic segmentation can be considered the basis of instance segmentation to some extent. Based on this motivation, we designed a voxel-based branch based on convertible sparse convolution and residual optimization modules. We design a point-based branch so that the network can maintain high-resolution representation. Then the two branches are combined to optimize the semantic segmentation results. Breadth-first search (BFS) performs well in indoor point clouds and is simple to operate. Therefore, we use this clustering operation to group the points of the same instance to obtain the instance segmentation result. The proposed method was tested on the indoor dataset Scan-Net v2 and achieved relatively good instance segmentation precision.

Improving performance of deep learning models for 3D point cloud semantic segmentation via attention mechanisms

• Semi-automated framework for reconstructing the finite element model of plastic greenhouses was proposed. • Proposed segmentation model (GHFrameFormer) extracts steel frame from plastic film with 65.97 % mIoU. • Reconstructed FE model showed below 0.7 % (MAPE) geometric error. • Predicted linear load–displacement in FE analysis matched the on-site loading test with 3.6 % NRMSE. • Proposed framework establishes a foundation for digital-twin safety monitoring of horticulture. Plastic greenhouses (PG) are widespread, low-cost agricultural structures that are highly vulnerable to extreme weather events. However, reliable on-site methods for evaluating structural safety are still lacking. Furthermore, no prior study has presented a pipeline for reconstructing finite element (FE) model of a PG from its point cloud data (PCD) using semantic segmentation. This study proposes and validates a semi-automated framework to address this gap through: 1) automated segmentation of steel frames from PCD extracted through LiDAR scanner; 2) conversion of segmented PCD into as-built FE model; and 3) evaluation of structural response. A transformer-based semantic segmentation model, GHFrameFormer, was trained on a diverse dataset of four PG and evaluated on a completely independent, large-scale test PG. GHFrameFormer achieved the highest mean intersection-over-union of 65.97 %, outperforming baseline models (e.g., Point Transformer v1) and maintaining high computational efficiency. The segmented PCD was converted into a FE model via a density-controlled iterative clustering algorithm. Geometric validation across three independent PG confirmed high fidelity, with a mean absolute percentage error remaining below 0.7 % compared to on-site measurements. Finally, on-site loading test results were compared with the FE analysis results of the reconstructed FE model. The reconstructed FE model reliably reproduced the structural response (load–displacement) in a linear-elastic range with a normalized root mean squared error of 3.6 %. The proposed framework demonstrates that an AI-based approach can dramatically reduce manual post-processing of raw PCD while delivering geometric fidelity and reliable structural response prediction, establishing a foundation for digital-twin-enabled structural monitoring in horticulture applications.

3D point cloud data captured from Oil and Gas (O&G) facilities are becoming increasingly abundant. This data supports the interactive visualization of complex plants, which is central to numerous industrial advancements. As industries seek to innovate and optimize their processes, the versatile use of 3D point cloud data is proving invaluable beyond traditional O&G applications. The precision and comprehensiveness of 3D point cloud data facilitate efficient design, construction, maintenance and operational monitoring across various industrial sectors. Digital Twin (DT) solutions enable this by maintaining 3D representations, such as 3D CAD models, that accurately reflect the current state of facilities. As installations change, the process of 3D scanning is often repeated to capture the latest updates. Engineers and designers then use modelling software to visualize the resulting point cloud and update the facility's 3D representation, incorporating any new modifications. However, this involves interpreting the visual point cloud data and structuring it into meaningful objects (such as valves, tanks, etc.), a process that remains manual, requiring labor-intensive tasks that are prone to errors and incur significant costs due to the need for specialized labor. We propose an Artificial Intelligence approach, that uses a deep learning-based algorithm to automatically segment 3D objects within a point cloud. Our workflow recognizes instances of usual components of production systems, such as control valves, pipes, and transmitters. A state-of-the-art 3D instance segmentation approach is adapted to the O&G domain. Given the lack of publicly available instance segmentation datasets in this domain, we manually annotate 54 scenes from an O&G offshore plant, each of them around 20 m2 in footprint. To further enhance model training, we also incorporated synthetic point clouds. Practical results on real point clouds, acquired from operational facilities and pre-processed as required by commercial DT solutions, indicated a performance of approximately 70% success (f1-score) in segmenting components of O&G production systems, reaching up to 85% for components such as control valves. Accordingly, a 52.4% reduction in costs and efforts associated with manually interpreting 3D point clouds is estimated. Point cloud instance segmentation has attracted much attention in the computer vision research community, and many approaches have been proposed. However, results in the literature are heavily biased toward a few benchmarking datasets, which were acquired in indoor environments like offices and rooms. Few studies have addressed the O&G domain, creating a gap in the literature for achieving more competitive results. This lack of comprehensive research limits the potential for innovation and optimization in the industry. This paper contributes by (i) assessing the performance of state-of-the-art algorithms in the domain of O&G facilities, (ii) proposing modifications to better generalize to components of O&G production systems, (iii) objectively measuring the impact of this algorithm in reducing manual effort for engineers, and (iv) advancing towards a new DT functionality that derives an asset 3D representation, that is faithful to physical reality, through the automatic interpretation of point cloud scans.

Along with increasingly popular virtual reality applications, the three-dimensional (3D) point cloud has become a fundamental data structure to characterize 3D objects and surroundings. To process 3D point clouds efficiently, a suitable model for the underlying structure and outlier noises is always critical. In this work, we propose a hypergraph-based new point cloud model that is amenable to efficient analysis and processing. We introduce tensor-based methods to estimate hypergraph spectrum components and frequency coefficients of point clouds in both ideal and noisy settings. We establish an analytical connection between hypergraph frequencies and structural features. We further evaluate the efficacy of hypergraph spectrum estimation in two common applications of sampling and denoising of point clouds for which we provide specific hypergraph filter design and spectral properties. Experimental results demonstrate the strength of hypergraph signal processing as a tool in characterizing the underlying properties of 3D point clouds.

Three-dimensional point clouds have become a cornerstone of single-plant phenotyping, with deep learning serving as the main driver of high-throughput trait analysis. However, repeated downsampling operations in U-shaped and pyramidal architectures have been shown to oversmooth features and erode fine-grained details in 3D single-plant point clouds. To address this, the present study proposes SMFCA-Net, a frequency-aware framework integrating a Multi-Frequency Fourier Cross-Attention (MFCA) module into the sparse U-Net bottleneck. The MFCA bottleneck, a combination of spectral partitioning and cross-band attention, has been shown to capture both long-range contextual dependencies and local structures. This achieves a better balance between global and local information. The MFCA module comprises a three-stage pipeline: The first stage is frequency decomposition (FD), which uses a 3D fast Fourier transform (3D FFT) to separate features into low-, mid-, and high-frequency bands. The second is frequency-aware cross-attention (CA), which facilitates cross-band interactions anchored on mid-frequency semantics. The third and final stage is cross-frequency adaptive fusion (AF), which aggregates the enhanced bands using learnable weights. Extensive experimentation on a range of crops and acquisition scenarios has demonstrated consistent improvements in single-plant classification, semantic segmentation, and instance segmentation. This highlights the effectiveness of SMFCA-Net for high-throughput phenotyping. The source code can be accessed at the following URL: https: //github.com/yangxin6/SMFCA.git.

Three-Dimensional (3D) semantic segmentation is an essential building block for interactive Augmented Reality (AR). However, existing Deep Neural Network (DNN) models for segmenting 3D objects are not only computation-intensive but also memory heavy, hindering their deployment on resource-constrained mobile devices. We present the design, implementation and evaluation of Slimmer, a generic and model-independent framework for accelerating 3D semantic segmentation and facilitating its real-time applications on mobile devices. In contrast to the current practice that directly feeds a point cloud to DNN models, Slimmer is motivated by our observation that these models remain high accuracy even if we remove a fraction of points from the input, which can significantly reduce the inference time and memory usage of these models. Our design of Slimmer faces two key challenges. First, the simplification method of point clouds should be lightweight. Otherwise, the reduced inference time may be canceled out by the incurred overhead of input-data simplification. Second, Slimmer still needs to accurately segment the removed points from the input to create a complete segmentation of the original input, again, using a lightweight method. Our extensive performance evaluation demonstrates that, by addressing these two challenges, Slimmer can dramatically reduce the resource utilization of a representative DNN model for 3D semantic segmentation. For example, if we can tolerate 1% accuracy loss, the reduction could be ~20% for inference time and ~9% for memory usage. The reduction increases to around ~27% for inference time and ~15% for memory usage when we can tolerate 2% accuracy loss.

Deep learning network for indoor point cloud semantic segmentation with transferability

In recent years, there has been significant growth in the ubiquity and popularity of three-dimensional (3D) point clouds, with an increasing focus on the classification of 3D point clouds. To extract richer features from point clouds, many researchers have turned their attention to various point set regions and channels within irregular point clouds. However, this approach has limited capability in attending to crucial regions of interest in 3D point clouds and may overlook valuable information from neighboring features during feature aggregation. Therefore, this paper proposes a novel 3D point cloud classification method based on global attention and adaptive graph convolution (Att-AdaptNet). The method consists of two main branches: the first branch computes attention masks for each point, while the second branch employs adaptive graph convolution to extract global features from the point set. It dynamically learns features based on point interactions, generating adaptive kernels to effectively and precisely capture diverse relationships among points from different semantic parts. Experimental results demonstrate that the proposed model achieves 93.8% in overall accuracy and 90.8% in average accuracy on the ModeNet40 dataset.