Robust Sea-Sky Line Detection in Complex Maritime Environments via Semantic Segmentation

Pixel-level labelling tasks, such as semantic segmentation, play a central role in image understanding. Recent approaches have attempted to harness the capabilities of deep learning techniques for image recognition to tackle pixel-level labelling tasks. One central issue in this methodology is the limited capacity of deep learning techniques to delineate visual objects. To solve this problem, we introduce a new form of convolutional neural network that combines the strengths of Convolutional Neural Networks (CNNs) and Conditional Random Fields (CRFs)-based probabilistic graphical modelling. To this end, we formulate mean-field approximate inference for the Conditional Random Fields with Gaussian pairwise potentials as Recurrent Neural Networks. This network, called CRF-RNN, is then plugged in as a part of a CNN to obtain a deep network that has desirable properties of both CNNs and CRFs. Importantly, our system fully integrates CRF modelling with CNNs, making it possible to train the whole deep network end-to-end with the usual back-propagation algorithm, avoiding offline post-processing methods for object delineation. We apply the proposed method to the problem of semantic image segmentation, obtaining top results on the challenging Pascal VOC 2012 segmentation benchmark.

The identification and classification of obstacles in navigable and non-navigable regions, as well as the measurement of distances, are crucial topics of investigation in the field of autonomous navigation for unmanned surface vehicles (USVs). Currently, USVs mostly rely on LiDAR and ultrasound technology for the purpose of detecting impediments that exist on water surfaces. However, it is worth noting that these approaches lack the capability to accurately discern the precise nature or classification of those obstacles. Nevertheless, the limited optical range of unmanned vessels hinders their ability to comprehensively perceive the entirety of the surrounding information. A cooperative USV-UAV system is proposed to ensure the visual perception ability of USVs. The multi-object recognition, semantic segmentation, and obstacle ranging through USV and unmanned aerial vehicle (UAV) perspectives are selected to validate the performance of a cooperative USV-UAV system. The you only look once-X (YOLOX) model, the proportional–integral–derivative-NET (PIDNet) model, and distance measurements based on a monocular camera are utilized to realize these problems. The results indicate that by integrating the viewpoints of USVs and UAVs, a collaborative USV-UAV system, employing the aforementioned methods, can successfully detect and classify different objects surrounding the USV. Additionally, it can differentiate between navigable and non-navigable regions for unmanned vessels through visual recognition, while accurately determining the distance between the USV and obstacles.

Lentic waters, such as lakes, lagoons, reservoirs, and wetlands are characterized by their absence of current. In recent decades, they have been threatened by pollution and scarcity due to various environmental factors. Therefore, they require frequent monitoring to ensure their health and purity, especially to control the proliferation of harmful cyanobacteria (pollutants). Machine Vision Systems (MVS) on board Autonomous Surface Vehicles (ASVs) is a good option for automatic image processing in this context. ASVs must navigate safely, and obstacle detection is essential. In addition, the segmentation of pollutants in water is crucial. We propose an architecture based on convolutional neural networks that integrates both object detection and semantic segmentation. The goal is to simultaneously extract all available global information to detect objects and amorphous textures (cyanobacterial patches and water bodies), considering their variations in size, pose, and appearance. The architecture includes two branches: object detection and semantic segmentation, sharing the same backbone and neck. We evaluate the model on our dataset and the results show that it can holistically understand lentic water scenes with high accuracy, and the integration of the attention mechanism improves its overall performance.

Cyanobacterial Harmful Algal Blooms (CyanoHABs) in lakes and reservoirs have increased substantially in recent decades due to different environmental factors. Its early detection is a crucial issue to minimize health effects, particularly in potential drinking and recreational water bodies. The use of Autonomous Surface Vehicles (ASVs) equipped with machine vision systems (cameras) onboard, represents a useful alternative at this time. In this regard, we propose an image Semantic Segmentation approach based on Deep Learning with Convolutional Neural Networks (CNNs) for the early detection of CyanoHABs considering an ASV perspective. The use of these models is justified by the fact that with their convolutional architecture, it is possible to capture both, spectral and textural information considering the context of a pixel and its neighbors. To train these models it is necessary to have data, but the acquisition of real images is a difficult task, due to the capricious appearance of the algae on water surfaces sporadically and intermittently over time and after long periods of time, requiring even years and the permanent installation of the image capture system. This justifies the generation of synthetic data so that sufficiently trained models are required to detect CyanoHABs patches when they emerge on the water surface. The data generation for training and the use of the semantic segmentation models to capture contextual information determine the need for the proposal, as well as its novelty and contribution.Three datasets of images containing CyanoHABs patches are generated: (a) the first contains real patches of CyanoHABs as foreground and images of lakes and reservoirs as background, but with a limited number of examples; (b) the second, contains synthetic patches of CyanoHABs generated with state-of-the-art Style-based Generative Adversarial Network Adaptive Discriminator Augmentation (StyleGAN2-ADA) and Neural Style Transfer as foreground and images of lakes and reservoirs as background, and (c) the third set, is the combination of the previous two. Four model architectures for semantic segmentation (UNet++, FPN, PSPNet, and DeepLabV3+), with two encoders as backbone (ResNet50 and EfficientNet-b6), are evaluated from each dataset on real test images and different distributions. The results show the feasibility of the approach and that the UNet++ model with EfficientNet-b6, trained on the third dataset, achieves good generalization and performance for the real test images.

Due to the phenomenon of mixed pixels in low-resolution remote sensing images, the green tide spectral features with low Enteromorpha coverage are not obvious. Super-resolution technology based on deep learning can supplement more detailed information for subsequent semantic segmentation tasks. In this paper, a novel green tide extraction method for MODIS images based on super-resolution and a deep semantic segmentation network was proposed. Inspired by the idea of transfer learning, a super-resolution model (i.e., WDSR) is first pre-trained with high spatial resolution GF1-WFV images, and then the representations learned in the GF1-WFV image domain are transferred to the MODIS image domain. The improvement of remote sensing image resolution enables us to better distinguish the green tide patches from the surrounding seawater. As a result, a deep semantic segmentation network (SRSe-Net) suitable for large-scale green tide information extraction is proposed. The SRSe-Net introduced the dense connection mechanism on the basis of U-Net and replaces the convolution operations with dense blocks, which effectively obtained the detailed green tide boundary information by strengthening the propagation and reusing features. In addition, the SRSe-Net reducs the pooling layer and adds a bridge module in the final stage of the encoder. The experimental results show that a SRSe-Net can obtain more accurate segmentation results with fewer network parameters.

In this study, we investigated how shapes are classified based on local and global features by four representative convolutional neural networks (CNNs), i.e., AlexNet, VGG, ResNet and Inception. While the local features are based on simple components, such as orientation of line segment, the global features are based on the whole object, such as whether an object has a hole. For example, solid triangles and solid squares are differentiated by local features, solid circles and rings are differentiated by a global feature. Two sets of experiments were performed in this research. In the first experiment, we examined how the four CNNs pre-trained on ImageNet (with transfer learning) learned to differentiate the regular shapes (equilateral-triangles, squares, circles and rings). Our results showed that the pre-trained CNNs exhibited faster learning rates in the tasks discriminating the local features than in the tasks discriminating the global feature. However, the transfer learning of discriminating the global feature in regular shapes were better generalized to irregular shapes than the transfer learning of discriminating local features. In the second experiment, the CNNs were trained from scratch (with random weights initialization) to discriminate local and global features in regular and irregular shapes. Different from the transfer learning, the CNNs exhibited faster learning rates in discriminating the global feature than the local features. Similar to transfer learning, the CNNs exhibited excellent learning generalization to discriminating the global feature of irregular shapes, but poor learning generalization to discriminating the local features in the irregular shapes. The overarching goal of this research is to create a paradigm and benchmark to directly compare how the CNNs and primate visual systems process geometrical invariants. In contrast to the ImageNet approach which employs natural images to train CNNs, we employed the “ShapeNet” approach which features geometrical shapes with well-defined properties. The ShapeNet approach will not only help elucidate the strengths and limitations of CNN computation, but also provide insights into visual information processing in the primates.

Developing computational algorithms to model the biological vision system has challenged researchers in the computer vision field for several decades. As a result, state-of-the-art algorithms such as the Convolutional Neural Network (CNN) have emerged for image classification and recognition tasks with promising results. CNNs however remain view-specific, producing good results when the variation between test and train data is small. Making CNNs learn invariant features to effectively recognise objects that undergo appearance changes as a result of transformations such as scaling remains a technical challenge. Recent physiological studies of the visual system are suggesting new paradigms. Firstly, our visual system uses both local features and global features in its recognition function. Secondly, cells tuned to global features respond quickly to visual stimuli for recognising objects. Thirdly, information from modalities that handle local features, global features and color are integrated in the brain for performing recognition tasks. While CNNs rely on aggregation of local features for recognition, these theories provide the potential for using global features to solve transformation invariance problems in CNNs. In this paper we realise these paradigms into a computational model, named as global features improved CNN (GCNN), and test it on classification of scaled images. We experiment combining Histogram of Gradients (HOG) global features, CNN local features and color information and test our technique on benchmark data sets. Our results show GCNN outperforms traditional CNN on classification of scaled images indicating potential effectiveness of our model towards improving scale-invariance in CNN based networks.

Panoptic Segmentation aims to provide an understanding of background (stuff) and instances of objects (things) at a pixel level. It combines the separate tasks of semantic segmentation (pixel level classification) and instance segmentation to build a single unified scene understanding task. Typically, panoptic segmentation is derived by combining semantic and instance segmentation tasks that are learned separately or jointly (multi-task networks). In general, instance segmentation networks are built by adding a foreground mask estimation layer on top of object detectors or using instance clustering methods that assign a pixel to an instance center. In this work, we present a fully convolution neural network that learns instance segmentation from semantic segmentation and instance contours (boundaries of things). Instance contours along with semantic segmentation yield a boundary aware semantic segmentation of things. Connected component labeling on these results produces instance segmentation. We merge semantic and instance segmentation results to output panoptic segmentation. We evaluate our proposed method on the CityScapes dataset to demonstrate qualitative and quantitative performances along with several ablation studies. Our overview video can be accessed from https://youtu.be/wBtcxRhG3e0.

Achieving accurate spatial awareness is a fundamental requirement for intelligent vision systems operating in complex and dynamic environments, such as autonomous navigation, robotic manipulation, and augmented reality. While Convolutional Neural Networks (CNNs) have demonstrated exceptional performance in tasks such as image classification and semantic segmentation, their inherently two-dimensional structure limits their ability to model and reason about three-dimensional spatial relationships. Specifically, CNNs are constrained by local receptive fields, a lack of explicit geometric context, and their dependence on appearance-based cues, which often results in inaccurate understanding of object boundaries, depth discontinuities, and occlusions in real-world scenes. To address these limitations, this paper investigates the fusion of RGB visual data with depth information through a multi-modal intermediate fusion framework. We propose a lightweight experimental prototype that integrates parallel feature extraction pipelines for RGB images and corresponding depth maps, followed by feature-level fusion to enhance semantic and geometric understanding. The experiment is conducted on the NYU Depth V2 dataset, which provides densely labeled indoor scenes with aligned RGB and depth data. A comparative analysis is performed between a baseline CNN model trained solely on RGB input and a modified model utilizing intermediate fusion of RGB and depth features. Experimental results indicate that the inclusion of depth information significantly improves the model’s ability to delineate object boundaries, resolve foreground-background ambiguities, and maintain semantic coherence across varying spatial scales. The depth-enhanced model demonstrates increased robustness to occlusions and illumination changes, highlighting the practical benefits of integrating geometric cues into visual perception pipelines. These findings provide empirical support for the theoretical premise that multi-modal feature fusion can substantially enhance spatial reasoning in CNN-based architectures. This study contributes both a conceptual understanding and an applied perspective on the design of multi-modal spatial systems. The results serve as a foundation for further development of robust, depth-aware visual perception models with applications in real-time robotics, autonomous systems, and immersive AR/VR environments.

Autonomous transportation is a new paradigm of an Industry 5.0 cyber-physical system that provides a lot of opportunities in smart logistics applications. The safety and reliability of deep learning-driven systems are still a question under research. The safety of an autonomous guided vehicle is dependent on the proper selection of sensors and the transmission of reflex data. Several academics worked on sensor-based difficulties by developing a sensor correction system and fine-tuning algorithms to regulate the system’s efficiency and precision. In this paper, the introduction of vision sensor and its scene terrain classification using a deep learning algorithm is performed with proposed datasets during sensor failure conditions. The proposed classification technique is to identify the mobile robot obstacle and obstacle-free path for smart logistic vehicle application. To analyze the information from the acquired image datasets, the proposed classification algorithm employs segmentation techniques. The analysis of proposed dataset is validated with U-shaped convolutional network (U-Net) architecture and region-based convolutional neural network (Mask R-CNN) architecture model. Based on the results, the selection of 1400 raw image datasets is trained and validated using semantic segmentation classifier models. For various terrain dataset clusters, the Mask R-CNN classifier model method has the highest model accuracy of 93%, that is, 23% higher than the U-Net classifier model algorithm, which has the lowest model accuracy nearly 70%. As a result, the suggested Mask R-CNN technique has a significant potential of being used in autonomous vehicle applications.

Medical Image Semantic Segmentation (MISS), the process of assigning a semantic label to each pixel in an image, is a foundational task in computational medicine, critical for quantitative diagnostics and treatment planning. However, developing robust MISS models faces two intertwined challenges. First, there is an architectural dilemma: Convolutional Neural Networks (CNNs), like U-Net, excel at learning local features but are limited by their receptive fields, failing to capture global context essential for segmenting organs with large deformations. Conversely, Vision Transformers (ViTs) effectively model long-range dependencies but lack the inductive biases of CNNs, leading to poor generalization on the small datasets typical in medicine without extensive pre-training. Second, the prohibitive cost and expertise required for creating pixel-level annotations create a severe data scarcity bottleneck. While Semi-Supervised Learning (SSL) aims to mitigate this by leveraging unlabeled data, existing methods often fail to learn high-level semantic relations and are susceptible to confirmation bias from noisy pseudo-labels, class imbalance, and suboptimal contrastive sample selection. This thesis presents a comprehensive investigation to systematically address these challenges, delivering a cohesive suite of novel deep learning frameworks. The contributions are four-fold: First, to resolve the architectural trade-off, this work introduces CS-Unet, a pure Transformer network built upon a U-Net-like architecture. Its core innovation is the Convolutional Swin Transformer (CST) block, which integrates convolutions directly within the Multi-Head Self-Attention and Feed-Forward Network modules. This design imbues the Transformer with inherent localized spatial context and strong inductive biases, enabling it to efficiently learn both local and global features. Without pre-training, CS-Unet outperforms existing Transformer and CNN-based models on multi-organ and cardiac datasets, achieving state-of-the-art performance with fewer parameters. Second, to address data scarcity, a novel Multi-Scale Cross Supervised Contrastive Learning (MCSC) framework for SSL is developed. MCSC jointly trains CNN and Transformer models, using a cross-teaching paradigm where each network provides pseudo-labels for the other. Crucially, it moves beyond simple output consistency by applying a contrastive loss to feature maps at multiple scales, enforcing hierarchical semantic consistency. To handle the class imbalance endemic to medical imaging, a class-prevalence-aware loss is used to ensure features for infrequent classes are learned robustly. Third, to fortify SSL against noisy pseudo-labels, a certainty-guided contrastive learning strategy is proposed. This approach mitigates the impact of inaccurate pseudo-labels by using a certainty metric to guide the selection of samples for contrastive learning. The framework’s computational efficiency is enhanced through novel sampling strategies that select a few representative samples for contrasting, and a negative memory bank is used to increase sample diversity and eliminate dependence on batch size. Fourth, this thesis introduces a new paradigm for SSL by leveraging external anatomical priors through the Contrastive Cross-Teaching with Registration (CCT-R) framework. CCT-R is the first method to integrate spatial registration transforms into the learning process. It features two novel modules: a Registration Supervision Loss (RSL), which uses transforms between labeled and unlabeled volumes to generate an additional, highly reliable source of pseudo-labels, and Registration-Enhanced Positive Sampling (REPS), which uses registration to identify anatomically-corresponding positive pairs across volumes for contrastive learning. Overall, these contributions provide a powerful toolkit that significantly alleviates the annotation bottleneck in medical AI. The proposed methods demonstrate state-of-the-art performance on challenging segmentation benchmarks, delivering a pathway to develop accurate, data-efficient models for real-world clinical applications and opening new avenues for research into fusing geometric priors with semantic segmentation.

The classification and detection of maritime targets are widely used in shipping navigation and military fields. With the development of spaceborne synthetic aperture radar (SAR) technology, more and more very high-resolution SAR images can be acquired for maritime targets recognition. However, due to the different imaging mechanisms between SAR images and optical images, it is difficult and laborious to interpret SAR images manually. This study uses a modified Alexnet structure to realise maritime targets classification on the Gaofen-3 spaceborne SAR images. The maritime targets dataset (MTD), including boats, cargo ships, container ships, windmills, oil tankers, and iron towers, is conducted. Moreover, the proposed convolution neural networks (CNNs) structure is trained and tested on the MTD. Experimental results show that the model can get an accuracy of 92.10% in classifying the six kinds of targets, and the performance is superior compared with other CNNs and traditional supportive vector machine algorithms.

The preliminary step in the navigation of Unmanned Vehicles is to detect and identify the horizon line. One method to locate the horizon and obstacles in an image is through a supervised learning, semantic segmentation algorithm using Neural Networks. Unmanned Aerial Vehicles (UAVs) are rapidly gaining prominence in military, commercial and civilian applications. For the safe navigation of UAVs, there poses a requirement for an accurate and efficient obstacle detection and avoidance. The position of the horizon and obstacles can also be used for adjusting flight parameters and estimating altitude. It can also be used for the navigation of Unmanned Ground Vehicles (UGV), by neglecting the part of the image above the horizon to reduce the processing time. Locating the horizon and identifying the various obstacles in an image can help in minimizing collisions and high costs due to failure of UAVs and UGVs. To achieve a robust and accurate system to aid navigation of autonomous vehicles, the efficiency and accuracy of Convolutional Neural Networks (CNN) and Recurrent-CNNs (RCNN) are analysed. It is observed via experimentation that the RCNN model classifies test images with higher accuracy.

Colorectal cancer typically affects the gastrointestinal tract within the human body. Colonoscopy is one of the most accurate methods of detecting cancer. The current system facilitates the identification of cancer by computer-assisted diagnosis (CADx) systems with a limited number of deep learning methods. It does not imply the depiction of mixed datasets for the functioning of the system. The proposed system, called ColoRectalCADx, is supported by deep learning (DL) models suitable for cancer research. The CADx system comprises five stages: convolutional neural networks (CNN), support vector machine (SVM), long short-term memory (LSTM), visual explanation such as gradient-weighted class activation mapping (Grad-CAM), and semantic segmentation phases. Here, the key components of the CADx system are equipped with 9 individual and 12 integrated CNNs, implying that the system consists mainly of investigational experiments with a total of 21 CNNs. In the subsequent phase, the CADx has a combination of CNNs of concatenated transfer learning functions associated with the machine SVM classification. Additional classification is applied to ensure effective transfer of results from CNN to LSTM. The system is mainly made up of a combination of CVC Clinic DB, Kvasir2, and Hyper Kvasir input as a mixed dataset. After CNN and LSTM, in advanced stage, malignancies are detected by using a better polyp recognition technique with Grad-CAM and semantic segmentation using U-Net. CADx results have been stored on Google Cloud for record retention. In these experiments, among all the CNNs, the individual CNN DenseNet-201 (87.1% training and 84.7% testing accuracies) and the integrated CNN ADaDR-22 (84.61% training and 82.17% testing accuracies) were the most efficient for cancer detection with the CNN+LSTM model. ColoRectalCADx accurately identifies cancer through individual CNN DesnseNet-201 and integrated CNN ADaDR-22. In Grad-CAM's visual explanations, CNN DenseNet-201 displays precise visualization of polyps, and CNN U-Net provides precise malignant polyps.

Remote sensing images contain a wealth of geospatial information. To accurately identify different geospatial categories and extract relevant data, image semantic segmentation plays a crucial role. In recent years, deep learning technology has brought significant breakthroughs to semantic segmentation of remote sensing images, significantly enhancing its performance. This paper investigates the application of deep learning technologies in remote sensing image semantic segmentation, based on Convolutional Neural Networks (CNN) and Transformer-based semantic segmentation methods. It conducts an in-depth comparison of their structural characteristics and applicable scenarios, summarizes the achievements and shortcomings of existing research, and provides technical references and theoretical support for future studies, thereby contributing to the further development of deep learning technology in the field of remote sensing. Research results indicate that CNN-based semantic segmentation methods still hold advantages in extracting local features and achieving efficient segmentation, whereas Transformers address CNN's limitations in global context modeling and long-range dependency capture. Therefore, the collaborative integration of CNN and Transformers will become an important research direction for enhancing model performance in the future.