PmcaNet: Pyramid multiscale channel attention network for electron microscopy image segmentation

Accurate segmentation of three-dimensional serial-section electron microscopy(EM) images is of great significance for neuroscience research. For anisotropic 3D EM image segmentation, it is necessary to obtain information in all dimensions in a balanced manner. However, many existing methods default that images are isotropic, leading to difficulty in obtaining high quality results in anisotropic EM images. In this work, we proposed Depth Swin Transformer Unet(DSTUnet), aU-shaped neural network based on Depth Swin Transformer (DST) block and convolution layers, to improve the segmentation of 3D EM images. DST block is composed of a Swin Transformer block with a flattened window to get a larger receptive field in the X-Y direction and a Depth Overlapped Attention (DOA) module to enable the transmission of neighborhood pixel information in depth-wise. The proposed DOA module employed a slightly larger and overlapped patch in the key and value vector, which significantly promotes the model performance. Taking advantage of DSTUnet, we realized segmentation of mitochondria in the MitoEM-R, Lucchi++ dataset, and neural cell from optical images, achieving superior performance to existing methods.

The use of Gibbs distribution-based Bayesian segmentation of electron microscopy images for visualizing nanostructures is investigated. Bayesian segmentation involves dividing an image into nonoverlapping areas that correspond as closely as possible to the observed image. A quantitative characteristic of this correspondence is the a posteriori probability of one variant of division or another. The most likely version is always the division with the greatest a posteriori probability. The Metropolis algorithm for stochastic relaxation is used to obtain Bayesian estimates of the a posteriori probability of a division. Our study of Bayesian segmentation requires visualization of nanostructures on an electron microscopy image of a film made of NiW nanocrystalline alloy.

Accurate segmentation of ore images plays a significant role in automatic geometric parameter detecting and composition analyzing of ore dressing progress. Semantic segmentation based on deep learning is a promising method for accurate ore image segmentation. However, the similar appearance with low contrast and blurry boundary of ores in image hamper segmentation accuracy. Moreover, it is difficult to train a deep network due to limited available ore images. In this work, an improved encoder-decoder network based on U-Net is proposed to handle above challenges. A contour awareness loss (CAL) is proposed to improve model sensitivity to misclassified pixels, pixels of similar appearance, and pixels near the boundary. The proposed scheme is verified on ore images by benchmarking against state-of-the-art segmentation methods. Experiment results show that the proposed scheme achieves 88.6% pixel-wise accuracy (PA) and 66.0% mean intersection over union (mIoU).

Semantic segmentation has been a fundamental task in interpreting remote sensing imagery (RSI) for various downstream applications. Due to the high intra-class variants and inter-class similarities, inflexibly transferring natural image-specific networks to RSI is inadvisable. To enhance the distinguishability of learnt representations, attention modules were developed and applied to RSI, resulting in satisfactory improvements. However, these designs capture contextual information by equally handling all the pixels regardless of whether they around edges. Therefore, blurry boundaries are generated, rising high uncertainties in classifying vast adjacent pixels. Hereby, we propose an edge distribution attention module (EDA) to highlight the edge distributions of leant feature maps in a self-attentive fashion. In this module, we first formulate and model column-wise and row-wise edge attention maps based on covariance matrix analysis. Furthermore, a hybrid attention module (HAM) that emphasizes the edge distributions and position-wise dependencies is devised combing with non-local block. Consequently, a conceptually end-to-end neural network, termed as EDENet, is proposed to integrate HAM hierarchically for the detailed strengthening of multi-level representations. EDENet implicitly learns representative and discriminative features, providing available and reasonable cues for dense prediction. The experimental results evaluated on ISPRS Vaihingen, Potsdam and DeepGlobe datasets show the efficacy and superiority to the state-of-the-art methods on overall accuracy (OA) and mean intersection over union (mIoU). In addition, the ablation study further validates the effects of EDA.

PET-guided radiation therapy treatment planning, clinical diagnosis, assessment of tumor growth, and therapy response rely on the accurate delineation of the tumor volume and quantification of tracer uptake. Most PET image segmentation techniques proposed thus far are suboptimal in the presence of heterogeneity of tracer uptake within the lesion. This work presents an active contour model approach based on the method of Chan and Vese ["Active contours without edges," IEEE Trans. Image Process. 10, 266-277 (2001)] designed to take into account the high level of statistical uncertainty (noise) and to handle the heterogeneity of tumor uptake typically present in PET images. In the proposed method, the fitting terms in the Chan-Vese formulation are modified by introducing new input images, including the smoothed version of the original image using anisotropic diffusion filtering (ADF) and the contourlet transform of the image. The advantage of utilizing ADF for image smoothing is that it avoids blurring the object's edges and preserves the average activity within a region, which is important for accurate PET quantification. Moreover, incorporating the contourlet transform of the image into the fitting terms makes the energy functional more effective in directing the evolving curve toward the object boundaries due to the enhancement of the tumor-to-background ratio (TBR). The proper choice of the energy functional parameters has been formulated by making a clear consensus based on tumor heterogeneity and TBR levels. This cautious parameter selection leads to proper handling of heterogeneous lesions. The algorithm was evaluated using simulated phantom and clinical studies, where the ground truth and histology, respectively, were available for accurate quantitative analysis of the segmentation results. The proposed technique was also compared to a number of previously reported image segmentation techniques. The results were quantitatively analyzed using three evaluation metrics, including the spatial overlap index (SOI), the mean relative error (MRE), and the mean classification error (MCE). Although the performance of the proposed method was analogous to other methods for some datasets, overall the proposed algorithm outperforms all other techniques. In the largest clinical group comprising nine datasets, the proposed approach improved the SOI from 0.41±0.14 obtained using the best-performing algorithm to 0.54±0.12 and reduced the MRE from 54.23±103.29 to 0.19±16.63 and the MCE from 112.86±69.07 to 60.58±18.43. The proposed segmentation technique is superior to other representative segmentation techniques in terms of highest overlap between the segmented volume and the ground truth∕histology and minimum relative and classification errors. Therefore, the proposed active contour model can result in more accurate tumor volume delineation from PET images.

In the summer of 2017, I contacted Shawn, then working at the Max Planck Institute in Martinsried, to ask him to join the Editorial Board of the Journal of Comparative Neurology (JCN). I was very impressed by his scientific acumen and his research on high-throughput electron microscopy that he wanted to apply at the level of an entire brain. Shawn immediately accepted and became a member of our journal's board in January 2018. Even before he was officially a member of the board, with characteristic enthusiasm, he began to assemble a group of contributors for a special issue of JCN dedicated to cutting-edge methods and approaches to structural and functional mapping of the mouse brain. Most unfortunately, his tenure as a member of JCN's board was to be a short one. With great sadness, we learned in July 2018 Shawn's tragic passing. We lost a great friend, a highly dedicated and dynamic colleague, and one of the most talented and promising young investigator in neuroscience at a point when he was developing his career independently at Keio University in Japan. Shawn's career was brief and cut short, but remarkably productive. He received his College Education at the University of Texas in Austin, where he graduated in 1998 with no fewer than three BS degrees in Mathematics, Physics, and Biochemistry. He then completed his PhD in neuroscience in 2005 at Johns Hopkins University School of Medicine at the Krieger Mind-Brain Institute in Baltimore, MD, under the mentorship of Ernst Niebur. Afterward, he was a postdoctoral researcher with Ted Jones at the University of California Davis where he developed the BrainMaps project, an internet-enabled multiresolution histochemical and tract-tracing brain atlas for various mammalian species. He moved on to Germany in 2009 for further postdoctoral work with Winfried Denk at the Max Planck Institute for Medical Research in Heidelberg, to pursue his work on comprehensive mapping the mammalian whole-brain circuitry and structure at the synaptic level using high-throughput volume electron microscopy. He became a Researcher at the Max Planck Institute for Neurobiology in Martinsried in 2013 where he developed methods to prepare small mammalian brains for electron microscopy and to image them using high-throughput multibeam scanning electron microscopy. In 2018, he moved to Japan, first at the National Institute for Physiological Sciences in Okazaki, and a few months later to Keio University School of Medicine in Tokyo, where he had just been appointed Assistant Professor at the time he passed away, working on a comprehensive ultrastructural mapping of the circuitry of the whole mouse brain at a single-synapse level of resolution. His goals were eventually to apply these approaches to small nonhuman primate brains. A list of Shawn's major scientific contributions appears at the end of this editorial. This special issue reflects Shawn's last efforts as an editor for our journal. It represents our recognition of his outstanding work and a memory of his contributions to understanding the mammalian whole-brain connectome and developing new techniques “to reconstruct the whole brain's wiring diagram.” The articles assembled in the present issue summarize recent efforts to attain a functional and structural connectome map of the mouse brain through multimodal and multiscale approaches. I am personally indebted to the many contributors who participated in the completion of this issue after Shawn's death and graciously helped me finalize it in a timely manner to celebrate the memory of our friend and colleague. We also express our most sincere and heartfelt condolences to his friends and family.

Abstract. This study explores the enhancement of UNet-based semantic segmentation for photovoltaic (PV) panels in remote sensing images by integrating attention mechanisms. Given the critical role of solar energy in achieving global sustainability, accurate PV panel detection is essential for effective energy management. Using the high-resolution PV01 dataset, which includes UAV-captured rooftop PV samples, we evaluate the impact of four distinct attention modules: Convolutional Block Attention Module (CBAM), Squeeze-and-Excitation Networks (SE-Net), Efficient Channel Attention (ECA-Net), and Coordinate Attention (CA) on segmentation performance. Comparative analysis demonstrates that UNet models with SE and CA modules substantially outperform the baseline, achieving the highest scores in Average Accuracy (AA), Average Precision (AP), Average Recall (AR), mean Intersection over Union (mIoU), and Average F1-score (AF). In particular, UNet + SE achieved an AA of 0.9809, AP of 0.9756, AR of 0.9629, AF of 0.9692, and mIoU of 0.9403, highlighting the efficacy of attention mechanisms in refining feature representations and advancing PV panel segmentation, thereby contributing to large-scale solar energy monitoring and deployment.

Geo-information extraction from satellite imagery has become crucial to carry out large-scale ground surveys in a short amount of time. With the increasing number of commercial satellites launched into orbit in recent years, high-resolution RGB color remote sensing imagery has attracted a lot of attention. However, because of the high cost of image acquisition and even more complicated annotation procedures, there are limited high-resolution satellite datasets available. Compared to close-range imagery datasets, existing satellite datasets have a much lower number of images and cover only a few scenarios (cities, background environments, etc.). They may not be sufficient for training robust learning models that fit all environmental conditions or be representative enough for training regional models that optimize for local scenarios. Instead of collecting and annotating more data, using synthetic images could be another solution to boost the performance of a model. This article proposes a GAN-assisted training scheme for road segmentation from high-resolution RGB color satellite images, which includes three critical components: (a) synthetic training sample generation, (b) synthetic training sample selection, and (c) assisted training strategy. Apart from the GeoPalette and cSinGAN image generators introduced in our prior work, this article explains in detail how to generate new training pairs using OpenStreetMap (OSM) and introduces a new set of evaluation metrics for selecting synthetic training pairs from a pool of generated samples. We conduct extensive quantitative and qualitative experiments to compare different image generators and training strategies. Our experiments on the downstream road segmentation task show that (1) our proposed metrics are more aligned with the trained model performance compared to commonly used GAN evaluation metrics such as the Fréchet inception distance (FID); and (2) by using synthetic data with the best training strategy, the model performance, mean Intersection over Union (mean IoU), is improved from 60.92% to 64.44%, when 1,000 real training pairs are available for learning, which reaches a similar level of performance as a model that is standard-trained with 4,000 real images (64.59%), i.e., enabling a 4-fold reduction in real dataset size.

This paper examines the use of deep learning methods for accurately segmenting breast cancer in ultrasound images, which is an important step in computer-aided cancer diagnosis. To improve border location and region-level accuracy, the proposed framework presents a hybrid attention-based segmentation model, combining the DenseNet121 encoder and three specially designed attention modules: global spatial attention (GSA), temporary spatial attention (TSA) and spatial feature enhancement block (SFEB). Its unique approach is a multi-level attention integration strategy that improves both local features and global awareness, allowing models to understand complex breast injuries more effectively. After training and evaluating the model using the Breast Ultrasound pictures (BUSI) dataset, which comprises 780 grayscale ultrasound pictures with matching binary masks and labels such as benign, malignant, or normal. With a mean Intersection over Union (IoU) of 0.94, overall segmentation accuracy of 0.89, Dice coefficient of 0.94, precision of 0.93, and recall of 0.94, the suggested method outperforms traditional Convolutional Neural Network (CNN) based techniques and shows great promise for supporting radiological diagnoses.

To solve the problems of rough edge and poor segmentation accuracy of traditional neural networks in small nucleus image segmentation, a nucleus image segmentation technology based on U-Net network is proposed. First, the U-Net network is used to segment the nucleus image, which stitches the feature images in the channel dimension to achieve feature fusion, and the skip structure is used to combine the low- and high-level features. Then, the subregional average pooling is proposed to improve the global average pooling in the attention module, and an attention channel expansion module is designed to improve the accuracy of image segmentation. Finally, the improved attention module is integrated into the U-Net network to achieve accurate segmentation of the nuclear image. Based on the Python platform, the experimental results show that the proposed segmentation technology can achieve fast convergence, and the mean intersection over union (MIoU) is 85.02%, which is better than other comparison technologies and has a good application prospect.

OCT layer segmentation using U-NET semantic segmentation and RESNET34 encoder-decoder

Cracks affect the robustness and adaptability of various infrastructures, including buildings, bridge piers, pavement, and pipelines. Therefore, the robustness and the reliability of automated crack detection are essential. In this study, we conducted image segmentation using various crack datasets by applying the advanced architecture of U-Net. First, we collected and integrated crack datasets from prior studies, including the cracks in buildings and pavements. For effective localization and detection of cracks, we used U-Net-based neural networks, ResU-Net, VGGU-Net, and EfficientU-Net. The models were evaluated by the five-fold cross-validation using several evaluation metrics including mean pixel accuracy (MPA), mean intersection over union (MIoU), and confusion matrix. The results of the integrated dataset showed that ResU-Net (68.47%) achieves the highest MIoU with a relatively low number of parameters compared to VGGU-Net (67.71%) and EfficientU-Net (68.07%). In addition to the performance, ResU-Net showed the lowest test runtime, 40 milliseconds per single image, and the highest true positive rate of 45.00% in the pixel-wise recognition test. As the models were trained and validated with diverse surfaces, the proposed approach can be used as a pre-trained model in the task with relatively few data sources. Furthermore, both practical and managerial implications are discussed herein.

The posterior parietal cortex exhibits choice-selective activity during perceptual decision-making tasks1-10. However, it is not known how this selective activity arises from the underlying synaptic connectivity. Here we combined virtual-reality behaviour, two-photon calcium imaging, high-throughput electron microscopy and circuit modelling to analyse how synaptic connectivity between neurons in the posterior parietal cortex relates to their selective activity. We found that excitatory pyramidal neurons preferentially target inhibitory interneurons with the same selectivity. In turn, inhibitory interneurons preferentially target pyramidal neurons with opposite selectivity, forming an opponent inhibition motif. This motif was present even between neurons with activity peaks in different task epochs. We developed neural-circuit models of the computations performed by these motifs, and found that opponent inhibition between neural populations with opposite selectivity amplifies selective inputs, thereby improving the encoding of trial-type information. The models also predict that opponent inhibition between neurons with activity peaks in different task epochs contributes to creating choice-specific sequential activity. These results provide evidence for how synaptic connectivity in cortical circuits supports a learned decision-making task.

In the domain of road inspection and structural health monitoring, precise crack identification and segmentation are essential for structural safety and disaster prediction. Traditional image processing technologies encounter difficulties in detecting cracks due to their morphological diversity and complex background noise. This results in low detection accuracy and poor generalization. To overcome these challenges, this paper introduces MixSegNet, a novel deep learning model that enhances crack recognition and segmentation by integrating multi-scale features and deep feature learning. MixSegNet integrates convolutional neural networks (CNNs) and transformer architectures to enhance the detection of small cracks through the extraction and fusion of fine-grained features. Comparative evaluations against mainstream models, including LRASPP, U-Net, Deeplabv3, Swin-UNet, AttuNet, and FCN, demonstrate that MixSegNet achieves superior performance on open-source datasets. Specifically, the model achieved a precision of 95.2%, a recall of 88.2%, an F1 score of 91.5%, and a mean intersection over union (mIoU) of 84.8%, thereby demonstrating its effectiveness and reliability for crack segmentation tasks.

EMBANet: A flexible efficient multi-branch attention network.