Deep learning for photoacoustic image reconstruction from incomplete data (Conference Presentation)

Image reconstruction with under-sampled data is usually achieved by an iterative algorithm, which minimizes an objective function. The objective function commonly contains a data fidelity term and one or more Bayesian terms. A popular Bayesian term is the total variation (TV) norm of the image. This paper suggests an addition Bayesian term that is generated by a neural network. This neural network is essentially a classifier. This classifier can recognize the artifacts caused by the incomplete data. This classifier is trained by patient images reconstructed by complete and incomplete data sets. This newly introduced Bayesian term is referred to as the CNN score, which is a real number in (-∞, ∞). Patient studies show the good correlation between the CNN score and the severeness of the artifacts due to the incomplete measurements. A neural network can extract features from images that are suffering from incomplete measurements and convert the features into a CNN score. An iterative image reconstruction algorithm can be developed to minimize this CNN score to suppress the artifacts in the reconstructed image.

Deep learning (also known as deep structured learning, hierarchical learning, or deep machine learning) is a branch of machine learning based on a set of algorithms that attempt to model high-level abstractions in data by using a deep graph with multiple processing layers, composed of multiple linear and nonlinear transformations. Deep learning has been characterized as a class of machine learning algorithms with the following characteristics [257]: • They use a cascade of many layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input. The algorithms may be supervised or unsupervised and applications include pattern analysis (unsupervised) and classification (supervised). • They are based on the (unsupervised) learning of multiple levels of features or representations of the data. Higher-level features are derived from lower-level features to form a hierarchical representation. • They are part of the broader machine learning field of learning representations of data. • They learn multiple levels of representations that correspond to different levels of abstraction; the levels form a hierarchy of concepts. These definitions have in common: multiple layers of nonlinear processing units and the supervised or unsupervised learning of feature representations in each layer, with the layers forming a hierarchy from low-level to high-level features. Various deep learning architectures such as deep neural networks, convolutional deep neural networks, deep belief networks (DBN), and recurrent neural networks have been applied to fields like computer vision, automatic speech recognition, natural language processing, audio recognition, and bioinformatics where they have been shown to produce state-of-the-art results on various tasks. In this chapter we start in Section 7.1 with an introduction, giving a brief history of this field, the relevant literature, and its applications. Then we study some basic concepts of deep learning such as convolutional neural networks, recurrent neural networks, backpropagation algorithm, restricted Boltzmann machines, and deep learning networks in Section 7.2. Then we illustrate three examples for Apache Spark implementation for mobile big data (MBD), user moving pattern extraction, and combination with nonparametric Bayesian learning, respectively, in Sections 7.3 through 7.5. Finally, we have summary in Section 7.6.

Magnetic resonance imaging (MRI) acceleration is usually achieved by data undersampling, while reconstruction from undersampled data is a challenging ill-posed problem for data-missing and noisy measurements introduce various artifacts. In recent years, deep learning methods have been extensively studied for MRI reconstruction, and most of work treat the reconstruction problem as a denoising problem or replace the regularization subproblem with a deep neural network (DNN) in an optimization unrolling scheme. In this work, we proposed to directly complete the missing and corrupted k-space data by a specially designed interpolation deep neural networks combined with some convolution layers in both frequency and spatial domains. Specifically, for every missing and corrupted frequency, we use a \(K-\) nearest neighbors estimation with learnable weights. Then, two convolution neural networks (CNNs) are applied to regularize the data in both k-space and image space. The proposed DNN structures have clear interpretability for solving this undersampling problem. Extensive experiments on MRI reconstruction with diverse sampling patterns and ratios, under noiseless and noise settings demonstrate the accuracy of the proposed method compared to other learning based algorithms, while being computationally more efficient for both training and reconstruction processes.KeywordsMRI reconstructionDeep learningCompressed sensingLearnable interpolation

Photoacoustic imaging is a new imaging technology in recent years, which combines the advantages of high resolution and rich contrast of optical imaging with the advantages of high penetration depth of acoustic imaging. Photoacoustic imaging has been widely used in biomedical fields, such as brain imaging, tumor detection and so on. The signal-tonoise ratio (SNR) of image signals in photoacoustic imaging is generally low due to the limitation of laser pulse energy, electromagnetic interference in the external environment and system noise. In order to solve the problem of low SNR of photoacoustic images, we use feedforward denoising convolutional neural network to further process the obtained images, so as to obtain higher SNR images and improve image quality. We use Python language to manage the referenced Python external library through Anaconda, and build a feedforward noise-reducing convolutional neural network on Pycharm platform. We first processed and segmated a training set containing 400 images, and then used it for network training. Finally, we tested it with a series of cerebrovascular photoacoustic microscopy images. The results show that the peak signal-to-noise ratio (PSNR) of the image increases significantly before and after denoising. The experimental results verify that the feed-forward noise reduction convolutional neural network can effectively improve the quality of photoacoustic microscopic images, which provides a good foundation for the subsequent biomedical research.

1 - Congruence of deep learning in biomedical engineering: future prospects and challenges

Photoacoustic tomography involves absorption of pulsed light and subsequent generation of ultrasound, which when detected using an array of sensors can produce clinically useful images. Practical considerations limit the number of sensors and their “view” of the region of interest (ROI), which can result in significant reconstruction artifacts. Iterative-reconstruction methods can improve image quality but are computationally expensive. Another approach to improve reconstructed images is to use convolution neural networks (CNN) as a post-processing step for removing artifacts. However, missing or heavily obscured features typically cannot be recovered using this approach. We present a new pixel-wise deep learning (PDL) approach that employs pixel-wise interpolation to window ROI-specific raw photoacoustic data and then directly performs the image reconstruction within the CNN framework. The utility of this approach was demonstrated on simulated photoacoustic data from a 64-element semi-circular sensor array. The training and testing datasets comprised of 500 images from a synthetic vasculature phantom and 50 images of an anatomically realistic vasculature obtained from micro-CT images, respectively. The structural similarity index of the PDL-reconstructed images (0.91 ± 0.03) indicated superior image quality compared to those obtained using the iterative reconstruction (0.82 ± 0.09) and CNN-based artifact removal (0.79 ± 0.07).

Reconstructing magnetic resonance images from undersampled k-space data is a challenging problem. This paper introduces a novel method of image reconstruction from undersampled k-space data based on the concept of singularizing operators and a novel singular k-space model. Exploring the sparsity of an image in the k-space, the singular k-space model (SKM) is proposed in terms of the k-space functions of a singularizing operator. The singularizing operator is constructed by combining basic difference operators. An algorithm is developed to reliably estimate the model parameters from undersampled k-space data. The estimated parameters are then used to recover the missing k-space data through the model, subsequently achieving high-quality reconstruction of the image using inverse Fourier transform. Experiments on physical phantom and real brain MR images have shown that the proposed SKM method constantly outperforms the popular total variation (TV) and the classical zero-filling (ZF) methods regardless of the undersampling rates, the noise levels, and the image structures. For the same objective quality of the reconstructed images, the proposed method requires much less k-space data than the TV method. The SKM method is an effective method for fast MRI reconstruction from the undersampled k-space data. Graphical abstract Two Real Images and their sparsified images by singularizing operator.

Deep-learning-based image reconstruction and noise reduction methods (DLIR) have been increasingly deployed in clinical CT. Accurate image quality assessment of these methods is challenging as the performance measured using physical phantoms may not represent the true performance of DLIR in patients since DLIR is trained mostly on patient images. In this work, we aim to develop a patient-data-based virtual imaging trial framework and, as a first application, use it to measure the spatial resolution properties of a DLIR method. The patient-data-based virtual imaging trial framework consists of five steps: (1) insertion of lesions into projection domain data using the acquisition geometry of the patient exam to simulate different lesion characteristics; (2) insertion of noise into projection domain data using a realistic photon statistical model of the CT system to simulate different dose levels; (3) creation of DLIR-processed images from projection or image data; (4) creation of ensembles of DLIR-processed patient images from a large number of noise and lesion realizations; and (5) evaluation of image quality using ensemble DLIR images. This framework was applied to measure the spatial resolution of a ResNet based deep convolutional neural network (DCNN) trained on patient images. Lesions in a cylindrical shape and different contrast levels (-500, -100, -50, -20, -10 HU) were inserted to the lower right lobe of the liver in a patient case. Multiple dose levels were simulated (50%, 25%, 12.5%). Each lesion and dose condition had 600 noise realizations. Multiple reconstruction and denoising methods were used on all the noise realizations, including the original filtered-backprojection (FBP), iterative reconstruction (IR), and the DCNN method with three different strength setting (DCNN-weak, DCNN-medium, and DCNN-strong). Mean lesion signal was calculated by performing ensemble averaging of all the noise realizations for each lesion and dose condition and then subtracting the lesion-present images from the lesion absent images. Modulation transfer functions (MTFs) both in-plane and along the z-axis were calculated based on the mean lesion signals. The standard deviations of MTFs at each condition were estimated with bootstrapping: randomly sampling (with replacement) all the DLIR/FBP/IR images from the ensemble data (600 samples) at each condition. The impact of varying lesion contrast, dose levels, and denoising strengths were evaluated. Statistical analysis with paired t-test was used to compare the z-axis and in-plane spatial resolution of five algorithms for five different contrasts and three dose levels. The in-plane and z-axis spatial resolution degradation of DCNN becomes more severe as the contrast or radiation dose decreased, or DCNN denoising strength increased. In comparison with FBP, a 59.5% and 4.1% reduction of in-plane and z-axis MTF (in terms of spatial frequencies at 50% MTF), respectively, was observed at low contrast (-10 HU) for DCNN with the highest denoising strength at 25% routine dose level. When the dose level reduces from 50% to 12.5% of routine dose, the in-plane and z-axis MTFs reduces from 92.1% to 76.3%, and from 98.9% to 95.5%, respectively, at contrast of -100 HU, using FBP as the reference. For most conditions of contrasts and dose levels, significant differences were found among the five algorithms, with the following relationship in both in-plane and cross-plane spatial resolution: FBP>DCNN-Weak>IR>DCNN-Medium>DCNN-Strong. The spatial resolution difference among algorithms decreases at higher contrast or dose levels. A patient-data-based virtual imaging trial framework was developed and applied to measuring the spatial resolution properties of a DCNN noise reduction method at different contrast and dose levels using real patient data. As with other non-linear image reconstruction and post-processing techniques, the evaluated DCNN method degraded the in-plane and z-axis spatial resolution at lower contrast levels, lower radiation dose, and higher denoising strength.

Deep learning (DL) has shown explosive growth in its application to bioinformatics and has demonstrated thrillingly promising power to mine the complex relationship hidden in large-scale biological and biomedical data. A number of comprehensive reviews have been published on such applications, ranging from high-level reviews with future perspectives to those mainly serving as tutorials. These reviews have provided an excellent introduction to and guideline for applications of DL in bioinformatics, covering multiple types of machine learning (ML) problems, different DL architectures, and ranges of biological/biomedical problems. However, most of these reviews have focused on previous research, whereas current trends in the principled DL field and perspectives on their future developments and potential new applications to biology and biomedicine are still scarce. We will focus on modern DL, the ongoing trends and future directions of the principled DL field, and postulate new and major applications in bioinformatics.

An approach to Magnetic Resonance (MR) image reconstruction from undersampled data is proposed. Undersampling artifacts are removed using an iterative thresholding algorithm applied to nonlinearly transformed image block arrays. Each block array is transformed using kernel principal component analysis where the contribution of each image block to the transform depends in a nonlinear fashion on the distance to other image blocks. Elimination of undersampling artifacts is achieved by conventional principal component analysis in the nonlinear transform domain, projection onto the main components and back-mapping into the image domain. Iterative image reconstruction is performed by interleaving the proposed undersampling artifact removal step and gradient updates enforcing consistency with acquired k-space data. The algorithm is evaluated using retrospectively undersampled MR cardiac cine data and compared to k-t SPARSE-SENSE, block matching with spatial Fourier filtering and k-t ℓ1-SPIRiT reconstruction. Evaluation of image quality and root-mean-squared-error (RMSE) reveal improved image reconstruction for up to 8-fold undersampled data with the proposed approach relative to k-t SPARSE-SENSE, block matching with spatial Fourier filtering and k-t ℓ1-SPIRiT. In conclusion, block matching and kernel methods can be used for effective removal of undersampling artifacts in MR image reconstruction and outperform methods using standard compressed sensing and ℓ1-regularized parallel imaging methods.

The study presents the results of the development and testing of deep learning neural network architectures, which demonstrate high accuracy rates in classifying neurophysiological data, in particular hemodynamic brain activation patterns obtained by functional near-infrared spectroscopy, during solving mathematical problems on spatial-numerical associations. The analyzed signal represents a multidimensional time series of oxyhemoglobin and deoxyhemoglobin dynamics. Taking the specificity of the fNIRS signal into account, a comparative analysis of 2 types of neural network architectures was carried out: (1) architectures based on recurrent neural networks: recurrent neural network with long short-term memory, recurrent neural network with long short-term memory with fully connected layers, bidirectional recurrent neural network with long short-term memory, convolutional recurrent neural network with long short-term memory; (2) architectures based on convolutional neural networks with 1D convolutions: convolutional neural network, fully convolutional neural network, residual neural network. Trained long short-term memory recurrent neural network architectures showed worse results in accuracy in comparison with 1D convolutional neural network architectures. Residual neural network (model_Resnet) demonstrated the highest accuracy rates in three experimental conditions more than 88% in detecting age-related differences in brain activation during spatial-numerical association tasks considering the individual characteristics of the respondents’ signal.

Inspired by recent advances in deep learning, we propose a framework for reconstructing dynamic sequences of 2-D cardiac magnetic resonance (MR) images from undersampled data using a deep cascade of convolutional neural networks (CNNs) to accelerate the data acquisition process. In particular, we address the case where data are acquired using aggressive Cartesian undersampling. First, we show that when each 2-D image frame is reconstructed independently, the proposed method outperforms state-of-the-art 2-D compressed sensing approaches, such as dictionary learning-based MR image reconstruction, in terms of reconstruction error and reconstruction speed. Second, when reconstructing the frames of the sequences jointly, we demonstrate that CNNs can learn spatio-temporal correlations efficiently by combining convolution and data sharing approaches. We show that the proposed method consistently outperforms state-of-the-art methods and is capable of preserving anatomical structure more faithfully up to 11-fold undersampling. Moreover, reconstruction is very fast: each complete dynamic sequence can be reconstructed in less than 10 s and, for the 2-D case, each image frame can be reconstructed in 23 ms, enabling real-time applications.

This research paper explores the rising interest in utilising deep learning methods, specifically Convolution Neural Network (CNN), to improve the reconstruction of Compressively Sampled Magnetic Resonance Imaging (CS MRI) images from under-sampled data. By training deep learning architectures on extensive data sets of paired under sampled and fully sampled MR images, these models aim to capture intricate patterns and structures, ultimately enhancing the accuracy of MR image reconstructions. A novel deep learning model is proposed dubbed as “Deep Iterative Shrinkage Thresholding Algorithm-Compressed Sensing Network” (DISTA-CSNet), specifically designed for efficient recovery of CS MRI. Our model showcases impressive results with only 20 epochs and can be effortlessly trained on diverse datasets. To ensure robustness across different datasets, the dropouts are incorporated into the model, evident from testing results. The trained DISTA-CSNet exhibits remarkable performance in recovering CS MRI from various data sets, surpassing several advanced deep learning techniques with changing compression ratios consistently. The proposed model demonstrates significant enhancements in both Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) metrics, affirming the efficacy of our proposed model. The capability of our DISTA-CSNet in accurately reconstructing CS MRI images from 5-fold undersampled data shows promise in improving medical imaging applications and advancing the field of compressed sensing MRI.

Deep Learning and Beyond: AI’s New Horizons” is an expansive monograph that delves into the dynamic and rapidly evolving field of deep learning, a key area of artificial intelligence. This comprehensive work provides an insightful exploration of the revolutionary developments, foundational concepts, and future directions of deep learning, positioning itself as a crucial resource for understanding this transformative technology. The monograph commences with an introductory chapter that lays the groundwork by elucidating the background of machine learning and chronicling the ascent of deep learning. This foundational perspective is vital for comprehending the underlying principles and methodologies that have propelled deep learning to the forefront of technological innovation. Subsequent chapters delve into the “Historical Evolution” of AI, tracing the origins of neural networks from their early inception to the Renaissance period of deep learning. This historical analysis is pivotal in understanding the field’s trajectory and significant milestones. The “Foundations of Deep Learning” chapter presents an in-depth analysis of neural networks, covering their basic architecture, activation functions, layers, and the critical process of backpropagation and gradient descent. This is followed by examining “Advanced Neural Network Architectures,” where the focus shifts to state-of-the-art models such as convolutional neural networks, recurrent neural networks, and groundbreaking transformers and attention mechanisms. In addressing the “Training Deep Models,” the monograph explores regularization techniques, optimization strategies, and the increasingly significant role of transfer learning and pre-trained models. The application-oriented chapter, “Applications of Deep Learning,” showcases real-world implementations in domains like image recognition, natural language viii processing, and autonomous systems, illustrating deep learning technologies’ vast potential and versatility. The monograph also tackles the “Challenges in Deep Learning,” including overfitting, interpretability, and the ethical considerations surrounding AI deployment. It then ventures into the “Future of Deep Learning,” contemplating emerging trends such as integrating quantum computing, neural architecture search, and the convergence of neuroscience and deep learning. “Case Studies” provide practical insights and empirical evidence of the impact of deep learning. At the same time, the concluding chapter synthesizes the current state of deep learning, offers predictions for the future, and provides reflective thoughts on this transformative field. “Deep Learning and Beyond: AI’s New Horizons” educates and informs readers about the vast potential and future possibilities of deep learning and artificial intelligence. Keywords: Deep Learning, Artificial Intelligence, Neural Networks, Machine Learning, Convolutional Neural Networks, Recurrent Neural Networks, Transformers, Quantum Computing, Ethical AI, Neural Architecture Search, Natural Language Processing, Autonomous Systems, AI in Healthcare, Future of AI, AI Applications, AI Challenges.

paper, we focus on exploring the relationship between weave patterns and their mechanical properties in woven fiber composites through Machine Learning. Specifically, we explore the interactions between woven architectures and in-plane stiffness properties through Deep Convolutional Neural Network (DCNN) and Generative Adversarial Network (GAN). Our research is important for exploring how woven composite’s pattern is related to its mechanical properties and accelerating woven composite design as well as optimization. We focus on two tasks: (1) Stiffness prediction: Predicting in-plane stiffness properties for given weave patterns. Our DCNN extracts high-level features through several convolutional and fully connected layers to determine the final predictions. (2) Weave pattern prediction: Predicting weave patterns for target stiffness properties, which can be treated as the reverse task of the first one. Due to many-to-one mapping between weave patterns and the composite properties, we utilize a Decoder Neural Network as our baseline model and compare its performance with GAN and Genetic Algorithm. We represent the weave patterns as 2D checkerboard models and use finite element analysis (FEA) to determine in-plane stiffness properties, which serve as input data for our ML framework. We show that: (1) for stiffness prediction, DCNN can predict stiffness values for a given weave pattern with relatively high accuracy (above 93%); (2) for weave pattern prediction, the GAN model gives the best prediction accuracy (above 92%) while Decoder Neural Network has the best time efficiency. HAOTIAN FENG