Convolutional Neural Network Training with Distributed K-FAC

Information extraction and coding of free-text pathology reports is an important activity for cancer registries to support national cancer surveillance. Cancer registrars must process high volumes of pathology reports on an annual basis. In this study, we investigated an automated approach using a coarse-to-fine training of convolutional neural networks (CNNs) for extracting the primary site, histological grade and laterality from unstructured cancer pathology text reports. Our proposed training scheme consists of two stages. In the first stage, the multi-task learning (MTL) with hard parameter sharing approach is used to train a multi-task MT-CNN model for all the tasks. Then, the TM-CNN model parameters are used to initialize a CNN model for each task to be fine trained individually using its corresponding dataset. The performance of our proposed approach was compared against a state-of-the-art CNN and the commonly used SVM classifier. We observed that the proposed model consistently outperformed the base line models, especially for the less prevalent classes. Specifically, the proposed training approach achieved a micro-F score of 0.7749 over 12 ICD-O-3 topography codes which is a significant improvement as compared with state-of-the-art CNN (0.7101) and the SVM (0.6019) classifiers. Also, the results demonstrate the potential of the proposed method for handling class imbalance within each task. It significantly improves macro-F score by 24% and 12% of the primary site and histology grade tasks, respectively.

Convolutional Neural Networks (CNNs) have been shown to have high accuracy for classification tasks in numerous applications, which has resulted in their widespread adoption. However, the high accuracy of CNNs comes at the cost of high compute and bandwidth requirements for both classification and training. In this work we discuss an FPGA-based CNN training engine: FCTE, implemented using High-Level Synthesis (HLS), targeting the Xilinx Kintex Ultrascale XCKU115 device. Furthermore, we detail custom-precision floating-point (CPFP) cores for multiplication and addition implemented using HLS, which allows for reduced area utilization. We use these cores with our engine to train networks to demonstrate that an exponent width of 6 and mantissa width of 5 achieves accuracy comparable to single-precision floating-point for the MNIST and CIFAR-10 datasets. These results are achieved using round-to-zero for the CPFP multipliers and round-to-nearest for the CPFP adders, allowing for LUT savings of 32.6% for the multipliers and 21.7% for the adders when compared to half-precision floating-point, while using the same number of DSPs.

Abstract. In recent years, there has been a significant improvement in the detection, identification and classification of objects and images using Convolutional Neural Networks. To study the potential of the Convolutional Neural Network, in this paper three approaches are investigated to train classifiers based on Convolutional Neural Networks. These approaches allow Convolutional Neural Networks to be trained on datasets containing only a few hundred training samples, which results in a successful classification. Two of these approaches are based on the concept of transfer learning. In the first approach features, created by a pretrained Convolutional Neural Network, are used for a classification using a support vector machine. In the second approach a pretrained Convolutional Neural Network gets fine-tuned on a different data set. The third approach includes the design and training for flat Convolutional Neural Networks from the scratch. The evaluation of the proposed approaches is based on a data set provided by the IEEE Geoscience and Remote Sensing Society (GRSS) which contains RGB and LiDAR data of an urban area. In this work it is shown that these Convolutional Neural Networks lead to classification results with high accuracy both on RGB and LiDAR data. Features which are derived by RGB data transferred into LiDAR data by transfer learning lead to better results in classification in contrast to RGB data. Using a neural network which contains fewer layers than common neural networks leads to the best classification results. In this framework, it can furthermore be shown that the practical application of LiDAR images results in a better data basis for classification of vehicles than the use of RGB images.

Successful training of convolutional neural networks (CNNs) requires a substantial amount of data. With small datasets, networks generalize poorly. Data Augmentation techniques improve the generalizability of neural networks by using existing training data more effectively. Standard data augmentation methods, however, produce limited plausible alternative data. Generative Adversarial Networks (GANs) have been utilized to generate new data and improve the performance of CNNs. Nevertheless, data augmentation techniques for training GANs are underexplored compared to CNNs. In this work, we propose a new GAN architecture for augmentation of chest X-rays for semi-supervised detection of pneumonia and COVID-19 using generative models. We show that the proposed GAN can be used to effectively augment data and improve classification accuracy of disease in chest X-rays for pneumonia and COVID-19. We compare our augmentation GAN model with Deep Convolutional GAN and traditional augmentation methods (rotate, zoom, etc.) on two different X-ray datasets and show our GAN-based augmentation method surpasses other augmentation methods for training a GAN in detecting anomalies in X-ray images.

Recently, the deep learning methods have achieved great success in pattern recognition tasks. Especially for character recognition, most of the state-of-the-art results belong to the deep learning models. Among those models, the convolutional neural network (CNN) becomes the most popular due to its outstanding performance. Therefore, many trials were made in order to make improvements on CNN. However, most of the trials only focused on the network structure or training skills, the inter-class information is usually ignored. In this paper, we have proposed a novel CNN model with two training feedbacks: the reconstruction feedback and the classification feedback. By using the reconstruction feedback, the inter-class information (for example, shape similarity) of the characters is taken into account. Consequently, without enlarging the network structure, our model can outperform those state-of-the-art improved CNN models, which is proved by the experimental results.

In the training process of convolutional neural networks (CNN), a batch normalization (BN) layer is often inserted after a convolution layer to accelerate the convergence of CNN training. In this work, we propose a BN processor that supports both the training and inference processes. To speed up the training of CNN, the proposed work develops an efficient dataflow integrating a novel BN processor design and the processing elements for convolution acceleration. We exploited the similarities in the calculations required for the BN forward and backward passes by sharing hardware elements between both passes, therefore reducing the area overhead. In addition to functional verification of the BN processor, we also completed Automatic Placement & Routing (APR) and conducted post-APR simulation on neural network training. Finally, the proposed solution not only significantly speeds up the CNN training process, but also achieves hardware saving.

To solve the problem of low accuracy of the traditional monitoring method, a deep learning based monitoring method is designed. A sensor is installed at the part that may cause the circuit breaker to run in non-full phase and the state data of the circuit breaker is collected through the sensor. Filter and digital-to-analog converter are used to reduce noise and digital-to-analog conversion of the signal transmitted by the sensor. After the data are standardized and normalized, input into the convolutional neural network. The parameters and weights of convolutional neural network are determined by training the sample data. After the training of convolutional neural network, the input neural network data are processed according to the determined parameters and weights. Output the processed results to the database, in the database classification algorithm after the data classification, sent to the circuit breaker operation control module, to achieve the circuit breaker operation status monitoring. Through the comparison with the traditional monitoring methods, it is verified that the designed deep learning-based state monitoring method has higher accuracy and can make up for the shortcomings of the traditional methods.

The article is devoted to the solution of the problem of developing a neural network system for calculating the yield of sunflower. The developed system is designed to simplify, automate and speed up the inspector's work on calculating the biological yield of sunflower. When assessing the biological yield, most of the time is taken by counting the seeds of a sunflower head. It is proposed to solve this problem using a trained convolutional neural network. The developed system allows you to add the name of a new calculation area, use phone cameras to obtain images of sunflowers (for mobile applications), add images to the system, search for and select the required subspecies of sunflower from the directory, obtain the values of the average weight of seeds of the selected subtype of sunflower, set the area of the determined sowing field, calculate the yield based on the added images, receive the result of calculating the biological yield of sunflower, and also receive previously calculated yield values by name from the device's memory and by the year of the assessment. The Python programming language was chosen to solve the problem of counting seeds in a sunflower head and developing a neural network system. The Roboflow system was used to create a sample with segmented images of sunflower heads. TenserFlow was used as a machine learning library. Keras and OpenCV libraries were used for segmentation and object recognition in the image. Google Colab was used as the development environment for the neural network system. The mobile application was implemented in Java using the Android Studio development environment. The desktop application was implemented in C# using the Visual Studio development environment on Windows Forms technology. The developed neural network system, based on the image transferred to the input of the neural network, detects a sunflower head, segments part of the image with the head, cutting out this area, then counts the seeds in the head and calculates the yield. The YOLOv8 convolutional neural network was used to segment the sunflower head. Using the Roboflow service, a data sample of 10,000 sunflower images was formed. Part of the data (70%) was used for training, and the rest of the data was used for testing the model. Testing of the neural network system showed its efficiency and the possibility of further practical use. Thus, the developed system allows accelerating the assessment of sunflower yield by automating this process.

In this study, a method of predicting the seismic responses of building structures based on a convolutional neural network (CNN) is proposed. In the method, the time histories of acceleration responses previously measured in a building during earthquakes are used in the CNN input layer, with the corresponding time histories of the displacement responses being used in the CNN output layer. The correlations between the features automatically extracted from the acceleration responses by the convolution and pooling operations in the various CNN layers and the displacement responses in the CNN output layer are established through CNN training. The trained CNN predicts the displacement responses for a future earthquake event using only the measured acceleration responses. To verify the validity of the proposed method, a numerical study on the ASCE benchmark model and an experimental study on a reinforced concrete frame structure are conducted. In the numerical study, the structural responses of the ASCE model subject to artificial earthquakes are utilized for CNN training, and the performance of the trained CNN is verified through seismic response prediction. In the experimental study, shaking table tests are performed for various earthquakes to measure the seismic responses of the reinforced concrete test structure. The seismic response prediction performance of the proposed method is examined using CNN trained with the measured seismic responses. Furthermore, the performances of CNNs trained with different numbers of datasets generated by the proposed data generation method based on data overlapping with the same data pool are discussed with related to the number of CNN training iteration number.

Large Format Additive Manufacturing (LFAM) enables the rapid production of composite components using short fiber-reinforced thermoplastics. This process offers high strength-to-weight ratios and efficient material deposition. However, the mechanical performance of printed parts is strongly dependent on fiber orientation. Reliable characterisation of orientation remains challenging due to the limitations of manual microscopy analysis. It is further hindered by the scarcity of large, annotated datasets required for deep learning inference (i.e., the automatic prediction of fiber orientation from image data using trained neural networks). This study addresses this gap by proposing a physics-informed methodology for the synthetic generation of microscopy-like images that replicate the microstructure of LFAM-processed composites. The simulation pipeline, implemented in Python, creates large-scale datasets by generating non-overlapping ellipsoidal fiber distributions with known geometrical and statistical parameters. Each synthetic image is paired with an analytically computed orientation tensor. This enables the supervised training of convolutional neural networks (CNNs) without the need for manual annotation. The system allows full control over fiber orientation, aspect ratio, and area fraction. It is calibrated using real micrographs obtained via X-ray Microscopy (XRM). The resulting synthetic images closely reproduce the statistical features of real microstructures and provide robust ground truth for model training and validation. Overall, this scalable approach enables rapid dataset generation and facilitates the development of accurate AI models for automated orientation analysis. It also supports future applications in digital twin frameworks, in-line quality control, and material characterisation in LFAM.

Person identification through chest X-ray radiographs stands as a vanguard in both healthcare and biometrical security domains.In contrast to traditional biometric modalities, such as facial recognition, fingerprints and iris scans, the research orientation towards chest X-ray recognition has been spurred by its remarkable recognition rates.Capturing the intricate anatomical nuances of an individual's rib cage, lungs and heart, chest X-ray images emerge as a focal point for identification, even in scenarios where the human body is entirely damaged.Concerning the field of deep learning, a paradigm is exemplified in contemporary generations, with promising outcomes in classification and image similarity challenges.However, the training of convolutional neural networks (CNNs) requires copious labelled data and is time-consuming.In this study, we delve into the rich repository of the NIH ChestX-ray14 dataset, comprising 112,120 frontal-view chest radiographs from 30,805 unique patients.Our methodology is nuanced, employing the potency of Siamese neural networks and the triplet loss in conjunction with refined CNN models for feature extraction.The Siamese networks facilitate robust image similarity comparison, while the triplet loss optimizes the embedding space, mitigating intraclass variations and amplifying inter-class distances.A meticulous examination of our experimental results reveals profound insights into our model performance.Noteworthy is the remarkable accuracy achieved by the VGG-19 model, standing at an impressive 97%.This achievement is underpinned by a well-balanced precision of 95.3% and an outstanding recall of 98.4%.Surpassing other CNN models utilized in our research and outshining existing state-of-the-art models, our approach establishes itself as a vanguard in the pursuit of person identification through chest X-ray images.

Glass forming ability prediction of bulk metallic glasses based on fused strategy

Machine learning algorithms for early fault detection of wind turbines using 10-minute SCADA data are attracting attention in the wind energy community due to their cost-effectiveness. It has been recently shown that convolutional neural networks (CNNs) can significantly improve the performance of such algorithms. One practical aspect in the deployment of these algorithms is that they require a large amount of historical SCADA data for training. These are not always available, for example in the case of newly installed turbines. Here we suggest a cross-turbine training scheme for CNNs: we train a CNN model on a turbine with abundant data and use the trained network to detect faults in a different wind turbine for which only little data are available. We show that this scheme is able to detect faults with an accuracy and robustness which are very similar to the single-turbine scheme, in which training and detection are both done on the same turbine. We demonstrate this for two different fault types: abrupt and slowly evolving faults and perform a sensitivity analysis in order to compare the performance of the two training schemes. We show that the scheme works successfully also when training on turbines from another farm and with different measured variables than the target turbine.

Deep learning allows us to automatize the acquisition of large amounts of behavioural animal data with applications for fisheries and aquaculture. In this work, we have trained an image-based deep learning algorithm, the Faster R-CNN (Faster region-based convolutional neural network), to automatically detect and track the gilthead seabream, Sparus aurata, to search for individual differences in behaviour. We collected videos using a novel Raspberry Pi high throughput recording system attached to individual experimental behavioural arenas. From the continuous recording during behavioural assays, we acquired and labelled a total of 14,000 images and used them, along with data augmentation techniques, to train the network. Then, we evaluated the performance of our network at different training levels, increasing the number of images and applying data augmentation. For every validation step, we processed more than 52,000 images, with and without the presence of the gilthead seabream, in normal and altered (i.e., after the introduction of a non-familiar object to test for explorative behaviour) behavioural arenas. The final and best version of the neural network, trained with all the images and with data augmentation, reached an accuracy of 92,79% ± 6.78% [89.24–96.34] of correct classification and 10.25 ± 61.59 pixels [6.59-13.91] of fish positioning error. Our recording system based on a Raspberry Pi and a trained convolutional neural network provides a valuable non-invasive tool to automatically track fish movements in experimental arenas and, using the trajectories obtained during behavioural tests, to assay behavioural types.

Objective: this work proposed the development of an application for identifying types of wound tissue, through images, using a convolutional neural network trained with photographs. Method: study of technological production of the prototyping type. In the first phase, image preparation, training and evaluation of the convolutional neural network were carried out. The images were collected from a database of medical records of people treated at a reference wound treatment clinic, in a city in the north of Minas Gerais. In the second stage, an application was developed responsible for acquiring the image (photo) of the wound through the mobile device's camera, resizing and cropping the image and sending it to the cloud service so that it can predict the tissue type classification to the user. present in the wound. Results: the trained model obtained an accuracy of 76%, considered a good result and the application for classifying tissue types in a wound will be available in the Play Store and Apple virtual stores. Final Considerations: the mobility provided by the application will allow healthcare professionals to respond quickly when classifying a wound, as the indication of appropriate treatment and the direction towards complete resolution of the injury depend on this.