Automatic Segmentation of Bulk Material Heaps Using Color, Texture, and Topography from Aerial Data and Deep Learning-Based Computer Vision

Deep learning based computer vision approaches for smart agricultural applications

This is the first of a two-part lesson introducing deep learning based computer vision methods for humanities research. Using a dataset of historical newspaper advertisements and the fastai Python library, the lesson walks through the pipeline of training a computer vision model to perform image classification.

Stock Watch

Computer vision algorithms can quickly analyze numerous images and identify useful information with high accuracy. Recently, computer vision has been used to identify 2D materials in microscope images. 2D materials have important fundamental properties allowing for their use in many potential applications, including many in quantum information science and engineering. One such material is hexagonal boron nitride (hBN), an isomorph of graphene with a very indistinguishable layered structure. In order to use these materials for research and product development, the most effective method is mechanical exfoliation where single-layer 2D crystallites must be prepared through an exfoliation procedure and then identified using reflected light optical microscopy. Performing these searches manually is a time-consuming and tedious task. Deploying deep learning-based computer vision algorithms for 2D material search can automate the flake detection task with minimal need for human intervention. In this work, we have implemented a new deep learning pipeline to classify crystallites of hBN based on coarse thickness classifications in reflected-light optical micrographs. We have used DetectoRS as the object detector and trained it on 177 images containing hexagonal boron nitride (hBN) flakes of varying thickness. The trained model achieved a high detection accuracy for the rare category of thin flakes (<50 atomic layers thick). Further analysis shows that our proposed pipeline could be generalized to various microscope settings and is robust against changes in color or substrate background.

Deep learning-based detection of depression by fusing auditory, visual and textual clues.

InfraGAN: A GAN architecture to transfer visible images to infrared domain

GUI design is an integral part of software development. The process of designing a mobile application typically starts with the ideation and inspiration search from existing designs. However, existing information-retrieval based, and database-query based methods cannot efficiently gain inspirations in three requirements: design practicality, design granularity and design knowledge discovery. In this paper we propose a web application, called Gallery D.C. that aims to facilitate the process of user interface design through real world GUI component search. Gallery D.C. indexes GUI compo-nent designs using reverse engineering and deep learning based computer vision techniques on millions of real world applications. To perform an advanced design search and knowledge discovery, our approach extracts information about size, color, component type, and text information to help designers explore multi-faceted design space and distill higher-order of design knowledge. Gallery D.C. is well received via an informal evaluation with 7 professional designers.Web Link: http://mui-collection.herokuapp.com/.Demo Video Link: https://youtu.be/zVmsz_wY5OQ.

This paper demonstrates that the instrumented indentation test (IIT), together with a trained artificial neural network (ANN), has the capability to characterize the mechanical properties of the local parts of a welded steel structure such as a weld nugget or heat affected zone. Aside from force-indentation depth curves generated from the IIT, the profile of the indented surface deformed after the indentation test also has a strong correlation with the materials’ plastic behavior. The profile of the indented surface was used as the training dataset to design an ANN to determine the material parameters of the welded zones. The deformation of the indented surface in three dimensions shown in images were analyzed with the computer vision algorithms and the obtained data were employed to train the ANN for the characterization of the mechanical properties. Moreover, this method was applied to the images taken with a simple light microscope from the surface of a specimen. Therefore, it is possible to quantify the mechanical properties of the automotive steels with the four independent methods: (1) force-indentation depth curve; (2) profile of the indented surface; (3) analyzing of the 3D-measurement image; and (4) evaluation of the images taken by a simple light microscope. The results show that there is a very good agreement between the material parameters obtained from the trained ANN and the experimental uniaxial tensile test. The results present that the mechanical properties of an unknown steel can be determined by only analyzing the images taken from its surface after pushing a simple indenter into its surface.

Deep learning based computer vision models like convolutional neural networks (CNN) and Vision Transformer (ViT) are more and more applied for the automatic analysis of sonar images. Since sonar image datasets typically have a limited number of samples, transfer-learning is used to train these models. However, commonly used pre-training datasets, like ImageNet, have a large domain gap to sonar images, i.e., images in these two datasets are fundamentally different. The selection of the pre-training dataset and the related domain gap have shown to have an impact on the final performance of the model. In this work, different datasets are analysed for applying transfer-learning to deep learning models for the classification of sidescan sonar images. In addition, the study is conducted for shallow CNNs, deeper CNNs as well as ViT. We quantify the domain gap using a variational autoencoder (VAE) and the t-distributed stochastic neighbor embedding t-SNE and link these values to the classification performance of the models after fine-tuning. Our results show that while no dataset leads to an improvement of all models, the Fetal dataset works well for most investigated models, while ImageNet and its grayscaled version led to a worse performance.

Cost-effective ecological monitoring in shallow waters using amphibious unmanned aerial vehicles (AUAV) and deep learning-based computer vision.

Background. High-throughput screening of immunotherapies is crucial for identifying promising candidates against various cancer indications and accelerating the pre-clinical development of cell therapies. Two technologies have shown promise in quantifying therapy effectiveness: patient-derived organoid (PDO) models co-cultured with cell therapies, and deep learning-based computer vision, which facilitates large-scale, automated image analysis to extract quantitative metrics of treatment efficacy and cell biology effects. Here, we use deep networks to predict PDO viability from brightfield images and compute explainable features which are used to stratify a large cohort of cell therapies, tested on multiple tumoroid lines and indications. Methods. Time-lapse confocal microscopy was used to record images of a large cohort of 27 cell therapies co-cultured with 15 different patient-derived tumor organoids (PDO) lines, across 8 cancer indications (breast, colorectal, endometrial, gastric, head and neck, liver, lung, pancreas) over the course of 72 hours, and at 2 different effector-to-target concentrations. PDO viability was measured by terminal TO-PRO-3 staining. A convolutional deep network was trained to perform label-free predictions of PDO viability from brightfield images at each timepoint (N=11, 110) and evaluated on a held-out test set (N=2, 112). Deep learning segmentation models were further used to co-localize tumoroid and immune cells, and extract a set of 6 explainable features - or spatial phenotypes - which quantified 1) tumoroid area, 2) cell apoptosis intensity and 3) temporal dynamics, as well as 4) immune proliferation, 5) immune infiltration and its 6) temporal dynamics. We fitted a linear model to measure their contribution in predicting terminal viability, and used hierarchical clustering to assess associations with exposure to different therapies. Results. Our bright-field model of viability was found to be highly concordant with ground truth viability from terminal vital dye stain on a held-out test set, representing 11 TO lines across 8 distinct cancer types (Pearson’s r=0.76). A linear model of spatial phenotypes showed high correlation with terminal viability (Pearson’s r=0.70), while clustering of spatial phenotypes stratified samples into groups of non-engineered and engineered therapies with and without an additional compound. Conclusion. Our study highlights that imaging-derived metrics from brightfield imaging can quantify treatment potency and cell phenotypes in ex-vivo screens. The approach pursued here is highly scalable and permits the analysis of large scale screening experiments to assess the impact of cell therapies on PDOs. Using explainable features further allows to stratify therapies into biologically meaningful groups based on the measured phenotypic readouts. Citation Format: Luca Lonini, Madhavi Kannan, Geoffrey Schau, Stanislaw Szydlo, Vignesh Krishnaraja, Sonal Khare, Daniel Rabe, Michael Streit, Mary Flaherty, Brandon Mapes, Mark Prytyskach, Stefan Kiesgen, Gurpanna Saggu, Megan McAfee, Kate Sasser, Justin Guinney, Richard Klinghoffer, Chi-Sing Ho. Stratification of cell therapies in solid tumor organoids using deep learning-derived imaging metrics [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2025; Part 1 (Regular Abstracts); 2025 Apr 25-30; Chicago, IL. Philadelphia (PA): AACR; Cancer Res 2025;85(8_Suppl_1):Abstract nr 7446.

In many medical domains recent advances in deep learning-based computer vision have moved the model performance towards (or exceeding in some cases) the level of human experts. On object classification and localization tasks, state-of-the-art classifiers based on convolutional neural networks (CNNs) achieve great accuracy. According to its rapid spread to numerous areas of the body, skin cancer is the most irritating sort of cancer diagnosis. Melanoma is a fatal kind of skin cancer that, if found early enough, can be treated with simple surgery. Therefore, Dermatologists can treat patients and save their lives by accurately classifying skin lesions in their early stages. Doctors and patients might both profit greatly from quick and precise diagnosis. As a result, computer-assisted diagnostic support systems is required. Motivated by this, this paper introduce a proposed model for melanoma classification. ISIC 2020, the most recent well-known public challenge dataset, is utilized to test the suggested model's capacity to classify Melanoma. Moreover, with less than 2% malignant samples, the dataset is significantly unbalanced. We utilise random over-sampling and data augmentation for the handling of unbalanced data. In addition, class weight technique was used to provide a weight for each class, so that the result is a classification that can equally learn from all classes, which places greater focus on the minority class. The suggested model for melanoma classification uses EfficientNet-B6 which it deep Convolutional Neural Networks (CNN) for classifying patients’ skin lesions into malignant or benign. The results of the research demonstrated that the suggested 97.84% system accuracy ratio is competitive with other models.

Allergen30: Detecting Food Items with Possible Allergens Using Deep Learning-Based Computer Vision

Object detection and tracking has always been one of the important research directions in computer vision. The purpose is to determine whether the object is contained in the input image and enclose the object with a bounding box. However, most object detection and tracking methods are applied to daytime objects, and the processing of nighttime objects is imprecise. In this paper, a spectral-spatial feature enhancement algorithm for nighttime object detection and tracking is proposed, which is inspired by symmetrical neural networks. The proposed method consists of the following steps. First, preprocessing is performed on unlabeled nighttime images, including low-light enhancement, object detection, and dynamic programming. Second, object features for daytime and nighttime times are extracted and modulated with a domain-adaptive structure. Third, the Siamese network can make full use of daytime and nighttime object features, which is trained as a tracker by the above images. Fourth, the test set is subjected to feature enhancement and then input to the tracker to obtain the final detection and tracking results. The feature enhancement step includes low-light enhancement and Gabor filtering. The spatial-spectral features of the target are fully extracted in this step. The NAT2021 dataset is used in the experiments. Six methods are employed as comparisons. Multiple judgment indicators were used to analyze the research results. The experimental results show that the method achieves excellent detection and tracking performance.

The proportion of composite materials (such as CFRP) to the metal used on the modern aircrafts is rising, imposing different kind of failure modes. Since the composite structures are known to be sensitive to the impact loading, there is need for means to assess the sub-surface damage in the structure rapidly. Royal Netherlands Aerospace Centre NLR has an extensive track record on contactless non-destructive inspection (NDI) methods based on optical sensors, such as 3D surface scanning, lock-in thermography and laser shearography. The combination of these methods (multi-domain inspection) enables us to assess the structural integrity of an aircraft outer surface in a short time, reducing inspection costs and the “down time” of the aircraft. Recently, NLR is working towards a 3D oriented mesh environment of an object, enhanced with NDI data, providing sub-surface damage information. By automatically stitching the 3D object images, it is possible to expand this method for a complete scan of an aircraft surface. Furthermore, the thermographic and shearography imaging information has been integrated into the 3D surface accounting for the image distortion from the different measurement angles. In this paper, the results from the various studies will be presented involving integration of the 2D measurements with the 3D scan mesh. Keywords: Thermography, Shearography, 3D scanning, Data fusion