Weakly-supervised learning to automatically count cotton flowers from aerial imagery

Plant Phenotyping is a set of methodologies for measuring and analyzing characteristic traits of a plant. While traditional plant phenotyping techniques are labor-intensive and destructive, modern imaging technologies have provided faster, non-invasive, and more cost-effective capabilities for plant phenotyping. Among different image-based phenotyping platforms, I focus on phenotyping with image data captured by Unmanned Aerial Vehicle (UAV) and ground vehicles. The crop plant used in my study is sorghum [Sorghum bicolor (L.) Moench]. In this thesis, I present multiple methods to estimate plot-level and plant-level plant traits from data collected by various platforms, including UAV and ground vehicles. I propose an image plant phenotyping system that provides end-to-end RGB data analysis for plant scientists. I describe a plant segmentation method using HSV color information. I introduce two methods to locate the center of the plants using Multiple Instance Learning (MIL) and Convolutional Neural Networks (CNN). I present three methods to segment individual leaves by shape-based approaches in both Cartesian coordinates and Polar coordinates. I propose a method to estimate leaf length and width for overhead leaf images. I describe a method to estimate leaf angle from data collected by a modified wheel-based sprayer with a sensor boom vehicle, Phenorover. Methods are tested and verified on image data collected by UAV and ground vehicle platforms in sorghum fields in West Lafayette, Indiana, USA. Estimated phenotypic traits include plant locations, the number of plants per plot, leaf area, canopy cover, Leaf Area Index (LAI), leaf count, leaf angle, leaf length, and leaf width.

Automatic computer-aided detection plays an important role in biomedical image analysis. Many studies have focused on weak supervised learning as annotation tasks are time-consuming and tedious. Compared with pixel-wise annotation by particular software on the scanned digital high-resolution images, an alternative method of marking out of suspicious regions on microscopy slides is significantly more convenient for pathologists. Additionally, with a focus on dysplasias in the central area, there is a high likelihood of the similar tissues to be found around in clusters. In this paper, for weak annotation on microscopy images, we propose an efficient Foveated Imaging based Multiple Instance Learning (FIMIL) framework to classify weakly-labeled microscopy images. The model also provides multi-scale algorithm for arbitrary image size, in which the patches with highest possibility to contain dysplasia are considered as ”fixation points” in the image. The developed model combines deep convolutional neural networks (CNNs) with multiple instance learning (MIL) for dysplasias detection with only image-level labeling. The benchmark tests are carried out on the marked regions of 40x magnified whole-slide cytology images and the normal/abnormal label and their corresponding possibilities are predicted. Evaluated on the real-life clinical data, our proposed model shows high accuracy and efficiency by weakly-supervised learning. 1

The computer aided diagnosis (CAD) problems of detecting potentially diseased structures from medical images are typically distinguished by the following challenging characteristics: extremely unbalanced data between negative and positive classes; stringent real-time requirement of online execution; multiple positive candidates generated for the same malignant structure that are highly correlated and spatially close to each other. To address all these problems, we propose a novel learning formulation to combine cascade classification and multiple instance learning (MIL) in a unified min-max framework, leading to a joint optimization problem which can be converted to a tractable quadratically constrained quadratic program and efficiently solved by block-coordinate optimization algorithms. We apply the proposed approach to the CAD problems of detecting pulmonary embolism and colon cancer from computed tomography images. Experimental results show that our approach significantly reduces the computational cost while yielding comparable detection accuracy to the current state-of-the-art MIL or cascaded classifiers. Although not specifically designed for balanced MIL problems, the proposed method achieves superior performance on balanced MIL benchmark data such as MUSK and image data sets.

We benchmarked histopathology foundation encoders paired with attention-based multiple instance learning (MIL) against convolutional neural networks (CNNs) to assess their robustness for endometrial cancer molecular classification (MMR-deficient, p53 aberrant, POLE pathogenic mutation, and no specific molecular profile) from whole-slide images (WSIs) in a real-world cohort. A public cohort of 815 patients (1195 WSIs) was assembled for model development. Generalizability was evaluated using an external cohort of 720 patients (1357 WSIs). Models were trained using five-fold cross-validation and tested on the external cohort. Performance was summarized using macro-area under the receiver operating characteristic curve (AUC), macro-F1 score, and balanced accuracy. In cross-validation, foundation encoder models outperformed CNNs (macro-AUC 0.799-0.860 vs 0.715-0.829). The best configuration (Virchow2 with CLAM MIL) achieved macro-AUC 0.860, macro-F1 score 0.607, and balanced accuracy 0.647. On external validation, CNN performance degraded substantially, whereas foundation models retained higher discrimination. UNI2 with CLAM MIL achieved the highest external macro-AUC 0.780 with a macro-F1 score of 0.416 and balanced accuracy of 0.507. Subtype-level performance was highest for p53abn (AUC 0.851). When evaluated within a benchmarking framework, foundation encoders paired with attention-based MIL demonstrate improved generalization for endometrial cancer molecular subtyping from WSIs compared with CNNs, supporting their potential for subtype inference.

PurposeTo evaluate whether open-source histopathology foundation model pipelines, paired with attention-based multiple instance learning (MIL), can accurately classify molecular subtypes of endometrial cancer (EC) from whole-slide images (WSIs) and maintain performance in a real-world, independent cohort.MethodsWe assembled a public discovery cohort of 815 patients (1,195 WSIs) from The Cancer Genome Atlas and Clinical Proteomic Tumor Analysis Consortium, and an independent external cohort of 720 patients (1,357 WSIs) with molecular subtyping determined by mismatch repair immunohistochemistry plus TP53 and POLE sequencing. Four ImageNet-pretrained convolutional neural networks (CNNs) and six open-source foundation encoders using two MIL aggregation strategies (TransMIL and CLAM) were benchmarked within the STAMP pipeline. Models were trained with five-fold cross-validation and evaluated on an independent cohort. Macro–area under the receiver operating characteristic curve (AUC) was the primary outcome.ResultsIn cross-validation, foundation models outperformed CNNs (macro-AUC 0.799–0.860 vs 0.715–0.829). The best configuration (Virchow2 with CLAM) achieved macro-AUC 0.860 (95%CI, 0.839–0.880), macro-F1 score 0.607, and balanced accuracy 0.647. External validation showed substantial degradation for CNNs, while foundation models retained higher discrimination (macro-AUC 0.667–0.780). UNI2 with CLAM had the highest external macro-AUC (0.780), and Virchow2 with CLAM had the best balanced accuracy (0.525). Subtype-level AUCs for UNI2 with CLAM were highest for p53abn (0.851).ConclusionsOpen-source foundation model pipelines with attention-based MIL can deliver accurate and generalizable molecular subtyping of EC directly from WSIs. These models outperform CNNs in real-world validation, supporting their potential as scalable, cost-effective tools to guide precision oncology and triage confirmatory molecular testing.

Breast cancer has the highest mortality among cancers in women. Computer-aided pathology to analyze microscopic histopathology images for diagnosis with an increasing number of breast cancer patients can bring the cost and delays of diagnosis down. Deep learning in histopathology has attracted attention over the last decade of achieving state-of-the-art performance in classification and localization tasks. The convolutional neural network, a deep learning framework, provides remarkable results in tissue images analysis, but lacks in providing interpretation and reasoning behind the decisions. We aim to provide a better interpretation of classification results by providing localization on microscopic histopathology images. We frame the image classification problem as weakly supervised multiple instance learning problem where an image is collection of patches i.e. instances. Attention-based multiple instance learning (A-MIL) learns attention on the patches from the image to localize the malignant and normal regions in an image and use them to classify the image. We present classification and localization results on two publicly available BreakHIS and BACH dataset. The classification and visualization results are compared with other recent techniques. The proposed method achieves better localization results without compromising classification accuracy.

Multiple instance learning (MIL) has shown prominent success in analyzing whole slide histopathology images (WSIs). However, existing MIL methods often suffer from overfitting due to weak supervision and the "needle-in-a-haystack" nature of WSIs. Additionally, most deterministic approaches lack a mechanism for uncertainty quantification. While Bayesian neural networks (BNNs) have emerged as a promising solution to mitigate overfitting and enable uncertainty estimation by imposing prior constraints, commonly used Gaussian BNNs exhibit unstable posterior predictive distributions under weak supervision and suffer from high prediction variance. To tackle these challenges, we propose a sparsity-induced Bayesian Neural Network to be adopted in the MIL scheme, named SIB-MIL, for robust WSI prediction. Instead of using Gaussian prior distributions, we place a sparsity-induced prior, the Horseshoe prior, on the BNN parameters to address the variance overflowing issue. Such sparsity also filters unimportant noise and highlights salient regions, which only occupy a small proportion in WSIs. Empirical evaluations on cancer classification and subtyping tasks corroborate that not only can our method improve the existing MIL networks, but it also performs well in uncertainty quantification. Codes are available at https://github.com/HKU-MedAI/SIB-MIL.

Unmanned aerial vehicles (UAV) are now used in a large number of applications. In order to accomplish autonomous navigation, UAVs must be equipped with robust and accurate localization systems. Most localization solutions available today rely on global navigation satellite systems (GNSS). However, such systems are known to introduce instabilities as a result of interference. More advanced solutions now use computer vision. While deep learning has now become the state-of-the-art in many areas, few attempts were made to use it for localization. In this paper, we present an entirely new type of approach based on convolutional neural networks (CNN). The network is trained with a new purpose-built dataset constructed using publicly available aerial imagery. Features extracted with the model are integrated in a particle filter for localization. Initial validation using real-world data, indicated that the approach is able to accurately estimate the localization of a quadcopter.

The study aimed to determine the efficacy and capabilities of using high-resolution aerial imagery and a convolutional neural network (CNN) to identify plant species and monitor land cover and land change in the context of remote sensing. The full capabilities of a CNN were examined, including testing whether the platform could be used for land cover and the evaluation of land change over time. An unmanned aerial vehicle (UAV) was used to collect the aerial data of the study area. The CNN was encoded and operated in RStudio, while digitised data from the input imagery were used by the programme as training and validation data. The object in this respect was to learn about the relevant features of the landscape, and thereafter to classify the Opuntia invasive plant species. Accuracy assessments were carried out on the results to test the efficacy of the aerial imagery in terms of its accuracy and reliability. The classification achieved an overall accuracy of 93%, while the kappa coefficient score was 0.86. CNN was also able to predict the land coverage area of Opuntia to be within four percent (4%) of the ground truthing data. A change in land cover over time was detected by the programme after the manual clearing of the plant had been undertaken. This research has determined that the use of a CNN in remote sensing is a very powerful tool for supervised image classifications. It can be used for monitoring land cover in that it is able to accurately estimate the spatial distribution of plant species and to monitor the growth or decline in the species over time. As such, it is an efficient methodology and its use in remote sensing could be extended.

Histopathological Whole Slide Imaging (WSI) has become a standard in the detection of breast cancer. Automated image analysis methods attempt to reduce the workload from the clinicians and Convolutional Neural Networks (CNNs) are a popular choice for this purpose. However, size of a WSI image typically is approximately <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$40,000\times 40.000$ </tex-math></inline-formula> pixels (can reach up to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$100,000\times 100.000$ </tex-math></inline-formula> pixels). CNNs cannot handle such large images. Moreover, downscaling a WSI image causes degradation of small-scale visual information. Hence, a large number of small patches (containing critical visual information) from a WSI image are extracted by a trained pathologist and are used for training. However, it requires massive amounts of time to precisely search and label appropriate class-representative patches. To address this issue, a Deep Multiple Instance Learning (MIL) based CNN framework has been introduced in this paper. In the proposed framework every slide is represented as a bag of extracted patches. Only the bag label is used for training, thus eliminating the requirement to provide patchwise labels. The patches inherit the label of the bag containing them. A WSI image (i.e. a bag) is labeled benign if all its patches are benign and labeled malignant even if a single patch contains malignant cells. Learning can be carried out at the bag level even with noisy patch labels. Performance of this method was evaluated using the BreakHis, IUPHL and UCSB breast cancer datasets where 93.06%, 96.63%, 95.83% accuracy was achieved respectively.

The detection of human action in videos of busy natural scenes with dynamic background is of interest for applications such as video surveillance. Taking a conventional fully supervised approach, the spatio-temporal locations of the action of interest have to be manually annotated frame by frame in the training videos, which is tedious and unreliable. In this paper, for the first time, a weakly supervised action detection method is proposed which only requires binary labels of the videos indicating the presence of the action of interest. Given a training set of binary labelled videos, the weakly supervised learning (WSL) problem is recast as a multiple instance learning (MIL) problem. A novel MIL algorithm is developed which differs from the existing MIL algorithms in that it locates the action of interest spatially and temporally by globally optimising both interand intra-class distance. We demonstrate through experiments that our WSL approach can achieve comparable detection performance to a fully supervised learning approach, and that the proposed MIL algorithm significantly outperforms the existing ones.

The application of deep learning, specifically deep convolutional neural networks (DCNNs), to the classification of remotely-sensed imagery of natural landscapes has the potential to greatly assist in the analysis and interpretation of geomorphic processes. However, the general usefulness of deep learning applied to conventional photographic imagery at a landscape scale is, at yet, largely unproven. If DCNN-based image classification is to gain wider application and acceptance within the geoscience community, demonstrable successes need to be coupled with accessible tools to retrain deep neural networks to discriminate landforms and land uses in landscape imagery. Here, we present an efficient approach to train/apply DCNNs with/on sets of photographic images, using a powerful graphical method called a conditional random field (CRF), to generate DCNN training and testing data using minimal manual supervision. We apply the method to several sets of images of natural landscapes, acquired from satellites, aircraft, unmanned aerial vehicles, and fixed camera installations. We synthesize our findings to examine the general effectiveness of transfer learning to landscape-scale image classification. Finally, we show how DCNN predictions on small regions of images might be used in conjunction with a CRF for highly accurate pixel-level classification of images.

&amp;lt;p&amp;gt;With consumer-grade unmanned aerial vehicles (UAVs) on the rise, which enable easy, time-flexible, and cost-effective acquisition of very high-resolution RGB data, the mapping of forest tree species using solely RGB imagery is of high interest, as it does not rely on sophisticated sensors, does not require extensive calibration and preprocessing and, therefore, enables the application by a wide audience. In combination with convolutional neural networks (CNNs), which particularly exploit spatial patterns and, therefore, highly benfit from very high-resolution remote sensing, this offers great potential for accurately mapping forest tree species.&amp;lt;/p&amp;gt;&amp;lt;p&amp;gt;Here, we present the findings of our recent study, in which we used very high-resolution RGB imagery from UAVs in combination with CNNs for the mapping of forest tree species. In this study, we used multicopter UAVs to obtain very high-resolution (&amp;lt;2 cm) RGB imagery over 51&amp;amp;#160;ha of temperate forests in the Southern Black Forest region, and the Hainich National Park in Germany. To fully harness the end-to-end learning capabilities of CNNs, we used a semantic segmentation approach (U-net) that concurrently segments and classifies tree species from imagery. With a diverse dataset in terms of study areas, site conditions, illumination properties, and phenology, we accurately mapped nine tree species, three genus-level classes, deadwood, and forest floor (mean F1-score 0.73). We found that a coarser spatial resolution substantially reduced the model accuracy (mean F1-score of 0.26 at 32&amp;amp;#160;cm resolution) and that larger tile sizes during CNN training negatively affected the model accuracies for underrepresented classes. Additional height information from normalized digital surface models slightly increased the model accuracy but simultaneously increased computational complexity and data requirements. Our results highlight the key role that UAVs can play in the mapping of forest tree species, given that air- and spaceborne remote sensing currently does not provide comparable spatial resolutions. Given the end-to-end learning capabilites of CNNs extensive preprocessing becomes partly obsolete, whereas the use of a large and diverse dataset facilitates a high degree of generalization of the CNN, thus fostering transferability. The synergy of high-resolution UAV imagery and CNN provide a fast and flexible yet accurate means of mapping forest tree species.&amp;lt;/p&amp;gt;&amp;lt;p&amp;gt;In this contribution, we will give an outlook on how the combination of UAV imagery and CNNs can be integrated with multitemporal satellite imagery (Sentinel-1 and Sentinel-2) in order to extrapolate the UAV-based tree species maps to larger areas.&amp;lt;/p&amp;gt;

Unmanned Aerial Vehicle (UAV) imagery are being assessed for analyzing within field spatial variability for agricultural precision management, because unmanned aerial vehicle imagery may be acquired quickly during critical periods of rapid crop growth. The objective of this study was to evaluate the use of unmanned aerial vehicle for the monitoring barley growth. Unmanned aerial vehicle imagery obtained from middle February to late June in Wanju, Jeollabuk-do. Unmanned aerial vehicle imagery corrected geometrically and atmospherically to calculate normalized difference vegetation index (NDVI). We analyzed the relationships between NDVI UAV of barley and biophysical measurements such as plant height, number of tiller and shoot dry weight over an entire barley growth period. The similar trend between NDVI UAV and growth parameters was shown. Correlation analysis between NDVI UAV and barley growth parameters revealed that NDVI UAV was highly correlated with shoot dry weight (r=0.932) and plant height (r=0.879). According to the relationship among growth parameters and NDVI UAV , the temporal variation of NDVIUAV was significant to interpret barley growth. The spatial distribution map of barley growth was in strong agreement with the field measurements in terms of geographical variation and relative numerical values when NDVI UAV was applied to power function. From these results, NDVI UAV can be used as a new tool for monitoring barley growth.

The presence of microvascular invasion (MVI) is a crucial indicator of postoperative recurrence in patients with hepatocellular carcinoma (HCC). Detecting MVI before surgery can improve personalized surgical planning and enhance patient survival. However, existing automatic diagnosis methods for MVI have certain limitations. Some methods only analyze information from a single slice and overlook the context of the entire lesion, while others require high computational resources to process the entire tumor with a three-dimension (3D) convolutional neural network (CNN), which could be challenging to train. To address these limitations, this paper proposes a modality-based attention and dual-stream multiple instance learning(MIL) CNN. In this retrospective study, 283 patients with histologically confirmed HCC who underwent surgical resection between April 2017 and September 2019 were included. Five magnetic resonance (MR) modalities including T2-weighted, arterial phase, venous phase, delay phase and apparent diffusion coefficient images were used in image acquisition of each patient. Firstly, Each two-dimension (2D) slice of HCC magnetic resonance image (MRI) was converted into an instance embedding. Secondly, modality attention module was designed to emulates the decision-making process of doctors and helped the model to focus on the important MRI sequences. Thirdly, instance embeddings of 3D scans were aggregated into a bag embedding by a dual-stream MIL aggregator, in which the critical slices were given greater consideration. The dataset was split into a training set and a testing set in a 4:1 ratio, and model performance was evaluated using five-fold cross-validation. Using the proposed method, the prediction of MVI achieved an accuracy of 76.43% and an AUC of 74.22%, significantly surpassing the performance of the baseline methods. Our modality-based attention and dual-stream MIL CNN can achieve outstanding results for MVI prediction.