Fine-Grained Recognition: Accounting for Subtle Differences between Similar Classes

Advancements in artificial intelligence for biometrics: A deep dive into model-based gait recognition techniques

The Spondyloarthritis Research Consortium of Canada (SPARCC) scoring system is a sacroiliitis grading system. To develop a deep learning-based pipeline for grading sacroiliitis using the SPARCC scoring system. Prospective. The study included 389 participants (42.2-year-old, 44.6% female, 317/35/37 for training/validation/testing). A pretrained algorithm was used to differentiate image with/without sacroiliitis. 3-T, short tau inversion recovery (STIR) sequence, fast spine echo. The regions of interest as ground truth for models' training were identified by a rheumatologist (HYC, 10-year-experience) and a radiologist (KHL, 6-year-experience) using the Assessment of Spondyloarthritis International Society definition of MRI sacroiliitis independently. Another radiologist (YYL, 4.5-year-experience) solved the discrepancies. The bone marrow edema (BME) and sacroiliac region models were for segmentation. Frangi-filter detected vessels used as intense reference. Deep learning pipeline scored using SPARCC scoring system evaluating presence and features of BMEs. A rheumatologist (SCWC, 6-year-experience) and a radiologist (VWHL, 14-year-experience) scored using the SPARCC scoring system once. The radiologist (YYL) scored twice with 5-day interval. Independent samples t-tests and Chi-squared tests were used. Interobserver and intraobserver reliability by intraclass correlation coefficient (ICC) and Pearson coefficient evaluated consistency between readers and the deep learning pipeline. We evaluated the performance using sensitivity, accuracy, positive predictive value, and Dice coefficient. A P-value <0.05 was considered statistically significant. The ICC and the Pearson coefficient between the SPARCC scores from three readers and the deep learning pipeline were 0.83 and 0.86, respectively. The sensitivity in identifying BME and accuracy of identifying SI joints and blood vessels was 0.83, 0.90, and 0.88, respectively. The dice coefficients were 0.82 (sacrum) and 0.80 (ilium). The high consistency with human readers indicated that deep learning pipeline may provide a SPARCC-informed deep learning approach for scoring of STIR images in spondyloarthritis. 1 TECHNICAL EFFICACY: Stage 2.

Ultrasound imaging provides information from a large part of the muscle. It has recently been shown that ultrafast ultrasound imaging can be used to record and analyze the mechanical response of individual MUs using blind source separation. In this work, we present an alternative method - a deep learning pipeline - to identify active MUs in ultrasound image sequences, including segmentation of their territories and signal estimation of their mechanical responses (twitch train). We train and evaluate the model using simulated data mimicking the complex activation pattern of tens of activated MUs with overlapping territories and partially synchronized activation patterns. Using a slow fusion approach (based on 3D CNNs), we transform the spatiotemporal image sequence data to 2D representations and apply a deep neural network architecture for segmentation. Next, we employ a second deep neural network architecture for signal estimation. The results show that the proposed pipeline can effectively identify individual MUs, estimate their territories, and estimate their twitch train signal at low contraction forces. The framework can retain spatio-temporal consistencies and information of the mechanical response of MU activity even when the ultrasound image sequences are transformed into a 2D representation for compatibility with more traditional computer vision and image processing techniques. The proposed pipeline is potentially useful to identify simultaneously active MUs in whole muscles in ultrasound image sequences of voluntary skeletal muscle contractions at low force levels.

In spite of the large-scale production and widespread distribution of vaccines and antiviral drugs, viruses remain a prominent human disease. Recently, the discovery of antiviral peptides (AVPs) has become an influential antiviral agent due to their extraordinary advantages. With the avalanche of newly-found peptide sequences in the post-genomic era, there is a great demand to develop a sequence-based predictor for timely identifying AVPs as this information is very useful for both basic research and drug development. In this study, we propose a novel sequence-based meta-predictor with an effective feature representation, called Meta-iAVP, for the accurate prediction of AVPs from given peptide sequences. Herein, the effective feature representation was extracted from a set of prediction scores derived from various machine learning algorithms and types of features. To the best of our knowledge, the model proposed herein represents the first meta-based approach for the prediction of AVPs. An overall accuracy and Matthews correlation coefficient of 95.20% and 0.90, respectively, was achieved from the independent test set on an objective benchmark dataset. Comparative analysis suggested that Meta-iAVP was superior to that of existing methods and therefore represents a useful tool for AVP prediction. Finally, in an effort to facilitate high-throughput prediction of AVPs, the model was deployed as the Meta-iAVP web server and is made freely available online at http://codes.bio/meta-iavp/ where users can submit query peptide sequences for determining the likelihood of whether or not these peptides are AVPs.

In the field of few-shot learning, different methods are proposed to optimize the model by changing the network structure or optimizing the algorithm. Although, by designing the end-to-end algorithms, the classification performance on a specific task can be improved. But when the amount of data is limited, it is still difficult to obtain generalization ability for different tasks. At the same time, a single loss function in the end-to-end training is always ineffective, because it is difficult for a shallow network to learn an effective image feature representation from complex natural images. In this paper, a constructive representation algorithm for few-shot learning without end-to-end training is proposed, which is suitable for few-shot learning in the natural images classification task. Through self-supervised representation learning, the proposed encoding model generates an effective feature representation. Then, a few-shot learning model is further constructed and trained for supervised classification tasks. Without the end-to-end training, the proposed learning method at different training stages use different loss functions. In the experiments, on the classification task of the public competition data set ISIC2018[11], our method has a 30% performance improvement over the state-of-the-art methods.

Colorectal cancer (CRC) stands as one of the deadliest cancer types worldwide. Identifying specific gene mutations crucial for targeted therapy is pivotal for CRC treatment. However, the current genetic testing method involves liquid biopsy, which is often cumbersome and cost-intensive. Histopathological whole-slide images (WSIs) are routinely generated for diagnostic purposes and can offer valuable insights into cellular heterogeneity. In this study, we aim to develop an integrated deep learning pipeline for predicting TP53 mutations in CRC, one of the most frequently mutated genes in CRC. We commenced by collecting 424 diagnostic WSIs from the TCGA-COAD cohorts. Each WSI was split as small patches at a resolution of 0.5 micrometers per pixel for following analyses. Quality control was implemented using Otsu thresholding and Gaussian blur filtering, both at the slide-level masking and patch-level filtering stages. Additionally, we applied the Macenko algorithm for color normalization to eliminate systematic errors. Following image preprocessing, we constructed a tissue-type classification model based on VGG19, which was trained and tested using the NCT-CRC-HE-100K and CRC-VAL-HE-7K datasets, both containing pixels with tissue types labeled by human experts. The trained classifier was then used to identify adenocarcinoma epithelium patches from TCGA-COAD. Subsequently, we applied multiple pre-trained Convolutional Neural Network (CNN) models, including ResNet-101, DenseNet-201, and Inception-ResNet-v2, to predict TP53 mutations for the identified adenocarcinoma epithelium patches from TCGA-COAD. A novel integration approach was employed to aggregate patch-level predictions into slide-level outcomes. We evaluated the trained models using various metrics, including accuracy, precision, recall, F1 score, AUC (Area Under the Curve), and Cohen's Kappa coefficient. The tissue-type classification model exhibited robust performance, achieving an overall accuracy of 94.7%, a Cohen's Kappa coefficient of 94%, a precision of 95.3%, a recall of 99.76%, and an F1 score of 97.48% for identifying adenocarcinoma epithelium patches. Regarding the TP53 mutation prediction models, ResNet-101, DenseNet-201, and Inception-ResNet-v2 exhibit the predictive ability to both patch level (AUC=65%) and slide level (AUC=70%). In summary, the proposed deep learning pipeline offers an efficient way to acquire gene mutation information from WSIs. Moreover, it demonstrates a capability to discover potential molecular aberration from the clinical images. Citation Format: Geng-Yun Tien, Liang-Chuan Lai, Tzu-Pin Lu, Mong-Hsun Tsai, Hsiang-Han Chen, Eric Y. Chuang. Prediction of gene mutation from colorectal adenocarcinoma whole slide images via integrated deep learning pipeline [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 4938.

Recently, with the emergence of retrieval requirements for certain individual in the same superclass, e.g., birds, persons, cars, fine-grained recognition task has attracted a significant amount of attention from academia and industry. In fine-grained recognition scenario, the inter-class differences are quite diverse and subtle, which makes it challenging to extract all the discriminative cues. Traditional training mechanism optimizes the overall discriminativeness of the whole feature. It may stop early when some feature elements has been trained to distinguish training samples well, leaving other elements insufficiently trained for a feature. This would result in a less generalizable feature extractor that only captures major discriminative cues and ignores subtle ones. Therefore, there is a need for a training mechanism that enforces the discriminativeness of all the elements in the feature to capture more the subtle visual cues. In this paper, we propose a Discrimination-Aware Mechanism (DAM) that iteratively identifies insufficiently trained elements and improves them. DAM is able to increase the number of well learned elements, which captures more visual cues by the feature extractor. In this way, a more informative representation is learned, which brings better generalization performance. We show that DAM can be easily applied to both proxy-based and pair-based loss functions, and thus can be used in most existing fine-grained recognition paradigms. Comprehensive experiments on CUB200-2011, Cars196, Market-1501, and MSMT17 datasets demonstrate the advantages of our DAM based loss over the related state-of-the-art approaches.

BackgroundVariability in the bony morphology of pathologic hips/knees is a challenge in automating preoperative computed tomography (CT) scan measurements. With the increasing prevalence of CT for advanced preoperative planning, processing this data represents a critical bottleneck in presurgical planning, research, and development. The purpose of this study was to demonstrate a reproducible and scalable methodology for analyzing CT-based anatomy to process hip and knee anatomy for perioperative planning and execution. MethodsOne hundred patients with preoperative CT scans undergoing total knee arthroplasty for osteoarthritis were processed. A two-step deep learning pipeline of classification and segmentation models was developed that identifies landmark images and then generates contour representations. We utilized an open-source computer vision library to compute measurements. Classification models were assessed by accuracy, precision, and recall. Segmentation models were evaluated using dice and mean Intersection over Union (IOU) metrics. Contour measurements were compared against manual measurements to validate posterior condylar axis angle, sulcus angle, trochlear groove-tibial tuberosity distance, acetabular anteversion, and femoral version. ResultsClassifiers identified landmark images with accuracy of 0.91 and 0.88 for hip and knee models, respectively. Segmentation models demonstrated mean IOU scores above 0.95 with the highest dice coefficient of 0.957 [0.954-0.961] (UNet3+) and the highest mean IOU of 0.965 [0.961-0.969] (Attention U-Net). There were no statistically significant differences for the measurements taken automatically vs manually (P > 0.05). Average time for the pipeline to preprocess (48.65 +/− 4.41 sec), classify/retrieve landmark images (8.36 +/− 3.40 sec), segment images (<1 sec), and obtain measurements was 2.58 (+/− 1.92) minutes. ConclusionsA fully automated three-stage deep learning and computer vision-based pipeline of classification and segmentation models accurately localized, segmented, and measured landmark hip and knee images for patients undergoing total knee arthroplasty. Incorporation of clinical parameters, like patient-reported outcome measures and instability risk, will be important considerations alongside anatomic parameters.

Alcohol craving can cause many problems for people's life. However, there are very few related works doing alcohol prediction based on physiological data, except some from our lab. The goal of this research is to predict whether people have alcohol drinking or not from real physiological data in order to help them with drinking problems. The raw physiological data in this work include skin temperature, heart rate, galvanic skin response (GSR), steps, and calories. The data was collected from 29 users with basic watch and the reading frequency is one record per minute. In this thesis, three data analysis pipelines, drinking record prediction pipeline, drinking episode statistical pipeline, and drinking episode deep learning pipeline, are implemented. The drinking record prediction pipeline is doing prediction based on oneminute record. The drinking episode pipeline is doing prediction based on thirty-minute episode. Statistical features are extracted from the thirty-minute data blocks. The drinking episode deep learning pipeline is doing prediction based on thirty-minute episode as well. In this deep learning pipeline, one dimensional signal is converted to spectrum graph. Then use Cifar 10 model to extract deep learning features from the spectrum graph. After that apply machine learning methods on the deep learning features to do the classification. Within-user and cross user experiments are conducted in this thesis because different users may have different reaction to alcohol. Different models are found for different users and general model is discovered for cross-users. Balanced data is used for training and testing, so the baseline accuracy is 50 percent. The accuracy for within-user is up to 88.89 Percent and the accuracy for cross-user is 75.68 Percent, which indicates that the withinuser result is much better than cross-user result. In order to find the most significant feature in alcohol drinking prediction, experiments are also conducted on skin temperature only features, heart rate only features, and GSR only features. The results show that heart rate contributes most in the alcohol drinking prediction.

The SARS-CoV-2 (COVID-19) has propagated rapidly around the world, and it became a global pandemic. It has generated a catastrophic effect on public health. Thus, it is crucial to discover positive cases as early as possible to treat touched patients fastly. Chest CT is one of the methods that play a significant role in diagnosing 2019-nCoV acute respiratory disease. The implementation of advanced deep learning techniques combined with radiological imaging can be helpful for the precise detection of the novel coronavirus. It can also be assistive to surmount the difficult situation of the lack of medical skills and specialized doctors in remote regions. This paper presented Deep Transfer Learning Pipelines with Apache Spark and KerasTensorFlow combined with the Logistic Regression algorithm for automatic COVID-19 detection in chest CT images, using Convolutional Neural Network (CNN) based models VGG16, VGG19, and Xception. Our model produced a classification accuracy of 85.64, 84.25, and 82.87 %, respectively, for VGG16, VGG19, and Xception. HIGHLIGHTS Deep Transfer Learning Pipelines with Apache Spark and Keras TensorFlow combined with Logistic Regression using CT images to screen for Corona Virus Disease (COVID-19) Automatic detection of COVID-19 in chest CT images Convolutional Neural Network (CNN) based models VGG16, VGG19, and Xception to predict COVID-19 in Computed Tomography image GRAPHICAL ABSTRACT

BackgroundIdentifying biomarkers that predict patient response to immunotherapy is critical for optimizing treatment strategies and improving clinical outcomes. Despite the success of immunotherapy, a significant proportion of patients do not...

Literature listing

BackgroundMultiplex immunofluorescence (mIF) can provide invaluable insights on spatial biology and the complexities of the immune tumor microenvironment (iTME), however, current analysis methods are laborious and user dependent. We applied...

The objective of video re-localization (VRL) is to localize a successive sequence of frames, namely, the target moment, from untrimmed reference videos that semantically correspond to a given query video. During training, the weakly supervised setting of VRL provides only coarse-grained video-level rather than frame-level annotations. For the weakly supervised VRL (WS-VRL) task, obtaining effective video feature representations that can be used to evaluate the relevance between videos and localizing the accurate temporal boundaries of the target moment remain challenging. In this paper, a novel multi-agent-reinforced switchable network (MARS) is proposed to address these challenges. MARS can adaptively guide video feature encoding and moment localization using multiple learned agents. Specifically, an agent-controlled switchable encoder is used to obtain effective video feature representations, and an agent-reinforced boundary localizer is used to determine accurate localized moments through progressive refinement. Furthermore, a relevance-oriented reward generator was designed to estimate the relevance of the localized moment to the query video and assign a reward to multiple agents. The effectiveness of the proposed MARS model was verified through extensive experiments on the ActivityNet-VRL dataset.

Deep learning models have demonstrated favorable performance on many medical image classification tasks. However, they rely on expensive hand-labeled datasets that are time-consuming to create. In this work, we explore a new supervision source to training deep learning models by using gaze data that is passively and cheaply collected during a clinician’s workflow. We focus on three medical imaging tasks, including classifying chest X-ray scans for pneumothorax and brain MRI slices for metastasis, two of which we curated gaze data for. The gaze data consists of a sequence of fixation locations on the image from an expert trying to identify an abnormality. Hence, the gaze data contains rich information about the image that can be used as a powerful supervision source. We first identify a set of gaze features and show that they indeed contain class-discriminative information. Then, we propose two methods for incorporating gaze features into deep learning pipelines. When no task labels are available, we combine multiple gaze features to extract weak labels and use them as the sole source of supervision (Gaze-WS). When task labels are available, we propose to use the gaze features as auxiliary task labels in a multi-task learning framework (Gaze-MTL). On three medical image classification tasks, our Gaze-WS method without task labels comes within 5 AUROC points (1.7 precision points) of models trained with task labels. With task labels, our Gaze-MTL method can improve performance by 2.4 AUROC points (4 precision points) over multiple baselines.