Joint learning with diverse knowledge for re-identification

Previous research in active learning has primarily focused on selecting examples from closed-set data, which consists solely of unlabeled examples from the target classes. However, this approach overlooks the more prevalent scenario of open-set data in real-world applications. Open-set data encompasses examples from both target classes and non-target classes. To fill this gap, we propose a novel framework called JOSAL, which enhances the accuracy of the classifier by precisely selecting the target class examples from open-set data. The JOSAL framework introduces the concept of joint learning, where the Sampler and Classifier components perform sampling and classification tasks, respectively, by sharing example features extracted from a pre-trained Encoder. To maximize the classification accuracy of the Classifier, the framework adopts a novel joint learning strategy. This strategy initially prioritizes optimizing the Sampler and gradually shifts the optimization attention to the Classifier. The experimental results demonstrate that, compared to baselines, our approach exhibits stronger sampling precision and achieves higher classification accuracy. To the best of our knowledge, this is the first work to address the open-set active learning problem using the joint learning paradigm.

In drug development, protocols define how clinical trials are conducted, and are therefore of paramount importance. They contain key patient-, investigator-, medication-, and study-related information, often elaborated in different sections in the protocol texts. Granular-level parsing on large quantity of existing protocols can accelerate clinical trial design and provide actionable insights into trial optimization. Here, we report our progresses in using deep learning NLP algorithms to enable automated protocol analytics. In particular, we combined a pre-trained BERT transformer model with joint-learning strategies to simultaneously identify clinically relevant entities (i.e. Named Entity Recognition) and extract the syntactic relations between these entities (i.e. Relation Extraction) from the eligibility criteria section in protocol texts. When comparing to standalone NER and RE models, our joint-learning strategy can effectively improve the performance of RE task while retaining similarly high NER performance, likely due to the synergy of optimizing toward both tasks’ objectives via shared parameters. The derived NLP model provides an end-to-end solution to convert unstructured protocol texts into structured data source, which will be embedded into a comprehensive clinical analytics workflow for downstream trial design missions such like patient population extraction, patient enrollment rate estimation, and protocol amendment prediction.

This chapter will discuss approaches to 2D human pose estimation and tracking in a non-overlapping camera network. It will demonstrate the limitations of current approaches and suggest strategies to overcome them. In particular, computational intractability due to high dimensional limb space, violation of articulation constraints, and view-point dependence. The chapter is divided into three major components; namely, search space reduction, pose validation, and view-invariant pose tracking in a non-overlapping camera network. Firstly, we present approaches for search space reduction, such as Kinematic Tree based sub-region selection for each limb, Mean-Shift based maxima search on the likelihood surface, and temporal based reduction of search in parameter space. Secondly, we devise a PCA based Pose Validation strategy to prune out anatomically incorrect hypotheses. Thirdly, we propose to incorporate articulation constraints while keeping the problem tractable. Finally, we enable view-invariance through the fusion of only two pose detectors and an articulated skeleton tracker.

Despite the fact that nonlinear subspace learning techniques (e.g. manifold learning) have successfully applied to data representation, there is still room for improvement in explainability (explicit mapping), generalization (out-of-samples), and cost-effectiveness (linearization). To this end, a novel linearized subspace learning technique is developed in a joint and progressive way, called joint and progressive learning strategy (J-Play), with its application to multi-label classification. The J-Play learns high-level and semantically meaningful feature representation from high-dimensional data by (1) jointly performing multiple subspace learning and classification to find a latent subspace where samples are expected to be better classified; (2) progressively learning multi-coupled projections to linearly approach the optimal mapping bridging the original space with the most discriminative subspace; (3) locally embedding manifold structure in each learnable latent subspace. Extensive experiments are performed to demonstrate the superiority and effectiveness of the proposed method in comparison with previous state-of-the-art methods.

Semantic change detection (SCD) in high-resolution (HR) remote sensing images faces two issues: (1) isolated network branch for binary change detection (BCD) within multi-task architecture result in suboptimal SCD performance; (2) false alarms or missed detections caused by illumination differences or seasonal transform. To address these issues, this study proposes a bi-temporal binary change enhancement network (Bi-BCENet). Specifically, we introduce a binary change enhancement (BCE) strategy based on multi-network joint learning to achieve superior SCD via improving change areas prediction. Within the network’s reasoning process, we develop a cross-attention fusion module (CAFM) to enhance the global similarity modeling via cross-network prompt fusion, and we employ a cosine similarity-based auxiliary loss to optimize non-change’s semantic consistency. The experiments on SECOND and CINA-FX datasets demonstrate that Bi-BCENet outperforms representative SCD networks, achieving 62.08%, 84.95% in FSCD and 66.88%, 83.10% in mIoUsCD, respectively. And the ablation analysis of network validates Bi-BCENet’s effectiveness in reducing false alarms and missed detections in SCD results. Moreover, for specific SCD of cropland, Bi-BCENet shows its strong potential in single-to-multi SCD.

This paper proposes a novel probabilistic approach for appearance-based person reidentification in non-overlapping camera networks. It accounts for varying illumination, varying camera gain and has low computational complexity. More specifically, we present a graphical model where we model the person’s appearance in addition to camera illumination and gain. We analytically derive the solutions for the person’s appearance and camera properties, and use a novel constant time Gibbs sampling scheme to estimate the identification labels. We validate our algorithm on two indoor datasets and perform a comparative analysis with existing algorithms. We demonstrate significantly increased re-identification accuracy in addition to significantly reducing the computational complexity on our datasets.

One of the goals of person re-identification systems is to support video-surveillance operators and forensic investigators to find an individual of interest in videos acquired by a network of non-overlapping cameras. This is attained by sorting images of previously observed individuals for decreasing values of their similarity with a given probe individual. Existing appearance descriptors, together with their similarity measures, are mostly aimed at improving ranking quality. We propose two fuzzy-based descriptors which are fast in terms of the processing time on descriptor generation and matching score computation. We then evaluate our approach on three benchmark data sets (VIPeR, i-LIDS, and ETHZ) with comparison of some descriptors in the state-of-the-art.

Person Re-identification is the task of recognizing comparable subjects across a network of nonoverlapping cameras. This is typically achieved by extracting from the source image a vector of characteristic features of the specific person captured by the camera. Learning a good set of robust, invariant and discriminative features is a complex task, often leveraging contrastive learning. In this article, we explore a different approach, learning the representation of an individual as the conditioning information required to generate images of the specific person starting from random noise. In this way we decouple the identity of the individual from any other information relative to a specific instance (pose, background, etc.), allowing interesting transformations from one identity to another. As generative models, we use the recent diffusion models that have already proven their sensibility to conditioning in many different contexts. The results presented in this article serve as a proof-of-concept. While our current performance on common benchmarks is lower than state-of-the-art techniques, the approach is intriguing and rich of innovative insights, suggesting a wide range of potential improvements along various lines of investigation.

Morph detection is of paramount significance when the integrity of Automatic Face Recognition (AFR) systems is concerned. Considering the risks incurred by morphing attacks, a robust automated morph detector is required which can distinguish authentic bona fide samples from altered morphed images. We leverage the wavelet sub-band decomposition of an input image, yielding the fine-grained spatial-frequency content of the input image. To enhance detection of morphed images, our goal is to find the most discriminative information across frequency channels and spatial domain. To this end, we propose an end-to-end attention-based deep morph detector which assimilates the most discriminative wavelet sub-bands of a given image which are obtained by a group sparsity representation learning scheme. Specifically, our group sparsity-constrained Deep Neural Network (DNN) learns the most discriminative wavelet sub-bands (channels) of an input image while the attention mechanism captures the most discriminative spatial regions of input images for the downstream task of morph detection. To this end, we adopt three attention mechanisms to diversify our refined features for morph detection. As the first attention mechanism, we employ the Convolutional Block Attention Module (CBAM) which provides us with refined feature maps. As the second attention mechanism, compatibility scores across spatial locations and output of our DNN highlights the most discriminative regions, and lastly, the multiheaded self-attention augmented convolutions account for our third attention mechanism. We evaluate the efficiency of our proposed framework through extensive experiments using multiple morph datasets that are compiled using bona fide images available in the FERET, FRLL, FRGC, and WVU Twin datasets. Most importantly, our proposed methodology has resulted in reduction in detection error rates when compared with state-of-the-art results. Finally, to further assess our multi-attentional morph detection, we delve into different combinations of attention mechanisms via a comprehensive ablation study.

Segmenting medical images via explicit–implicit attention aggregation

Fully-automated person re-identification in multi-camera surveillance system with a robust kernel descriptor and effective shadow removal method

Person re-identification refers to match the same pedestrian across disjoint views in non-overlapping camera networks. Lots of local and global features in the literature are put forward to solve the matching problem, where color feature is robust to viewpoint variance and gradient feature provides a rich representation robust to illumination change. However, how to effectively combine the color and gradient features is an open problem. In this paper, to effectively leverage the color-gradient property in multiple color spaces, we propose a novel Second Order Histogram feature (SOH) for person reidentification in large surveillance dataset. Firstly, we utilize discrete encoding to transform commonly used color space into Encoding Color Space (ECS), and calculate the statistical gradient features on each color channel. Then, a second order statistical distribution is calculated on each cell map with a spatial partition. In this way, the proposed SOH feature effectively leverages the statistical property of gradient and color as well as reduces the redundant information. Finally, a metric learned by KISSME [1] with Mahalanobis distance is used for person matching. Experimental results on three public datasets, VIPeR, CAVIAR and CUHK01, show the promise of the proposed approach.

Multi-target tracking in a non-overlapping camera network is an active research field, and one of the important problems in it is the person re-identification problem. In this study, the authors propose an approach to improve the performance of the backbone model in the person re-identification. Their approach focuses on training a fusion model with a shallow model and making hard triplets with relationship matrices quickly and efficiently. The proposed approach is simple, but it improves the performance of the backbone. In addition, the hard triplet mining in their process is much faster than the conventional approach. Experimental evaluation shows that the proposed approach can improve the performances of the backbone model. The proposed approach improves rank-1 and mean average precision (mAP) performance by more than 12.54 and 15.44%, respectively, over the backbone models in the Market1501 and DukeMTMC-reID dataset. The approach also achieves competitive performances compared with state-of-the-art approaches.

To address the challenges of target occlusion and background clutter that result in missed and false detections during safety helmet detection on construction sites, this article proposes a novel ISE-YOLOv11 (Improved Squeeze-and-Excitation based YOLOv11) model. Firstly, a C3K2_a deformable convolution module is designed to capture diverse features of the target and reduce missed detections. To achieve rapid target localization, an improved attention mechanism ISE (Improved Squeeze-and-Excitation) is introduced to replace the classical attention module in YOLOv11. This enhancement enables more accurate target positioning, extracts richer edge and texture features, and reduces the number of parameters simultaneously. Furthermore, a cross-scale feature fusion module termed MDFF (Multi-scale Deformable Feature Fusion) is constructed to significantly improve the model’s perception capability of targets in occluded backgrounds and enhance detection accuracy. Experimental results show that on the SHWD (Safety Helmet Wearing) dataset, the improved model maintains real-time performance at 38 FPS. The improved model maintains real-time performance while achieving recall and accuracy of 90.2% and 92.4%, respectively, which is 0.6% and 1.5% higher than the original YOLOv11 model. In addition, ISE-YOLOv11 exhibits superior detection performance across various scenarios, demonstrating better adaptability to complex and diverse environments. It provides an efficient and reliable technical solution for safety supervision on construction sites and offers new insights into the integration of attention mechanisms with single-stage detectors.

Facial expression recognition (FER) in uncontrolled, in-the-wild settings is a challenging task due to variations in pose, illumination, occlusion, and background clutter. The state-of-the-art methods on FER employ fixed size pooling in attention networks, resulting in suboptimal robustness in unconstrained environments. In this work, a novel Multi-Scale Attention Network (MSAN) is integrated with the YOLOv8 backbone architecture for expression classification. The YOLOv8 backbone provides efficient feature extraction with a lightweight, detection-oriented architecture, enabling real-time processing and strong spatial feature representation. The Convolutional Block Attention Mechanism (CBAM) is employed in the proposed work. The standard convolutional blocks in YOLOv8 backbone are replaced with MSAN, wherein the fixed-size pooling in the spatial attention mechanism of CBAM is replaced with a Pyramid Pooling Mechanism (PPM). The PPM employs adaptive average pooling at multiple resolutions to capture multi-scale contextual information, which is subsequently fused into a unified feature representation. This enables the spatial attention mechanism in MSAN to capture both global facial context and local region-level cues simultaneously, leading to superior robustness to scale variations, partial occlusions, and diverse expression variations. The work is evaluated on benchmark wild datasets including RAF-DB, FER2013, and AffectNet, with all experiments performed using 5-fold cross-validation. Experimental results demonstrate that the multi-scale attention network with PPM in YOLOv8 backbone achieved improvement in classification accuracy and robustness when compared with the state-of-the-art methods for expression classification in the wild. These findings highlight the effectiveness of integrating MSAN into lightweight detection-based backbones for real-world FER applications.