Progressive Cross-Media Correlation Learning

Existing cross-media retrieval methods are usually conducted under the supervised setting, which need lots of annotated training data. Generally, it is extremely labor-consuming to annotate cross-media data. So unsupervised cross-media retrieval is highly demanded, which is very challenging because it has to handle heterogeneous distributions across different media types without any annotated information. To address the above challenge, this paper proposes Domain Adaptation with Scene Graph (DASG) approach, which transfers knowledge from the source domain to improve cross-media retrieval in the target domain. Our DASG approach takes Visual Genome as the source domain, which contains image knowledge in the form of scene graph. The main contributions of this paper are as follows: First, we propose to address unsupervised cross-media retrieval by domain adaptation. Instead of using the labor-consuming annotated information of cross-media data in the training stage, our DASG approach learns cross-media correlation knowledge from Visual Genome, and then transfers the knowledge to cross-media retrieval through media alignment and distribution alignment. Second, our DASG approach utilizes fine-grained information via scene graph representation to enhance generalization capability across domains. The generated scene graph representation builds (subject$\rightarrow $ relationship$\rightarrow $ object) triplets by exploiting objects and relationships within image and text, which makes the cross-media correlation more precise and promotes unsupervised cross-media retrieval. Third, we exploit the related tasks including object and relationship detection for learning more discriminative features across domains. Leveraging the semantic information of objects and relationships improves cross-media correlation learning for retrieval. Experiments on two widely-used cross-media retrieval datasets, namely Flickr-30K and MS-COCO, show the effectiveness of our DASG approach.

Cross-media retrieval is becoming increasingly important nowadays. To address this challenging problem, most existing approaches project heterogeneous features into a unified feature space to facilitate their similarity computation. However, this unified feature space usually has no explicit semantic meanings, which might ignore the hints contained in the original media content, and thus is not able to fully measure the similarities among different media types. By considering the above issues, we propose a new approach to cross-media retrieval via semantic entity projection SEP in this paper. Our approach consists of three main steps. Firstly, an entity level with fine-grained semantics between low-level features and high-level concepts are constructed, so as to help bridge the semantic gap to a certain extent. Then, an entity projection is learned by minimizing both cross-media correlation error and single-media reconstruction error from low-level features to the entity level, with which a unified feature space with explicit semantic meanings can be obtained from low-level features. Finally, the semantic abstraction of high-level concepts is generated by using logistic regression to conduct cross-media retrieval. Experimental results on the Wikipedia dataset show the effectiveness of the proposed approach.

Emerging multimedia content including images and texts are always jointly utilized to describe the same semantics. As a result, cross-media retrieval becomes increasingly important, which is able to retrieve the results of the same semantics with the query but with different media types. In this paper, we propose a novel heterogeneous similarity measure with nearest neighbors (HSNN). Unlike traditional similarity measures which are limited in homogeneous feature space, HSNN could compute the similarity between media objects with different media types. The heterogeneous similarity is obtained by computing the probability for two media objects belonging to the same semantic category. The probability is achieved by analyzing the homogeneous nearest neighbors of each media object. HSNN is flexible so that any traditional similarity measure could be incorporated, which is further regarded as the weak ranker. An effective ranking model is learned from multiple weak rankers through AdaRank for cross-media retrieval. Experiments on the wikipedia dataset show the effectiveness of the proposed approach, compared with state-of-the-art methods. The cross-media retrieval also shows to outperform image retrieval systems on a unimedia retrieval task.

Data plays a fundamental role in the training of Large Language Models (LLMs). While attention has been paid to the collection and composition of datasets, determining the data sampling strategy in training remains an open question. Most LLMs are trained with a simple strategy, random sampling. However, this sampling strategy ignores the unbalanced nature of training data distribution, which can be sub-optimal. In this paper, we propose Clus-terClip Sampling to balance the text distribution of training data for better model training. Specifically, ClusterClip Sampling utilizes data clustering to reflect the data distribution of the training set and balances the common samples and rare samples during training based on the cluster results. A repetition clip operation is introduced to mitigate the overfitting issue led by samples from certain clusters. Extensive experiments validate the effectiveness of Clus-terClip Sampling, which outperforms random sampling and other cluster-based sampling variants under various training datasets and large language models 1 .

Automated monitoring of the movements and behaviour of animals is a valuable research tool. Recently, machine learning tools were applied to many species to classify units of behaviour. For the monitoring of wild species, collecting enough data for training models might be problematic, thus we examine how machine learning models trained on one species can be applied to another closely related species with similar behavioural conformation. We contrast two ways to calculate accuracies, termed here as overall and threshold accuracy, because the field has yet to define solid standards for reporting and measuring classification performances. We measure 21 dogs and 7 wolves, and find that overall accuracies are between 51 and 60% for classifying 8 behaviours (lay, sit, stand, walk, trot, run, eat, drink) when training and testing data are from the same species and between 41 and 51% when training and testing is cross-species. We show that using data from dogs to predict the behaviour of wolves is feasible. We also show that optimising the model for overall accuracy leads to similar overall and threshold accuracies, while optimizing for threshold accuracy leads to threshold accuracies well above 80%, but yielding very low overall accuracies, often below the chance level. Moreover, we show that the most common method for dividing the data between training and testing data (random selection of test data) overestimates the accuracy of models when applied to data of new specimens. Consequently, we argue that for the most common goals of animal behaviour recognition overall accuracy should be the preferred metric. Considering, that often the goal is to collect movement data without other methods of observation, we argue that training data and testing data should be divided by individual and not randomly.

Existing domain generalization (DG) methods typically rely on source domain data for model training and unknown target domain data to evaluate generalization ability. However, collecting target domain data is challenging because it must be out-of-distribution from the source domain data. Thus, developing Target-Free Evaluation (TFEval) methods is essential to assess the generalization performance of deep models without requiring unknown target domain data, relying only on trained data. In this paper, we estimate generalization performance for TFEval by training a regression predictor. The predictor takes model representations as input and predicts generalization performance. Inspired by the observation that Batch Normalization (BN) parameters strongly reflect domain/distribution information, we extract model representations by calculating BN parameter differences between the trained model and its proxy models. The proxy models are trained on a significantly smaller subset of the source domain data. We construct multiple TFEval datasets by collecting numerous deep models, and experimental results on these datasets demonstrate that our evaluation method effectively predicts the generalization performance of deep models.

A deep semantic segmentation model-based method can achieve state-of-the-art accuracy and high computational efficiency in large-scale crop mapping. However, the model cannot be widely used in actual large-scale crop mapping applications, mainly because the annotation of ground truth data for deep semantic segmentation model training is time-consuming. At the operational level, it is extremely difficult to obtain a large amount of ground reference data by photointerpretation for the model training. Consequently, in order to solve this problem, this study introduces a workflow that aims to extract rice distribution information in training sample shortage regions, using a deep semantic segmentation model (i.e., U-Net) trained on pseudo-labels. Based on the time series Sentinel-1 images, Cropland Data Layer (CDL) and U-Net model, the optimal multi-temporal datasets for rice mapping were summarized, using the global search method. Then, based on the optimal multi-temporal datasets, the proposed workflow (a combination of K-Means and random forest) was directly used to extract the rice-distribution information of Jiangsu (i.e., the K–RF pseudo-labels). For comparison, the optimal well-trained U-Net model acquired from Arkansas (i.e., the transfer model) was also transferred to Jiangsu to extract local rice-distribution information (i.e., the TF pseudo-labels). Finally, the pseudo-labels with high confidences generated from the two methods were further used to retrain the U-Net models, which were suitable for rice mapping in Jiangsu. For different rice planting pattern regions of Jiangsu, the final results showed that, compared with the U-Net model trained on the TF pseudo-labels, the rice area extraction errors of pseudo-labels could be further reduced by using the U-Net model trained on the K–RF pseudo-labels. In addition, compared with the existing rule-based rice mapping methods, he U-Net model trained on the K–RF pseudo-labels could robustly extract the spatial distribution information of rice. Generally, this study could provide new options for applying a deep semantic segmentation model to training sample shortage regions.

To further improve the accuracy of recurrent neural network in predicting the gas concentration in the upper corner of the mine tunnel, this paper proposes a method to construct a gas concentration prediction model based on multiple sequence long and short memory network, considering the spatial correlation between the gas concentration in the return airway and upper corner. The reliability of the model construction is improved by using the white noise test and smoothness test to verify the interpretability of the data in this paper and constructing supervised learning type data for gas concentration prediction model training and testing by means of data set division and data windowing. Through experimental comparison, grid search, and time series decomposition, the model algorithm, training parameters, and experimental results were combined to make an in-depth analysis of the influence of each parameter on the model training and the prediction. A training model of the spatially fused gas concentration prediction model with a network layer of 1 and a number of neurons of 32 as the model structure, Adam as the optimization algorithm, and a learning rate of 0.001 and a batch size of 32 as the training parameters was finally determined. The gas concentration prediction model trained in this paper performed well in the test set with a mean square error (MSE) of 0.0013, and its superiority was verified by comparing it with other models to provide some experience and basis for subsequent studies on gas concentration prediction in the upper corner.

As scaling up training data has significantly improved the general multimodal capabilities of Large Vision-Language Models (LVLMs), they still suffer from the hallucination issue, generating text that is inconsistent with the visual input. This phenomenon motivates us to systematically investigate the role of training data in hallucination. We introduce a new benchmark, POPEv2, which consists of counterfactual images collected from the training data of LVLMs with certain objects masked. Through comprehensive evaluation on POPEv2, we find that current LVLMs suffer from training bias: they fail to fully leverage their training data and hallucinate more frequently on images seen during training. Specifically, they perform poorly on counterfactual images, often incorrectly answering “Yes” to questions about masked objects. To understand this issue, we conduct probing experiments on the models’ internal components, revealing that this training bias is primarily located in the language modeling (LM) head, which fails to correctly translate accurate visual representations into textual outputs. Based on these findings, we propose Obliviate, an efficient and lightweight unlearning method designed to mitigate object hallucination via training bias unlearning. Obliviate identifies the discrepancy between ground-truth labels and model outputs on the training data as a proxy for bias and adopts a parameter- and data-efficient fine-tuning strategy that only updates the LM head. Extensive experiments demonstrate the effectiveness of our approach. While only reusing the training data and updating approximately 2% of the parameters, Obliviate significantly reduces hallucination across both discriminative and generative tasks. Furthermore, it demonstrates strong scalability with respect to both model size (2B to 72B) and training data volume, and exhibits promising generalization to hallucination types beyond object-level hallucination.

Modern face recognition systems leverage datasets containing images of hundreds of thousands of specific individuals' faces to train deep convolutional neural networks to learn an embedding space that maps an arbitrary individual's face to a vector representation of their identity. The performance of a face recognition system in face verification (1:1) and face identification (1:N) tasks is directly related to the ability of an embedding space to discriminate between identities. Recently, there has been significant public scrutiny into the source and privacy implications of large-scale face recognition training datasets such as MS-Celeb-1M and MegaFace, as many people are uncomfortable with their face being used to train dual-use technologies that can enable mass surveillance. However, the impact of an individual's inclusion in training data on a derived system's ability to recognize them has not previously been studied. In this work, we audit ArcFace, a state-of-the-art, open source face recognition system, in a large-scale face identification experiment with more than one million distractor images. We find a Rank-1 face identification accuracy of 79.71% for individuals present in the model's training data and an accuracy of 75.73% for those not present. This modest difference in accuracy demonstrates that face recognition systems using deep learning work better for individuals they are trained on, which has serious privacy implications when one considers all major open source face recognition training datasets do not obtain informed consent from individuals during their collection.

Multilingual speaker recognition involves multilingual speech data in model training, which empowers the system to handle recognition requests in multiple languages. The multilingual training approach augments data from multiple languages, but inevitably introduces probability dispersion, due to the more complex language conditions. This paper proposes a language-aware training approach for PLDA which involves language information when training the PLDA model. The proposed approach has been evaluated with the i-vector/PLDA framework using the CSLT-CUDGT2014 Chinese-Uyghur bilingual speech database. The experimental results show that the language PLDA training resulted in a relative EER reduction of 15.38% in the Chinese test and 20.07% in the Uyghur test.

Active learning has recently attracted increasing attention in the task of person re-identification, due to its unique scalability that not only maximally reduces the annotation cost but also retains the satisfying performance. Although some preliminary active learning methods have been explored in scalable person re-identification task, they have the following two problems: 1) the inefficiency in the selection process of image pairs due to the huge search space, and 2) the ineffectiveness caused by ignoring the impact of unlabeled data in model training. Considering that, we propose a Multi-grained Active Semi-Supervised learning framework, named MASS, to address the scalable person re-identification problem existing in the practical scenarios. Specifically, we firstly design a cluster-scatter procedure to alleviate the inefficiency problem, which consists of two components: cluster step and scatter step. The cluster step shrinks the search space into individual small clusters by a coarse-grained clustering method, and the subsequent scatter step further mines the hard distinguished image pairs from unlabelled set to purify the learned clusters by a novel centrality-based adaptive purification strategy. Afterward, we introduce a customized purification loss for the purified clustering, which utilizes the complementary information in both labeled and unlabeled data to optimize the model for solving the ineffectiveness problem. The cluster-scatter procedure and the model optimization are performed in an iterative fashion to achieve the promising performance while greatly reducing the annotation cost. Extensive experimental results have demonstrated that MASS can even achieve a competitive performance with fully supervised methods in the case of extremely less annotation requirements.

Global oceans have absorbed a substantial portion of the anthropogenic carbon dioxide (CO2) emitted into the atmosphere. Data-based machine learning (DML) estimates for the oceanic CO2 sink have become an import part of the Global Carbon Budget in recent years. Although DML models are considered objective as they impose very few subjective conditions in optimizing model parameters, they face the challenge of data scarcity problem when applied to mapping ocean CO2 concentrations, from which air-sea CO2 fluxes can be computed. Data scarcity forces DML models to pool multiple years’ data for model training. When the time span extends to a few decades, the result could be largely affected by how ocean CO2 trends are obtained. This study extracted the trends using a new method and reconstructed monthly surface ocean CO2 concentrations and air-sea fluxes in 1980–2020 with a spatial resolution of 1×1 degree. Comparing with six other products, our results show a smaller oceanic sink and the sink in early and late year of the modelled period could be overestimated if ocean CO2 trends were not well processed by models. We estimated that the oceanic sink has increased from 1.79 PgC yr-1 in 1980s to 2.58 PgC yr-1 in 2010s with a mean acceleration of 0.027 PgC yr-2.

Global oceans have absorbed a substantial portion of the anthropogenic carbon dioxide (CO2) emitted into the atmosphere. Data-based machine learning (DML) estimates for the oceanic CO2 sink have become an import part of the Global Carbon Budget in recent years. Although DML models are considered objective as they impose very few subjective conditions in optimizing model parameters, they face the challenge of data scarcity problem when applied to mapping ocean CO2 concentrations, from which air-sea CO2 fluxes can be computed. Data scarcity forces DML models to pool multiple years’ data for model training. When the time span extends to a few decades, the result could be largely affected by how ocean CO2 trends are obtained. This study extracted the trends using a new method and reconstructed monthly surface ocean CO2 concentrations and air-sea fluxes in 1980–2020 with a spatial resolution of 1×1 degree. Comparing with six other products, our results show a smaller oceanic sink and the sink in early and late year of the modelled period could be overestimated if ocean CO2 trends were not well processed by models. We estimated that the oceanic sink has increased from 1.79 PgC yr-1 in 1980s to 2.58 PgC yr-1 in 2010s with a mean acceleration of 0.027 PgC yr-2.

Global oceans have absorbed a substantial portion of the anthropogenic carbon dioxide (CO2) emitted into the atmosphere. Data-based machine learning (DML) estimates for the oceanic CO2 sink have become an import part of the Global Carbon Budget in recent years. Although DML models are considered objective as they impose very few subjective conditions in optimizing model parameters, they face the challenge of data scarcity problem when applied to mapping ocean CO2 concentrations, from which air-sea CO2 fluxes can be computed. Data scarcity forces DML models to pool multiple years’ data for model training. When the time span extends to a few decades, the result could be largely affected by how ocean CO2 trends are obtained. This study extracted the trends using a new method and reconstructed monthly surface ocean CO2 concentrations and air-sea fluxes in 1980–2020 with a spatial resolution of 1×1 degree. Comparing with six other products, our results show a smaller oceanic sink and the sink in early and late year of the modelled period could be overestimated if ocean CO2 trends were not well processed by models. We estimated that the oceanic sink has increased from 1.79 PgC yr-1 in 1980s to 2.58 PgC yr-1 in 2010s with a mean acceleration of 0.027 PgC yr-2.