Self-supervised Contrastive BERT Fine-tuning for Fusion-Based Reviewed-Item Retrieval

Multisource and multimodal data fusion plays a pivotal role in large-scale artificial intelligence applications involving big data. However, the choice of fusion strategies for different scenarios is often based on experimental comparisons, which leads to increased computational costs during model training and suboptimal performance during testing. In this paper, we present a theoretical analysis of early fusion, late fusion, and gradual fusion methods. We derive equivalence conditions between early and late fusions within the framework of generalized linear models. Moreover, we analyze the failure conditions of early fusion in the presence of nonlinear feature-label relationships. Furthermore, we propose an approximate equation for evaluating the accuracy of early and late fusion methods as a function of sample size, feature quantity, and modality number. We also propose a critical sample size threshold at which the performance dominance of early fusion and late fusion models undergoes a reversal. Finally, we introduce a fusion method selection paradigm for selecting the most appropriate fusion method prior to task execution and demonstrate its effectiveness through extensive numerical experiments. Our theoretical framework is expected to solve the problems of computational and resource costs in model construction, improving the scalability and efficiency of data fusion methods.

Graph contrastive learning has emerged as a powerful unsupervised graph representation learning tool. The key to the success of graph contrastive learning is to acquire high-quality positive and negative samples as contrasting pairs to learn the underlying structural semantics of the input graph. Recent works usually sample negative samples from the same training batch with the positive samples or from an external irrelevant graph. However, a significant limitation lies in such strategies: the unavoidable problem of sampling false negative samples. In this paper, we propose a novel method to utilize Counterfactual mechanism to generate artificial hard negative samples for Graph Contrastive learning, namely CGC. We utilize a counterfactual mechanism to produce hard negative samples, ensuring that the generated samples are similar but have labels that differ from the positive sample. The proposed method achieves satisfying results on several datasets. It outperforms some traditional unsupervised graph learning methods and some SOTA graph contrastive learning methods. We also conducted some supplementary experiments to illustrate the proposed method, including the performances of CGC with different hard negative samples and evaluations for hard negative samples generated with different similarity measurements. The implementation code is available online to ease reproducibility1.

Siamese network-based visual tracking algorithms have achieved excellent performance in recent years, but challenges such as fast target motion, shape and scale variations have made the tracking extremely difficult. The regression of anchor-free tracking has low computational complexity, strong real-time performance, and is suitable for visual tracking. Based on the anchor-free siamese tracking framework, this paper firstly introduces balance factors and modulation coefficients into the cross-entropy loss function to solve the classification inaccuracy caused by the imbalance between positive and negative samples as well as the imbalance between hard and easy samples during the training process, so that the model focuses more on the positive samples and the hard samples that make the major contribution to the training. Secondly, the intersection over union (IoU) loss function of the regression branch is improved, not only focusing on the IoU between the predicted box and the ground truth box, but also considering the aspect ratios of the two boxes and the minimum bounding box area that accommodate the two, which guides the generation of more accurate regression offsets. The overall loss of classification and regression is iteratively minimized and improves the accuracy and robustness of visual tracking. Experiments on four public datasets, OTB2015, VOT2016, UAV123 and GOT-10k, show that the proposed algorithm achieves the state-of-the-art performance.

Despite the superior classification performance of support vector machines (SVMs), training SVMs on large datasets is still a challenging problem. Sample reduction methods have been proposed and shown to reduce the training complexity significantly, but more or less trade off the generalization performance. This paper presents an efficient sample reduction method for multi-class classification using one-vs-rest SVMs, called Multi-class Sample Selection (MUSS). For each binary one-vs-rest classification problem, positive samples and negative samples are selected based on the distances from the cluster centers of positive class, assuming that positive samples with large distances from the positive centers and negative samples with small distances from the positive centers are near the classification boundary. The intention of clustering is to improve the computation efficiency of sample selection, other than to select from cluster centers as previous methods did. Experiments on a wide variety of datasets demonstrate the superiority of the proposed MUSS over other competitive algorithms in respect of the tradeoff between reduced sample size and classification performance. The experimental results show that MUSS also works well for binary classification problems.

The aim of the study was to inspect the acoustic properties and sleep characteristics of a preapneic snoring sound. The feasibility of forecasting upcoming respiratory events by snoring sound was also investigated. Participants with habitual snoring or a heavy breathing sound during sleep were recruited consecutively. Polysomnography was conducted, and snoring-related breathing sound was recorded simultaneously. Acoustic features and sleep features were extracted from 30-second samples, and a machine learning algorithm was used to establish 2 prediction models. A total of 74 eligible participants were included. Model 1, tested by 5-fold cross-validation, achieved an accuracy of 0.92 and an area under the curve of 0.94 for respiratory event prediction. Model 2, with acoustic features and sleep information tested by Leave-One-Out cross-validation, had an accuracy of 0.78 and an area under the curve of 0.80. Sleep position was found to be the most important among all sleep features contributing to the performance of the 2 models. Preapneic sound presented unique acoustic characteristics, and snoring-related breathing sound could be deployed as a real-time apneic event predictor. The models, combined with sleep information, serve as a promising tool for an early warning system to forecast apneic events. Wang B, Yi X, Gao J, etal. Real-time prediction of upcoming respiratory events via machine learning using snoring sound signal. J Clin Sleep Med. 2021;17(9):1777-1784.

In this paper we explore automatic genre labelling of motion picture previews using audio-visual features present in movie trailers and the focus is on fusion techniques (early fusion and late fusion) and the resultant improvement on classification accuracy. This paper proposes a novel combination of deep learned features (from a pretrained VGG-16 model) obtained using a state-of-the-art shot detector and hand-crafted audio features. This combination of features and an associated comparison of early and late fusion with these features has not been attempted in the literature before. Furthermore, two popular fusion techniques and three distinct classification algorithms are investigated to determine the optimal fusion technique and classifier combination. The study uses a subset of the LMTD-9 movie trailer dataset with selected genres (action, comedy, drama and horror). The best performing low-level audio features are comprised of timbre features extracted using the MIRtoolbox followed by standalone mel-frequency cepstral coefficients. The best performing high-level audio feature is tonality. Audio features are augmented by visual features extracted using a pre-trained convolutional neural network (VGG-16). Feature fusion (early and late fusion) methods are investigated together with classification methods such as extreme gradient boosting, support vector machine and a neural network. Evaluation metrics such as precision, recall, confusion matrices and F1 score are used to measure classification accuracy. Early fusion methods outperform late fusion methods with a classification performance gain of approximately 10% for a four class classification problem. The best classification performance for early fusion obtained with a support vector machine is (73.12% accuracy), followed by the extreme gradient boosting classifier (69.37% accuracy) and neural network classifier (67.50% accuracy), whereas chance is 25%. It is shown that superior classification performance can be achieved by employing early feature fusion of low-level audio descriptors, high-level audio descriptors and high-level visual feature descriptors together with suitable classifiers.

In advertising, identifying the content safety of web pages is a significant concern since advertisers do not want brands to be associated with threatening content. At the same time, publishers would like to maximize the number of web pages on which they can place ads. Thus, a fine balance must be achieved while classifying content safety in order to satisfy both advertisers and publishers. In this paper, we propose a multimodal machine learning framework that fuses visual and textual information from web pages to improve current predictions of content safety. The primary focus is on late fusion, which involves combining final model outputs of separate modalities, such as images and text, to arrive at a single decision. This paper presents a fully automated machine learning framework that performs binary and multilabel classification using late fusion techniques. We also introduce additional work in early fusion, which involves extracting and fusing intermediate features from the two separate models. Our algorithms are applied to data extracted from relevant web pages in the advertising industry. Both of our late and early fusion methods obtain significant improvements over algorithms currently in use.

In this paper, a novel approach for discovering visual patterns associated with semantic concepts using web image resources is proposed.This approach can be used to improve the performance in image clustering and retrieval, image annotation, and other applications such as object recognition. Exploring the rich information in web images that represent semantic concepts as both visual content and text information, this research attempts to effectively learn intrinsic patterns related to semantic concepts. Because the quality of learning algorithms is strongly related to the selection of positive and negative samples, positive samples are first selected effectively, then negative samples are determined reliably based on the selected positive samples. Finally, a good quality visual model associated with a semantic concept is built through an unsupervised learning process. The proposed scheme is completely automatic,needing no human intervention,and is robust and reliable for generic images. Experimental results demonstrate the effectiveness of the proposed approach.

This paper presents a focused Web crawling system iSurfer for information retrieval from the Web. Different from other focused crawlers, iSurfer uses an incremental method to learn a page classification model and a link prediction model. It employs an online sample detector to incrementally distill new samples from crawled Web pages for online updating of the model learned. Other focused crawling systems use classifiers that are built from initial positive and negative samples and can not learn incrementally. The performances of these classifiers depend on the topical coverage of the initial positive and negative samples. However, the initial samples, particularly the negative ones, with a good coverage of target topics are difficult to find. Therefore, the iSurfer’s incremental learning strategy has an advantage. It starts from a few positive samples and gains more integrated knowledge about the target topics over time. Our experiments on various topics have demonstrated that the incremental learning method can improve the harvest rate with a few initial samples.KeywordsFocused CrawlerIncremental LearningPositive Sample Based LearningLink PredictionWeb Page Classification

Visual recognition inside the vehicle cabin leads to safer driving and more intuitive human-vehicle interaction but such systems face substantial obstacles as they need to capture different granularities of driver behaviour while dealing with highly limited body visibility and changing illumination. Multimodal recognition mitigates a number of such issues: prediction outcomes of different sensors complement each other due to different modality-specific strengths and weaknesses. While several late fusion methods have been considered in previously published frameworks, they constantly feature different architecture backbones and building blocks making it very hard to isolate the role of the chosen late fusion strategy itself.This paper presents an empirical evaluation of different paradigms for decision-level late fusion in video-based driver observation. We compare seven different mechanisms for joining the results of single-modal classifiers which have been both popular, (e.g. score averaging) and not yet considered (e.g. rank-level fusion) in the context of driver observation evaluating them based on different criteria and benchmark settings. This is the first systematic study of strategies for fusing outcomes of multimodal predictors inside the vehicles, conducted with the goal to provide guidance for fusion scheme selection.

The Covid-19 pandemic has not ended yet. Therefore, laboratory tests are very important for determining the diagnosis of Covid-19. At this case, PCR is the gold standard in laboratory tests. Therefore, the accuracy of the PCR assay should be a major concern. This study aims to evaluate the validation results of several Covid-19 PCR reagent kits. A total of 10 positive and negative clinical samples were used as test samples. DNA extraction was performed using a commercial reagent kit. PCR examination was carried out using four different blinded commercial reagent kits. The results of the examination were compared in each sample. The results showed that Reagents kit A, C, and D gave the same results for each sample, namely five positive and five negative samples. Reagent kit B showed different results, namely eight positive samples and two negative samples. A total of 3 (33.3%) of the 10 sample examinations indicated inaccurate results for Reagent kit B. The Reagent kit B test results using two different boxes showed different results with seven positive results by box 1, while eight positive results by another box. In conclusion, 1 out of 4 tested Covid-19 PCR kit reagents gave inaccurate results. Therefore, in covid-19 testing laboratories, it is recommended to choose a PCR reagent kit that has been approved by a trusted organization and perform reagent validation tests.

Live HIV rapid test kits (RTKs) are usually packaged with only a single test buffer vial. A major challenge with HIV testing is the problem of lack of the RTK buffer vial to complete the number of tests in an RTK pack due to misuse, misplacement or contamination of buffer with clients' blood during HIV testing. This has led to the use of sterile normal saline and tap water as substitute liquids in place of the buffer vial. This study compared the use of the commercially provided buffer vial and these substitute liquids in HIV rapid testing while using the commercially provided buffer as gold standard. One hundred (100) HIV positive samples from patients attending antiretroviral therapy (ART) clinic and 100 HIV negative blood samples from blood donors placed in EDTA were tested with Determine RTK using the commercially provided buffer and substitute liquids (Sterile normal saline and tap water). The Determine RTK was also tested using the commercially provided buffer, normal saline and tap water without addition of blood samples. Results of the tests with substitute liquids were compared with those obtained with that of RTK's commercially provided buffer. With HIV positive and negative blood samples, the expected test line and control line or only control line (for negative samples) were observed with the commercially provided buffer and sterile normal saline. When tested without blood samples using the RTK's buffer and normal saline, the control line was observed but no test line for all 100 tests done. Invalid test results were recorded in 48% of the negative samples and 40% of test with no blood added when with tap water was used. This study also shows varied colour intensity for the buffer vial and substitute liquids used. For samples tested with sterile normal saline, 52% of the tested samples showed stronger colour intensity in test line that control line, 37% had equal colour intensity while 11% had higher colour intensity in control than test line. Seventy seven (77%) of Samples tested with tap water had the colour intensity greater in test line than control line while 23% had equal colour intensity in both test and control line. With sterile normal saline used with HIV positive and negative blood samples, a 100% clear background like that of the gold standard was obtained. However, tap water when used as a substitute buffer to run all 100 HIV positive samples had an unclear background. An unclear background was also observed in 100% and 28% of the test ran with tap water only (without blood samples) and with HIV Negative samples respectively.

Contrastive learning is an important technique in the field of machine learning for learning data representations. In the field of self-supervised visual representation learning, the strategy of positive and negative sample selection is key to improving the efficiency and effectiveness of model learning. Traditional self-supervised learning methods often employ random sampling strategies to select positive and negative samples, but this approach may result in uneven sample quality when dealing with complex datasets, thereby affecting the learning outcomes. To alleviate this problem, this study is dedicated to exploring more effective strategies for positive and negative sample selection and processing to optimize the self-supervised learning process. To this end, we propose an improved method of self-supervised learning called contrastive learning with enhanced diversity. On the one hand, this method utilizes the weight parameters of the DINO pre-trained model to initialize the feature extraction network of SimCLR, providing more accurate calculations of feature similarity. On the other hand, by setting a threshold on the feature similarity matrix and penalizing (subtracting 0.5 from) similarity scores that do not exceed this threshold, we reduce the excessive impact of high similarity scores on model training, thereby helping the model better distinguish between positive and negative samples. In downstream image classification tasks, we conducted detailed evaluations of the improved model, specifically including fine-tuning and linear evaluation aspects. Experiments show that the proposed approach improves the performance of the loss functions and improves the accuracy of the proposed SimCLR model.

Learning discriminative features between abnormal and normal instances is crucial for video anomaly detection within the multiple instance learning framework. Existing methods primarily focus on instances with the highest anomaly scores, neglecting the identification and differentiation of hard samples, leading to misjudgments and high false alarm rates. To address these challenges, we propose a dual triplet contrastive loss strategy. This approach employs dual memory units to extract four key feature categories: hard samples, negative samples, positive samples, and anchor samples. Contrastive loss is utilized to constrain the distance between hard samples and other samples, enabling accurate identification of hard samples and enhancing the discriminative ability of hard samples and abnormal features. Additionally, a multi-scale feature perception module is designed to capture feature information at different levels, while an adaptive global-local feature fusion module constructs complementary feature enhancement through feature fusion. Experimental results demonstrate the effectiveness of our method, achieving AUC scores of 87.16% on the UCF-Crime dataset and AP scores of 83.47% on the XD-Violence dataset.

Reliable and rapid detection of severe acute respiratory syndrome coronavirus 2 in laboratory setting is critical to control the pandemic. We aimed to an evaluated polymerase chain reaction (PCR) efficiency of nasopharyngeal swabs stored in viral transport medium (VTM) in different temperatures. Ninety swabs taken into VTM were analyzed at the first hour, then divided into two groups with similar numbers of positive and negative samples. Positive samples of each group were also subgrouped according to Fam CT values as low CT (<25), medium CT (25-32), and high CT (32-38) groups. One group was stored at 4°C, while the other was stored at room temperature, PCR analyses were repeated every 24 h for 5 days and on Day 12. There was a total of 30 positive samples (12 low CT, 11 medium CT, and 7 high CT). The CT values of both groups remained unchanged in first 3 days while the CT values of the room temperature group increased after the third day. All of the positive samples remained positive in both groups for the first 5 days. On the 12th day, the total number of positives decreased to 8 in the room temperature group and 11 in the 4°C groups. All the low CT samples remained positive in both groups. In conclusion, it is safe to store positive samples in room temperature for up to 5 days. Only samples with high viral loads remain positive for 12 days, regardless of whether stored at room temperature or 4°C. Negative samples don't turn to invalid if stored in VTM.