DeepFake detection with multi-scale convolution and vision transformer

The advancement of generative models has made it easier to create highly realistic Deepfake videos. This accessibility has led to a surge in research on Deepfake detection to mitigate potential misuse. Typically, Deepfake detection models utilize binary backbones, even though the training dataset contains additional exploitable information, such as the Deepfake generation method employed for each video. However, recent findings suggest that inferring a binary class from a multi-class backbone yields superior performance compared to directly employing a binary backbone. Building upon this research, our article introduces two novel methods to infer a binary class from a multi-class backbone. The first method, named root dummies , leverages the dummy triplet loss, which employs fixed vectors (i.e., dummies) instead of mined positives and negatives in the triplet loss. By training the multi-class backbone with these dummies, we can easily infer a binary class during testing by adjusting the number of dummies (from six during training to two during inference). Through this approach, we achieve an accuracy improvement of 0.23% compared to the existing inference method, without requiring additional training. The second proposed method is transfer learning. It involves training a classifier, such as a support vector machine, to predict binary classes based on the image embeddings generated by the multi-class backbone. Although this method necessitates additional training, it further enhances the model’s performance, resulting in an accuracy increase of 1.79%. In summary, our proposed methods improve the accuracy of Deepfake detection by simply modifying the number of classes during training, making them suitable for integration into a variety of existing Deepfake training pipelines. Additionally, to foster reproducible research, we have made the source code of our solution publicly available at https://github.com/beuve/DmyT .

The rapid development and prevalence of generative AI have made it easy for people to create high-quality deepfake images and videos, but their abuses have also increased exponentially. To mitigate potential social disruption, it is crucial to quickly detect the authenticity of each deepfake content hidden in a sea of information. While researchers have worked on developing deep learning-based methods, the deepfake datasets utilized in these studies are far from the real world in terms of their qualities; most popular deepfake datasets are human-distinguishable. To address this problem, we present a novel deepfake dataset, HiDF, a high-quality and human-indistinguishable deepfake dataset consisting of 62K images and 8K videos. HiDF is a meticulously curated dataset that includes diverse subjects that have undergone rigorous quality checks. A comparison of the quality between HiDF and existing deepfake datasets demonstrates that HiDF is human-indistinguishable. Hence, it can be a valuable benchmark dataset for deepfake detection tasks. Data and code (https://github.com/DSAIL-SKKU/HiDF) are publicly available for future deepfake detection research.

The rapid development of deepfake technology has led to the generation of a large amount of tampered video and image content, posing a major challenge to content authenticity verification. In particular, detecting deepfakes in image sequences (e.g., agricultural product packaging) is particularly difficult because the anomalies introduced by the tampering techniques are often subtle and temporally continuous. In this paper, we propose a new deepfake detection method based on time series, combining independent component analysis (FastICA) with anomaly detection techniques. We first apply FastICA to extract independent components from image sequences to identify anomalous visual patterns that are unique to deepfake tampering. In addition, we use an efficient anomaly detection algorithm, LSHiforest, to achieve scalable and accurate identification of suspicious sequences. Experimental results show that the proposed method can still detect deepfake content with high accuracy in challenging scenarios with complex temporal dynamics. Our work provides a promising solution for real‐time and large‐scale detection of deepfake content in dynamic media.

Remote photoplethysmography is susceptible to motion artifacts and individual physiological variations in complex environments. This paper proposes a remote heart rate estimation method based on frequency regulation and multi-scale spatio-temporal modeling. To address artifact noise issues, a frequency-regulated normalization module is designed to emphasize the dominant heart rate frequency while suppressing noise. To address the issue of individual physiological variations, the proposed method introduces a multi-level spatio-temporal feature fusion module to comprehensively capture physiological information through multi-scale convolutions and cross-layer integration. Subsequently, a dynamic weighting spatio-temporal feature module is introduced during spatio-temporal modeling to enhance long-term dependency modeling. Experimental results demonstrate that the proposed method achieves superior performance in cross-dataset evaluation. When trained on the PURE dataset and tested on the UBFC-rPPG dataset, the mean absolute error decreases from 1.31 to 1.28. Conversely, when trained on the UBFC-rPPG dataset and tested on the PURE dataset, the mean absolute error further decreases from 0.97 to 0.82. These results significantly outperform existing state-of-the-art methods, demonstrating the strong generalization capability and outstanding performance of our model across datasets. From the perspectives of frequency-regulated and multi-scale spatio-temporal modeling, this work enriches the modeling methodology for remote photoplethysmography pulse wave-based heart rate estimation, enhancing the stability and usability of remote heart rate estimation under complex interference and cross-scenario conditions.

One of the first stages of the three‐dimensional (3D) subsurface modeling process involves collation and analysis of available borehole and/or outcrop data to identify individual subsurface units, usually distinguished by the grain size of the sediment, and the elevation of their bounding contacts. Input data can come from a variety of sources and may be categorized according to their reliability and/or quality. The output from the 3D model is a prediction of subsurface conditions based on these data and the reliability of the output model is highly dependent on both the quality of input data and the types of interpolation methods used. This article presents a new quality weighting methodology that allows the user to assign a differential weighting factor to data points of variable quality in the modeling process. Input data are categorized into high and low quality datasets which are then recombined using a grid math process in which a differential “weighting” factor is applied. This allows the 3D modeling program to maximize the use and effectiveness of data from all available sources while giving high quality data greater influence on the final model output, and will result in the generation of more accurate and reliable 3D subsurface models.

Learning a deep dual-level network for robust DeepFake detection

The number of new falsified video contents is dramatically increasing, making the need to develop effective deepfake detection methods more urgent than ever. Even though many existing deepfake detection approaches show promising results, the majority of them still suffer from a number of critical limitations. In general, poor generalization results have been obtained under unseen or new deepfake generation methods. Consequently, in this paper, we propose a deepfake detection method called HCiT, which combines Convolutional Neural Network (CNN) with Vision Transformer (ViT). The HCiT hybrid architecture exploits the advantages of CNN to extract local information with the ViT's self-attention mechanism to improve the detection accuracy. In this hybrid architecture, the feature maps extracted from the CNN are feed into ViT model that determines whether a specific video is fake or real. Experiments were performed on Faceforensics++ and DeepFake Detection Challenge preview datasets, and the results show that the proposed method significantly outperforms the state-of-the-art methods. In addition, the HCiT method shows a great capacity for generalization on datasets covering various techniques of deepfake generation. The source code is available at: https://github.com/KADDAR-Bachir/HCiT

The rapid proliferation of deepfake technology, driven by advanced generative models like GANs and Diffusion Models, has enabled the creation of hyper-realistic manipulated media that threatens information integrity, personal reputation, and national security. As these tools become more accessible, the need for automated, robust, and efficient detection systems is critical. Existing detection methodologies largely rely on Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), or Vision Transformers (ViTs). However, each approach faces significant limitations. CNN-based methods suffer from "temporal blindness," failing to detect inconsistencies across video frames. Hybrid CNN-RNN models address this but are hindered by slow training speeds and limited capacity for modeling long-range dependencies. While Transformer-based models excel at capturing global context, their quadratic computational complexity (O(N2)) makes them computationally expensive and memory-intensive, rendering them inefficient for processing long, high-resolution video sequences. Furthermore, many current models struggle to generalize effectively to unseen datasets and novel manipulation techniques. To overcome these challenges, this paper proposes a novel hybrid deepfake detection framework that integrates an EfficientNet backbone with the Mamba Selective State Space Model (SSM). This architecture leverages EfficientNet for robust spatial feature extraction and utilizes Mamba for temporal sequence modeling. Mamba introduces a content-aware selection mechanism and operates with linear-time complexity (O(N2)), allowing it to efficiently capture subtle long-range temporal inconsistencies that traditional models miss, without the heavy computational cost of Transformers. Experimental results demonstrate that this proposed system offers a scalable, efficient solution with superior generalization capabilities, making it highly suitable for real-time deepfake detection in diverse and complex video environments. Keywords: Deepfake Detection, GAN, Selective State Space Models (SSM), Mamba model, Vision Mamba (Vim), Temporal Sequence Modeling, EfficientNet Backbone, machine learning, deep learning, convolutional neural networks, transformers, attention mechanisms, benchmarking systems, datasets, Synthetic Media Detection, Linear-Time Complexity, Video Anomaly Detection, Computational Efficiency, Hybrid CNN-SSM, Facial Forgery Detection, Long-Range Dependency Modeling.

Detection of fake audio and video is a challenging problem. Deepfake is popularly used for creating fake audio and video content using deep learning. Deepfakes, artificially created audiovisual interpretations can be used to degrade the reputation of a renowned person, hate-speech, or affect public belief. The development of novel methods for identifying various deep fake video types has received a significant amount of research throughout the years. In this research, we present a thorough comparative analysis of current state-of-the-art deepfake detection methods. The primary goal of our research is to identify the factors that contribute to the performance degradation of deepfake detection models currently being used when tested against a comprehensive dataset.

Auditory attention switch detection (AASD) poses significant challenges for adaptive neurotechnologies, particularly under electroencephalogram (EEG) with low signal-to-noise ratios (SNRs). However, the performance of existing methods is limited due to insufficient feature discriminability and high detection delay. To solve the problem, this paper proposes a Hierarchical Spatiotemporal Network (HSTN) for detecting auditory attention switch from EEG signals. The model employs a hierarchical spatiotemporal encoder to extract spatiotemporal features of EEG signals, integrates short-term transient and long-term dependency information through a multi-scale gated attention mechanism, and synchronously optimizes auditory attention switch detection and auditory attention decoding tasks via a multi-task joint training strategy. Experimental results demonstrate that HSTN significantly outperforms baseline models in both auditory attention switch detection (AASD F1=0.89, accuracy 88.6%) and auditory attention decoding tasks (AAD accuracy 89.3%), with superior model parameter efficiency and inference time. Ablation experiments further validate the critical roles of multi-task learning, gated attention, and multi-scale convolutions. This study provides an efficient solution for auditory attention switch detection in complex auditory scenarios.Clinical Relevance-The study confirms that spatiotemporal feature encoding combined with multi-task joint training significantly enhances performance in EEG attention switch detection, providing a practical technical framework for enabling dynamic sound source enhancement in intelligent hearing aids and auditory brain-computer interface systems.

In the field of ophthalmic surgery, many clinicians nowadays record their microscopic procedures with a video camera and use the recorded footage for later purpose, such as forensics, teaching, or training. However, in order to efficiently use the video material after surgery, the video content needs to be analyzed automatically. Important semantic content to be analyzed and indexed in these short videos are operation instruments, since they provide an indication of the corresponding operation phase and surgical action. Related work has already shown that it is possible to accurately detect instruments in cataract surgery videos. However, their underlying dataset (from the CATARACTS challenge) has very good visual quality, which is not reflecting the typical quality of videos acquired in general hospitals. In this paper, we therefore analyze the generalization performance of deep learning models for instrument recognition in terms of dataset change. More precisely, we trained such models as ResNet-50, Inception v3 and NASNet Mobile using a dataset of high visual quality (CATARACTS) and test it on another dataset with low visual quality (Cataract-101), and vice versa. Our results show that the generalizability is surprisingly low in general, but slightly worse for the model trained on the high-quality dataset.

Computer vision is an interdisciplinary scientific field that deals with how computers obtain, store, interpret and understand digital images or videos using artificial intelligence based on neural networks, machine learning and deep learning methodologies. They are used in countless applications such as image retrieval and classification, driving and transport monitoring, medical diagnostics and aerial monitoring. Written by a team of international experts, this edited book covers the state-of-the-art of advanced research in the fields of computer vision and recognition systems from fundamental concepts to methodologies and technologies and real world applications including object detection, biometrics, Deepfake detection, sentiment and emotion analysis, traffic enforcement camera monitoring, vehicle control and aerial remote sensing imagery. The book will be useful for industry and academic researchers, scientists and engineers in the fields of computer vision, machine vision, image processing and recognition, multimedia, AI, machine and deep learning, data science, biometrics, security, and signal processing. It will also make a great course reference for advanced students and lecturers in these fields of research.

With the wide application of data-driven machine learning in various fields, the impact of data quality on model performance, reliability, and generalization ability has become increasingly prominent. High-quality datasets not only enhance the stability of model training and the accuracy of predictions but also support more effective feature utilization and modeling of complex tasks. However, existing dataset evaluation methods primarily focus on a single metric, such as prediction accuracy or annotation quality, lacking a comprehensive, multi-dimensional assessment of the dataset's overall quality. In response to this issue, this study proposes the Comprehensive Data Quality Index (CDQI), which uniformly quantifies the four dimensions of prediction performance (P), model complexity (C), mean feature information (MFE), and model robustness (R). To comprehensively evaluate the contribution of the dataset to model training. This paper first introduces the model performance indicators, feature information quantity measurement, and cross-validation methods, laying a theoretical foundation for the construction of CDQI. Subsequently, an empirical analysis was conducted on four types of classic datasets, and a visual comparison of CDQI and each index was carried out, accompanied by tables and radar charts. The experimental results demonstrate that CDQI can effectively distinguish between simple and complex datasets, accurately reflect the comprehensive performance of the model on various data, and provide a reliable basis for dataset selection, model optimization, and experimental design. The CDQI framework proposed in this study has good universality and can provide a reference for data quality assessment.

Although deep convolutional neural networks (CNNs) have received great attention in image denoising, most CNN-based methods do not take full account of the hierarchical features of the original low-quality images, including the spatial feature information within channels, which decreases the representational capacity of the network. To solve this problem, we propose a dense activation network (DAN) with a dense activation block (DAB) consisting of a dense information fusion (DIF) net and a spatial activation (SA) net. Specifically, DIF takes into account both local feature information in the current block and global feature information across blocks to enhance the propagation of feature information, and it adopts dilated convolution to enlarge the receptive field of a network. Furthermore, we focus on the intra-channel spatial relationships and propose a novel SA net that adaptively recalibrates features by considering the spatial relationships within channels. Experiments on benchmark datasets show that our DAN models achieve favorable performance against state-of-the-art methods.

A fast-paced development of DeepFake generation techniques challenge the detection schemes designed for known type DeepFakes. A reliable Deepfake detection approach must be agnostic to generation types, which can present diverse quality and appearance. Limited generalizability across different generation schemes will restrict the wide-scale deployment of detectors if they fail to handle unseen attacks in an open set scenario. We propose a new approach, Multi-Channel Xception Attention Pairwise Interaction (MCX-API), that exploits the power of pairwise learning and complementary information from different color space representations in a fine-grained manner. We first validate our idea on a publicly available dataset in a intra-class setting (closed set) with four different Deepfake schemes. Further, we report all the results using balanced-open-set-classification (BOSC) accuracy in an inter-class setting (open-set) using three public datasets. Our experiments indicate that our proposed method can generalize better than the state-of-the-art Deepfakes detectors. We obtain 98.48% BOSC accuracy on the <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$FF++$</tex> dataset and 90.87% BOSC accuracy on the CelebDF dataset suggesting a promising direction for generalization of DeepFake detection. We further utilize t-SNE and attention maps to interpret and visualize the decision-making process of our proposed network.