Eff-YNet: A Dual Task Network for DeepFake Detection and Segmentation

Despite the promising advances in deepfake detection on current datasets, detecting visual deepfakes in real-world scenarios (e.g., deepfake videos and live streaming on YouTube) remains a challenge due to the inherent quality degradation such as unpredictable compression employed by social media platforms. Such degradation perturbs discernible forgery clues and diminishes the effectiveness of deepfake detection methods, raising a critical safety concern to the misuse of forgery faces in real-world scenarios. In this paper, we aim to understand the impacts of real-world degradation on the robustness of deepfake detection. Particularly, we investigate the risk of degraded deepfakes towards their detection on two real-world scenarios (i.e., deepfake videos and deepfake live streaming on social media platforms). By measuring the effects of real-world degradations on the performance and representation capabilities of detection models, we reveal that real-world deepfakes can be simulated via common degradation operations (e.g., JPEG compression) as they are perceptually similar to deepfake detectors. By analyzing the training dynamics under different sequences of training samples, we observe that the training order of deepfakes progressing from non-degraded (easy) to heavily degraded (hard) enhances the adaptability of detection models to various degradation in real-world scenarios. Drawing from these observations, we present a novel deepfake detection method ProFake to enhance the robustness of deepfake detection against real-world quality degradations. ProFake enables quality-adaptive learning via progressively degrade, detect and assign weights for the training samples driven by the feedback of model performance and image quality, which ensures that our model gradually focuses on more challenging samples to achieve quality-adaptive deepfake detection. Extensive experiments show that compared with existing methods, ProFake improves deepfake detection accuracy by an average of over 10 % in real-world scenarios and by an average of over 30 % in heavily degraded scenarios, while maintaining comparable performance in detecting high-quality deepfakes.

AI-synthesized face-swapping videos, commonly known as DeepFakes, is an emerging problem threatening the trustworthiness of online information. The need to develop and evaluate DeepFake detection algorithms calls for large-scale datasets. However, current DeepFake datasets suffer from low visual quality and do not resemble DeepFake videos circulated on the Internet. We present a new large-scale challenging DeepFake video dataset, Celeb-DF, which contains 5, 639 high-quality DeepFake videos of celebrities generated using an improved synthesis process. We conduct a comprehensive evaluation of DeepFake detection methods and datasets to demonstrate the escalated level of challenges posed by Celeb-DF. Then we introduce Landmark Breaker, the first dedicated method to disrupt facial landmark extraction, and apply it to the obstruction of the generation of DeepFake videos. The experiments are conducted on three state-of-the-art facial landmark extractors using our Celeb-DF dataset.

In recent years, the abuse of a face swap technique called deepfake has raised enormous public concerns. So far, a large number of deepfake videos (known as deepfakes) have been crafted and uploaded to the internet, calling for effective countermeasures. One promising countermeasure against deepfakes is deepfake detection. Several deepfake datasets have been released to support the training and testing of deepfake detectors, such as DeepfakeDetection [1] and FaceForensics++ [23]. While this has greatly advanced deepfake detection, most of the real videos in these datasets are filmed with a few volunteer actors in limited scenes, and the fake videos are crafted by researchers using a few popular deepfake softwares. Detectors developed on these datasets may become less effective against real-world deepfakes on the internet. To better support detection against real-world deepfakes, in this paper, we introduce a new dataset WildDeepfake, which consists of 7,314 face sequences extracted from 707 deepfake videos collected completely from the internet. WildDeepfake is a small dataset that can be used, in addition to existing datasets, to develop and test the effectiveness of deepfake detectors against real-world deepfakes. We conduct a systematic evaluation of a set of baseline detection networks on both existing and our WildDeepfake datasets, and show that WildDeepfake is indeed a more challenging dataset, where the detection performance can decrease drastically. We also propose two (eg. 2D and 3D) Attention-based Deepfake Detection Networks (ADDNets) to leverage the attention masks on real/fake faces for improved detection. We empirically verify the effectiveness of ADDNets on both existing datasets and WildDeepfake. The dataset is available at: https://github.com/deepfakeinthewild/deepfake-in-the-wild.

The proliferation of the capability for producing and distributing deepfake videos threatens the integrity of systems of justice, democratic processes, and the general ability to critically assess evidence. This study sought to identify individual differences that predict one’s ability to detect these forgeries. It was hypothesized that measures of affect detection and political orientation would correlate with performance on a deepfake detection task. Within a sample (N = 173) of college undergraduates and participants from Amazon’s M-turk, affect detection ability was shown to correlate with deepfake detection ability, r(171) = .73, p < .001, and general orientation to the political left was shown to correlate with deepfake detection ability, r(171) = .42, p < .001. The results of this study serve to identify populations who are particularly susceptible to deception via deepfake video and to inform the development of interventions that may help defend against attempts to influence them.

The rapid proliferation of deepfake technology, driven by advanced generative models like GANs and Diffusion Models, has enabled the creation of hyper-realistic manipulated media that threatens information integrity, personal reputation, and national security. As these tools become more accessible, the need for automated, robust, and efficient detection systems is critical. Existing detection methodologies largely rely on Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), or Vision Transformers (ViTs). However, each approach faces significant limitations. CNN-based methods suffer from "temporal blindness," failing to detect inconsistencies across video frames. Hybrid CNN-RNN models address this but are hindered by slow training speeds and limited capacity for modeling long-range dependencies. While Transformer-based models excel at capturing global context, their quadratic computational complexity (O(N2)) makes them computationally expensive and memory-intensive, rendering them inefficient for processing long, high-resolution video sequences. Furthermore, many current models struggle to generalize effectively to unseen datasets and novel manipulation techniques. To overcome these challenges, this paper proposes a novel hybrid deepfake detection framework that integrates an EfficientNet backbone with the Mamba Selective State Space Model (SSM). This architecture leverages EfficientNet for robust spatial feature extraction and utilizes Mamba for temporal sequence modeling. Mamba introduces a content-aware selection mechanism and operates with linear-time complexity (O(N2)), allowing it to efficiently capture subtle long-range temporal inconsistencies that traditional models miss, without the heavy computational cost of Transformers. Experimental results demonstrate that this proposed system offers a scalable, efficient solution with superior generalization capabilities, making it highly suitable for real-time deepfake detection in diverse and complex video environments. Keywords: Deepfake Detection, GAN, Selective State Space Models (SSM), Mamba model, Vision Mamba (Vim), Temporal Sequence Modeling, EfficientNet Backbone, machine learning, deep learning, convolutional neural networks, transformers, attention mechanisms, benchmarking systems, datasets, Synthetic Media Detection, Linear-Time Complexity, Video Anomaly Detection, Computational Efficiency, Hybrid CNN-SSM, Facial Forgery Detection, Long-Range Dependency Modeling.

Medical image segmentation is a crucial step in diagnosis and analysis of diseases for clinical applications. Deep convolutional neural network methods such as DeepLabv3+ have successfully been applied for medical image segmentation, but multi-level features are seldom integrated seamlessly into different attention mechanisms, and few studies have fully explored the interactions between medical image segmentation and classification tasks. Herein, we propose a feature-compression-pyramid network (FCP-Net) guided by game-theoretic interactions with a hybrid loss function (HLF) for the medical image segmentation. The proposed approach consists of segmentation branch, classification branch and interaction branch. In the encoding stage, a new strategy is developed for the segmentation branch by applying three modules, e.g., embedded feature ensemble, dilated spatial mapping and channel attention (DSMCA), and branch layer fusion. These modules allow effective extraction of spatial information, efficient identification of spatial correlation among various features, and fully integration of multi-receptive field features from different branches. In the decoding stage, a DSMCA module and a multi-scale feature fusion module are used to establish multiple skip connections for enhancing fusion features. Classification and interaction branches are introduced to explore the potential benefits of the classification information task to the segmentation task. We further explore the interactions of segmentation and classification branches from a game theoretic view, and design an HLF. Based on this HLF, the segmentation, classification and interaction branches can collaboratively learn and teach each other throughout the training process, thus applying the conjoint information between the segmentation and classification tasks and improving the generalization performance. The proposed model has been evaluated using several datasets, including ISIC2017, ISIC2018, REFUGE, Kvasir-SEG, BUSI, and PH2, and the results prove its competitiveness compared with other state-of-the-art techniques.

Deepfake techniques can forge the visual or audio signals in the video, which leads to inconsistencies between visual and audio (VA) signals. Therefore, multimodal detection methods expose deepfake videos by extracting VA inconsistencies. Recently, deepfake technology has started VA collaborative forgery to obtain more realistic deepfake videos, which poses new challenges for extracting VA inconsistencies. Recent multimodal detection methods propose to first extract natural VA correspondences in real videos in a self-supervised manner, and then use the learned real correspondences as targets to guide the extraction of VA inconsistencies in the subsequent deepfake detection stage. However, the inherent VA relations are difficult to extract due to the modality gap, which leads to the limited auxiliary performance of the aforementioned self-supervised methods. In this paper, we propose Predictive Visual-audio Alignment Self-supervision for Multimodal Deepfake Detection (PVASS-MDD), which consists of PVASS auxiliary and MDD stages. In the PVASS auxiliary stage in real videos, we first devise a three-stream network to associate two augmented visual views with corresponding audio clues, leading to explore common VA correspondences based on cross-view learning. Secondly, we introduce a novel cross-modal predictive align module for eliminating VA gaps to provide inherent VA correspondences. In the MDD stage, we propose to the auxiliary loss to utilize the frozen PVASS network to align VA features of real videos, to better assist multimodal deepfake detector for capturing subtle VA inconsistencies. We conduct extensive experiments on existing widely used and latest multimodal deepfake datasets. Our method obtains a significant performance improvement compared to state-of-the-art methods.

In this paper we propose a novel human-centered approach for detecting forgery in face images, using dynamic prototypes as a form of visual explanations. Currently, most state-of-the-art deepfake detections are based on black-box models that process videos frame-by-frame for inference, and few closely examine their temporal inconsistencies. However, the existence of such temporal artifacts within deepfake videos is key in detecting and explaining deepfakes to a supervising human. To this end, we propose Dynamic Prototype Network (DPNet) - an interpretable and effective solution that utilizes dynamic representations (i.e., prototypes) to explain deepfake temporal artifacts. Extensive experimental results show that DPNet achieves competitive predictive performance, even on unseen testing datasets such as Google's DeepFakeDetection, DeeperForensics, and Celeb-DF, while providing easy referential explanations of deepfake dynamics. On top of DPNet's prototypical framework, we further formulate temporal logic specifications based on these dynamics to check our model's compliance to desired temporal behaviors, hence providing trustworthiness for such critical detection systems.

This article investigates deepfake detection by comparing the strategies used by young adults and seniors, focusing on identifying similarities and differences between the two age groups. Semi-structured interviews were conducted with 40 participants, including 20 young adults aged 21–35 and 20 seniors aged 55–70, to explore how they evaluate the authenticity of deepfake videos. The interview data were analyzed using qualitative content analysis to develop an in-depth understanding of age-related detection strategies. The findings reveal five common strategies employed by both groups: analysis of character appearance, scrutiny of non-human visual elements, assessment of audio cues, reliance on personal knowledge, and consultation with external sources. While both age groups draw on similar types of cues, seniors tend to rely more on accumulated life experience, whereas young adults place greater emphasis on information gained through digital media exposure. By highlighting these shared and divergent approaches, this study contributes to the deepfake detection literature and offers valuable insights into the factors shaping individuals’ strategies for identifying deepfake videos.

The rapid advancement of generative adversarial networks (GANs) and other AI-driven synthesis techniques, deepfake videos have emerged as a significant threat to digital media integrity, enabling the creation of highly realistic but fake video content. These manipulated videos can be used maliciously in disinformation campaigns, identity theft, and other cybercrimes, making their detection a critical challenge. This paper presents a deep learning-based approach for deepfake video detection that leverages both spatial artifacts and temporal inconsistencies introduced during the manipulation process. The proposed method utilizes a hybrid architecture combining convolutional neural networks (CNNs) for frame-level feature extraction and recurrent neural networks (RNNs), specifically Long Short-Term Memory (LSTM) networks, to capture temporal dependencies across frames. We evaluate our method on widely used benchmark datasets including FaceForensics++, Celeb-DF, and DFDC, demonstrating superior accuracy, robustness, and generalization capability against multiple types of deepfake generation techniques. The experimental results validate the effectiveness of our approach in real-world scenarios, highlighting its potential for deployment in digital forensics, content moderation, and media verification systems. Keywords: Deepfake detection, Synthetic media, Deepfake forensics, AI-generated media,Face manipulation Video

Deepfake technology, allowing manipulations of audiovisual content by means of artificial intelligence, is on the rise. This has sparked concerns about a weaponization of manipulated videos for malicious ends. A theory on deepfake detection is presented and three preregistered studies examined the detection of deepfakes in the political realm (featuring UK’s Prime Minister Boris Johnson, Studies 1–3, or former U.S. President Barack Obama, Study 2). Based on two system models of information processing as well as recent theory and research on fake news, individual differences in analytic thinking and political interest were examined as predictors of correctly detecting deepfakes. Analytic thinking (Studies 1 and 2) and political interest (Study 1) were positively associated with identifying deepfakes and negatively associated with the perceived accuracy of a fake news piece about a leaked video (whether or not the deepfake video itself was presented, Study 3). Implications for research and practice are discussed.

The advancement of generative models has made it easier to create highly realistic Deepfake videos. This accessibility has led to a surge in research on Deepfake detection to mitigate potential misuse. Typically, Deepfake detection models utilize binary backbones, even though the training dataset contains additional exploitable information, such as the Deepfake generation method employed for each video. However, recent findings suggest that inferring a binary class from a multi-class backbone yields superior performance compared to directly employing a binary backbone. Building upon this research, our article introduces two novel methods to infer a binary class from a multi-class backbone. The first method, named root dummies , leverages the dummy triplet loss, which employs fixed vectors (i.e., dummies) instead of mined positives and negatives in the triplet loss. By training the multi-class backbone with these dummies, we can easily infer a binary class during testing by adjusting the number of dummies (from six during training to two during inference). Through this approach, we achieve an accuracy improvement of 0.23% compared to the existing inference method, without requiring additional training. The second proposed method is transfer learning. It involves training a classifier, such as a support vector machine, to predict binary classes based on the image embeddings generated by the multi-class backbone. Although this method necessitates additional training, it further enhances the model’s performance, resulting in an accuracy increase of 1.79%. In summary, our proposed methods improve the accuracy of Deepfake detection by simply modifying the number of classes during training, making them suitable for integration into a variety of existing Deepfake training pipelines. Additionally, to foster reproducible research, we have made the source code of our solution publicly available at https://github.com/beuve/DmyT .

Large annotated datasets are required for training deep learning models, but in medical imaging data sharing is often complicated due to ethics, anonymization and data protection legislation. Generative AI models, such as generative adversarial networks (GANs) and diffusion models, can today produce very realistic synthetic images, and can potentially facilitate data sharing. However, in order to share synthetic medical images it must first be demonstrated that they can be used for training different networks with acceptable performance. Here, we therefore comprehensively evaluate four GANs (progressive GAN, StyleGAN 1–3) and a diffusion model for the task of brain tumor segmentation (using two segmentation networks, U-Net and a Swin transformer). Our results show that segmentation networks trained on synthetic images reach Dice scores that are 80%–90% of Dice scores when training with real images, but that memorization of the training images can be a problem for diffusion models if the original dataset is too small. Our conclusion is that sharing synthetic medical images is a viable option to sharing real images, but that further work is required. The trained generative models and the generated synthetic images are shared on AIDA data hub.

Deepfake technology, which uses artificial intelligence to create realistic fake images, audio, and videos, has raised significant concerns due to its potential for misuse and manipulation. The emergence of deepfake technology poses a significant threat to the integrity of digital content, necessitating robust detection mechanisms. This paper proposes a novel method for deepfake detection by combining Overlapping Multiple Dynamic Images (OMDI) and Inversed Overlapping Multiple Dynamic Images (I‐OMDI). Both representations capture temporal inconsistencies and subtle visual artifacts in fake videos by effectively utilizing spatial–temporal information. Our approach employs EfficientNetB7 as the backbone for feature extraction, enabling the model to distinguish between real and fake videos with high accuracy. By combining OMDI and I‐OMDI with a weighted average strategy, we amplify the strengths of each method. Specifically, we assign equal weights of 0.5 to OMDI and I‐OMDI based on their individual contributions to performance metrics. This balance yields substantial performance improvements across multiple datasets. When evaluated on the Celeb‐DF v2 and DFDC datasets, our proposed model achieves state‐of‐the‐art results, with an AUC score of 0.9952 on Celeb‐DF v2 and 0.9947 on DFDC. These results underscore the robustness of the combined OMDI and I‐OMDI methods in identifying deepfake videos. Furthermore, our model demonstrates superior performance compared to existing methods, including those by Tran et al. and Heo et al., underscoring its effectiveness in practical deepfake detection applications. © 2025 Institute of Electrical Engineers of Japan and Wiley Periodicals LLC.

Generalizing deepfake detection across diverse manipulation techniques remains a critical challenge. In this paper, we propose a hybrid deepfake detection framework that combines global and patch-based local modeling to improve generalization. Our architecture integrates a Global Blending Detector (GBD), trained on self-blended images to capture coarse blending artifacts from full-face inputs, with a Patch-based Artifact Detector (PAD) that focuses on five strategically selected overlapping 2×2 facial patches. These patches are designed to extract subtle and localized forgery cues, particularly in semantically significant facial regions. The PAD module leverages both a frozen CLIP-ViT encoder for semantic context and a trainable EfficientNet-B4 to capture fine-grained visual anomalies. Features from both global and local branches are concatenated and fed into a unified classifier. Extensive experiments conducted on five benchmark datasets demonstrate the effectiveness of our approach. Our method achieves an average AUC of 91.4%, with top performance on CDF-v2 (95.6%), DFDCP (93.7%), and FFIW (93.9%), outperforming or matching state-of-the-art models in cross-dataset evaluations. These results confirm that the combination of global and region-specific features significantly enhances the robustness and generalizability of deepfake detection.