Emotions Don't Lie

The advancement of generative models has made it easier to create highly realistic Deepfake videos. This accessibility has led to a surge in research on Deepfake detection to mitigate potential misuse. Typically, Deepfake detection models utilize binary backbones, even though the training dataset contains additional exploitable information, such as the Deepfake generation method employed for each video. However, recent findings suggest that inferring a binary class from a multi-class backbone yields superior performance compared to directly employing a binary backbone. Building upon this research, our article introduces two novel methods to infer a binary class from a multi-class backbone. The first method, named root dummies , leverages the dummy triplet loss, which employs fixed vectors (i.e., dummies) instead of mined positives and negatives in the triplet loss. By training the multi-class backbone with these dummies, we can easily infer a binary class during testing by adjusting the number of dummies (from six during training to two during inference). Through this approach, we achieve an accuracy improvement of 0.23% compared to the existing inference method, without requiring additional training. The second proposed method is transfer learning. It involves training a classifier, such as a support vector machine, to predict binary classes based on the image embeddings generated by the multi-class backbone. Although this method necessitates additional training, it further enhances the model’s performance, resulting in an accuracy increase of 1.79%. In summary, our proposed methods improve the accuracy of Deepfake detection by simply modifying the number of classes during training, making them suitable for integration into a variety of existing Deepfake training pipelines. Additionally, to foster reproducible research, we have made the source code of our solution publicly available at https://github.com/beuve/DmyT .

Detecting multimodal deepfakes has become a pressing concern due to the rising sophistication of generative techniques capable of creating highly convincing visual-speech synchronized deepfakes. As these manipulated contents proliferate on social media, remote education, smart meetings, smart homes, and virtual reality (VR), their increasing realism significantly impacts individuals, underscoring the urgent need for effective detection methods to curb their deceptive spread. Several research efforts have aimed to detect visual–audio deepfakes by examining inter-modal correlation and inter-class consistency. However, optimal detection results have yet to be achieved. Studies have indicated that detectors learn forged characteristics specific to particular generative methods and hide the real class in the visual modality, leading to a bias towards forgery distributions and degrading detection performance. Since audio modality data is a 2-D signal, we consider this bias in the audio modality to be more serious than in the visual modality, a conclusion we have verified through experiments. Therefore, an audio encoder that learns a universal audio distribution is indispensable for effective audio modality deepfake detection. In this work, we propose a new visual–audio deepfake detection method visual–audio multitask transformer with specialty audio encoder (VATS), designed to learn a more robust and comprehensive representation, thereby mitigating overfitting. The structure includes unimodal encoders for preliminary encoding and a dual-stream structure based on a multimodal framework for multilevel interaction and precise detection. To prevent the audio encoder from focusing excessively on specific identities or generator textures, we employ a pretrained speech encoder in the audio stream, while utilizing a linear projection encoder in the visual stream to focus on the fine-grained features of each video frame rather than the semantic information of the graphic. Our method comprises three key components: the fast temporal multimodal alignment and embedding module (FTAE), the modality interaction block (MIB), and the interpretable manipulated location module (IML). Experimental results on FakeAVCeleb demonstrate that our proposed method outperforms previous works in multimodal deepfake detection.

The rapid advancement of digital content generation technologies has led to the proliferation of deepfake media, posing significant security and privacy risks on social networks. Conventional deepfake detection methods struggle with accuracy, computational efficiency, and adaptability to evolving fake image generation techniques. To address these challenges, we propose a novel deepfake detection framework that integrates an Adaptive deep convolutional generative adversarial network (Adaptive-DCGAN) with a lightweight convolutional neural network (Lite-CNN). The Adaptive-DCGAN generates high-quality synthetic images, enhancing the training dataset, while the Lite-CNN efficiently classifies real and fake images with minimal computational overhead. Experimental results demonstrate that the proposed model achieves 95% accuracy, significantly outperforming existing GAN-CNN-based detection systems, which only reach 53% accuracy. Additionally, the framework exhibits superior performance metrics, including 95% precision, recall, and F1-score, ensuring robust and reliable deepfake detection. The proposed system offers a scalable and efficient solution for securing digital media platforms against deepfake threats. Future work will focus on improving generalization across diverse datasets, incorporating advanced synthetic image generation techniques, and enabling real-time deepfake detection for large-scale social media applications.

The rapid growth of deepfake technology, generating realistic manipulated media, poses a significant threat due to potential misuse. Therefore, effective detection methods are urgently needed to prevent malicious use, as current approaches often focus on single modalities or the simple fusion of audio–visual signals, limiting their accuracy. To solve this problem, we propose a deepfake detection scheme based on bimodal temporal feature prediction, which innovatively introduces the idea of temporal feature prediction into the audio–video bimodal deepfake detection task, aiming at fully exploiting the temporal laws of audio–visual modalities. First, pairs of adjacent audio–video sequence clips are used to construct input quadruples, and a dual-stream network is employed to extract temporal feature representations from video and audio, respectively. A video prediction module and an audio prediction module are designed to capture the temporal inconsistencies within each single modality by predicting future temporal features and comparing them with reference features. Then, a projection layer network is designed to align the audio–visual features, using contrastive loss functions to perform contrastive learning and maximize the differences between real and fake video modalities. Experiments on the FakeAVCeleb dataset demonstrate superior performance with an accuracy of 84.33% and an AUC of 89.91%, outperforming existing methods and confirming the effectiveness of our approach in deepfake detection.

Learning a deep dual-level network for robust DeepFake detection

The current challenge of Deepfake detection is the cross-domain performance on unseen Deepfake data. Instead of extracting forgery artifacts that are robust to the cross-domain scenarios as most previous works, we propose a novel method named Single-sIde Guided Meta-leArning framework for DeepFake detection (SIGMA-DF) which simulates the cross-domain scenarios during training by synthesizing virtual testing domain through meta-learning. In addition, SIGMA-DF integrates the meta-learning algorithm with a new ensemble meta-learning framework, which separately trains multiple meta-learners in the meta-train phase to aggregate multiple domain shifts in each iteration. Hence multiple cross-domain scenarios are simulated, better leveraging the domain knowledge. In addition, considering the contribution of hard samples in single-side distribution optimization, a novel weighted single-side loss function is proposed to only narrow the intra-class distance between real faces and enlarge the inter-class distance for both real and fake faces in embedding space with the awareness of sample weights. Extensive experiments are conducted on several standard Deepfake detection datasets to demonstrate that the proposed SIGMA-DF achieves state-of-the-art performance. In particular, in the cross-domain evaluation from FF++ to Celeb-DF and DFDC, our SIGMA-DF outperforms the baselines by 4.4% and 4.5% in terms of AUC, respectively.

Deepfake technology poses significant challenges to digital trust by generating highly convincing but synthetic multimedia content. This work focuses on advanced deepfake detection methodologies utilising deep convolutional neural networks (CNNs), specifically leveraging the EfficientNet-B6 architecture. The research employs the comprehensive DeepFake Detection Challenge (DFDC) dataset, known for its extensive diversity and scale. A robust preprocessing phase involved strategic frame extraction and precise facial cropping, ensuring optimal data quality, for training the detection model. Evaluation metrics, including Area Under the Curve (AUC) and Log Loss, were systematically applied to assess model performance. The results indicated strong performance of the EfficientNet-B6 model in accurately distinguishing manipulated videos from authentic content within the DFDC dataset. This project highlights the critical need for continually evolving deepfake detection algorithms to effectively respond to increasingly sophisticated generation techniques, thereby contributing to the broader initiative of combating digital misinformation.

- Deepfakes are artificially generated multimedia content that can convincingly mimic real human faces and voices using advanced AI techniques such as Generative Adversarial Networks (GANs). This poses serious ethical, social, and security challenges in digital communication. To address this issue, the proposed project presents a Multimodal Deepfake Detection System that integrates image, video, and audio analysis pipelines within a unified framework. The system employs Efficient Net-based CNN for image forgery detection, CNN combined with Bi-LSTM for temporal video analysis, and a 1D-CNN with LSTM for detecting manipulated or cloned audio. The predictions from these modalities are combined using a Fuzzy Fusion Engine, which intelligently weights each confidence score to produce a final verdict with high accuracy and interpretability. The model is trained using public Deepfake datasets such as Face Forensics++, Celeb-DF, and DFDC, with binary cross-entropy loss, data augmentation, and early stopping to ensure stable convergence and better generalization. The trained models are deployed on Hugging Face Spaces, while the web interface is hosted on Vercel, enabling real-time Deepfake detection for users through a browser interface. This approach enhances detection accuracy by leveraging multimodal evidence (visual, temporal, and auditory), improves generalization across datasets, and provides an explainable and efficient solution to combat the growing threat of Deepfakes. Key Words: Deepfake Detection, Multimodal Learning, Bi-LSTM, Fuzzy Fusion, Convolutional Neural Network (CNN), Face Forensics++, Celeb DF, DFDC, Machine Learning, Real-Time Detection, Hugging Face.

Benefitting from the development of deep generative networks, modern fake news generation methods called Deepfake rapidly go viral over the Internet, calling for efficient detection methods. Existing Deepfake detection methods basically use binary classification networks trained on frame-level inputs and lack leveraging temporal information in videos. Besides, the accuracy of these methods will rapidly decrease when processing low-quality data. In this work, we propose a two-stream network to detect Deepfake in video level with the capability of handling low-quality data. The proposed architecture firstly divides the input video into segments and then feeds selected frames of each segment into two streams: The first stream takes RGB information as input and tries to learn the semantic inconsistency. The second stream parallelly leverages noise features extracted by spatial rich model (SRM) filters. Additionally, our experiments found that traditional SRM filters with fixed weights contribute insignificant improvement, we thus design novel learnable SRM filters, which can better fit the noise inconsistency in tampered regions. Segmental fusion and stream fusion are conducted at last to combine the information from segments and streams. We evaluate our algorithm on the existing largest Deepfake dataset FaceForensics++ and the experimental results show that we obtain state-of-the-art performance.

Nowadays, less technical individuals can create false videos by only source and target images, using deepfakes generation tools and methodologies. Distributing false information on social media and other concerns related to the deepfakes have thus significantly increased. To deal with the challenges posed by incorrect details, efficient Deepfakes detection algorithms must be developed considering the tremendous advancement in deepfakes generating techniques. Existing techniques are not reliable enough to find deepfakes, especially when the videos are made with various deepfakes generation methods. The Swin Y-Net Transformers (SWYNT) architecture we created in this paper can visually discriminate between natural and artificial faces. The architecture uses a Swin transformer, encoder, and decoder in a U -Net architecture with a classification branch to build a model that can classify and segment deepfakes. The segmentation process creates segmentation masks and helps train the classifier. We have evaluated our suggested method using the extensive, standard, and diverse FaceForensics++ (FF++) and the Celeb-DF dataset. The generalizability evaluation of our process, which is part of the performance evaluation, reveals the model's promising performance in identifying deepfakes videos generated using various methodologies on both large-scale datasets.

The immanent dependencies between audio and visual modalities extracted from video content and the well-established film grammar (i.e., domain knowledge) are important for emotion video recognition and regression. However, these tools have yet to be exploited successfully. Therefore, we propose a multimodal deep regression Bayesian network (MMDRBN) to capture the relationship between audio and visual modalities for emotion video tagging. We then modify the structure of the MMDRBN to incorporate domain knowledge. A regression Bayesian network (RBN) is formed from one latent layer, one visible layer and directed links from the latent layer to the visible layer. RBN is able to fully represent the data, since it captures the dependencies not only among the visible variables but also among the latent variables given visible variables. For the MMDRBN, first, we learn several layers of RBNs using audio and visual modalities, and then stack these RBNs to form two deep networks. A joint representation is obtained from the top layers of the two deep networks, capturing the deep dependencies between audio and visual modalities. We also summarize the main audio and visual elements used by filmmakers to convey emotions and formulate them as semantical meaningful middle-level representation, i.e., attributes. Through these attributes, we construct the knowledge-augmented MMDRBN, which learns a hybrid middle-level video representation using video data and the summarized attributes. Experimental results of both emotion recognition and regression from videos on the LIRIS-ACCEDE database demonstrate that the proposed model can successfully capture the intrinsic connections between audio and visual modalities, and integrate the middle-level representation learning from video data and semantical attributes summarized from film grammar. Thus, it achieves superior performance on emotion video tagging compared to state-of-the-art methods.

In the same vein of discriminative one-shot learning, Siamese networks allow recognizing an object from a single exemplar with the same class label. However, they do not take advantage of the underlying structure of the data and the relationship among the multitude of samples as they only rely on the pairs of instances for training. In this paper, we propose a new quadruplet deep network to examine the potential connections among the training instances, aiming to achieve a more powerful representation. We design a shared network with four branches that receive a multi-tuple of instances as inputs and are connected by a novel loss function consisting of pair loss and triplet loss. According to the similarity metric, we select the most similar and the most dissimilar instances as the positive and negative inputs of triplet loss from each multi-tuple. We show that this scheme improves the training performance. Furthermore, we introduce a new weight layer to automatically select suitable combination weights, which will avoid the conflict between triplet and pair loss leading to worse performance. We evaluate our quadruplet framework by model-free tracking-by-detection of objects from a single initial exemplar in several visual object tracking benchmarks. Our extensive experimental analysis demonstrates that our tracker achieves superior performance with a real-time processing speed of 78 frames/s. Our source code is available.

Sports data is more readily available and consequently, there has been an increase in the amount of sports analysis, predictions and rankings in the literature. Sports are unique in their respective stochastic nature, making analysis, and accurate predictions valuable to those involved in the sport. In response, we focus on Siamese Neural Networks (SNN) in unison with LightGBM and XGBoost models, to predict the importance of matches and to rank teams in Rugby and Basketball. Six models were developed and compared, a LightGBM, a XGBoost, a LightGBM (Contrastive Loss), LightGBM (Triplet Loss), a XGBoost (Contrastive Loss), XGBoost (Triplet Loss). The models that utilise a Triplet loss function perform better than those using Contrastive loss. It is clear LightGBM (Triplet loss) is the most effective model in ranking the NBA, producing a state of the art (SOTA) mAP (0.867) and NDCG (0.98) respectively. The SNN (Triplet loss) most effectively predicted the Super 15 Rugby, yielding the SOTA mAP (0.921), NDCG (0.983), and $r_s$ (0.793). Triplet loss produces the best overall results displaying the value of learning representations/embeddings for prediction and ranking of sports. Overall there is not a single consistent best performing model across the two sports indicating that other Ranking models should be considered in the future.

Despite the increased availability of UAV / drone imagery in Low- to upper  Middle-Income countries and the demonstrated potential of deep learning to support the interpretation of these images for sustainable development purposes, practical operations in these countries are constrained by the need for sufficient labeled data-sets which are often difficult to obtain (especially for tropical forest). This makes it difficult to train suitable networks and assess whether the model is performing well. One such example is the use of drones to monitor the Atlantic Forest in Sao Paulo, Brazil. Here, members of the Sao Paulo Municipal Green and Environment Secretariat (Secretaria do Verde e do Meio Ambiente - SVMA) are starting to use drones to  identify some native and invasive species in their forests. Deep learning will quickly speed up this process, but there is little training data available. This refers to the so called ‘small-data problem’ commonly found in DL for remote sensing applications [1]. A workflow was designed to support this application through a novel zero-shot learning technique and explainable AI methods. A pre-trained tree-crown detection model ‘DeepForest’ [2] is used to identify individual tree crowns in the UAV imagery. The detected tree-crowns are further classified using a Siamese network architecture using zero-shot learning – the model is trained on relevant data-sets but not exposed to species found in the test data-set. A Siamese network architecture is motivated by the need for explainability in DL models – the results will be used for making administrative decision for forest management. A more intricate DL model (such as image segmentation) could be more accurate but at the cost of transparency/explainability. In particular, we apply a variation of the ‘What I Know’ (WIK) explainability method [3] which provides examples from the training set along with the test sample increasing transparency and understanding of the model results.[1] Safonova, Anastasiia, et al. "Ten deep learning techniques to address small data problems with remote sensing." International Journal of Applied Earth Observation and Geoinformation 125 (2023): 103569.[2] Weinstein, Ben G., et al. "DeepForest: A Python package for RGB deep learning tree crown delineation." Methods in Ecology and Evolution 11.12 (2020): 1743-1751.[3] Ishikawa, Shin-nosuke, et al. "Example-based explainable AI and its application for remote sensing image classification." International Journal of Applied Earth Observation and Geoinformation 118 (2023): 103215.  

AI-synthesized face-swapping videos, commonly known as DeepFakes, is an emerging problem threatening the trustworthiness of online information. The need to develop and evaluate DeepFake detection algorithms calls for large-scale datasets. However, current DeepFake datasets suffer from low visual quality and do not resemble DeepFake videos circulated on the Internet. We present a new large-scale challenging DeepFake video dataset, Celeb-DF, which contains 5, 639 high-quality DeepFake videos of celebrities generated using an improved synthesis process. We conduct a comprehensive evaluation of DeepFake detection methods and datasets to demonstrate the escalated level of challenges posed by Celeb-DF. Then we introduce Landmark Breaker, the first dedicated method to disrupt facial landmark extraction, and apply it to the obstruction of the generation of DeepFake videos. The experiments are conducted on three state-of-the-art facial landmark extractors using our Celeb-DF dataset.