Assessment framework for deepfake detection in real-world situations

Due to the abuse of deep forgery technology, the research on forgery detection methods has become increasingly urgent. The corresponding relationship between the frequency spectrum information and the spatial clues, which is often neglected by current methods, could be conducive to a more accurate and generalized forgery detection. Motivated by this inspiration, we propose a wavelet‐based texture mining and enhancement framework for face forgery detection. First, we introduce a frequency‐guided texture enhancement (FGTE) module that mining the high‐frequency information to improve the network’s extraction of effective texture features. Next, we propose a global–local feature refinement (GLFR) module to enhance the model’s leverage of both global semantic features and local texture features. Moreover, the interactive fusion module (IFM) is designed to fully incorporate the enhanced texture clues with spatial features. The proposed method has been extensively evaluated on five public datasets, such as FaceForensics++ (FF++), deepfake (DF) detection (DFD) challenge (DFDC), Celeb‐DFv2, DFDC preview (DFDC‐P), and DFD, for face forgery detection, yielding promising performance within and cross dataset experiments.

The increased use of power- and space-constrained embedded processors in a wide variety of autonomous imaging and surveillance applications implies increased speed of computational resources that follow image acquisition in the processing stream. In such early vision applications, one typically processes an entire image data stream prior to spatial downselection operations such as focus-of-attention involving area-of-interest (AOI) selection. Downselection is especially useful in the emerging technologies of spatiotemporal adaptive processing (STA) and biomimetic automated target recognition (ATR). Here, progressive data reduction employs operations or sub-algorithms that process fewer data but in a more involved manner at each step of an algorithm. Implementationally, the STAP approach is amenable to embedded hardware using processors with deep pipelines and fine-grained parallelism. In part 1 of this series of two papers, we showed how compression of an image or sequence of images could facilitated more efficient image computation or ATR by processing fewer (compressed) data. This technique, called compressive processing or compressive computation, typically utilizes fewer operations than the corresponding ATR or image processing operation over noncompressed data, and can produce space, time, error, and power (STEP) savings on the order of the compression ratio (CR). Part 1 featured algorithms for edge detection in images compressed by vector quantization (VQ), visual pattern image coding (VPIC), and EBLAST, which is a recently-reported block-oriented high- compression transform. In this paper, we continue presentation of theory for compressive computation of early processing operations such as morphological erosion and dilation, as well as higher- level operations such as connected component labeling. We also discuss the algorithm and hardware modeling technique that supports analysis and verification of compressive processing efficiency. This methodology emphasizes 1) translation of each image processing algorithm or operation to a prespecified compressive format, 2) determination of the operation mix M for each algorithms produced in Step 1), and 3) simulation of M on various architectural models, to estimate performance. As in part 1, algorithms are expressed in image algebra, a rigorous, concise notation that unified linear and nonlinear mathematics in the image domain.© (2000) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.

Image classification is a fundamental problem in the field of computer vision and involves assigning a label to an image based on its content. In this paper, we survey both traditional machine learning-based image classification methods and deep learning-based image classification methods, specifically, we review two deep learning-based image classification methods: a convolutional neural network (CNN) method and a pre-trained CNN-based transfer learning method for image classification. We first briefly review traditional machine learning-based image classification methods and then deep learning-based image classification methods, discussing both the feature extraction methods and the classification methods used in the deep learning-based methods. We discuss several deep neural network architectures for image classification, including LeNet, AlexNet, VGGNet, GoogleNet, ResNet, and DenseNet, and finally, we conclude with a discussion on the applications of image classification and compare the different methods based on various factors both qualitatively and quantitatively, and present our experimental results, while also summarizing various methods of data augmentation, batch normalization, and regularization by dropout. We introduce techniques for practical training of deep networks, and discuss fine-tuning, pruning, and model quantization for efficient inference of trained neural networks, and finally, we introduce and discuss the runtime engines for the deployment of trained neural networks for efficient inference.

The rise of deepfake technology poses significant threats to the authenticity of content on social media. This research introduces Sach-AI, a pioneering framework for detecting various deepfakes in video, audio, and image data. Leveraging the power of deep neural networks, Sach-AI utilizes Eulerian Video Magnification combined with the ResNext architecture for enhanced detection. For video deepfakes, Long Short-Term Memory (LSTM) networks are integrated to improve classification tasks. This combination allows Sach-AI to effectively address the evolving, multimodal nature of deepfakes. The framework has been rigorously evaluated using diverse datasets, such as Celeb-DF and FaceForensics , demonstrating its robustness and accuracy. Sach-AI achieved 97.76% accuracy in video deepfake detection, surpassing Intel’s FakeCatcher, 99.13% accuracy in audio deepfake detection, and 93.64% accuracy in image deepfake detection. These results underscore Sach-AI’s reliability in safeguarding digital media integrity against deceptive synthetic content in an era increasingly dominated by artificial technologies.

Illumination estimation is a fundamental prerequisite for many computer vision applications. Various statistics and deep learning-based estimation methods have been proposed, and further studies are ongoing. In this study, we first perform a comparative analysis of representative statistics and deep learning-based methods and subsequently investigate combining them to improve the illumination estimation accuracy. We use hyperspectral images as the training data and support vector regression to combine the methods. Based on the results, we confirm that their combination enhances their accuracy.

Learning a deep dual-level network for robust DeepFake detection

Concrete surface crack detection plays a crucial role in infrastructure maintenance and safety. Deep learning-based methods have shown great potential in this task. However, under real-world conditions such as poor image quality, environmental interference, and complex crack patterns, existing models still face challenges in detecting fine cracks and often rely on large training parameters, limiting their practicality in complex environments. To address these issues, this paper proposes a crack detection model based on adaptive feature quantization, which primarily consists of a maximum soft pooling module, an adaptive crack feature quantization module, and a trainable crack post-processing module. Specifically, the maximum soft pooling module improves the continuity and integrity of detected cracks. The adaptive crack feature quantization module enhances the contrast between cracks and background features and strengthens the model’s focus on critical regions through spatial feature fusion. The trainable crack post-processing module incorporates edge-guided post-processing algorithms to correct false predictions and refine segmentation results. Experiments conducted on the Crack500 Road Crack Dataset show that, the proposed model achieves notable improvements in detection accuracy and efficiency, with an average F1-score improvement of 2.81% and a precision gain of 2.20% over the baseline methods. In addition, the model significantly reduces computational cost, achieving a 78.5–88.7% reduction in parameter size and up to 96.8% improvement in inference speed, making it more efficient and deployable for real-world crack detection applications.

The advancement of generative models has made it easier to create highly realistic Deepfake videos. This accessibility has led to a surge in research on Deepfake detection to mitigate potential misuse. Typically, Deepfake detection models utilize binary backbones, even though the training dataset contains additional exploitable information, such as the Deepfake generation method employed for each video. However, recent findings suggest that inferring a binary class from a multi-class backbone yields superior performance compared to directly employing a binary backbone. Building upon this research, our article introduces two novel methods to infer a binary class from a multi-class backbone. The first method, named root dummies , leverages the dummy triplet loss, which employs fixed vectors (i.e., dummies) instead of mined positives and negatives in the triplet loss. By training the multi-class backbone with these dummies, we can easily infer a binary class during testing by adjusting the number of dummies (from six during training to two during inference). Through this approach, we achieve an accuracy improvement of 0.23% compared to the existing inference method, without requiring additional training. The second proposed method is transfer learning. It involves training a classifier, such as a support vector machine, to predict binary classes based on the image embeddings generated by the multi-class backbone. Although this method necessitates additional training, it further enhances the model’s performance, resulting in an accuracy increase of 1.79%. In summary, our proposed methods improve the accuracy of Deepfake detection by simply modifying the number of classes during training, making them suitable for integration into a variety of existing Deepfake training pipelines. Additionally, to foster reproducible research, we have made the source code of our solution publicly available at https://github.com/beuve/DmyT .

Deepfake technology, driven by advancements in artificial intelligence, enables the creation of highly realistic manipulated videos, posing significant threats to security, privacy, and misinformation. Traditional detection methods struggle to keep pace with the evolving sophistication of deepfake techniques. This study proposes a hybrid deep learning approach that leverages Convolutional Neural Networks (CNN) for feature extraction and Long Short-Term Memory (LSTM) networks for temporal sequence analysis to enhance deepfake detection accuracy. The CNN model captures spatial inconsistencies and artifacts in individual frames, while the LSTM network analyzes sequential dependencies to detect temporal anomalies indicative of deepfakes. Experimental evaluations on benchmark datasets demonstrate the effectiveness of the approach, achieving high accuracy in distinguishing real from fake videos. The proposed model offers a robust and scalable solution for deepfake detection, contributing to the fight against digital media manipulation and misinformation.

Abstract—Deepfake technology, driven by generative adversarial networks (GANs), poses significant challenges in digital security, misinformation, and privacy. Detecting deepfakes in images and videos requires advanced deep learning models. This study explores deepfake detection using convolutional neural networks (CNNs), recurrent neural networks (RNNs), and transformer-based architectures like Vision Transformers (ViTs). We employ Meso4_DF deepfake detection pipeline that uses TensorFlow/Keras, PyTorch, OpenCV for processing, with Dlib, Scikit-Image, and NumPy for feature extraction, leveraging transfer learning for enhanced accuracy. Video analysis includes frame extraction and temporal feature learning using LSTMs and 3D CNNs. Experimental results demonstrate that deep learning-based methods achieve high accuracy in distinguishing real and manipulated media, offering a robust approach to deepfake detection. Keywords—Deepfake Detection,CNN,GAN,Meso4 model, Image Analysis,Video Analysis.

The rapid advancement of digital content generation technologies has led to the proliferation of deepfake media, posing significant security and privacy risks on social networks. Conventional deepfake detection methods struggle with accuracy, computational efficiency, and adaptability to evolving fake image generation techniques. To address these challenges, we propose a novel deepfake detection framework that integrates an Adaptive deep convolutional generative adversarial network (Adaptive-DCGAN) with a lightweight convolutional neural network (Lite-CNN). The Adaptive-DCGAN generates high-quality synthetic images, enhancing the training dataset, while the Lite-CNN efficiently classifies real and fake images with minimal computational overhead. Experimental results demonstrate that the proposed model achieves 95% accuracy, significantly outperforming existing GAN-CNN-based detection systems, which only reach 53% accuracy. Additionally, the framework exhibits superior performance metrics, including 95% precision, recall, and F1-score, ensuring robust and reliable deepfake detection. The proposed system offers a scalable and efficient solution for securing digital media platforms against deepfake threats. Future work will focus on improving generalization across diverse datasets, incorporating advanced synthetic image generation techniques, and enabling real-time deepfake detection for large-scale social media applications.

Despite the promising advances in deepfake detection on current datasets, detecting visual deepfakes in real-world scenarios (e.g., deepfake videos and live streaming on YouTube) remains a challenge due to the inherent quality degradation such as unpredictable compression employed by social media platforms. Such degradation perturbs discernible forgery clues and diminishes the effectiveness of deepfake detection methods, raising a critical safety concern to the misuse of forgery faces in real-world scenarios. In this paper, we aim to understand the impacts of real-world degradation on the robustness of deepfake detection. Particularly, we investigate the risk of degraded deepfakes towards their detection on two real-world scenarios (i.e., deepfake videos and deepfake live streaming on social media platforms). By measuring the effects of real-world degradations on the performance and representation capabilities of detection models, we reveal that real-world deepfakes can be simulated via common degradation operations (e.g., JPEG compression) as they are perceptually similar to deepfake detectors. By analyzing the training dynamics under different sequences of training samples, we observe that the training order of deepfakes progressing from non-degraded (easy) to heavily degraded (hard) enhances the adaptability of detection models to various degradation in real-world scenarios. Drawing from these observations, we present a novel deepfake detection method ProFake to enhance the robustness of deepfake detection against real-world quality degradations. ProFake enables quality-adaptive learning via progressively degrade, detect and assign weights for the training samples driven by the feedback of model performance and image quality, which ensures that our model gradually focuses on more challenging samples to achieve quality-adaptive deepfake detection. Extensive experiments show that compared with existing methods, ProFake improves deepfake detection accuracy by an average of over 10 % in real-world scenarios and by an average of over 30 % in heavily degraded scenarios, while maintaining comparable performance in detecting high-quality deepfakes.

Small mammals, and particularly rodents, are common inhabitants of farmlands, where they play key roles in the ecosystem, but when overabundant, they can be major pests, able to reduce crop production and farmers’ incomes, with tangible effects on the achievement of Sustainable Development Goals no 2 (SDG2, Zero Hunger) of the United Nations. Farmers do not currently have a standardized, accurate method of detecting the presence, abundance, and locations of rodents in their fields, and hence do not have environmentally efficient methods of rodent control able to promote sustainable agriculture oriented to reduce the environmental impacts of cultivation. New developments in unmanned aerial system (UAS) platforms and sensor technology facilitate cost-effective data collection through simultaneous multimodal data collection approaches at very high spatial resolutions in environmental and agricultural contexts. Object detection from remote-sensing images has been an active research topic over the last decade. With recent increases in computational resources and data availability, deep learning-based object detection methods are beginning to play an important role in advancing remote-sensing commercial and scientific applications. However, the performance of current detectors on various UAS-based datasets, including multimodal spatial and physical datasets, remains limited in terms of small object detection. In particular, the ability to quickly detect small objects from a large observed scene (at field scale) is still an open question. In this paper, we compare the efficiencies of applying one- and two-stage detector models to a single UAS-based image and a processed (via Pix4D mapper photogrammetric program) UAS-based orthophoto product to detect rodent burrows, for agriculture/environmental applications as to support farmer activities in the achievements of SDG2. Our results indicate that the use of multimodal data from low-cost UASs within a self-training YOLOv3 model can provide relatively accurate and robust detection for small objects (mAP of 0.86 and an F1-score of 93.39%), and can deliver valuable insights for field management with high spatial precision able to reduce the environmental costs of crop production in the direction of precision agriculture management.

Detecting multimodal deepfakes has become a pressing concern due to the rising sophistication of generative techniques capable of creating highly convincing visual-speech synchronized deepfakes. As these manipulated contents proliferate on social media, remote education, smart meetings, smart homes, and virtual reality (VR), their increasing realism significantly impacts individuals, underscoring the urgent need for effective detection methods to curb their deceptive spread. Several research efforts have aimed to detect visual–audio deepfakes by examining inter-modal correlation and inter-class consistency. However, optimal detection results have yet to be achieved. Studies have indicated that detectors learn forged characteristics specific to particular generative methods and hide the real class in the visual modality, leading to a bias towards forgery distributions and degrading detection performance. Since audio modality data is a 2-D signal, we consider this bias in the audio modality to be more serious than in the visual modality, a conclusion we have verified through experiments. Therefore, an audio encoder that learns a universal audio distribution is indispensable for effective audio modality deepfake detection. In this work, we propose a new visual–audio deepfake detection method visual–audio multitask transformer with specialty audio encoder (VATS), designed to learn a more robust and comprehensive representation, thereby mitigating overfitting. The structure includes unimodal encoders for preliminary encoding and a dual-stream structure based on a multimodal framework for multilevel interaction and precise detection. To prevent the audio encoder from focusing excessively on specific identities or generator textures, we employ a pretrained speech encoder in the audio stream, while utilizing a linear projection encoder in the visual stream to focus on the fine-grained features of each video frame rather than the semantic information of the graphic. Our method comprises three key components: the fast temporal multimodal alignment and embedding module (FTAE), the modality interaction block (MIB), and the interpretable manipulated location module (IML). Experimental results on FakeAVCeleb demonstrate that our proposed method outperforms previous works in multimodal deepfake detection.

SA-MVSNet: Self-attention-based multi-view stereo network for 3D reconstruction of images with weak texture