Hybrid spatio-temporal models for deepfake detection using facial movement analysis

Deepfake detection is an emerging field dedicated to identifying synthetic media where individuals’ likenesses are manipulated through advanced machine learning techniques. While deepfakes hold potential for innovation in entertainment and creative industries, they also pose serious threats such as misinformation, privacy invasion, and digital fraud. Effective deepfake detection is crucial for mitigating these risks. Current detection methods face several challenges. One significant issue is the over-reliance on spatial information, which limits the ability to detect subtle manipulations that may not be obvious in individual frames but become apparent over time. Furthermore, existing techniques often struggle to integrate both spatial and temporal information effectively, which is necessary for a comprehensive analysis. The development of robust and computationally efficient models is essential to ensure practical implementation and widespread adoption of deepfake detection solutions. A deeper examination of eye blinking sequences, for example, could reveal subtle inconsistencies that are indicative of manipulation, while leveraging both spatial and temporal facial data can enhance detection accuracy. This thesis explores two innovative approaches for detecting deepfake videos. The first approach focuses on analyzing eye blinking patterns, capitalizing on the observation that deepfake generators struggle to replicate natural blinking dynamics accurately. By compiling a robust dataset through meticulous frame-by-frame analysis of genuine and deepfake videos, significant disparities in blinking patterns were identified. The second approach delves into the spatial and temporal analysis of facial features. Recognizing that deepfakes, despite their sophistication, often leave subtle traces of manipulation in both individual frames and frame transitions, this research employs 3D data representation to capture these inconsistencies. Leveraging the insights from the first approach, an initial fully connected model achieved 90.8% accuracy. This promising result was further improved by integrating Convolutional Neural Networks (CNNs) with Bidirectional Long Short-Term Memory (BiLSTM) networks, boosting the accuracy by 4.6%, reaching an impressive 95.4%. For the second approach, a baseline 3D CNN model processing sequences of images initially achieved an accuracy of 95.5%. This performance was enhanced by incorporating a specialized called NuanceFusion layer, resulting in an improvement of 1.8%. Further enhancement was achieved with an advanced model integrating 2D CNN and BiLSTM networks, increasing accuracy by 3.4% and achieving a perfect recall of 100%. This streamlined approach effectively combines spatial and temporal analysis for robust deepfake detection. Through comprehensive experimentation and analysis, this thesis demonstrates the effectiveness of both blinking pattern analysis and spatial-temporal facial feature analysis for deepfake detection. The research not only highlights the strengths of each approach but also explores the trade-offs between accuracy, computational efficiency, and resource utilization, offering valuable insights for tailoring solutions based on specific application requirements. The findings provide a solid foundation for future research, emphasizing the need for continuously adapting and refining deepfake detection methodologies in the face of an ever-evolving digital landscape.