There have been a lot of attempts and models which look at detecting deepfake videos. Specifically, a Kaggle compitition in 2019 offered a $1,000,000 USD prize to create a model that successfully detects deepfake videos based on the large data set which they provided. After looking at both the dataset and the model used for this competition, it became clear that the deepfake videos used were lower quality where it is evident that the face is swapped in the video. You can see this from images in the reports folder on GitHub, showing two examples of deepfake videos which were used in their model.
Therefore, it is unknown how this model will perform for high quality deepfakes, as well as deepfakes which do not swap full faces. In other words, will the model perform well with the dataset that is saved locally? The third image in the imgs folder under reports is an example of a deepfake used in a larger project of mine. Looking at the difference between the three, it becomes clear that there is a signficant difference in terms of quality based on each dataset, as well as swapping method used. The technique used by the Kaggle researchers seems to full face swaps. However, deepfakes can swap out specific facial features such as mouth movement, or vocal patterns. Therefore, it is unknown how this model will perform in terms of different manipulation techniques.
In addition, when looking at their model further in the file saved as train.R, they did not tune their model and only had one hidden layer. Even though their model accuracy is high, it may be only specific to the validation set which was created. Therefore, it will be interesting to first run the model to create a baseline for comparison purposes, then train our data on the same exact model to see how well the model performs. Following, it will also be of interest to tune the model on our data to see if we can increase the accuracy further.
This project can be of practical interest. If their model performs well without tuning on our data, we can assume that the model which was created can be a powerful tool to detect deepfakes online. Companies and governments have been struggling with the right answer to this as of late. Facebook created their own deepfake detection software (Shead, 2020), and decided to remove all deepfakes from their platform, regardless of content or intent. This approach, in my eyes, is relatively extreme as deepfakes can have signficant benefits in certain areas (Chesney and Citron, 2019).
The full data set for both the kaggle data as well as my locally saved data can be found at: this link.
Essentially, what this project attempts to do is to see whether or not we can generalize the results of an already successful model on a dataset which was created by professionals in a higher quality manner. In other words, I want to see if the model can classify the deepfakes I have as either fake or real, or will tuning be required to increase the accuracy of the model?
As I said, the starting point for this project is to essentially re-create and use the existing model on Kaggle as a baseline to compare both our model without tuning, as well as our model with tuning. The researchers in this case used a CNN model, with accuracy as their metric and a relatively low learning rate to begin. They address this in their blog post as they said they already knew a low learning rate was necessary. In addition, they used only one hidden layer, whereas it is usually better to have multiple hidden layers.
In order to recreate their model. The steps are as followed:
I began by downloading their data locally prior to sending it to the Google Cloud.
magick and opencv packages in R.image_read_video at 1 fps.ocv_readand ocv_face functions in the opencv package.The full procedure can be found in prepare_data.
In terms of building their model, they began by defining the number of epochs and size. They then created a training and validation set, as well as creating a convolutional base using application_vgg16 with imagenet as weights. They defined their model with one flatten layer and one hidden layer consisting of 256 neurons and relu as an activation function. Their output layer uses sigmoid since it is a binary classification. Their compiled model also consisted of a loss function of binary_crossentropy, a learning rate of 2e-5, and a metric of accuracy. The full model is saved as train.R.
As I said above, I ran this model to obtain a baseline for my own data. As the accuracy shows, we successfully replicated the findings from the original model such that the final accuracy was 0.8848. You can access the full run using this link.
In the next steps, I will prepare the videos which I have saved locally in a similar manner, and rerun this model exactly without tuning to see if the model, in its current form, will be able to sucessfully detect the fake videos from the real videos in my dataset.
##
## Attaching package: 'dplyr'## The following objects are masked from 'package:stats':
##
## filter, lag## The following objects are masked from 'package:base':
##
## intersect, setdiff, setequal, union## $ metric_loss: num 0.254
## $ metric_acc : num 0.885## $ run_dir : chr "/Users/admin/Documents/GitHub/projg06jeremy/scripts/runs//cloudml_2021_06_03_123500113"
## $ metric_loss : num 0.254
## $ metric_acc : num 0.885
## $ flag_n_neurons : int NA
## $ flag_n_lr : num NA
## $ flag_n_layers : int NA
## $ flag_n_epochs : int NA
## $ samples : int 143
## $ epochs : int 10
## $ epochs_completed : int 10
## $ metrics : chr "(metrics data frame)"
## $ model : chr "(model summary)"
## $ loss_function : chr "binary_crossentropy"
## $ optimizer : chr "<tensorflow.python.keras.optimizer_v2.rmsprop.RMSprop>"
## $ learning_rate : num 2e-05
## $ script : chr "train.R"
## $ start : POSIXct[1:1], format: "2021-06-03 12:38:31"
## $ end : POSIXct[1:1], format: "2021-06-03 12:45:19"
## $ completed : logi TRUE
## $ output : chr "(script ouptut)"
## $ source_code : chr "(source archive)"
## $ context : chr "cloudml"
## $ type : chr "training"
## $ cloudml_created : POSIXct[1:1], format: "2021-06-03 12:35:26"
## $ cloudml_end : POSIXct[1:1], format: "2021-06-03 12:47:42"
## $ cloudml_job : chr "cloudml_2021_06_03_123500113"
## $ cloudml_log_url : chr "https://console.cloud.google.com/logs?resource=ml_job%2Fjob_id%2Fcloudml_2021_06_03_123500113&project=melodic-furnace-314214"
## $ cloudml_master_type: chr "standard_gpu"
## $ cloudml_ml_units : num 0.28
## $ cloudml_start : POSIXct[1:1], format: "2021-06-03 12:37:10"
## $ cloudml_state : chr "SUCCEEDED"To begin, I had to prepare my data in a similar manner as above. First, I created a dataframe to classify which video was fake and which was real. I used this dataframe to extract the frames with 1 fps again, resized the images to 1000x1000 and created two additional files for each real image and fake image. I ran the same code to detect the faces and saved the images as png files in their respected folder. You can find the full procedure of preparation done on my videos in the file prepare_data_local.
The same exact model was used on this data set to begin, you can find the script for this model in train_3. The idea is to see if the model can be successful without tuning the model. The accuracy of the model proves higher than that of their dataset at 0.8919. This result shows that they were able to create a model which can be applied to videos outside of their own videos used, and to videos created by other individuals (i.e., different techniques). Again, use this link to obtain the full run data. In the next steps, I will still tune the model to see if there are opportunities to increase the accuracy on the locally saved dataset.
## $ metric_loss: num 0.317
## $ metric_acc : num 0.892## $ run_dir : chr "/Users/admin/Documents/GitHub/projg06jeremy/scripts/runs//cloudml_2021_06_03_122316592"
## $ metric_loss : num 0.317
## $ metric_acc : num 0.892
## $ flag_n_neurons : int NA
## $ flag_n_lr : num NA
## $ flag_n_layers : int NA
## $ flag_n_epochs : int NA
## $ samples : int 19
## $ epochs : int 10
## $ epochs_completed : int 10
## $ metrics : chr "(metrics data frame)"
## $ model : chr "(model summary)"
## $ loss_function : chr "binary_crossentropy"
## $ optimizer : chr "<tensorflow.python.keras.optimizer_v2.rmsprop.RMSprop>"
## $ learning_rate : num 2e-05
## $ script : chr "train_3.R"
## $ start : POSIXct[1:1], format: "2021-06-03 12:26:53"
## $ end : POSIXct[1:1], format: "2021-06-03 12:28:05"
## $ completed : logi TRUE
## $ output : chr "(script ouptut)"
## $ source_code : chr "(source archive)"
## $ context : chr "cloudml"
## $ type : chr "training"
## $ cloudml_created : POSIXct[1:1], format: "2021-06-03 12:23:49"
## $ cloudml_end : POSIXct[1:1], format: "2021-06-03 12:30:35"
## $ cloudml_job : chr "cloudml_2021_06_03_122316592"
## $ cloudml_log_url : chr "https://console.cloud.google.com/logs?resource=ml_job%2Fjob_id%2Fcloudml_2021_06_03_122316592&project=melodic-furnace-314214"
## $ cloudml_master_type: chr "standard_gpu"
## $ cloudml_ml_units : num 0.28
## $ cloudml_start : POSIXct[1:1], format: "2021-06-03 12:25:33"
## $ cloudml_state : chr "SUCCEEDED"Up until this point, I've been able to show that the model as is will already be successful when trained on the locally saved data. However, there is still a lot which can be done when it comes to tuning the model. Specifically, in it's current state, this model only uses one hidden layer, 10 epochs, a low starting learning rate, and a fixed number of neurons. Therefore, the purpose of this section is to tune the model, such that I tune the number of layers, number of neurons per layer, number of epochs, and the learning rate. The values are as followed:
n_neuronsn_layersn_lrn_epochsThe model for the inserted tuned values is found in train_2 and tuning.yml.
The results show that the model that increased the accuracy the most is as followed:
1 hidden layer
128 neurons
1e-5 learning rate
30 epochs
This results in an accuracy of 0.9351. The code for all tuning runs, the code is posted below.
What I was interested in doing now was to continue to increase the best models epochs, while adding an early stopping criteria. Therefore, I reran the model with 1 hidden layer,128 neurons, 1e-5 learning rate, but included 100 epochs and an early stopping criteria with patience = 3 and restoring best weights. This model can be found in train_4. The results show an even greater increase in accuracy such that we obtained a final accuracy of 0.9892. The full number of epochs was used, therefore, the early stopping criteria was not needed with this model. The full list of runs can be viewed in runs using the function below. For the results of the tuned model with 100 epochs use this link.
## Data frame: 243 x 6
## metric_loss metric_acc flag_n_neurons flag_n_lr flag_n_layers flag_n_epochs
## 1 0.0535 0.9892 NA NA NA NA
## 89 0.1492 0.9351 128 1e-05 1 30
## 29 0.1898 0.9297 256 1e-05 1 30
## 203 0.1767 0.9297 32 1e-05 3 30
## 26 0.2156 0.9189 256 1e-05 2 30
## 27 0.2427 0.9189 256 1e-05 2 20
## 86 0.1757 0.9189 128 1e-05 2 30
## 90 0.2269 0.9135 128 1e-05 1 20
## 143 0.2198 0.9135 64 1e-05 3 30
## 146 0.2042 0.9135 64 1e-05 2 30
## # ... with 233 more rows## Data frame: 243 x 31
## run_dir
## 1 /Users/admin/Documents/GitHub/projg06jeremy/scripts/runs//cloudml_2021_06_08_052754581
## 89 /Users/admin/Documents/GitHub/projg06jeremy/scripts/runs//cloudml_2021_06_07_153326991-153
## 29 /Users/admin/Documents/GitHub/projg06jeremy/scripts/runs//cloudml_2021_06_07_153326991-213
## 203 /Users/admin/Documents/GitHub/projg06jeremy/scripts/runs//cloudml_2021_06_07_153326991-039
## 26 /Users/admin/Documents/GitHub/projg06jeremy/scripts/runs//cloudml_2021_06_07_153326991-216
## 27 /Users/admin/Documents/GitHub/projg06jeremy/scripts/runs//cloudml_2021_06_07_153326991-215
## 86 /Users/admin/Documents/GitHub/projg06jeremy/scripts/runs//cloudml_2021_06_07_153326991-156
## 90 /Users/admin/Documents/GitHub/projg06jeremy/scripts/runs//cloudml_2021_06_07_153326991-152
## 143 /Users/admin/Documents/GitHub/projg06jeremy/scripts/runs//cloudml_2021_06_07_153326991-099
## 146 /Users/admin/Documents/GitHub/projg06jeremy/scripts/runs//cloudml_2021_06_07_153326991-096
## metric_acc metric_loss
## 1 0.9892 0.0535
## 89 0.9351 0.1492
## 29 0.9297 0.1898
## 203 0.9297 0.1767
## 26 0.9189 0.2156
## 27 0.9189 0.2427
## 86 0.9189 0.1757
## 90 0.9135 0.2269
## 143 0.9135 0.2198
## 146 0.9135 0.2042
## # ... with 233 more rows
## # ... with 28 more columns:
## # flag_n_neurons, flag_n_lr, flag_n_layers, flag_n_epochs, samples,
## # epochs, epochs_completed, metrics, model, loss_function, optimizer,
## # learning_rate, script, start, end, completed, output, source_code,
## # context, type, cloudml_created, cloudml_end, cloudml_job,
## # cloudml_log_url, cloudml_master_type, cloudml_ml_units, cloudml_start,
## # cloudml_stateOverall, this work contains three aspects which are important. First, I replicate the findings from the Kaggle model. This was an important step to use these results as a baseline for this project. Second, I was able to use my local data to extract the faces and frames using as similar method, and show that the model which they used without tuning is going to be successful on various datasets with higher quality levels. This is an important finding as the results show that their model can be generalized to other videos that are made by other professionals with higher quality and different manipulation techniques. The deepfakes which were used by the Kaggle researchers used full face swap, while the deepfakes created online use other methods such as vocal swaps, mouth swaps, and the combination of the two in terms of the Nixon deepfake. Therefore, the results prove that regardless of the deepfake method (i.e., face swap, mouth swap, vocal swap), the model will still be able to detect the manipulation. And finally, what this project shows is tuning the model on my locally saved data will increase the accuracy even further.
As deepfake videos become more and more sophisticated and the quality continues to improve, such as the recent viral deepfake of Tom Cruise, it will be interesting to see how well these models will be at detecting such deepfakes. As for now, these models prove to be successful at detecting a variety of manipulations.