This study sought to test the hypothesis that anime films are more likely than non-anime films to be “over-rated” by user ratings on IMDB. That is, that the disagreement between the IMDB user rating and the metascore would be greater for anime films than for non-anime films.
The pre-registration for this study is available at https://aspredicted.org/3sh38.pdf
IMDB was searched to obtain a sample of anime and non-anime films. The 100 films with the highest number of user votes was retreieved for each subset and the title, user rating, and metascore was retained for each film.
The disagreement score was calculated by subtracting the metascore from the IMDB user rating for each film (after first rescaling so all ratings were out of 100). A positive score indicates more positive assessment of the film from IMDB users than critics. The magnitude of the disagreement score indicates the average extent of disagreement between users and critics for each film.
Means and Standard Deviations are reported in Table 1 for each of the two types of films.
Table 1
| Film Type | n | Mean | SD |
|---|---|---|---|
| Anime | 36 | 10.28 | 12.95 |
| Not Anime | 100 | 6.43 | 10.28 |
Disagreement scores are also plotted using a violin plot (below). The horizontal line shows equipoise, where there is no disagreement in the between IMDB users’ and critics’ rating for a film. Dots above the line indicate films that were rated more positively by users than critics; dots below the line were rated more negatively by users than critics.
The film that was most “over-rated” was Digimon: The Movie with IMDB users rating that film 39 points higher than critics. The most “under-rated” film was Gravity with IMDB users rating that film 18 points lower than critics. 5 films demonstrated agreement between reviewers and IMDB user ratings (see Table 2).
Table 2
| Title | Rating | Metascore |
|---|---|---|
| The Departed | 8.5 | 85 |
| Iron Man | 7.9 | 79 |
| Raiders of the Lost Ark | 8.5 | 85 |
| Star Wars: Episode VII - The Force Awakens | 8.1 | 81 |
| Spider-Man | 7.3 | 73 |
An independent samples t-test was conducted to evaluate the extent of disagreement between metascore and IMDB user ratings for anime vs. non-anime films. The difference in diagreement scores between the two genres was not statistically significant (t(51.74) = -1.61, p = 0.057).
Contrary to expectations, anime films were no more consistently over-rated than non-anime films