Most frequently used words in the series

To find the frequency of all the words in the entire series we used the table function to count up all times each word was used throughout the series.

word	frequency
the	51593
and	27430
to	26985
of	21802
a	20966
he	20322
harry	16557
was	15631
said	14398
his	14264

As you can see, this initial step is not very interesting. It is interesting that Harry makes the top list of words.

Displaying frequency of non “stop words” in the whole series

We thought that our analysis would become more interesting if we removed “stopwords”. Stopwords are words that alone don’t carry very much meaning. For example, on its own the word “the” doesn’t carry any meaning, but the word “death” carries significant meaning.

word	frequency
harry	16557
ron	5750
hermione	4912
dumbledore	2873
looked	2344
professor	2006
hagrid	1732
time	1713
wand	1639
eyes	1604

Plotting word frequency by book

Character frequency of whole series

Spell frequency of whole series

To find which spells are used most often, we compared our dataframe containing information about how many times each word is used and a list of the first word in each spell. We did this because the only times that the first word in a spell is used is when they are saying the entire spell. This allowed us to use inner_join to get our frequencies of each spell. It is interesting to note that expecto patronum is the most used spell in the series, and the least used in a four-way tie between confundo, liberacorpus, morsmordre, and rictusempra.

word	frequency
expecto patronum	43
accio	33
expelliarmus	27
stupefy	27
avada kedavra	23
lumos	22
riddikulus	17
crucio	15
impedimenta	13
disillusionment	11

Location Frequency Full Series

To find the frequency of the locations, we had a slightly more complicated process. Since not all of the locations can be boiled down to one word, we had to compare the list of locations to a dataframe of sentences. From here, we used regular expressions to extract the locations from each sentence, then used table to count up how often each location was mentioned. As you would expect, Hogwarts is the location that is most mentioned. It is also interesting that Diagon Alley comes up very often even though it doesn’t seem like a lot of the books are set in Diagon Alley.

location	occurences
Hogwarts	856
Azkaban	177
Hogsmeade	151
Ministry of Magic	148
Durmstrang	65
Burrow	56
Diagon Alley	56
Godric’s Hollow	54
Beauxbatons	53
Honeydukes	28

Sentiment Analysis

Trials and Tribulations

There were a few things that we tried that didn’t work or didn’t give us any additional insights. The main thing we tried was Named Entity Recognition using natural language processing. We found the r package NLP and openNLP, which together were supposed to be able to predict the part of speech of every word in a sentence and be able to tell us if each word is a name, location, organization, or none of the above. This, however, did not work at all. It would only recognize some names that were very common (like John), but not slightly less common ones (like Anna). So, when we looked at names in Harry Potter, it didn’t work at all. We aren’t certain whether we didn’t implement it right or whether the underlying method it uses is really bad, but we couldn’t seem to get it to work right.

Another thing that we tried was clustering to see if there were any underlying patterns. We started with kmeans clustering, which didn’t yield anything interesting. Next, we used DBSCAN, since we didn’t need to choose how many clusters to force the data into. This was slightly more interesting but the process was very time consuming, and we weren’t sure what radius to pick and how many points to have in each cluster. It was interesting though.

Lastly, our data is located under ~/Mscs 264 F18/Project/Jack_Abe/HarryPotterDataScience/Data and our work is under ~/Mscs 264 F18/Project/Jack_Abe/HarryPotterDataScience/Work.

Top 10 most frequent words in each book