Import the Data

Import the included dataset BNC_wordfreq.csv. This file contains the compiled word frequencies for the British National Corpus.

BNC <- read.csv("BNC_wordfreq.csv", header = TRUE)

Load the Libraries + Functions

Load all the libraries or functions that you will use to for the rest of the assignment. It is helpful to define your libraries and functions at the top of a report, so that others can know what they need for the report to compile correctly.

Continuous Basic Statistics

Calculate the following statistics on the word frequencies from the BNC data: - Dispersion: min/max, 1st/3rd quantile, standard deviation - Location: mean, median, mode

summary(BNC$frequency)
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##     800    1282    2335   13567    6050 6187267
sd(BNC$frequency)
## [1] 123948.4
mean(BNC$frequency)
## [1] 13566.67
median(BNC$frequency)
## [1] 2335
mode(BNC$frequency)
## [1] "numeric"

Continuous Graphical Displays

Create a histogram, density plot, and qqnorm plot of the frequencies from the British National Corpus.

par(mfrow = c(1, 3))

hist(BNC$frequency, main = "Histogram of BNC Frequency", 
     xlab = "frequencies from the British National Corpuss")

plot(density(BNC$frequency), main = "Density Plot of BNC Frequency", 
     xlab = "frequencies from the British National Corpus")

{qqnorm(BNC$frequency)
qqline(BNC$frequency)}

Zipf’s Law

Create a plot displaying Zipf’s Law on the BNC data.

plot(sort(BNC$frequency, decreasing = TRUE), 
     type = "b", main = "Zipf's Law", ylab = "Word Frequency")

Interpretation of the Frequency Data

Using your results from above, answer the following: - What does the dispersion of the data look like? How much does the data span from minimum to maximum, 1st-3rd quartile, etc. In this question, look at the dispersion statistics and explain what they mean to naive audience. - What do the mean, median, and mode tell us about the use of words in the English Language? How frequent are words in general (mean/median), and what is the most common frequency in the data? - In looking at the pictures of the frequency data, does word frequency appear to be normally distributed (explain, not just yes/no)? - Does the frequency in the BNC follow Zipf’s Law (explain, not just yes/no)?

Answer: 1. The dipersion of the data looks like the data is not normally distributed. The data is skewed to the right side, since the maximum of the frequency is 6187267, comparing to the min of 800 and mean of 13565. The min is closer to the mean of the dataset.

  1. The mean/median/mode tells us some words are used far more frequent than other words.In general, the words are used by 13k times. The most frequent word in data is “the”

  2. No. Since QQ plot tells us about the normality of data, and it shows from the QQ plot is far from normality.

  3. Yes, because the frequency is consistent with theoritical distribution of Zipf’s law.

Categorical Basic Statistics

Included in the BNC frequency data is the part of speech of each word. Create a summary of the types of parts of speech.

A = Adjective, Adv = Adverb, Conj = Conjunction, Det = Determinant, N = Noun, Prep = Preposition, Pron = Pronoun, V = Verb

head(BNC)
##   frequency word part_of_speech
## 1   6187267  the            det
## 2   4239632   be              v
## 3   3093444   of           prep
## 4   2687863  and           conj
## 5   2186369    a            det
## 6   1924315   in           prep
summary(BNC$part_of_speech)
##                 a               adv              conj               det 
##              1124               427                34                47 
## infinitive-marker      interjection             modal                 n 
##                 1                13                12              3262 
##              prep              pron                 v 
##                71                46              1281

Categorical Graphical Displays

Create a pie chart of the top 5 parts of speech for the BNC data. You can use base plot or ggplot2.

pos_table = table(BNC$part_of_speech)
pos_table_top5 = sort(pos_table, decreasing = TRUE)[1:5] #this gives you top 5
pie(pos_table, main = "Pie Chart of Verb Types", col = c("green", "red", "yellow","blue","purple"),labels = paste(names(pos_table)), prop.table(pos_table)*1)

Categorical Dispersion

The nouns shaving and descriptor have very similar frequencies in the Corpus of Contemporary American English (COCA), namely, 513 and 515. Do you think these words are equally spread across writing types? Use the information below (and the function from class) to calculate their deviations.

freqreg = c(95385672, 90344134, 91044778, 187245672) #word count of COCA categories
shaving = c(25, 175, 40, 273) #frequencies for shaving
descriptor = c(6, 7, 462, 40) #frequencies for descriptor
names(freqreg) = names(shaving) = names(descriptor) = c("Spoken", "Fiction", "Academic", "Press")
dis <- rbind(shaving, descriptor) 
dis
##            Spoken Fiction Academic Press
## shaving        25     175       40   273
## descriptor      6       7      462    40
#categories of the corpus
dev_prop <- function(observed_count, expected_count) {
  DP_value <- sum(abs(prop.table(observed_count) - prop.table(expected_count)))/2
  DP_normal <- DP_value / (1 - min(prop.table(expected_count)))
  return(DP_normal)
}
dev_prop(dis['shaving', ], freqreg)
## [1] 0.3415698
dev_prop(dis['descriptor', ], freqreg)
## [1] 0.8703311

Interpretation of the Categorical Data

Using your results from above, answer the following: - What are the most frequent parts of speech by type (i.e., we are only considering each word individually, not the frequency of the words as well)? - What are the least frequent parts of speech by type? - Was shaving or descriptor more “dispersed” throughout the texts? Remember, that values close to zero mean that the concept is represented evenly across corpora, while values closer to one are more likely to appear in one corpus over another.

Answer: 1.The most frequent part of speech is Noun

  1. The least frequent part is infinitive-marker

  2. descriptor is more ‘dispersed’ throughout the texts than shaving since the deviation of descriptor is higher.