Quizes and Solutions

Quiz1 - Group1

Please find the book and line number in Jane Austen books which has the most number of trust words.

• use nrc sentiments
• please remove stop words
• hint: count(book,linenumber, sentiment)

Expected outcome:

book	linenumber	sentiment	n
…	…	trust	5
…	…	…	.

Solution

library(tidytext)
library(tidyverse)
library(janeaustenr)
austen_books() %>%
  group_by(book) %>%
  mutate(linenumber = row_number()) %>%
  ungroup() %>%
  unnest_tokens(word, text) %>%
  anti_join(stop_words) %>%
  inner_join(get_sentiments("nrc")) %>%
  filter(sentiment=="trust") %>% 
  count(book,linenumber, sentiment) %>% 
  arrange(desc(n)) %>% 
  head(2)

Quiz1 - Group2

Please find the book and line number in Jane Austen books which has the most number of positive words.

• use bing sentiments
• please remove stop words
• hint: count(book,linenumber, sentiment)

Expected outcome:

book	linenumber	sentiment	n
…	…	positive	6
…	…	…	.

Solution

library(tidytext)
library(janeaustenr)
austen_books() %>%
  group_by(book) %>%
  mutate(linenumber = row_number()) %>%
  ungroup() %>%
  unnest_tokens(word, text) %>%
  anti_join(stop_words) %>%
  inner_join(get_sentiments("bing")) %>%
  filter(sentiment=="positive") %>% 
  count(book,linenumber, sentiment) %>% 
  arrange(desc(n)) %>% 
  head(2)

Quiz2 - Group1

Find the highest correlated “positive” (nrc) words in “Emma” (sections of 20 lines, word count threshold is 10)

Expected output:

item1	item2	correlation
lucky		0.6329502

Solution

library(janeaustenr)
library(tidytext)
library(widyr)
library(tidyverse)
austen_books() %>%
  filter(book=="Emma") %>% 
  mutate(section = row_number() %/% 20) %>%
  filter(section > 0) %>%
  unnest_tokens(word, text) %>%
  filter(!word %in% stop_words$word) %>% 
  inner_join(get_sentiments("nrc")) %>% 
  filter(sentiment=="positive") %>% 
  group_by(word) %>%
  filter(n() >= 10) %>%   # try lower numbers and see what happens
  pairwise_cor(word, section, sort = TRUE) %>% 
  head(2)

Joining, by = "word"

Note: this quiz gave different result in most of student computers, most likely due to a bug in widyr package

Quiz2 - Group2

Find the top 2 highest correlated words with the word pride in sections of 20 lines in all Jane Austen books. (Word count threshold is 10)

Expected output:

item1	item2	correlation
pride		0.2301
pride		0.1817

Solution

library(janeaustenr)
library(tidytext)
library(widyr)
library(tidyverse)
austen_books() %>%
  mutate(section = row_number() %/% 20) %>%
  filter(section > 0) %>%
  unnest_tokens(word, text) %>%
  filter(!word %in% stop_words$word) %>% 
  group_by(word) %>%
  filter(n() >= 10) %>%   # try lower numbers and see what happens
  pairwise_cor(word, section, sort = TRUE) %>% 
  filter(item1=="pride") %>% 
  head(2)

Note: this quiz gave different result in most of student computers, most likely due to a bug in widyr package

Quiz3 - Group1 and Group2

Please make a linear model of displacement versus mpg from mtcars lm(mpg ~ disp) and answer the questions:

1. if a car has 250 displacement, what would its mpg be?

2. which car has the highest (positive) residual?

3. if lm(mpg ~ cyl) has R squared of 0.726, which model, disp or cyl is better for predicting mpg

Solution

library(broom)
# a
lmfit <- lm(mpg ~ disp, mtcars)
dummy <- data_frame(disp=250)
predict(lmfit,dummy)

       1 
19.29607 

# b
augment(lmfit) %>% 
  arrange(-.resid) %>% 
  head(1)

# residual is difference between actual value and predicted value (actual-prediction)
# thus Corolla has 7.2 more mileage than expected mileage for its displacement
# c
summary(lmfit)$r.squared   #OR glance(lmfit)

[1] 0.7183433

# higher R-squared means better fit/model

Quiz4 - Group1 and Group2

Please do two different kmeans cluster on mtcars

1. regular data, 3 clusters, 10 starts
2. scaled data, 3 clusters, 10 starts

then show the table of cluster matches

	1	2	3
1
2
3

Solution

set.seed(1)
k1 <- kmeans(mtcars, centers= 3, nstart = 10)
k2 <- kmeans(scale(mtcars), centers= 3, nstart = 10 )
table(k1$cluster, k2$cluster)

   
     1  2  3
  1  3 13  0
  2  2  0  7
  3  0  2  5