Three pillars of critical realism in Great Britain
- Thomas Hardy (1840-1928), an English novelist and poet, Victorian realist.
Many of his novels concern tragic characters struggling against their passions and social circumstances, and they are often set in the semi-fictional region of Wessex.
- H. G. Wells (1866-1946), an English writer.
He is now best remembered for his science fiction novels and is often called the “father of science fiction (notably social science fiction)”.
- D. H. Lawrence (1885-1930), an English writer and poet.
His collected works represent, among other things, an extended reflection upon the dehumanising effects of modernity and industrialisation.
The tidy text format
Tidy data has a specific structure: each variable is a column, each observation is a row, each type of observational unit is a table.
We thus define the tidy text format as being a table with one-token-per-row.
This one-token-per-row structure is in contrast to the ways text is often stored in current analyses, perhaps as strings or in a document-term matrix.
For tidy text mining, the token that is stored in each row is most often a single word, but can also be an n-gram, sentence, or paragraph.
Comparing frequencies across texts of different authors
- Tidying other books.
- Creating a proportion.
library(tidyr)
library(stringr)
frequency <- bind_rows(mutate(tidy_books, author = "Thomas Hardy"),
mutate(tidy_hgwells, author = "H.G.Wells"),
mutate(tidy_lawrence, author = "D.H.Lawrence")) %>%
mutate(word = str_extract(word, "[a-z']+")) %>%
count(author, word) %>%
group_by(author) %>%
mutate(proportion = n / sum(n)) %>%
select(-n) %>%
spread(author, proportion) %>%
gather(author, proportion, `D.H.Lawrence`:`H.G.Wells`)
library(scales)
library(ggplot2)
ggplot(frequency, aes(x = proportion, y = `Thomas Hardy`, color = abs(`Thomas Hardy` - proportion))) +
geom_abline(color = "gray40", lty = 2) +
geom_jitter(alpha = 0.1, size = 2.5, width = 0.3, height = 0.3) +
geom_text(aes(label = word), check_overlap = TRUE, vjust = 1.5) +
scale_x_log10(labels = percent_format()) +
scale_y_log10(labels = percent_format()) +
scale_color_gradient(limits = c(0, 0.001), low = "darkslategray4", high = "gray75") +
facet_wrap(~author, ncol = 2) +
theme(legend.position="none") +
labs(y = "Thomas Hardy", x = NULL, title= "Comparing word frequencies")
- Calculating correlation.
##
## Pearson's product-moment correlation
##
## data: proportion and Thomas Hardy
## t = 113.2, df = 11784, p-value < 2.2e-16
## alternative hypothesis: true correlation is not equal to 0
## 95 percent confidence interval:
## 0.7130095 0.7303097
## sample estimates:
## cor
## 0.7217723
##
## Pearson's product-moment correlation
##
## data: proportion and Thomas Hardy
## t = 84.574, df = 8220, p-value < 2.2e-16
## alternative hypothesis: true correlation is not equal to 0
## 95 percent confidence interval:
## 0.6703905 0.6935119
## sample estimates:
## cor
## 0.6821216
Analyzing bigrams
So far we’ve considered words as individual units, and considered their relationships to sentiments or to documents. However, many interesting text analyses are based on the relationships between words.
## # A tibble: 217,182 x 2
## bigram n
## <chr> <int>
## 1 of the 3798
## 2 in the 2627
## 3 to the 1734
## 4 it was 1267
## 5 to be 1167
## 6 and the 1060
## 7 at the 1052
## 8 on the 1033
## 9 he had 1030
## 10 of a 990
## # ... with 217,172 more rows
A lot of the most common bigrams are pairs of common (uninteresting) words, such as of the and to be: what we call “stop-words”. Let’s separate pairs into two columns, “word1” and “word2”, at which point we can remove cases where either is a stop-word.
## # A tibble: 47,956 x 2
## book bigram
## <chr> <chr>
## 1 Desperate Remedies desperate remedies
## 2 Desperate Remedies thomas hardy
## 3 Desperate Remedies hardy contents
## 4 Desperate Remedies contents prefatory
## 5 Desperate Remedies prefatory note
## 6 Desperate Remedies fortnight iii
## 7 Desperate Remedies days iv
## 8 Desperate Remedies day vi
## 9 Desperate Remedies twelve hours
## 10 Desperate Remedies hours vii
## # ... with 47,946 more rows
Our sentiment analysis approach simply counted the appearance of positive or negative words, according to a reference lexicon. One of the problems with this approach is that a word’s context can matter nearly as much as its presence.
Now that we have the data organized into bigrams, it’s easy to examine the most frequent words that were preceded by “not” and were associated with a sentiment.
## # A tibble: 965 x 3
## word1 word2 n
## <chr> <chr> <int>
## 1 not to 239
## 2 not a 177
## 3 not be 121
## 4 not the 87
## 5 not so 82
## 6 not been 81
## 7 not at 80
## 8 not know 63
## 9 not have 58
## 10 not much 56
## # ... with 955 more rows
## # A tibble: 163 x 3
## word2 value n
## <chr> <dbl> <int>
## 1 help 2 23
## 2 care 2 17
## 3 love 3 17
## 4 like 2 16
## 5 allow 1 15
## 6 true 2 11
## 7 worth 2 11
## 8 want 1 10
## 9 good 3 9
## 10 wish 1 9
## # ... with 153 more rows
It’s worth asking which words contributed the most in the “wrong” direction. To compute that, we can multiply their value by the number of times they appear (so that a word with a value of +3 occurring 10 times has as much impact as a word with a sentiment value of +1 occurring 30 times).
Visualising a network of bigrams
We may be interested in visualizing all of the relationships among words simultaneously, rather than just the top few at a time. As one common visualization, we can arrange the words into a network, or “graph.”
library(igraph)
bigram_graph <- bigram_counts %>%
filter(n > 20) %>%
graph_from_data_frame()
library(ggraph)
set.seed(2016)
a <- grid::arrow(type = "closed", length = unit(.15, "inches"))
ggraph(bigram_graph, layout = "fr") +
geom_edge_link(aes(edge_alpha = n), show.legend = FALSE,
arrow = a, end_cap = circle(.07, 'inches')) +
geom_node_point(color = "lightblue", size = 5) +
geom_node_text(aes(label = name), vjust = 1, hjust = 1) +
theme_void()
We may instead want to examine correlation among words, which indicates how often they appear together relative to how often they appear separately.
section_words <- all_books %>%
filter(book == "Far From The Madding Crowd") %>%
mutate(section = row_number() %/% 10) %>%
filter(section > 0) %>%
unnest_tokens(word, text) %>%
filter(!word %in% stop_words$word)
library(widyr)
word_pairs <- section_words %>%
pairwise_count(word, section, sort = TRUE)
word_cors <- section_words %>%
group_by(word) %>%
filter(n() >= 20) %>%
pairwise_cor(word, section, sort = TRUE)
library(ggplot2)
word_cors %>%
filter(item1 %in% c("fanny", "oak", "troy", "love")) %>%
group_by(item1) %>%
top_n(6) %>%
ungroup() %>%
mutate(item2 = reorder(item2, correlation)) %>%
ggplot(aes(item2, correlation)) +
geom_bar(stat = "identity") +
facet_wrap(~ item1, scales = "free") +
theme_minimal(base_size = 13) +
labs(title = "Correlation among words")+
coord_flip()
Just as we used ggraph to visualize bigrams, we can use it to visualize the correlations and clusters of words that were found by the widyr package.
Calculating term frequency and inverse document frequency.
The idea of tf-idf is to find the important words for the content of each document by decreasing the weight for commonly used words and increasing the weight for words that are not used very much in a collection or corpus of documents. Calculating tf-idf attempts to find the words that are important (i.e., common) in a text, but not too common.
## # A tibble: 43,323 x 6
## book word n tf idf tf_idf
## <chr> <chr> <int> <dbl> <dbl> <dbl>
## 1 Jude The Obscure jude 840 0.00573 1.39 0.00795
## 2 Desperate Remedies cytherea 529 0.00370 1.39 0.00514
## 3 Far From The Madding Crowd bathsheba 483 0.00346 1.39 0.00480
## 4 Desperate Remedies manston 429 0.00300 1.39 0.00416
## 5 The Wonderlanders fitzpiers 368 0.00268 1.39 0.00372
## 6 The Wonderlanders melbury 361 0.00263 1.39 0.00365
## 7 Far From The Madding Crowd gabriel 337 0.00242 1.39 0.00335
## 8 Desperate Remedies aldclyffe 322 0.00225 1.39 0.00313
## 9 The Wonderlanders winterborne 300 0.00219 1.39 0.00303
## 10 Far From The Madding Crowd boldwood 280 0.00201 1.39 0.00278
## # ... with 43,313 more rows
Here we see all proper nouns, names that are in fact important in these novels. None of them occur in all of novels, and they are important, characteristic words for each text within the corpus of Hardy’s novels.
