We have a text corpus composed of documents. (In CLA 3206A, typically a collection of novels. So the corpus is the collection; the documents, the individual novels). TF-IDF, term frequency–inverse document frequency, measures “how important a word is to a document in a collection” {2}.

We know that the frequently used terms in English, words such as “the”, “to”, “and”, and “of”, provide us with little insight about the document’s topics or distinct content. So low information value. But we also sense that if a term occurs often in one document, but not nearly as much in the other documents, that word likely does both relate to the content and help distinguish the document. So higher information value. TF-IDF, a widely used statistical measure for text-mining and informational retrieval, provides a formal mathematical expression of that intuition. It balances the document TF score against how often the term occurs in the rest of the corpus, the IDF. If a term is used often in the document and corpus, it has a low to (effectively) zero TF-IDF score. If the term appears often in the document, but rarely in the corpus: high TF-IDF score. So by this method, TF-IDF indicates the information value of a term.

(See also Silge & Robinson, “Analyzing word and document frequency: tf-idf”).

Source: CLA 3206A TM Guide. TJ Haslam, CC-BY-4.0, 2020. In-text citations by URL link.

To cite a standard definition:

A bigram or digram is a sequence of two adjacent elements from a string of tokens, which are typically letters, syllables, or words. A bigram is an n-gram for n=2. The frequency distribution of every bigram in a string is commonly used for simple statistical analysis of text in many applications, including in computational linguistics, cryptography, speech recognition, and so on. {4}

The CLA 3206A bigram analysis methods focused on adjacent words, also known as word pairs (when separated). In some cases, as is standard Tidytext TM practice, stop-words were used to produce cleaned data sets of bigrams and word pairs {5}.

Bigrams and word pairs typically provide valuable insight into the major topics of a document or corpus. Word pairs also typically provide insight about the document style, and about word-associations. These in turn may reveal important linkages between content and ideas.

Source: CLA 3206A TM Guide. TJ Haslam, CC-BY-4.0, 2020. In-text citations by URL link.

A network graph can reveal the structure of relationships among the word pairs, and not just a simple count of frequency or even tf-idf value. It potentially reveals connections and linkages that we might otherwise miss if the word pairs were plotted in a bar graph, or listed in a table, or if the text were read in a natural linear fashion.

Although relational, the Word Pairs network graph is also directional: it proceeds from term 1 to term 2, as indicated by the line and arrow. Indeed, as Silge & Robinson point out: the Word Pairs network graph is also a “visualization of a Markov chain, a common model in text processing. In a Markov chain, each choice of word depends only on the previous word” {8}.

In brief, the Word Pairs network graph provides us with an overview of key word-pairs that also indicates their relationships within the the document (or documents).

(See also Silge & Robinson, “Relationships between words: n-grams and correlations”).

Source: CLA 3206A TM Guide. TJ Haslam, CC-BY-4.0, 2020. In-text citations by URL link.

Word Pairs vs. Word Corrs:
Word Pairs consider adjacent words: term1, term2. This has a linear order: the way we naturally read a text. For some examples from Nathaniel Hawthorne’s The Scarlet Letter, designating term2 as “child”: “elf child”, “strange child”, “naughty child”, and “poor child”. In contrast, Word Correlations consider document sections {6}. If termX appears in a given section, what other terms are likely to appear in that same section? This can be anywhere in the section: any place before or after termX, not just adjacent.

Word Corr Findings :
So a Word Correlation analysis tells us what words are associated with termX, and the strength of those associations {6}. These word clusters, centered on termX, reveal linkages of language and thought that might otherwise escape our attention since we typically read in a linear fashion. In the case of an individual author, they help show that author’s linguistic habits – the unconscious as well as conscious mind at work. For a corpus of different authors, they can help reveal underlying assumptions – assumptions perhaps even unknown to the authors being studied!

Word Correlation analyses discover which terms co-occur with each other, as per a specified section of a document, and the importance (statistical significance) of that co-occurrence as measured by the correlation value {9}.

A network graph of word correlations displays linkages and associations that are not as easily captured in a bar plot or data table, and are typically missed when reading the text in a natural linear fashion. Unlike a Word Pairs network graph, a Word Correlations network graph is not directional: rather, it is cluster-centered depending on the strength of correlation.

In brief, the Word Correlations network graph reveals valuable term clusters (empirically determined word associations) which provide information about the deeper layers of language-usage and thought in the document (or documents).

(See also Silge & Robinson, “Relationships between words: n-grams and correlations”).

Source: CLA 3206A TM Guide. TJ Haslam, CC-BY-4.0, 2020. In-text citations by URL link.

The Gender Analyses in CLA 3206A follow the lead of Julia Silge’s study, for which Silge herself credits the break-through academic study by Professors Matthew Jockers and Gabi Kirilloff, “Understanding Gender and Character Agency in the 19th Century Novel”, Journal of Cultural Analytics (2017).

Approaching the topic of “character identity [as realized through] character action”, Jockers and Kirilloff(2017) examined “character agency in the context of character gender” by examining “trends in behavior associated with male and female characters”. They did so by studying patterns of gender pronoun usage: she what? he what? Similar to Silge, CLA 3206A follows their general method, though with differences on the technical (coding) level.

She | He
By seeing what words – and particularly verbs – follow “she” or “he”, we can gain insight into how these texts portray gender and hence gender roles: “character agency in the context of character gender”, as Jockers and Kirilloff (2017) express it.

Source: CLA 3206A TM Guide. TJ Haslam, CC-BY-4.0, 2020. In-text citations by URL link.