1 Why R for linguistics?

As linguists, we work with language data — words, sentences, texts, corpora. R was built for data analysis, and it handles text beautifully.

Think about it: if you wanted to count the most frequent words in a 10,000-word text, how long would that take by hand? In R, it takes three lines of code.

Let’s find out how.

2 Part 1 — R basics: talking to your data

2.1 Your first R commands

R works like a calculator, but for language too.

# Basic math — R as a calculator
2 + 2
## [1] 4
100 / 4
## [1] 25
# Storing a value in a variable
my_number <- 42
my_number
## [1] 42

Try it yourself: Change 42 to your favourite number and run the chunk.

2.2 Working with text (strings)

In R, text is stored in strings — words or sentences in quotes.

# A single word
word <- "linguistics"
word
## [1] "linguistics"
# How many characters does it have?
nchar(word)
## [1] 11
# A sentence
sentence <- "Language is a window into the human mind."
sentence
## [1] "Language is a window into the human mind."
# Count characters
nchar(sentence)
## [1] 41

Try it yourself: Replace the sentence with your own and check its length.

2.3 Vectors — a list of words

A vector is R’s way of storing multiple items together.

# A vector of words
words <- c("syntax", "phonology", "morphology", "semantics", "pragmatics")
words
## [1] "syntax"     "phonology"  "morphology" "semantics"  "pragmatics"
# How many words?
length(words)
## [1] 5
# Access individual elements
words[1]   # first element
## [1] "syntax"
words[3]   # third element
## [1] "morphology"

3 Part 2 — Text analysis: a real linguistics task

3.1 Our text

We’ll work with the opening of Don Quijote (Chapter 1, 1605). This is one of the most analysed texts in the Spanish literary canon — let’s see what R finds.

text <- "In a village of La Mancha, the name of which I have no desire to call to mind, 
there lived not long since one of those gentlemen that keep a lance in the lance-rack, 
an old buckler, a lean hack, and a greyhound for coursing. 
An olla of rather more beef than mutton, a salad on most nights, scraps on Saturdays, 
lentils on Fridays, and a pigeon or so extra on Sundays, made away with three-quarters of his income. 
The rest of it went in a doublet of fine cloth and velvet breeches and shoes to match for holidays, 
while on week-days he made a brave figure in his best homespun."

3.2 Step 1 — Split the text into words (tokenization)

Tokenization means breaking text into individual units (tokens).

# Split by spaces and punctuation
words_raw <- unlist(strsplit(tolower(text), "[^a-zA-Z]+"))

# Remove empty strings
words_clean <- words_raw[words_raw != ""]

# Show the first 20 words
head(words_clean, 20)
##  [1] "in"      "a"       "village" "of"      "la"      "mancha"  "the"    
##  [8] "name"    "of"      "which"   "i"       "have"    "no"      "desire" 
## [15] "to"      "call"    "to"      "mind"    "there"   "lived"
# How many words total?
length(words_clean)
## [1] 114

Discussion: Why did we use tolower()? What happens if we don’t?

3.3 Steps 2–4 — Frequencies, stop words, and visualization

Run this whole chunk at once — it counts words, removes stop words, and plots the result.

# --- Step 2: Count word frequencies ---
freq_table <- sort(table(words_clean), decreasing = TRUE)

cat("Top 15 most frequent words (including function words):\n")
## Top 15 most frequent words (including function words):
print(head(freq_table, 15))
## words_clean
##     a    of    on   and    in   the    to    an   for   his lance  made  away 
##     8     7     5     4     4     3     3     2     2     2     2     2     1 
##  beef  best 
##     1     1
# --- Step 3: Remove stop words ---
# These are called "function words" or "closed-class words" in linguistics
stop_words <- c("a", "an", "the", "of", "in", "on", "and", "or", "is", "it",
                "he", "his", "her", "was", "were", "that", "to", "for", "at",
                "by", "with", "not", "so", "which", "i", "have", "no", "there",
                "while", "than", "made", "went", "more", "most", "rest")

content_words <- words_clean[!words_clean %in% stop_words]
freq_content  <- sort(table(content_words), decreasing = TRUE)

cat("\nTop 15 content words (after removing stop words):\n")
## 
## Top 15 content words (after removing stop words):
print(head(freq_content, 15))
## content_words
##    lance     away     beef     best    brave breeches  buckler     call 
##        2        1        1        1        1        1        1        1 
##    cloth coursing     days   desire  doublet    extra   figure 
##        1        1        1        1        1        1        1
# --- Step 4: Visualize ---
top_words <- head(freq_content, 12)
top_words <- sort(top_words)  # ascending so longest bar is at top

barplot(top_words,
        horiz  = TRUE,
        las    = 1,
        col    = "#74b9ff",
        border = "white",
        main   = "Most frequent content words",
        sub    = "Don Quijote, Chapter 1 (English translation)",
        xlab   = "Frequency")

Discussion: What do you notice after removing stop words? What words remain, and what do they tell us about the text?

Try it yourself: Change col = "#74b9ff" to col = "#fd79a8" — what changes?

4 Part 3 — Going further: what else can R do?

4.1 Type-Token Ratio (TTR)

A classic measure of lexical diversity — what proportion of the words are unique?

# Types = unique words; Tokens = total words
types  <- length(unique(words_clean))
tokens <- length(words_clean)

ttr <- types / tokens

cat("Types (unique words):", types, "\n")
## Types (unique words): 82
cat("Tokens (total words):", tokens, "\n")
## Tokens (total words): 114
cat("Type-Token Ratio:    ", round(ttr, 3), "\n")
## Type-Token Ratio:     0.719

Discussion: What does a high TTR mean? A low one? Which text would have a higher TTR — a children’s book or an academic article?

4.2 Average word length

word_lengths <- nchar(words_clean)

cat("Mean word length:", round(mean(word_lengths), 2), "characters\n")
## Mean word length: 3.94 characters
cat("Shortest word:  ", min(word_lengths), "characters\n")
## Shortest word:   1 characters
cat("Longest word:   ", max(word_lengths), "characters\n")
## Longest word:    9 characters
# Distribution
hist(word_lengths,
     main = "Distribution of word lengths",
     xlab = "Number of characters",
     col  = "#74b9ff",
     border = "white")

Discussion: What shape is the distribution? Is this what you expected?

5 Homework challenge (optional)

Try replacing the text variable with a different text — a paragraph from a Spanish novel, a news article, or even a song lyric you like.

Then answer: 1. What are the 10 most frequent content words? 2. What is the Type-Token Ratio? 3. What does the frequency chart tell you about the text?

6 Where to go next

Tool What it does
tidytext Tidy text analysis in R
quanteda Corpus analysis, concordances, collocations
udpipe Automatic POS tagging and dependency parsing
ggplot2 Beautiful data visualizations

A great free resource: Text Mining with R (Silge & Robinson) — available at tidytextmining.com

Happy coding! Remember: errors are not failures — they’re just R asking for clarification.