Lab Goal

This lab scrapes basic SEO metadata from several brand websites, converts the scraped text into a small SEO corpus, tokenizes the text into individual words, removes low-signal stop words, and calculates word co-occurrence pairs.

The goal is to answer:

Which words appear together across the scraped SEO metadata, and what do those pairings suggest about the semantic signals each brand is sending?

1. Load Required Packages

This version intentionally uses a smaller package stack than the full tutorial so it can run in a short in-class lab window.

required_packages <- c(
  "rvest",
  "tidytext",
  "widyr",
  "knitr"
)

missing_packages <- required_packages[
  !sapply(required_packages, requireNamespace, quietly = TRUE)
]

if (length(missing_packages) > 0) {
  stop(
    "Install missing packages before knitting: ",
    paste(missing_packages, collapse = ", ")
  )
}

library(rvest)
library(tidytext)
library(widyr)
library(knitr)

2. Define Target Websites

The professor’s starter code used LL Bean, Patagonia, and Nike. Use City websites instead and add a source label so the later text analysis can treat each website as a separate document.

sites <- data.frame(
  source = c("Moorpark", "Camarillo", "Oxnard", "Ventura"),
  url = c(
    "https://www.moorparkca.gov/",
    "https://www.cityofcamarillo.org/",
    "https://www.oxnard.gov/",
    "https://www.cityofventura.ca.gov/"
  ),
  stringsAsFactors = FALSE
)

knitr::kable(sites, caption = "Target Websites for SEO Metadata Scraping")
Target Websites for SEO Metadata Scraping
source url
Moorpark https://www.moorparkca.gov/
Camarillo https://www.cityofcamarillo.org/
Oxnard https://www.oxnard.gov/
Ventura https://www.cityofventura.ca.gov/

3. Helper Functions

These helper functions keep the scraping code safe and readable.

# Base-R replacement for stringr::str_squish()
squish_text <- function(x) {
  if (length(x) == 0 || all(is.na(x))) {
    return(NA_character_)
  }

  x <- paste(x, collapse = " | ")
  x <- gsub("\\s+", " ", x)
  x <- trimws(x)

  if (identical(x, "")) {
    return(NA_character_)
  }

  x
}

safe_text <- function(page, css_selector) {
  tryCatch({
    extracted <- page |>
      html_elements(css_selector) |>
      html_text2()

    squish_text(extracted)
  }, error = function(e) {
    NA_character_
  })
}

safe_attr <- function(page, css_selector, attr_name) {
  tryCatch({
    extracted <- page |>
      html_element(css_selector) |>
      html_attr(attr_name)

    squish_text(extracted)
  }, error = function(e) {
    NA_character_
  })
}

scrape_seo_meta <- function(url, source_name) {
  Sys.sleep(1.5) # polite delay between requests

  tryCatch({
    page <- read_html(url)

    data.frame(
      source = source_name,
      url = url,
      title = safe_text(page, "title"),
      meta_description = safe_attr(page, "meta[name='description']", "content"),
      h1 = safe_text(page, "h1"),
      h2 = safe_text(page, "h2"),
      h3 = safe_text(page, "h3"),
      scrape_status = "success",
      scrape_error = NA_character_,
      stringsAsFactors = FALSE
    )
  }, error = function(e) {
    data.frame(
      source = source_name,
      url = url,
      title = NA_character_,
      meta_description = NA_character_,
      h1 = NA_character_,
      h2 = NA_character_,
      h3 = NA_character_,
      scrape_status = "failed",
      scrape_error = conditionMessage(e),
      stringsAsFactors = FALSE
    )
  })
}

4. Scrape SEO Metadata

results <- list()

for (i in seq_len(nrow(sites))) {
  results[[i]] <- scrape_seo_meta(
    url = sites$url[i],
    source_name = sites$source[i]
  )
}

seo_data <- do.call(rbind, results)

write.csv(seo_data, file = "seo_data.csv", row.names = FALSE)

knitr::kable(
  seo_data[, c("source", "url", "title", "meta_description", "h1", "scrape_status")],
  caption = "Scraped SEO Metadata"
)
Scraped SEO Metadata
source url title meta_description h1 scrape_status
Moorpark https://www.moorparkca.gov/ Moorpark, CA - Official Website | Official Website | Arrow Left | Arrow Right | Slideshow Left Arrow | Slideshow Right Arrow NA | Calendars success
Camarillo https://www.cityofcamarillo.org/ Welcome to Camarillo, CA NA NA success
Oxnard https://www.oxnard.gov/ NA NA NA failed
Ventura https://www.cityofventura.ca.gov/ Ventura, CA | Official Website | Arrow Left | Arrow Right | Slideshow Left Arrow | Slideshow Right Arrow NA NA success

5. Scrape Quality Check

Some modern websites use JavaScript rendering, bot protection, or dynamic metadata. If a field is missing, the row is kept and documented instead of silently replacing it with fake text.

quality_check <- data.frame(
  source = seo_data$source,
  title_missing = is.na(seo_data$title),
  meta_description_missing = is.na(seo_data$meta_description),
  h1_missing = is.na(seo_data$h1),
  scrape_status = seo_data$scrape_status,
  scrape_error = seo_data$scrape_error,
  stringsAsFactors = FALSE
)

knitr::kable(quality_check, caption = "Scrape Quality Check")
Scrape Quality Check
source title_missing meta_description_missing h1_missing scrape_status scrape_error
Moorpark FALSE TRUE FALSE success NA
Camarillo FALSE TRUE TRUE success NA
Oxnard TRUE TRUE TRUE failed cannot open the connection
Ventura FALSE TRUE TRUE success NA

6. Build SEO Corpus

The corpus combines title tags, meta descriptions, and headings into one text field per brand.

seo_corpus <- seo_data

seo_corpus$text <- apply(
  seo_corpus[, c("title", "meta_description", "h1", "h2", "h3")],
  1,
  function(row_values) {
    squish_text(row_values[!is.na(row_values)])
  }
)

seo_corpus <- seo_corpus[
  !is.na(seo_corpus$text) & nchar(seo_corpus$text) > 0,
  c("source", "url", "text")
]

if (nrow(seo_corpus) == 0) {
  knitr::kable(
    data.frame(note = "No usable scraped text was returned. Try rerunning the scrape or checking the sites manually."),
    caption = "SEO Corpus"
  )
} else {
  knitr::kable(seo_corpus, caption = "SEO Corpus: Combined Title, Meta Description, and Headings")
}
SEO Corpus: Combined Title, Meta Description, and Headings
source url text
1 Moorpark https://www.moorparkca.gov/ Moorpark, CA - Official Website | Official Website | Arrow Left | Arrow Right | Slideshow Left Arrow | Slideshow Right Arrow | | Calendars | Loading | | City Hall Hours | Helpful Links
2 Camarillo https://www.cityofcamarillo.org/ Welcome to Camarillo, CA | Trending topics | City News | City Events | QUICK LINKS | ECONOMIC DEVELOPMENT | Share this page |
4 Ventura https://www.cityofventura.ca.gov/ Ventura, CA | Official Website | Arrow Left | Arrow Right | Slideshow Left Arrow | Slideshow Right Arrow | Citywide News | Calendar | Trending Topics | Loading | | July 2026 | Contact Us | Quick Links | Helpful Links

7. Tokenize and Clean Words

if (nrow(seo_corpus) > 0) {
  words <- tidytext::unnest_tokens(
    seo_corpus,
    output = word,
    input = text
  )

  data("stop_words", package = "tidytext")

  words_clean <- words[
    !(words$word %in% stop_words$word) &
      nchar(words$word) > 2 &
      !grepl("^[0-9]+$", words$word),
  ]

  knitr::kable(
    head(words_clean[, c("source", "word")], 30),
    caption = "First 30 Cleaned Word Tokens"
  )
} else {
  words_clean <- data.frame(
    source = character(),
    url = character(),
    word = character(),
    stringsAsFactors = FALSE
  )

  knitr::kable(
    data.frame(note = "No tokens generated because the SEO corpus was empty."),
    caption = "Tokenization Status"
  )
}
First 30 Cleaned Word Tokens
source word
1 Moorpark moorpark
3 Moorpark official
4 Moorpark website
5 Moorpark official
6 Moorpark website
7 Moorpark arrow
8 Moorpark left
9 Moorpark arrow
11 Moorpark slideshow
12 Moorpark left
13 Moorpark arrow
14 Moorpark slideshow
16 Moorpark arrow
17 Moorpark calendars
18 Moorpark loading
19 Moorpark city
20 Moorpark hall
21 Moorpark hours
22 Moorpark helpful
23 Moorpark links
26 Camarillo camarillo
28 Camarillo trending
29 Camarillo topics
30 Camarillo city
31 Camarillo news
32 Camarillo city
33 Camarillo events
34 Camarillo quick
35 Camarillo links
36 Camarillo economic

8. Word Frequency

if (nrow(words_clean) > 0) {
  word_freq <- sort(table(words_clean$word), decreasing = TRUE)

  word_freq_df <- data.frame(
    word = names(word_freq),
    count = as.integer(word_freq),
    stringsAsFactors = FALSE
  )

  top_words <- head(word_freq_df, 20)

  knitr::kable(
    top_words,
    caption = "Top 20 Words in Scraped SEO Metadata"
  )
} else {
  word_freq_df <- data.frame(
    word = character(),
    count = integer(),
    stringsAsFactors = FALSE
  )

  knitr::kable(
    data.frame(note = "No word frequency table available because no clean tokens were generated."),
    caption = "Word Frequency Status"
  )
}
Top 20 Words in Scraped SEO Metadata
word count
arrow 8
left 4
links 4
slideshow 4
city 3
official 3
website 3
helpful 2
loading 2
news 2
quick 2
topics 2
trending 2
calendar 1
calendars 1
camarillo 1
citywide 1
contact 1
development 1
economic 1

9. Word Frequency Chart

if (nrow(word_freq_df) > 0) {
  chart_words <- head(word_freq_df, 10)
  chart_words <- chart_words[order(chart_words$count), ]

  barplot(
    chart_words$count,
    names.arg = chart_words$word,
    horiz = TRUE,
    las = 1,
    main = "Top Words in Scraped SEO Metadata",
    xlab = "Count"
  )
} else {
  plot.new()
  text(0.5, 0.5, "No word frequency chart available")
}

10. Word Co-Occurrence Analysis

A co-occurrence pair means two words appeared in the same website document. With only three websites, many pairs may have a count of 1. Higher counts suggest stronger repeated association across the scraped brand metadata.

if (nrow(words_clean) > 0 && length(unique(words_clean$source)) > 0) {
  word_cooc <- widyr::pairwise_count(
    words_clean,
    item = word,
    feature = source,
    sort = TRUE,
    upper = FALSE
  )

  if (nrow(word_cooc) > 0) {
    knitr::kable(
      head(word_cooc, 25),
      caption = "Top 25 Word Co-Occurrence Pairs"
    )
  } else {
    knitr::kable(
      data.frame(note = "No co-occurrence pairs were generated. The corpus may be too small or too sparse."),
      caption = "Word Co-Occurrence Status"
    )
  }
} else {
  word_cooc <- data.frame(
    item1 = character(),
    item2 = character(),
    n = integer(),
    stringsAsFactors = FALSE
  )

  knitr::kable(
    data.frame(note = "No co-occurrence table available because no clean tokens were generated."),
    caption = "Word Co-Occurrence Status"
  )
}
Top 25 Word Co-Occurrence Pairs
item1 item2 n
official website 2
official arrow 2
website arrow 2
official left 2
website left 2
arrow left 2
official slideshow 2
website slideshow 2
arrow slideshow 2
left slideshow 2
official loading 2
website loading 2
arrow loading 2
left loading 2
slideshow loading 2
official helpful 2
website helpful 2
arrow helpful 2
left helpful 2
slideshow helpful 2
loading helpful 2
official links 2
website links 2
arrow links 2
left links 2

11. Brand-Level Keyword Snapshot

This table helps connect the word-level analysis back to each brand page.

if (nrow(words_clean) > 0) {
  brand_word_counts <- as.data.frame(table(words_clean$source, words_clean$word), stringsAsFactors = FALSE)
  names(brand_word_counts) <- c("source", "word", "count")
  brand_word_counts <- brand_word_counts[brand_word_counts$count > 0, ]
  brand_word_counts <- brand_word_counts[order(brand_word_counts$source, -brand_word_counts$count), ]

  brand_top_words <- do.call(
    rbind,
    lapply(
      split(brand_word_counts, brand_word_counts$source),
      function(x) head(x, 10)
    )
  )

  rownames(brand_top_words) <- NULL

  knitr::kable(
    brand_top_words,
    caption = "Top Terms by Brand Source"
  )
} else {
  knitr::kable(
    data.frame(note = "No brand-level keyword snapshot available."),
    caption = "Brand-Level Keyword Status"
  )
}
Top Terms by Brand Source
source word count
Camarillo city 2
Camarillo camarillo 1
Camarillo development 1
Camarillo economic 1
Camarillo events 1
Camarillo links 1
Camarillo news 1
Camarillo page 1
Camarillo quick 1
Camarillo share 1
Moorpark arrow 4
Moorpark left 2
Moorpark official 2
Moorpark slideshow 2
Moorpark website 2
Moorpark calendars 1
Moorpark city 1
Moorpark hall 1
Moorpark helpful 1
Moorpark hours 1
Ventura arrow 4
Ventura left 2
Ventura links 2
Ventura slideshow 2
Ventura calendar 1
Ventura citywide 1
Ventura contact 1
Ventura helpful 1
Ventura july 1
Ventura loading 1

12. SEO Interpretation

The co-occurrence table shows which words appear together across the scraped SEO metadata. These repeated pairings help reveal the semantic neighborhood each brand is signaling through title tags, meta descriptions, and page headings.

Strong pairs suggest terms that search engines and users may naturally associate with the brand or product category. This does not prove search ranking performance, but it does provide a practical way to inspect topical positioning, compare page messaging, and identify possible content gaps.

For example, if a brand repeatedly places product-category terms near brand-positioning terms, that page is sending a clearer topical signal. If important category terms are missing from the metadata and headings, the page may be leaving useful SEO context on the table.

13. Limitations

14. AI-Use / Learning Note

Category Notes
Task Built a minimum viable SEO metadata and word co-occurrence lab artifact.
AI support Helped simplify the package stack, preserve the professor’s scraping objective, and produce a full replacement R Markdown file.
Human decision-making Chose to prioritize a stable knit over optional network visualization packages that were slow to install.
Problems encountered Full package installation began compiling a large dependency tree, which was not practical for a short in-class lab.
Fixes Reduced the required packages to rvest, tidytext, widyr, and knitr; replaced httr2, dplyr, stringr, igraph, and ggraph dependencies.
Learning The analytical core is scraping, cleaning, tokenizing, co-occurrence counting, and interpretation; network visualization is useful but not mandatory for understanding the concept.
Portfolio value Demonstrates practical troubleshooting, reproducible text analytics, and business interpretation of SEO signals.