1 + 1[1] 2HW 5
Quarto
Quarto enables you to weave together content and executable code into a finished document. To learn more about Quarto see https://quarto.org.
Running Code
When you click the Render button a document will be generated that includes both content and the output of embedded code. You can embed code like this:
You can add options to executable code like this
[1] 4The echo: false option disables the printing of code (only output is displayed).
library(rvest)
library(tidyverse)── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr 1.2.1 ✔ readr 2.2.0
✔ forcats 1.0.1 ✔ stringr 1.6.0
✔ ggplot2 4.0.3 ✔ tibble 3.3.1
✔ lubridate 1.9.5 ✔ tidyr 1.3.2
✔ purrr 1.2.2
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ readr::guess_encoding() masks rvest::guess_encoding()
✖ dplyr::lag() masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errorsurl <- "https://books.toscrape.com"
webpage <- read_html(url)
webpage{html_document}
<html lang="en-us" class="no-js">
[1] <head>\n<title>\n All products | Books to Scrape - Sandbox\n</title>\n ...
[2] <body id="default" class="default">\n \n \n \n \n ...books <- webpage |>
html_elements("article.product_pod")
length(books)[1] 20
The first code loads the packages needed for web scraping and working with data. The second code uses read_html() to read the website. The third code uses a CSS selector to find the books on the page. The result should show 20 books on the first page.
titles <- books |>
html_element("h3 > a") |>
html_attr("title")
head(titles)[1] "A Light in the Attic"
[2] "Tipping the Velvet"
[3] "Soumission"
[4] "Sharp Objects"
[5] "Sapiens: A Brief History of Humankind"
[6] "The Requiem Red" ratings_raw <- books |>
html_element("p.star-rating") |>
html_attr("class")
ratings <- gsub("star-rating ", "", ratings_raw)
head(ratings)[1] "Three" "One" "One" "Four" "Five" "One" prices <- books |>
html_element("p.price_color") |>
html_text2()
prices <- as.numeric(gsub("£", "", prices))
head(prices)[1] 51.77 53.74 50.10 47.82 54.23 22.65These three code chunks extract the book titles, ratings, and prices from the website. The html_element() function selects specific information, and html_attr() extracts attributes. The gsub() function removes unnecessary text from the ratings and the pound symbol from the prices.
stock <- books |>
html_element("p.availability") |>
html_text2() |>
trimws()
head(stock)[1] "In stock" "In stock" "In stock" "In stock" "In stock" "In stock"images <- books |>
html_element("img.thumbnail") |>
html_attr("src") |>
gsub("../../", "https://books.toscrape.com/", x = _)
head(images)[1] "media/cachttps://books.toscrape.com/da/2cdad67c44b002e7ead0cc35693c0e8b.jpg"
[2] "media/cachttps://books.toscrape.com/0c/260c6ae16bce31c8f8c95daddd9f4a1c.jpg"
[3] "media/cachttps://books.toscrape.com/ef/3eef99c9d9adef34639f510662022830.jpg"
[4] "media/cachttps://books.toscrape.com/51/3251cf3a3412f53f339e42cac2134093.jpg"
[5] "media/cachttps://books.toscrape.com/a5/bea5697f2534a2f86a3ef27b5a8c12a6.jpg"
[6] "media/cachttps://books.toscrape.com/33/68339b4c9bc034267e1da611ab3b34f8.jpg"test_df <- tibble(
title = titles,
rating = ratings,
price = prices,
stock = stock,
image = images
)
head(test_df)# A tibble: 6 × 5
title rating price stock image
<chr> <chr> <dbl> <chr> <chr>
1 A Light in the Attic Three 51.8 In stock media/cachttps://…
2 Tipping the Velvet One 53.7 In stock media/cachttps://…
3 Soumission One 50.1 In stock media/cachttps://…
4 Sharp Objects Four 47.8 In stock media/cachttps://…
5 Sapiens: A Brief History of Humankind Five 54.2 In stock media/cachttps://…
6 The Requiem Red One 22.6 In stock media/cachttps://…Here this code collects stock and image information and combine everything into a data frame. To make it easier for us to see what is going on.
all_books <- list()
for (i in 1:50) {
if (i == 1) {
url <- "https://books.toscrape.com"
} else {
url <- paste0("https://books.toscrape.com/catalogue/page-", i, ".html")
}
page <- read_html(url)
books <- html_elements(page, "article.product_pod")
all_books[[i]] <- tibble(
title = html_attr(html_element(books, "h3 > a"), "title"),
rating = gsub("star-rating ", "", html_attr(html_element(books, "p.star-rating"), "class")),
price = html_text2(html_element(books, "p.price_color")),
stock = trimws(html_text2(html_element(books, "p.availability"))),
image = gsub("../../", "https://books.toscrape.com/",
html_attr(html_element(books, "img.thumbnail"), "src"))
)
Sys.sleep(0.5)
cat("Scraped page", i, "\n")
}Scraped page 1
Scraped page 2
Scraped page 3
Scraped page 4
Scraped page 5
Scraped page 6
Scraped page 7
Scraped page 8
Scraped page 9
Scraped page 10
Scraped page 11
Scraped page 12
Scraped page 13
Scraped page 14
Scraped page 15
Scraped page 16
Scraped page 17
Scraped page 18
Scraped page 19
Scraped page 20
Scraped page 21
Scraped page 22
Scraped page 23
Scraped page 24
Scraped page 25
Scraped page 26
Scraped page 27
Scraped page 28
Scraped page 29
Scraped page 30
Scraped page 31
Scraped page 32
Scraped page 33
Scraped page 34
Scraped page 35
Scraped page 36
Scraped page 37
Scraped page 38
Scraped page 39
Scraped page 40
Scraped page 41
Scraped page 42
Scraped page 43
Scraped page 44
Scraped page 45
Scraped page 46
Scraped page 47
Scraped page 48
Scraped page 49
Scraped page 50 full_df <- bind_rows(all_books)
nrow(full_df)[1] 1000And this code goes through all 50 pages and combines the book information into one dataset.
full_df$price <- as.numeric(gsub("£", "", full_df$price))
head(full_df)# A tibble: 6 × 5
title rating price stock image
<chr> <chr> <dbl> <chr> <chr>
1 A Light in the Attic Three 51.8 In stock media/cachttps://…
2 Tipping the Velvet One 53.7 In stock media/cachttps://…
3 Soumission One 50.1 In stock media/cachttps://…
4 Sharp Objects Four 47.8 In stock media/cachttps://…
5 Sapiens: A Brief History of Humankind Five 54.2 In stock media/cachttps://…
6 The Requiem Red One 22.6 In stock media/cachttps://…rating_map <- c(One=1, Two=2, Three=3, Four=4, Five=5)
full_df1 <- full_df |>
mutate(rating_num = rating_map[rating])
table1 <- full_df1 |>
count(rating, sort = TRUE)
table1# A tibble: 5 × 2
rating n
<chr> <int>
1 One 226
2 Three 203
3 Five 196
4 Two 196
5 Four 179full_df1 |>
group_by(rating) |>
summarise(avg_price = mean(price, na.rm = TRUE)) |>
arrange(desc(avg_price))# A tibble: 5 × 2
rating avg_price
<chr> <dbl>
1 Four 36.1
2 Five 35.4
3 Two 34.8
4 Three 34.7
5 One 34.6And then this code goes and converts prices and ratings into numbers, count books by rating, and calculate average prices.