Exam Review

Question 1 was designed to ensure that you did not lose track of why we were: (i) scraping data; (ii) cleaning data; (iii) manipulating data; (iv) data warehouse.

Scraping Data:

Purpose: Scraping is essential for gathering data that isn’t readily available in structured formats, like publicly displayed data on websites. It’s often used to collect up-to-date, specific, or large-scale datasets without manual entry.

Example: A business may scrape e-commerce sites for competitor pricing or gather social media mentions to analyze customer sentiment.

Cleaning Data:

Purpose: Raw data often contains errors, inconsistencies, duplicates, or irrelevant information. Data cleaning is the process of making data reliable and usable, ensuring analysis is accurate and actionable.

Example: Cleaning data might involve removing duplicate rows, handling missing values, or standardizing date formats to prepare data for meaningful insights.

Manipulating Data:

Purpose: Manipulation allows us to reshape, filter, aggregate, or calculate new variables to extract insights relevant to our questions or goals. Using tools like dplyr in R streamlines these transformations.

Example: With flight data, manipulation can involve filtering for specific airlines, summarizing delays by month, or merging datasets to analyze trends across years.

Data Warehouse:

Purpose: A data warehouse serves as a centralized repository for integrated data from multiple sources, supporting analysis across time and scale. Warehousing enables consistency, faster querying, and more efficient, large-scale analyses.

Example: A company might consolidate sales, marketing, and customer support data in a warehouse, allowing for a unified view of performance metrics and customer behavior over time.

# Question 2 evaluated learning outcomes of web scraping. Read HTML Tables with “rvest”. Exactly similar to the table extract that we did in class. (Assignment 3)

<!-- rnb-text-end -->


<!-- rnb-chunk-begin -->


<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuIyBFeGFtcGxlOlxuXG5saWJyYXJ5KHJ2ZXN0KVxuXG51cmwgPC0gXCJodHRwczovL3d3dy5pbWRiLmNvbS90aXRsZS90dDcyMzU0NjYvZnVsbGNyZWRpdHM/cmVmXz10dF9jbF9zbVwiXG5cbnBhZ2UgPC0gcmVhZF9odG1sKHVybClcblxudGFibGVzIDwtIHBhZ2UgJT4lIGh0bWxfdGFibGUoKVxuXG5zZXJpZXNfY2FzdF90YWJsZSA8LSB0YWJsZXNbWzNdXVxuXG5jbGVhbmVkX3RhYmxlIDwtIHNlcmllc19jYXN0X3RhYmxlICU+JSBzZWxlY3QoMiwgNClcblxuY2xlYW5lZF90YWJsZSA8LSBzdWJzZXQoY2xlYW5lZF90YWJsZSwgY2xlYW5lZF90YWJsZVssIDFdICE9IFwiXCIgJiBjbGVhbmVkX3RhYmxlWywgMl0gIT0gXCJcIilcblxuXG5jbGVhbmVkX3RhYmxlIDwtIGNsZWFuZWRfdGFibGVbIWFwcGx5KGNsZWFuZWRfdGFibGUgPT0gXCJcIiwgMSwgYWxsKSwgXVxuXG5cbmZpbmFsX3Jvd3MgPC0gbnJvdyhjbGVhbmVkX3RhYmxlKVxuZmluYWxfY29sdW1ucyA8LSBuY29sKGNsZWFuZWRfdGFibGUpXG5cbmNhdChcIlRoZSBjbGVhbmVkIGNhc3QgdGFibGUgaGFzXCIsIGZpbmFsX3Jvd3MsIFwib2JzZXJ2YXRpb25zIGFuZFwiLCBmaW5hbF9jb2x1bW5zLCBcImNvbHVtbnMuXFxuXCIpXG5cbnByaW50KGhlYWQoY2xlYW5lZF90YWJsZSkpXG5gYGAifQ== -->

```r
# Example:

library(rvest)

url <- "https://www.imdb.com/title/tt7235466/fullcredits?ref_=tt_cl_sm"

page <- read_html(url)

tables <- page %>% html_table()

series_cast_table <- tables[[3]]

cleaned_table <- series_cast_table %>% select(2, 4)

cleaned_table <- subset(cleaned_table, cleaned_table[, 1] != "" & cleaned_table[, 2] != "")


cleaned_table <- cleaned_table[!apply(cleaned_table == "", 1, all), ]


final_rows <- nrow(cleaned_table)
final_columns <- ncol(cleaned_table)

cat("The cleaned cast table has", final_rows, "observations and", final_columns, "columns.\n")

print(head(cleaned_table))

Question 3 is designed to extract data from a given API. I provide you with the package and function that were needed.Be able to index a list and/or data frame.

Example:

install.packages("jsonlite")
library(jsonlite)
BTC <- fromJSON("https://min-api.cryptocompare.com/data/v2/histoday?fsym=BTC&tsym=USD&limit=100")

str(BTC)

BTC_data <- BTC$Data$Data  

head(BTC_data)

max_close_price <- max(BTC_data$close, na.rm = TRUE)

max_close_price

Question 4 is designed to measure your ability to clean data. It was similar to what we have done with the iris datasets.The kNN procedure.

#In R, the class package has a knn function for classification, and packages like VIM or DMwR provide functions for kNN-based imputation.

Question 5 is designed to measure your ability to manipulate data. It was similar to what we have done with the nycflights13 datasets. The use of data manipulation functions from “dplyr” package to derive the metrics needed.

Example:
#install.packages("dplyr")
#install.packages("pacman")

pacman::p_load(nycflights13)

View(flights) # this View() function opens lets you directly view the whole dataset

glimpse(flights) # this glimpse() function provides a quick overview of the dataset
summary(flights)


select(flights, starts_with("dep"))

 
sortf <- arrange(flights,desc(dep_delay)) 
select(sortf, carrier, flight, tailnum, everything())