Getting and Cleaning Data: Week 1 Quiz

---

Question 1

The American Community Survey distributes downloadable data about United States communities. Download the 2006 microdata survey about housing for the state of Idaho using download.file() from here:

https://d396qusza40orc.cloudfront.net/getdata%2Fdata%2Fss06hid.csv

fileUrl1 <- "https://d396qusza40orc.cloudfront.net/getdata%2Fdata%2Fss06hid.csv"
download.file(fileUrl1, destfile = "./Dataset/Quiz1-01.csv", method = "curl")

and load the data into R. The code book, describing the variable names is here: https://d396qusza40orc.cloudfront.net/getdata%2Fdata%2FPUMSDataDict06.pdf

quiz1Data <- read.csv("./Dataset/Quiz1-01.csv")
head(quiz1Data)

How many properties are worth $1,000,000 or more?

Answer

sum(quiz1Data$VAL == 24, na.rm = TRUE)

[1] 53

---

Question 2

Use the data you loaded from Question 1. Consider the variable FES in the code book. Which of the “tidy data” principles does this variable violate?

Answer

Tidy data has one variable per column.

---

Question 3

Download the Excel spreadsheet on Natural Gas Aquisition Program here:

https://d396qusza40orc.cloudfront.net/getdata%2Fdata%2FDATA.gov_NGAP.xlsx (original data source: http://catalog.data.gov/dataset/natural-gas-acquisition-program)

fileUrl1 <- "https://d396qusza40orc.cloudfront.net/getdata%2Fdata%2FDATA.gov_NGAP.xlsx"
download.file(fileUrl1, destfile = "./Dataset/Quiz1-03.xlsx", method = "curl")

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
100 16197  100 16197    0     0   5436      0  0:00:02  0:00:02 --:--:--  5435

dateDownloaded <- date()
dateDownloaded

[1] "Wed Jun 20 21:33:55 2018"

Read rows 18-23 and columns 7-15 into R and assign the result to a variable called: dat

library(xlsx)
col <- 7:15
row <- 18:23
dat <- read.xlsx("./Dataset/Quiz1-03.xlsx", sheetIndex=1, colIndex = col, rowIndex = row)
dat

What is the value of:

Answer

sum(dat$Zip*dat$Ext, na.rm=T)

[1] 36534720

---

Question 4

Read the XML data on Baltimore restaurants from here:

https://d396qusza40orc.cloudfront.net/getdata%2Fdata%2Frestaurants.xml

library(XML)
fileUrl3 <- "https://d396qusza40orc.cloudfront.net/getdata%2Fdata%2Frestaurants.xml"
BalResto <- xmlTreeParse(sub("s", "", fileUrl3), useInternal=TRUE)
rootNode <- xmlRoot(BalResto)

How many restaurants have zipcode 21231?

Answer

zip <- xpathSApply(rootNode, "//zipcode", xmlValue)
sum(zip == 21231)

[1] 127

---

Question 5

The American Community Survey distributes downloadable data about United States communities. Download the 2006 microdata survey about housing for the state of Idaho using download.file() from here:

https://d396qusza40orc.cloudfront.net/getdata%2Fdata%2Fss06pid.csv

fileUrl4 <- "https://d396qusza40orc.cloudfront.net/getdata%2Fdata%2Fss06pid.csv"
download.file(fileUrl4, destfile = "./Dataset/Quiz1-05.csv", method = "curl")

using the fread() command load the data into an R object: DT

library(data.table)
DT <- fread("./Dataset/Quiz1-05.csv")
DT

The following are ways to calculate the average value of the variable: pwgtp15

broken down by sex. Using the data.table package, which will deliver the fastest user time?

Answer

• option a: rowMeans(DT[DT$SEX==1]); rowMeans(DT[DT$SEX==2])

system.time(rowMeans(DT[DT$SEX==1]), rowMeans(DT[DT$SEX==2]))

Error in rowMeans(DT[DT$SEX == 2]) : 'x' must be numeric

• option b: DT[DT$SEX==1,]\(pwgtp15), mean(DT[DT\)SEX==2,]$pwgtp15))

system.time(mean(DT[DT$SEX==1,]$pwgtp15), mean(DT[DT$SEX==2,]$pwgtp15))

   user  system elapsed 
  0.052   0.000   0.050 

• option c: DT[,mean(pwgtp15),by=SEX])

system.time(DT[,mean(pwgtp15),by=SEX])

   user  system elapsed 
  0.004   0.000   0.003 

• option d: sapply(split(DT\(pwgtp15,DT\)SEX),mean))

system.time(sapply(split(DT$pwgtp15,DT$SEX),mean))

   user  system elapsed 
  0.000   0.000   0.002 

• option e: tapply(DT\(pwgtp15,DT\)SEX,mean))

system.time(tapply(DT$pwgtp15,DT$SEX,mean))

   user  system elapsed 
  0.000   0.000   0.002 

• option f: mean(DT\(pwgtp15,by=DT\)SEX))

system.time(mean(DT$pwgtp15,by=DT$SEX))

   user  system elapsed 
  0.000   0.000   0.001 

---

END

---