EDA-dplyr study

Overview

EDA学习，根据 Exploratory Data Analysis with R 这本书，这里首先是dplyr包的学习。

Key “verbs

•select: return a subset of the columns of a data frame, using a flexible notation

•filter: extract a subset of rows from a data frame based on logical conditions

•arrange: reorder rows of a data frame

•rename: rename variables in a data frame

•mutate: add new variables/columns or transform existing variables

•summarise / summarize: generate summary statistics of different variables in the data frame, possibly within strata

•%>%: the “pipe” operator is used to connect multiple verb actions together into a pipeline

library(dplyr)

Attaching package: ‘dplyr’

The following objects are masked from ‘package:stats’:

    filter, lag

The following objects are masked from ‘package:base’:

    intersect, setdiff, setequal, union

chicago <- readRDS("chicago.rds")
dim(chicago)

[1] 6940    8

str(chicago)

'data.frame':   6940 obs. of  8 variables:
 $ city      : chr  "chic" "chic" "chic" "chic" ...
 $ tmpd      : num  31.5 33 33 29 32 40 34.5 29 26.5 32.5 ...
 $ dptp      : num  31.5 29.9 27.4 28.6 28.9 ...
 $ date      : Date, format:  ...
 $ pm25tmean2: num  NA NA NA NA NA NA NA NA NA NA ...
 $ pm10tmean2: num  34 NA 34.2 47 NA ...
 $ o3tmean2  : num  4.25 3.3 3.33 4.38 4.75 ...
 $ no2tmean2 : num  20 23.2 23.8 30.4 30.3 ...

select

The select() function can be used to select columns of a data frame that you want to focus on.

选择特定列[直接指定]

names(chicago)[1:3]

[1] "city" "tmpd" "dptp"

subset <- select(chicago, city:dptp)
head(subset)

选择特定列[通过不显示一些列，间接指定]

subset2 <- select(chicago, -(city:dptp))
head(subset2)

同样地，还有另一种方式

i <- match("city", names(chicago))  # 1
j <- match("dptp", names(chicago))  # 3
head(chicago[, -(i:j)])

通过列名字符来选取特定列

subset <- select(chicago, ends_with("2"))
str(subset)

'data.frame':   6940 obs. of  4 variables:
 $ pm25tmean2: num  NA NA NA NA NA NA NA NA NA NA ...
 $ pm10tmean2: num  34 NA 34.2 47 NA ...
 $ o3tmean2  : num  4.25 3.3 3.33 4.38 4.75 ...
 $ no2tmean2 : num  20 23.2 23.8 30.4 30.3 ...

subset <- select(chicago, starts_with("d"))
str(subset)

'data.frame':   6940 obs. of  2 variables:
 $ dptp: num  31.5 29.9 27.4 28.6 28.9 ...
 $ date: Date, format:  ...

filter

The filter() function is used to extract subsets of rows from a data frame.

chic.f <- filter(chicago, pm25tmean2 > 30 & tmpd > 80)
select(chic.f, date, tmpd, pm25tmean2)

summary(chic.f$pm25tmean2)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  31.20   32.70   33.90   36.68   38.85   51.54 

arrange

The arrange() function is used to reorder rows of a data frame according to one of the variables/columns.

chicago <- arrange(chicago, date)
head(select(chicago, date, pm25tmean2), 3)

tail(select(chicago, date, pm25tmean2), 3)

降序排列

chicago <- arrange(chicago, desc(date))
head(select(chicago, date, pm25tmean2), 3)

tail(select(chicago, date, pm25tmean2), 3)

rename

head(chicago[, 1:5], 3)

chicago <- rename(chicago, dewpoint=dptp, pm25=pm25tmean2)
head(chicago[, 1:5], 3)

注意，新的名字在前，旧的名字在后

mutate

The mutate() function exists to compute transformations of variables in a data frame.

chicago <- mutate(chicago, pm25detrend = pm25 - mean(pm25, na.rm = TRUE))
head(chicago)

There is also the related transmute() function, which does the same thing as mutate() but then drops all non-transformed variables.

head(transmute(chicago,
               pm10detrend = pm10tmean2 - mean(pm10tmean2, na.rm = TRUE),
               o3detrend = o3tmean2 - mean(o3tmean2, na.rm = TRUE)))

group_by

The group_by() function is used to generate summary statistics from the data frame within strata defined by a variable.

chicago <- mutate(chicago, year=as.POSIXlt(date)$year + 1900)
years <- group_by(chicago, year)
summarise(years, pm25=mean(pm25, na.rm = TRUE),
          o3 = max(o3tmean2, na.rm = TRUE),
          n02 = median(no2tmean2, na.rm = TRUE))

一个更复杂的例子： what are the average levels of ozone (o3) and nitrogen dioxide (no2) within quintiles of pm25?

qq <- quantile(chicago$pm25, seq(0, 1, 0.2), na.rm = TRUE)
chicago <- mutate(chicago, pm25.quint = cut(pm25, qq))
quint <- group_by(chicago, pm25.quint)
summarize(quint, o3=mean(o3tmean2, na.rm = TRUE),
          no2 = mean(no2tmean2, na.rm = TRUE))

%>%

The pipeline operater %>% is very handy for stringing together multiple dplyr functions in a sequence of operations.

使用管道操作符，使得我们可以一次执行一系列的命令，而免于嵌套和创建一些无用的中间变量。

mutate(chicago, pm25.quint=cut(pm25, qq)) %>%
    group_by(pm25.quint) %>%
    summarise(o3=mean(o3tmean2, na.rm = TRUE),
              no2 = mean(no2tmean2, na.rm=TRUE))

Notice in the above code that I pass the chicago data frame to the first call to mutate(), but then afterwards I do not have to pass the first argument to group_by() or summarize(). Once you travel down the pipeline with %>%, the first argument is taken to be the output of the previous element in the pipeline.

Another example might be computing the average pollutant level by month. This could be useful to see if there are any seasonal trends in the data.

mutate(chicago, month=as.POSIXlt(date)$mon+1)%>%
    group_by(month)%>%
    summarise(pm25=mean(pm25, na.rm=TRUE),
              o3 = max(o3tmean2, na.rm=TRUE),
              no2=median(no2tmean2, na.rm=TRUE))