Data Transformation

1.Load the required libraries and datasets

library(tidyverse)
library(nycflights13)

2.dplyr basics

The five important dplyr() verbs are :

1. filter()
2. arrange()
3. select()
4. mutate()
5. summarise()

all of these can be used in conjuction with group_by(). for a full documentation of dplyr() functions do ?dplyr()

2.1 filter rows with filter() :

by default filter() pipes ‘and’ operator to all the arguments you pass within it,and you get an output only if all the things are evaluated in your argument,if you need explict operations use them explicitly !

logical operations :

1. &
2. |
3. !
4. ==
5. !=

jan1 <- filter(flights, month == 1, day == 1)
filter(flights, month == 11 | month == 12)

nov_dec <- filter(flights, month %in% c(11, 12))
filter(flights, !(arr_delay > 120 | dep_delay > 120))

filter(flights, arr_delay <= 120, dep_delay <= 120)

filter(flights,dest == "IAH" | dest == "HOU")

 
filter(flights,arr_delay < 2 & dep_delay == 0)

filter(flights,dep_time >= 600 & dep_time <= 2400)

also check the doc for between() using ?between() ,look at what it does…

2.2 Arrange() :

arrange() works similarly to filter() except that instead of selecting rows, it changes their order. It takes a data frame and a set of column names (or more complicated expressions) to order by. If you provide more than one column name, each additional column will be used to break ties in the values of preceding columns:

arrange(flights,year,month,day)

arrange(flights,desc(arr_delay))

2.3 select() :

It’s not uncommon to get datasets with hundreds or even thousands of variables. In this case, the first challenge is often narrowing in on the variables you’re actually interested in. select() allows you to rapidly zoom in on a useful subset using operations based on the names of the variables.

select() is not terribly useful with the flights data because we only have 19 variables, but you can still get the general idea:

# Select columns by name
select(flights, year, month, day)

# Select all columns between year and day (inclusive)
select(flights, year:day)

# Select all columns except those from year to day (inclusive)
select(flights, -(year:day))

• There are a number of helper functions you can use within select():
  • starts_with(“abc”): matches names that begin with “abc”
  • ends_with(“xyz”): matches names that end with “xyz”.
  • contains(“ijk”): matches names that contain “ijk”.
  • matches(“(.)\1”): selects variables that match a regular expression.
  • num_range(“x”, 1:3) matches x1, x2 and x3.

See ?select for more details. also check ?rename() and ?everything() .

2.4 mutate() :

Besides selecting sets of existing columns, it’s often useful to add new columns that are functions of existing columns. That’s the job of mutate().

#creating a new dataset
flights_sml <- select(flights, 
  year:day, 
  ends_with("delay"), 
  distance, 
  air_time
)
#adding new variables to the dataset 
mutate(flights_sml,
  gain = arr_delay - dep_delay,
  speed = distance / air_time * 60
)

#adding new columns to your dataframe
mutate(flights_sml,
  gain = arr_delay - dep_delay,
  hours = air_time / 60,
  gain_per_hour = gain / hours
)

#If you only want to keep the new variables, use transmute():
transmute(flights,
  gain = arr_delay - dep_delay,
  hours = air_time / 60,
  gain_per_hour = gain / hours
)

2.5 summarise() :

The last key verb is summarise(). It collapses a data frame to a single row:

summarise(flights, delay = mean(dep_delay, na.rm = TRUE))

summarise() is not terribly useful unless we pair it with group_by(). This changes the unit of analysis from the complete dataset to individual groups. Then, when you use the dplyr verbs on a grouped data frame they’ll be automatically applied “by group”. For example, if we applied exactly the same code to a data frame grouped by date, we get the average delay per date:

by_day <- group_by(flights, year, month, day)
summarise(by_day, delay = mean(dep_delay, na.rm = TRUE))

this is not the only thing summarise() can do !!!! check ?summarise()

for an exhaustive documentation of dplyr,visit :

https://cran.rstudio.com/web/packages/dplyr/vignettes/introduction.html

https://cran.r-project.org/web/packages/dplyr/dplyr.pdf