Data Exploration Using R and Spark

I gonna use R and Spark for a sample data exploration task to explore the R capabilites for using spark distributed data processing

Setup

There is almost no installations needed for this tutorial but here is the requried tools for running this script in your machine.

• Java and i assumed it was already installed on your machine and why java and the answer is very simple ( SPARK written in Scala which is a JMV Java Virtual Machine language ) you can download java from here and this link will help you

• Spark lastest version 2.2 binaries and you can download it from here and follow the setup instruction if you don’t no worries i will do this step for you ;)

• R version 3

• R Studio to write this notebook

• ggplot2 for graphs

- SparkR is an R package that provides a light-weight frontend to use Apache Spark from R. In Spark 2.2.0, SparkR provides a distributed data frame implementation that supports operations like selection, filtering, aggregation etc. (similar to R data frames, dplyr) but on large datasets. SparkR also supports distributed machine learning using MLlib.

- The entry point into SparkR is the SparkSession which connects your R program to a Spark cluster. You can create a SparkSession using sparkR.session and pass in options such as the application name, any spark packages depended on.

Creating SparkDataFrames

With a SparkSession, applications can create SparkDataFrames from

• Local R data frame for example “df <- as.DataFrame(faithful)”.

• Hive table for example “results <- sql(‘FROM src SELECT key, value’)”.

• Other data sources and that is our case reading data from CSV file.

The following dataset represent the flights in 2015 for x airlines so my first step is to read the dataset i will depend on my analysis on Q/A style by asking some questions and trying to answer them by a simple graph

csvPath <- "data/flight-data/csv/2015-summary.csv"
df <- read.df(csvPath, "csv", header = "true", inferSchema = "true", na.strings = "NA")
head(df)

What is the top 10 country that people like to travel most?

new_df <- summarize(groupBy(df, df$DEST_COUNTRY_NAME), count = sum(df$count))
head(new_df)

plot_df <- as.data.frame(new_df) %>% dplyr::arrange(desc(count))
View(plot_df)
plot_df$DEST_COUNTRY_NAME <- factor(plot_df$DEST_COUNTRY_NAME)

p <- ggplot(plot_df[1:10,], aes(DEST_COUNTRY_NAME, count))
p + geom_point(aes(colour = DEST_COUNTRY_NAME, size = count))

What is the top 10 country that people hate to travel most?

plot_df$DEST_COUNTRY_NAME <- factor(plot_df$DEST_COUNTRY_NAME)
n <- nrow(plot_df)
p <- ggplot(plot_df[(n-10):n,], aes(DEST_COUNTRY_NAME, count))
p + geom_point(aes(colour = DEST_COUNTRY_NAME, size = count))

What is the top 10 country that people like to leave most?

new_df <- summarize(groupBy(df, df$ORIGIN_COUNTRY_NAME), count = sum(df$count))
head(new_df)

plot_df <- as.data.frame(new_df) %>% dplyr::arrange(desc(count))
View(plot_df)
plot_df$ORIGIN_COUNTRY_NAME <- factor(plot_df$ORIGIN_COUNTRY_NAME)

p <- ggplot(plot_df[1:10,], aes(ORIGIN_COUNTRY_NAME, count))
p + geom_point(aes(colour = ORIGIN_COUNTRY_NAME, size = count))

What is the top 10 country that people hate to leave most?

plot_df$ORIGIN_COUNTRY_NAME <- factor(plot_df$ORIGIN_COUNTRY_NAME)
n <- nrow(plot_df)
p <- ggplot(plot_df[(n-10):n,], aes(ORIGIN_COUNTRY_NAME, count))
p + geom_point(aes(colour = ORIGIN_COUNTRY_NAME, size = count))

What is the top 10 country that people like to travel to U.S ?

usa_flights_rdd <- filter(df, df$DEST_COUNTRY_NAME == "United States")

new_df <- summarize(groupBy(usa_flights_rdd, df$ORIGIN_COUNTRY_NAME), count = sum(df$count))
head(new_df)

plot_df <- as.data.frame(new_df) %>% dplyr::arrange(desc(count))
View(plot_df)
plot_df$ORIGIN_COUNTRY_NAME <- factor(plot_df$ORIGIN_COUNTRY_NAME)

p <- ggplot(plot_df[1:10,], aes(ORIGIN_COUNTRY_NAME, count))
p + geom_point(aes(colour = ORIGIN_COUNTRY_NAME, size = count))

What is the top 10 country that people hate to travel to U.S ?

plot_df$ORIGIN_COUNTRY_NAME <- factor(plot_df$ORIGIN_COUNTRY_NAME)
n <- nrow(plot_df)
p <- ggplot(plot_df[(n-10):n,], aes(ORIGIN_COUNTRY_NAME, count))
p + geom_point(aes(colour = ORIGIN_COUNTRY_NAME, size = count))

References

• SparkR Tutorials