Principles of Data Visualization and Introduction to ggplot2

library(tidyverse)
library(dplyr)
library(ggplot2)
library(scales)

I have provided you with data about the 5,000 fastest growing companies in the US, as compiled by Inc. magazine. lets read this in:

inc <- read.csv("https://raw.githubusercontent.com/charleyferrari/CUNY_DATA_608/master/module1/Data/inc5000_data.csv", header= TRUE)

And lets preview this data:

head(inc)
##   Rank                         Name Growth_Rate   Revenue
## 1    1                         Fuhu      421.48 1.179e+08
## 2    2        FederalConference.com      248.31 4.960e+07
## 3    3                The HCI Group      245.45 2.550e+07
## 4    4                      Bridger      233.08 1.900e+09
## 5    5                       DataXu      213.37 8.700e+07
## 6    6 MileStone Community Builders      179.38 4.570e+07
##                       Industry Employees         City State
## 1 Consumer Products & Services       104   El Segundo    CA
## 2          Government Services        51     Dumfries    VA
## 3                       Health       132 Jacksonville    FL
## 4                       Energy        50      Addison    TX
## 5      Advertising & Marketing       220       Boston    MA
## 6                  Real Estate        63       Austin    TX
summary(inc)
##       Rank                          Name       Growth_Rate     
##  Min.   :   1   (Add)ventures         :   1   Min.   :  0.340  
##  1st Qu.:1252   @Properties           :   1   1st Qu.:  0.770  
##  Median :2502   1-Stop Translation USA:   1   Median :  1.420  
##  Mean   :2502   110 Consulting        :   1   Mean   :  4.612  
##  3rd Qu.:3751   11thStreetCoffee.com  :   1   3rd Qu.:  3.290  
##  Max.   :5000   123 Exteriors         :   1   Max.   :421.480  
##                 (Other)               :4995                    
##     Revenue                                  Industry      Employees      
##  Min.   :2.000e+06   IT Services                 : 733   Min.   :    1.0  
##  1st Qu.:5.100e+06   Business Products & Services: 482   1st Qu.:   25.0  
##  Median :1.090e+07   Advertising & Marketing     : 471   Median :   53.0  
##  Mean   :4.822e+07   Health                      : 355   Mean   :  232.7  
##  3rd Qu.:2.860e+07   Software                    : 342   3rd Qu.:  132.0  
##  Max.   :1.010e+10   Financial Services          : 260   Max.   :66803.0  
##                      (Other)                     :2358   NA's   :12       
##             City          State     
##  New York     : 160   CA     : 701  
##  Chicago      :  90   TX     : 387  
##  Austin       :  88   NY     : 311  
##  Houston      :  76   VA     : 283  
##  San Francisco:  75   FL     : 282  
##  Atlanta      :  74   IL     : 273  
##  (Other)      :4438   (Other):2764

Think a bit on what these summaries mean. Use the space below to add some more relevant non-visual exploratory information you think helps you understand this data:

# overview of inc data frame
str(inc)
## 'data.frame':    5001 obs. of  8 variables:
##  $ Rank       : int  1 2 3 4 5 6 7 8 9 10 ...
##  $ Name       : Factor w/ 5001 levels "(Add)ventures",..: 1770 1633 4423 690 1198 2839 4733 1468 1869 4968 ...
##  $ Growth_Rate: num  421 248 245 233 213 ...
##  $ Revenue    : num  1.18e+08 4.96e+07 2.55e+07 1.90e+09 8.70e+07 ...
##  $ Industry   : Factor w/ 25 levels "Advertising & Marketing",..: 5 12 13 7 1 20 10 1 5 21 ...
##  $ Employees  : int  104 51 132 50 220 63 27 75 97 15 ...
##  $ City       : Factor w/ 1519 levels "Acton","Addison",..: 391 365 635 2 139 66 912 1179 131 1418 ...
##  $ State      : Factor w/ 52 levels "AK","AL","AR",..: 5 47 10 45 20 45 44 5 46 41 ...

Let’s check on some properties:

#Another important is understand where missing values are located since they might affect or skew our visualizations
colSums(is.na(inc))
##        Rank        Name Growth_Rate     Revenue    Industry   Employees 
##           0           0           0           0           0          12 
##        City       State 
##           0           0
sum(is.na(inc$Employees))
## [1] 12
#check on Standard deviation to see how the values are far from the mean
sd(inc$Growth_Rate)
## [1] 14.12369
sd(inc$Revenue)
## [1] 240542281
sd(inc$Employees, na.rm = TRUE)#A few companies have missing employee counts
## [1] 1353.128
#We can also do IQR in case the data is skewed
IQR(inc$Growth_Rate)
## [1] 2.52
IQR(inc$Revenue)
## [1] 23500000
IQR(inc$Employees, na.rm = TRUE)
## [1] 107

According to statistics class, we can create a correlation chart that can show if the variables in our data are related to one another. In this case, it does not seem that they have a simple linear relationship. There is a potential relationship between Employees and Revenue, as dipicted below in the chart produced by the following code:

library(corrgram)
## 
## Attaching package: 'corrgram'
## The following object is masked from 'package:lattice':
## 
##     panel.fill
corrgram(inc, order=TRUE, lower.panel=panel.ellipse,
  upper.panel=panel.pts, text.panel=panel.txt,
  diag.panel=panel.minmax)

Question 1

Create a graph that shows the distribution of companies in the dataset by State (ie how many are in each state). There are a lot of States, so consider which axis you should use. This visualization is ultimately going to be consumed on a ‘portrait’ oriented screen (ie taller than wide), which should further guide your layout choices.

# raw data summary
d1 <- inc %>% 
    group_by(State) %>% 
    tally() %>%
  rename(count = n) %>%
  arrange(desc(count))
#plot our data
g <- ggplot(d1, aes(x = reorder(State, count), y = count)) 
g <- g + geom_bar(stat = "identity", fill = 'darkred') + coord_flip() 
g <- g + ggtitle("Distribution of 5,000 Fastest Growing Companies")
g <- g + labs(subtitle = "by state", x = "state", y = "company count")
g <- g + theme(plot.title = element_text(hjust = 0.5), plot.subtitle = element_text(hjust = 0.5))
g

# save to file
ggsave("Figure1.png", height = 9, width = 7)

Quesiton 2

Lets dig in on the state with the 3rd most companies in the data set. Imagine you work for the state and are interested in how many people are employed by companies in different industries. Create a plot that shows the average and/or median employment by industry for companies in this state (only use cases with full data, use R’s complete.cases() function.) In addition to this, your graph should show how variable the ranges are, and you should deal with outliers.

# find state with third most companies (our data is already ordered in descendant, therefore the 3rd row)
state_3 <- d1[3,"State"]
state_3
## # A tibble: 1 x 1
##   State
##   <fct>
## 1 NY
# filter original data: complete cases and 3rd ranked state only   
data_3 <- inc[complete.cases(inc),] %>%
  inner_join(state_3, by = "State")

# find mean employees by industry
means <- aggregate(Employees ~ Industry, data_3, mean)

# find maximum average employee no.
means_max <- max(means$Employees)

# prepare plot data: box plots (with outliers removed) to show variation; dots for mean EEs
g <- ggplot(data_3, aes(x = reorder(Industry,Employees,mean), y = Employees))
g <- g + geom_boxplot(outlier.shape = NA, show.legend=F) + coord_flip()
g <- g + labs(x = "industry", y = "employees", title="Mean Employee Size by Industry")
g <- g + labs(subtitle = "with boxplots")
g <- g + geom_point(data = means, aes(x=Industry, y=Employees), color='darkred', size = 2)
g <- g + theme(plot.title = element_text(hjust = 0.5), plot.subtitle = element_text(hjust = 0.5))

# plot data, linear scale
g <- g +  scale_y_continuous(limits = c(0,means_max), breaks = seq(0, means_max, 200))
g

# save to file
ggsave("Figure2_part1.png", height = 9, width = 7)

According to the grapg above, we can see that the mean data are highly skewed, that is there is an outlier employee count in the Business Products & Services industry that is almost 2 times greater than the second ranked industry, Consumer Products & Services. i.e. more than 100%. Because of this significanct skew, we need to scale everything down and try to visualize the employee count data on a logarithmic scale, like so:

# plot data, log scale
g <- g + scale_y_log10(limits = c(1, means_max))
g <- g + labs(caption = "(logarithmic scale spacing)")
g <- g + theme(plot.caption = element_text(size = 8))
g

#save to file
ggsave("Figure2_part2.png", height = 9, width = 7)

Question 3

Now imagine you work for an investor and want to see which industries generate the most revenue per employee. Create a chart that makes this information clear. Once again, the distribution per industry should be shown.

d3 <- inc[complete.cases(inc$Revenue), ]
d3 <- d3[complete.cases(d3$Employees), ] 
d3 <- d3 %>%
    group_by(Industry) %>%
    summarise(TotRev = sum(Revenue, na.rm=T), TotEmp = sum(Employees, na.rm=T)) %>%
    mutate(AvgRevK = (TotRev/TotEmp)/1000)

ggplot(d3, aes(x=reorder(Industry, AvgRevK), y=AvgRevK)) + 
    geom_bar(stat="identity", width=.5, fill="darkred")+ 
    labs(title="Revenue Per Employee",
         subtitle="5000 Fastest Growing Companies by Industry", 
         caption="",
         y="Revenue/Employee - Thousands ($)", 
         x="Industry") + 
    theme_classic() +
    coord_flip()

ggsave('Figure3.png')
## Saving 7 x 5 in image