R Markdown

This is an R Markdown document. Markdown is a simple formatting syntax for authoring HTML, PDF, and MS Word documents. For more details on using R Markdown see http://rmarkdown.rstudio.com.

When you click the Knit button a document will be generated that includes both content as well as the output of any embedded R code chunks within the document. You can embed an R code chunk like this:

summary(cars)
##      speed           dist       
##  Min.   : 4.0   Min.   :  2.00  
##  1st Qu.:12.0   1st Qu.: 26.00  
##  Median :15.0   Median : 36.00  
##  Mean   :15.4   Mean   : 42.98  
##  3rd Qu.:19.0   3rd Qu.: 56.00  
##  Max.   :25.0   Max.   :120.00

Including Plots

You can also embed plots, for example:

Note that the echo = FALSE parameter was added to the code chunk to prevent printing of the R code that generated the plot.

  1. Anscombes quartet is a set of 4 \(x,y\) data sets that were published by Francis Anscombe in a 1973 paper Graphs in statistical analysis. For this first question load the anscombe data that is part of the library(datasets) in R. And assign that data to a new object called data.
data<-anscombe
x1<-data[,1]
x2<-data[,2]
x3<-data[,3]
x4<-data[,4]
y1<-data[,5]
y2<-data[,6]
y3<-data[,7]
y4<-data[,8]
  1. Summarise the data by calculating the mean, variance, for each column and the correlation between each pair (eg. x1 and y1, x2 and y2, etc) (Hint: use the fBasics() package!)
library("fBasics")
## Loading required package: timeDate
## Warning in as.POSIXlt.POSIXct(Sys.time()): unknown timezone 'default/
## America/New_York'
## Loading required package: timeSeries
## 
## Rmetrics Package fBasics
## Analysing Markets and calculating Basic Statistics
## Copyright (C) 2005-2014 Rmetrics Association Zurich
## Educational Software for Financial Engineering and Computational Science
## Rmetrics is free software and comes with ABSOLUTELY NO WARRANTY.
## https://www.rmetrics.org --- Mail to: info@rmetrics.org
mean(x1)
## [1] 9
var(x1)
## [1] 11
mean(x2)
## [1] 9
var(x2)
## [1] 11
mean(x3)
## [1] 9
var(x3)
## [1] 11
mean(x4)
## [1] 9
var(x4)
## [1] 11
mean(y1)
## [1] 7.500909
var(y1)
## [1] 4.127269
mean(y2)
## [1] 7.500909
var(y2)
## [1] 4.127629
mean(y3)
## [1] 7.5
var(y3)
## [1] 4.12262
mean(y4)
## [1] 7.500909
var(y4)
## [1] 4.123249
correlationTest(x1,y1)
## 
## Title:
##  Pearson's Correlation Test
## 
## Test Results:
##   PARAMETER:
##     Degrees of Freedom: 9
##   SAMPLE ESTIMATES:
##     Correlation: 0.8164
##   STATISTIC:
##     t: 4.2415
##   P VALUE:
##     Alternative Two-Sided: 0.00217 
##     Alternative      Less: 0.9989 
##     Alternative   Greater: 0.001085 
##   CONFIDENCE INTERVAL:
##     Two-Sided: 0.4244, 0.9507
##          Less: -1, 0.9388
##       Greater: 0.5113, 1
## 
## Description:
##  Mon Nov 13 01:40:37 2017
correlationTest(x2,y2)
## 
## Title:
##  Pearson's Correlation Test
## 
## Test Results:
##   PARAMETER:
##     Degrees of Freedom: 9
##   SAMPLE ESTIMATES:
##     Correlation: 0.8162
##   STATISTIC:
##     t: 4.2386
##   P VALUE:
##     Alternative Two-Sided: 0.002179 
##     Alternative      Less: 0.9989 
##     Alternative   Greater: 0.001089 
##   CONFIDENCE INTERVAL:
##     Two-Sided: 0.4239, 0.9506
##          Less: -1, 0.9387
##       Greater: 0.5109, 1
## 
## Description:
##  Mon Nov 13 01:40:37 2017
correlationTest(x3,y3)
## 
## Title:
##  Pearson's Correlation Test
## 
## Test Results:
##   PARAMETER:
##     Degrees of Freedom: 9
##   SAMPLE ESTIMATES:
##     Correlation: 0.8163
##   STATISTIC:
##     t: 4.2394
##   P VALUE:
##     Alternative Two-Sided: 0.002176 
##     Alternative      Less: 0.9989 
##     Alternative   Greater: 0.001088 
##   CONFIDENCE INTERVAL:
##     Two-Sided: 0.4241, 0.9507
##          Less: -1, 0.9387
##       Greater: 0.511, 1
## 
## Description:
##  Mon Nov 13 01:40:37 2017
correlationTest(x4,y4)
## 
## Title:
##  Pearson's Correlation Test
## 
## Test Results:
##   PARAMETER:
##     Degrees of Freedom: 9
##   SAMPLE ESTIMATES:
##     Correlation: 0.8165
##   STATISTIC:
##     t: 4.243
##   P VALUE:
##     Alternative Two-Sided: 0.002165 
##     Alternative      Less: 0.9989 
##     Alternative   Greater: 0.001082 
##   CONFIDENCE INTERVAL:
##     Two-Sided: 0.4246, 0.9507
##          Less: -1, 0.9388
##       Greater: 0.5115, 1
## 
## Description:
##  Mon Nov 13 01:40:37 2017
  1. Create scatter plots for each \(x, y\) pair of data.
plot(x1, y1, main="Scatterplot between x1,y1")

plot(x2, y2, main="Scatterplot between x2,y2")

plot(x3, y3, main="Scatterplot between x3,y3")

plot(x4, y4, main="Scatterplot between x4,y4")

  1. Now change the symbols on the scatter plots to solid circles and plot them together as a 4 panel graphic
par(mfrow=c(2,2))
plot(x1,y1, main="Scatterplot between x1,y1",pch=19) 
plot(x2,y2, main="Scatterplot between x2,y2",pch=19) 
plot(x3,y3, main="Scatterplot between x3,y3",pch=19) 
plot(x4,y4, main="Scatterplot between x4,y4",pch=19)

  1. Now fit a linear model to each data set using the lm() function.
fit1<-lm(y1~x1)
fit2<-lm(y2~x2)
fit3<-lm(y3~x3)
fit4<-lm(y4~x4)
  1. Now combine the last two tasks. Create a four panel scatter plot matrix that has both the data points and the regression lines. (hint: the model objects will carry over chunks!)
par(mfrow=c(2,2))
plot(x1,y1, main="Scatterplot between x1,y1",pch=19) 
abline(fit1, col="red") # regression line (y~x)
plot(x2,y2, main="Scatterplot between x2,y2",pch=19) 
abline(fit2, col="red") # regression line (y~x)
plot(x3,y3, main="Scatterplot between x3,y3",pch=19) 
abline(fit3, col="red") # regression line (y~x)
plot(x4,y4, main="Scatterplot between x4,y4",pch=19)

  1. Now compare the model fits for each model object.
anova(fit1)
## Analysis of Variance Table
## 
## Response: y1
##           Df Sum Sq Mean Sq F value  Pr(>F)   
## x1         1 27.510 27.5100   17.99 0.00217 **
## Residuals  9 13.763  1.5292                   
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
anova(fit2)
## Analysis of Variance Table
## 
## Response: y2
##           Df Sum Sq Mean Sq F value   Pr(>F)   
## x2         1 27.500 27.5000  17.966 0.002179 **
## Residuals  9 13.776  1.5307                    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
anova(fit3)
## Analysis of Variance Table
## 
## Response: y3
##           Df Sum Sq Mean Sq F value   Pr(>F)   
## x3         1 27.470 27.4700  17.972 0.002176 **
## Residuals  9 13.756  1.5285                    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
anova(fit4)
## Analysis of Variance Table
## 
## Response: y4
##           Df Sum Sq Mean Sq F value   Pr(>F)   
## x4         1 27.490 27.4900  18.003 0.002165 **
## Residuals  9 13.742  1.5269                    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
  1. In text, summarize the lesson of Anscombe’s Quartet and what it says about the value of data visualization.

Anscombe’s quartet is a classic example of why summary statistics are not sufficient to analyze data and tell the whole story about the data. Anscombe’s Quartet consists of four data set of similar kind of data. Each data set consists of eleven pairs of values for x and y. The summary statistics are almost similar foe each column values and correlations.

The mean of x = 9 The Variance of x = 11 The mean of y = 7.5 The variance of y = 4.12 Correlation between x and y = .816 Linear regression equation - y = 3 + 0.5x

The four data set appear to be identical. But, when we plot the four data sets on x, y plane we find the data sets are not similar. Each data set tells a different story. Dataset1 consists of a set of follows a rough linear relationship. Dataset2 does not show linear relationship, rather y has a smooth curve relation with x and with little residual variability. Dataset III looks like a tight linear relationship between x and y, except for one large outlier. Dataset IV looks like x remains constant, except for one outlier as well. From Anscombe’s Quartet we can conclude that, summary statists may not tell us all the story about the data whereas the graphical representation or visualization of data can give us a clear picture about what’s going on with the data. abline(fit4, col=“red”) # regression line (y~x)