DATA 605 - Final Project
Don Padmaperuma
library(kableExtra)Problem 1
Using R, generate a random variable X that has 10,000 random uniform numbers from 1 to N, where N can be any number of your choosing greater than or equal to 6. Then generate a random variable Y that has 10,000 random normal numbers with a mean of (N+1)/2.
set.seed(123)
N <- 10
X <- runif(10000, min=0, max=N)
Y <- rnorm(10000, mean=(N+1)/2, sd=(N+1)/2)# mean and standard deviation is (N+1)/2
df <- data.frame(cbind(X, Y))
summary(X)## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 0.000653 2.528918 4.945676 4.975494 7.433941 9.999414hist(X)
summary(Y)## Min. 1st Qu. Median Mean 3rd Qu. Max.
## -15.649 1.841 5.475 5.511 9.330 26.663hist(Y)
### Probability Calculate as a minimum the below probabilities a through c. Assume the small letter “x” is estimated as the median of the X variable, and the small letter “y” is estimated as the 1st quartile of the Y variable. Interpret the meaning of all probabilities.
#set variables
#median of the X variable
x<-median(X)
x## [1] 4.945676y <- quantile(Y, 0.25)
y## 25%
## 1.840529a. P(X>x | X>y)
pXXy <- nrow(subset(df, X > x & Y > y))/10000
pXy <- nrow(subset(df, X > y))/10000
Prob_a <- (pXXy/pXy)
Prob_a## [1] 0.4606328b. P(X>x, Y>y)
Prob_b <- nrow(subset(df, X > x & Y > y))/10000
Prob_b## [1] 0.3756c. P(X<x | X>y)
pXXy2 <- nrow(subset(df, X < x & X > y))/10000
pXy2<- nrow(subset(df, X > y))/10000
Prob_c<-pXXy2/pXy2
Prob_c## [1] 0.386804Investigate
Investigate whether P(X>x and Y>y)=P(X>x)P(Y>y) by building a table and evaluating the marginal and joint probabilities.
matrix<-matrix( c(sum(X>x & Y<y),sum(X>x & Y>y), sum(X<x & Y<y),sum(X<x & Y>y)), nrow = 2,ncol = 2)
matrix<-cbind(matrix,c(matrix[1,1]+matrix[1,2],matrix[2,1]+matrix[2,2]))
matrix<-rbind(matrix,c(matrix[1,1]+matrix[2,1],matrix[1,2]+matrix[2,2],matrix[1,3]+matrix[2,3]))
contingency<-as.data.frame(matrix)
names(contingency) <- c("X>x","X<x", "Total")
row.names(contingency) <- c("Y<y","Y>y", "Total")
kable(contingency) %>%
kable_styling(bootstrap_options = "bordered")| X>x | X<x | Total | |
|---|---|---|---|
| Y<y | 1244 | 1256 | 2500 |
| Y>y | 3756 | 3744 | 7500 |
| Total | 5000 | 5000 | 10000 |
prob_matrix<-matrix/matrix[3,3]
contingency_p<-as.data.frame(prob_matrix)
names(contingency_p) <- c("X>x","X<x", "Total")
row.names(contingency_p) <- c("Y<y","Y>y", "Total")
kable(round(contingency_p,2)) %>%
kable_styling(bootstrap_options = "bordered")| X>x | X<x | Total | |
|---|---|---|---|
| Y<y | 0.12 | 0.13 | 0.25 |
| Y>y | 0.38 | 0.37 | 0.75 |
| Total | 0.50 | 0.50 | 1.00 |
Compute P(X>x)P(Y>y)
prob_matrix[3,1]*prob_matrix[2,3]## [1] 0.375Compute P(X>x and Y>y)
round(prob_matrix[2,1],digits = 3)## [1] 0.376Since the values are so similar/close we would conclude that X and Y are indeed independent.
Fisher’s Exact Test and the Chi Square Test
Check to see if independence holds by using Fisher’s Exact Test and the Chi Square Test. What is the difference between the two? Which is most appropriate?
Fisher’s Exact Test
fisher.test(matrix,simulate.p.value=TRUE)##
## Fisher's Exact Test for Count Data with simulated p-value (based
## on 2000 replicates)
##
## data: matrix
## p-value = 0.9985
## alternative hypothesis: two.sidedChi Square Test
chisq.test(matrix, correct = TRUE)##
## Pearson's Chi-squared test
##
## data: matrix
## X-squared = 0.0768, df = 4, p-value = 0.9993P values obtained from both test seems to greater than 0.05 making null hypothesis H0 acceptable.
Fisher’s exact test is practically applied only in analysis of small samples but actually it is valid for all sample sizes. While the chi-squared test relies on an approximation, Fisher’s exact test is one of exact tests.
As Fisher’s exact test is used in analysis of small samples, chi-squared test is appropriate in this case.
Problem 2
The House Prices: Advanced Regression Techniques competition. https://www.kaggle.com/c/house-prices-advanced-regression-techniques.
# Import training data
train <- read.csv('https://raw.githubusercontent.com/gpadmaperuma/DATA-605/master/Final%20Project/train.csv')
test <- read.csv('https://raw.githubusercontent.com/gpadmaperuma/DATA-605/master/Final%20Project/test.csv')Descriptive and Inferential Statistics
Summary of Training data
summary(train)## Id MSSubClass MSZoning LotFrontage
## Min. : 1.0 Min. : 20.0 C (all): 10 Min. : 21.00
## 1st Qu.: 365.8 1st Qu.: 20.0 FV : 65 1st Qu.: 59.00
## Median : 730.5 Median : 50.0 RH : 16 Median : 69.00
## Mean : 730.5 Mean : 56.9 RL :1151 Mean : 70.05
## 3rd Qu.:1095.2 3rd Qu.: 70.0 RM : 218 3rd Qu.: 80.00
## Max. :1460.0 Max. :190.0 Max. :313.00
## NA's :259
## LotArea Street Alley LotShape LandContour
## Min. : 1300 Grvl: 6 Grvl: 50 IR1:484 Bnk: 63
## 1st Qu.: 7554 Pave:1454 Pave: 41 IR2: 41 HLS: 50
## Median : 9478 NA's:1369 IR3: 10 Low: 36
## Mean : 10517 Reg:925 Lvl:1311
## 3rd Qu.: 11602
## Max. :215245
##
## Utilities LotConfig LandSlope Neighborhood Condition1
## AllPub:1459 Corner : 263 Gtl:1382 NAmes :225 Norm :1260
## NoSeWa: 1 CulDSac: 94 Mod: 65 CollgCr:150 Feedr : 81
## FR2 : 47 Sev: 13 OldTown:113 Artery : 48
## FR3 : 4 Edwards:100 RRAn : 26
## Inside :1052 Somerst: 86 PosN : 19
## Gilbert: 79 RRAe : 11
## (Other):707 (Other): 15
## Condition2 BldgType HouseStyle OverallQual
## Norm :1445 1Fam :1220 1Story :726 Min. : 1.000
## Feedr : 6 2fmCon: 31 2Story :445 1st Qu.: 5.000
## Artery : 2 Duplex: 52 1.5Fin :154 Median : 6.000
## PosN : 2 Twnhs : 43 SLvl : 65 Mean : 6.099
## RRNn : 2 TwnhsE: 114 SFoyer : 37 3rd Qu.: 7.000
## PosA : 1 1.5Unf : 14 Max. :10.000
## (Other): 2 (Other): 19
## OverallCond YearBuilt YearRemodAdd RoofStyle
## Min. :1.000 Min. :1872 Min. :1950 Flat : 13
## 1st Qu.:5.000 1st Qu.:1954 1st Qu.:1967 Gable :1141
## Median :5.000 Median :1973 Median :1994 Gambrel: 11
## Mean :5.575 Mean :1971 Mean :1985 Hip : 286
## 3rd Qu.:6.000 3rd Qu.:2000 3rd Qu.:2004 Mansard: 7
## Max. :9.000 Max. :2010 Max. :2010 Shed : 2
##
## RoofMatl Exterior1st Exterior2nd MasVnrType MasVnrArea
## CompShg:1434 VinylSd:515 VinylSd:504 BrkCmn : 15 Min. : 0.0
## Tar&Grv: 11 HdBoard:222 MetalSd:214 BrkFace:445 1st Qu.: 0.0
## WdShngl: 6 MetalSd:220 HdBoard:207 None :864 Median : 0.0
## WdShake: 5 Wd Sdng:206 Wd Sdng:197 Stone :128 Mean : 103.7
## ClyTile: 1 Plywood:108 Plywood:142 NA's : 8 3rd Qu.: 166.0
## Membran: 1 CemntBd: 61 CmentBd: 60 Max. :1600.0
## (Other): 2 (Other):128 (Other):136 NA's :8
## ExterQual ExterCond Foundation BsmtQual BsmtCond BsmtExposure
## Ex: 52 Ex: 3 BrkTil:146 Ex :121 Fa : 45 Av :221
## Fa: 14 Fa: 28 CBlock:634 Fa : 35 Gd : 65 Gd :134
## Gd:488 Gd: 146 PConc :647 Gd :618 Po : 2 Mn :114
## TA:906 Po: 1 Slab : 24 TA :649 TA :1311 No :953
## TA:1282 Stone : 6 NA's: 37 NA's: 37 NA's: 38
## Wood : 3
##
## BsmtFinType1 BsmtFinSF1 BsmtFinType2 BsmtFinSF2
## ALQ :220 Min. : 0.0 ALQ : 19 Min. : 0.00
## BLQ :148 1st Qu.: 0.0 BLQ : 33 1st Qu.: 0.00
## GLQ :418 Median : 383.5 GLQ : 14 Median : 0.00
## LwQ : 74 Mean : 443.6 LwQ : 46 Mean : 46.55
## Rec :133 3rd Qu.: 712.2 Rec : 54 3rd Qu.: 0.00
## Unf :430 Max. :5644.0 Unf :1256 Max. :1474.00
## NA's: 37 NA's: 38
## BsmtUnfSF TotalBsmtSF Heating HeatingQC CentralAir
## Min. : 0.0 Min. : 0.0 Floor: 1 Ex:741 N: 95
## 1st Qu.: 223.0 1st Qu.: 795.8 GasA :1428 Fa: 49 Y:1365
## Median : 477.5 Median : 991.5 GasW : 18 Gd:241
## Mean : 567.2 Mean :1057.4 Grav : 7 Po: 1
## 3rd Qu.: 808.0 3rd Qu.:1298.2 OthW : 2 TA:428
## Max. :2336.0 Max. :6110.0 Wall : 4
##
## Electrical X1stFlrSF X2ndFlrSF LowQualFinSF
## FuseA: 94 Min. : 334 Min. : 0 Min. : 0.000
## FuseF: 27 1st Qu.: 882 1st Qu.: 0 1st Qu.: 0.000
## FuseP: 3 Median :1087 Median : 0 Median : 0.000
## Mix : 1 Mean :1163 Mean : 347 Mean : 5.845
## SBrkr:1334 3rd Qu.:1391 3rd Qu.: 728 3rd Qu.: 0.000
## NA's : 1 Max. :4692 Max. :2065 Max. :572.000
##
## GrLivArea BsmtFullBath BsmtHalfBath FullBath
## Min. : 334 Min. :0.0000 Min. :0.00000 Min. :0.000
## 1st Qu.:1130 1st Qu.:0.0000 1st Qu.:0.00000 1st Qu.:1.000
## Median :1464 Median :0.0000 Median :0.00000 Median :2.000
## Mean :1515 Mean :0.4253 Mean :0.05753 Mean :1.565
## 3rd Qu.:1777 3rd Qu.:1.0000 3rd Qu.:0.00000 3rd Qu.:2.000
## Max. :5642 Max. :3.0000 Max. :2.00000 Max. :3.000
##
## HalfBath BedroomAbvGr KitchenAbvGr KitchenQual
## Min. :0.0000 Min. :0.000 Min. :0.000 Ex:100
## 1st Qu.:0.0000 1st Qu.:2.000 1st Qu.:1.000 Fa: 39
## Median :0.0000 Median :3.000 Median :1.000 Gd:586
## Mean :0.3829 Mean :2.866 Mean :1.047 TA:735
## 3rd Qu.:1.0000 3rd Qu.:3.000 3rd Qu.:1.000
## Max. :2.0000 Max. :8.000 Max. :3.000
##
## TotRmsAbvGrd Functional Fireplaces FireplaceQu GarageType
## Min. : 2.000 Maj1: 14 Min. :0.000 Ex : 24 2Types : 6
## 1st Qu.: 5.000 Maj2: 5 1st Qu.:0.000 Fa : 33 Attchd :870
## Median : 6.000 Min1: 31 Median :1.000 Gd :380 Basment: 19
## Mean : 6.518 Min2: 34 Mean :0.613 Po : 20 BuiltIn: 88
## 3rd Qu.: 7.000 Mod : 15 3rd Qu.:1.000 TA :313 CarPort: 9
## Max. :14.000 Sev : 1 Max. :3.000 NA's:690 Detchd :387
## Typ :1360 NA's : 81
## GarageYrBlt GarageFinish GarageCars GarageArea GarageQual
## Min. :1900 Fin :352 Min. :0.000 Min. : 0.0 Ex : 3
## 1st Qu.:1961 RFn :422 1st Qu.:1.000 1st Qu.: 334.5 Fa : 48
## Median :1980 Unf :605 Median :2.000 Median : 480.0 Gd : 14
## Mean :1979 NA's: 81 Mean :1.767 Mean : 473.0 Po : 3
## 3rd Qu.:2002 3rd Qu.:2.000 3rd Qu.: 576.0 TA :1311
## Max. :2010 Max. :4.000 Max. :1418.0 NA's: 81
## NA's :81
## GarageCond PavedDrive WoodDeckSF OpenPorchSF EnclosedPorch
## Ex : 2 N: 90 Min. : 0.00 Min. : 0.00 Min. : 0.00
## Fa : 35 P: 30 1st Qu.: 0.00 1st Qu.: 0.00 1st Qu.: 0.00
## Gd : 9 Y:1340 Median : 0.00 Median : 25.00 Median : 0.00
## Po : 7 Mean : 94.24 Mean : 46.66 Mean : 21.95
## TA :1326 3rd Qu.:168.00 3rd Qu.: 68.00 3rd Qu.: 0.00
## NA's: 81 Max. :857.00 Max. :547.00 Max. :552.00
##
## X3SsnPorch ScreenPorch PoolArea PoolQC
## Min. : 0.00 Min. : 0.00 Min. : 0.000 Ex : 2
## 1st Qu.: 0.00 1st Qu.: 0.00 1st Qu.: 0.000 Fa : 2
## Median : 0.00 Median : 0.00 Median : 0.000 Gd : 3
## Mean : 3.41 Mean : 15.06 Mean : 2.759 NA's:1453
## 3rd Qu.: 0.00 3rd Qu.: 0.00 3rd Qu.: 0.000
## Max. :508.00 Max. :480.00 Max. :738.000
##
## Fence MiscFeature MiscVal MoSold
## GdPrv: 59 Gar2: 2 Min. : 0.00 Min. : 1.000
## GdWo : 54 Othr: 2 1st Qu.: 0.00 1st Qu.: 5.000
## MnPrv: 157 Shed: 49 Median : 0.00 Median : 6.000
## MnWw : 11 TenC: 1 Mean : 43.49 Mean : 6.322
## NA's :1179 NA's:1406 3rd Qu.: 0.00 3rd Qu.: 8.000
## Max. :15500.00 Max. :12.000
##
## YrSold SaleType SaleCondition SalePrice
## Min. :2006 WD :1267 Abnorml: 101 Min. : 34900
## 1st Qu.:2007 New : 122 AdjLand: 4 1st Qu.:129975
## Median :2008 COD : 43 Alloca : 12 Median :163000
## Mean :2008 ConLD : 9 Family : 20 Mean :180921
## 3rd Qu.:2009 ConLI : 5 Normal :1198 3rd Qu.:214000
## Max. :2010 ConLw : 5 Partial: 125 Max. :755000
## (Other): 9summary(test)## Id MSSubClass MSZoning LotFrontage
## Min. :1461 Min. : 20.00 C (all): 15 Min. : 21.00
## 1st Qu.:1826 1st Qu.: 20.00 FV : 74 1st Qu.: 58.00
## Median :2190 Median : 50.00 RH : 10 Median : 67.00
## Mean :2190 Mean : 57.38 RL :1114 Mean : 68.58
## 3rd Qu.:2554 3rd Qu.: 70.00 RM : 242 3rd Qu.: 80.00
## Max. :2919 Max. :190.00 NA's : 4 Max. :200.00
## NA's :227
## LotArea Street Alley LotShape LandContour
## Min. : 1470 Grvl: 6 Grvl: 70 IR1:484 Bnk: 54
## 1st Qu.: 7391 Pave:1453 Pave: 37 IR2: 35 HLS: 70
## Median : 9399 NA's:1352 IR3: 6 Low: 24
## Mean : 9819 Reg:934 Lvl:1311
## 3rd Qu.:11518
## Max. :56600
##
## Utilities LotConfig LandSlope Neighborhood Condition1
## AllPub:1457 Corner : 248 Gtl:1396 NAmes :218 Norm :1251
## NA's : 2 CulDSac: 82 Mod: 60 OldTown:126 Feedr : 83
## FR2 : 38 Sev: 3 CollgCr:117 Artery : 44
## FR3 : 10 Somerst: 96 RRAn : 24
## Inside :1081 Edwards: 94 PosN : 20
## NridgHt: 89 RRAe : 17
## (Other):719 (Other): 20
## Condition2 BldgType HouseStyle OverallQual OverallCond
## Artery: 3 1Fam :1205 1.5Fin:160 Min. : 1.000 Min. :1.000
## Feedr : 7 2fmCon: 31 1.5Unf: 5 1st Qu.: 5.000 1st Qu.:5.000
## Norm :1444 Duplex: 57 1Story:745 Median : 6.000 Median :5.000
## PosA : 3 Twnhs : 53 2.5Unf: 13 Mean : 6.079 Mean :5.554
## PosN : 2 TwnhsE: 113 2Story:427 3rd Qu.: 7.000 3rd Qu.:6.000
## SFoyer: 46 Max. :10.000 Max. :9.000
## SLvl : 63
## YearBuilt YearRemodAdd RoofStyle RoofMatl Exterior1st
## Min. :1879 Min. :1950 Flat : 7 CompShg:1442 VinylSd:510
## 1st Qu.:1953 1st Qu.:1963 Gable :1169 Tar&Grv: 12 MetalSd:230
## Median :1973 Median :1992 Gambrel: 11 WdShake: 4 HdBoard:220
## Mean :1971 Mean :1984 Hip : 265 WdShngl: 1 Wd Sdng:205
## 3rd Qu.:2001 3rd Qu.:2004 Mansard: 4 Plywood:113
## Max. :2010 Max. :2010 Shed : 3 (Other):180
## NA's : 1
## Exterior2nd MasVnrType MasVnrArea ExterQual ExterCond
## VinylSd:510 BrkCmn : 10 Min. : 0.0 Ex: 55 Ex: 9
## MetalSd:233 BrkFace:434 1st Qu.: 0.0 Fa: 21 Fa: 39
## HdBoard:199 None :878 Median : 0.0 Gd:491 Gd: 153
## Wd Sdng:194 Stone :121 Mean : 100.7 TA:892 Po: 2
## Plywood:128 NA's : 16 3rd Qu.: 164.0 TA:1256
## (Other):194 Max. :1290.0
## NA's : 1 NA's :15
## Foundation BsmtQual BsmtCond BsmtExposure BsmtFinType1
## BrkTil:165 Ex :137 Fa : 59 Av :197 ALQ :209
## CBlock:601 Fa : 53 Gd : 57 Gd :142 BLQ :121
## PConc :661 Gd :591 Po : 3 Mn :125 GLQ :431
## Slab : 25 TA :634 TA :1295 No :951 LwQ : 80
## Stone : 5 NA's: 44 NA's: 45 NA's: 44 Rec :155
## Wood : 2 Unf :421
## NA's: 42
## BsmtFinSF1 BsmtFinType2 BsmtFinSF2 BsmtUnfSF
## Min. : 0.0 ALQ : 33 Min. : 0.00 Min. : 0.0
## 1st Qu.: 0.0 BLQ : 35 1st Qu.: 0.00 1st Qu.: 219.2
## Median : 350.5 GLQ : 20 Median : 0.00 Median : 460.0
## Mean : 439.2 LwQ : 41 Mean : 52.62 Mean : 554.3
## 3rd Qu.: 753.5 Rec : 51 3rd Qu.: 0.00 3rd Qu.: 797.8
## Max. :4010.0 Unf :1237 Max. :1526.00 Max. :2140.0
## NA's :1 NA's: 42 NA's :1 NA's :1
## TotalBsmtSF Heating HeatingQC CentralAir Electrical
## Min. : 0 GasA:1446 Ex:752 N: 101 FuseA: 94
## 1st Qu.: 784 GasW: 9 Fa: 43 Y:1358 FuseF: 23
## Median : 988 Grav: 2 Gd:233 FuseP: 5
## Mean :1046 Wall: 2 Po: 2 SBrkr:1337
## 3rd Qu.:1305 TA:429
## Max. :5095
## NA's :1
## X1stFlrSF X2ndFlrSF LowQualFinSF GrLivArea
## Min. : 407.0 Min. : 0 Min. : 0.000 Min. : 407
## 1st Qu.: 873.5 1st Qu.: 0 1st Qu.: 0.000 1st Qu.:1118
## Median :1079.0 Median : 0 Median : 0.000 Median :1432
## Mean :1156.5 Mean : 326 Mean : 3.543 Mean :1486
## 3rd Qu.:1382.5 3rd Qu.: 676 3rd Qu.: 0.000 3rd Qu.:1721
## Max. :5095.0 Max. :1862 Max. :1064.000 Max. :5095
##
## BsmtFullBath BsmtHalfBath FullBath HalfBath
## Min. :0.0000 Min. :0.0000 Min. :0.000 Min. :0.0000
## 1st Qu.:0.0000 1st Qu.:0.0000 1st Qu.:1.000 1st Qu.:0.0000
## Median :0.0000 Median :0.0000 Median :2.000 Median :0.0000
## Mean :0.4345 Mean :0.0652 Mean :1.571 Mean :0.3777
## 3rd Qu.:1.0000 3rd Qu.:0.0000 3rd Qu.:2.000 3rd Qu.:1.0000
## Max. :3.0000 Max. :2.0000 Max. :4.000 Max. :2.0000
## NA's :2 NA's :2
## BedroomAbvGr KitchenAbvGr KitchenQual TotRmsAbvGrd
## Min. :0.000 Min. :0.000 Ex :105 Min. : 3.000
## 1st Qu.:2.000 1st Qu.:1.000 Fa : 31 1st Qu.: 5.000
## Median :3.000 Median :1.000 Gd :565 Median : 6.000
## Mean :2.854 Mean :1.042 TA :757 Mean : 6.385
## 3rd Qu.:3.000 3rd Qu.:1.000 NA's: 1 3rd Qu.: 7.000
## Max. :6.000 Max. :2.000 Max. :15.000
##
## Functional Fireplaces FireplaceQu GarageType GarageYrBlt
## Typ :1357 Min. :0.0000 Ex : 19 2Types : 17 Min. :1895
## Min2 : 36 1st Qu.:0.0000 Fa : 41 Attchd :853 1st Qu.:1959
## Min1 : 34 Median :0.0000 Gd :364 Basment: 17 Median :1979
## Mod : 20 Mean :0.5812 Po : 26 BuiltIn: 98 Mean :1978
## Maj1 : 5 3rd Qu.:1.0000 TA :279 CarPort: 6 3rd Qu.:2002
## (Other): 5 Max. :4.0000 NA's:730 Detchd :392 Max. :2207
## NA's : 2 NA's : 76 NA's :78
## GarageFinish GarageCars GarageArea GarageQual GarageCond
## Fin :367 Min. :0.000 Min. : 0.0 Fa : 76 Ex : 1
## RFn :389 1st Qu.:1.000 1st Qu.: 318.0 Gd : 10 Fa : 39
## Unf :625 Median :2.000 Median : 480.0 Po : 2 Gd : 6
## NA's: 78 Mean :1.766 Mean : 472.8 TA :1293 Po : 7
## 3rd Qu.:2.000 3rd Qu.: 576.0 NA's: 78 TA :1328
## Max. :5.000 Max. :1488.0 NA's: 78
## NA's :1 NA's :1
## PavedDrive WoodDeckSF OpenPorchSF EnclosedPorch
## N: 126 Min. : 0.00 Min. : 0.00 Min. : 0.00
## P: 32 1st Qu.: 0.00 1st Qu.: 0.00 1st Qu.: 0.00
## Y:1301 Median : 0.00 Median : 28.00 Median : 0.00
## Mean : 93.17 Mean : 48.31 Mean : 24.24
## 3rd Qu.: 168.00 3rd Qu.: 72.00 3rd Qu.: 0.00
## Max. :1424.00 Max. :742.00 Max. :1012.00
##
## X3SsnPorch ScreenPorch PoolArea PoolQC
## Min. : 0.000 Min. : 0.00 Min. : 0.000 Ex : 2
## 1st Qu.: 0.000 1st Qu.: 0.00 1st Qu.: 0.000 Gd : 1
## Median : 0.000 Median : 0.00 Median : 0.000 NA's:1456
## Mean : 1.794 Mean : 17.06 Mean : 1.744
## 3rd Qu.: 0.000 3rd Qu.: 0.00 3rd Qu.: 0.000
## Max. :360.000 Max. :576.00 Max. :800.000
##
## Fence MiscFeature MiscVal MoSold
## GdPrv: 59 Gar2: 3 Min. : 0.00 Min. : 1.000
## GdWo : 58 Othr: 2 1st Qu.: 0.00 1st Qu.: 4.000
## MnPrv: 172 Shed: 46 Median : 0.00 Median : 6.000
## MnWw : 1 NA's:1408 Mean : 58.17 Mean : 6.104
## NA's :1169 3rd Qu.: 0.00 3rd Qu.: 8.000
## Max. :17000.00 Max. :12.000
##
## YrSold SaleType SaleCondition
## Min. :2006 WD :1258 Abnorml: 89
## 1st Qu.:2007 New : 117 AdjLand: 8
## Median :2008 COD : 44 Alloca : 12
## Mean :2008 ConLD : 17 Family : 26
## 3rd Qu.:2009 CWD : 8 Normal :1204
## Max. :2010 (Other): 14 Partial: 120
## NA's : 1Univariate descriptive statistics
Provide univariate descriptive statistics and appropriate plots for the training data set
#Summary
summary(train$SalePrice)## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 34900 129975 163000 180921 214000 755000#Histogram
hist(train$SalePrice, main="Sale Price")
# QQ Plot
qqnorm(train$SalePrice)
qqline(train$SalePrice)
Scatter plots
rovide a scatterplot matrix for at least two of the independent variables and the dependent variable
#ScatterPlot
pairs(~SalePrice+LotArea+GrLivArea++GarageArea,data=train, main="Scatterplot Matrix")
#### Correlation Matrix
Derive a correlation matrix for any three quantitative variables in the dataset. Test the hypotheses that the correlations between each pairwise set of variables is 0 and provide an 80% confidence interval. Discuss the meaning of your analysis. Would you be worried about familywise error? Why or why not?
#Subsetting data
sub_df <- data.frame(train$LotArea,train$GrLivArea,train$GarageArea)
#Correlation
cormatrix <- cor(sub_df)
cormatrix## train.LotArea train.GrLivArea train.GarageArea
## train.LotArea 1.0000000 0.2631162 0.1804028
## train.GrLivArea 0.2631162 1.0000000 0.4689975
## train.GarageArea 0.1804028 0.4689975 1.0000000library(corrplot)## Warning: package 'corrplot' was built under R version 3.6.3## corrplot 0.84 loadedcorrplot(cormatrix, method="square")
#### Hypothesis Test
#GrLivArea & LotArea
cor.test(train$LotArea,train$GrLivArea,method = "pearson",conf.level = 0.80)##
## Pearson's product-moment correlation
##
## data: train$LotArea and train$GrLivArea
## t = 10.414, df = 1458, p-value < 2.2e-16
## alternative hypothesis: true correlation is not equal to 0
## 80 percent confidence interval:
## 0.2315997 0.2940809
## sample estimates:
## cor
## 0.2631162#GarageArea & LotArea
cor.test(train$LotArea,train$GarageArea,method = "pearson",conf.level = 0.80)##
## Pearson's product-moment correlation
##
## data: train$LotArea and train$GarageArea
## t = 7.0034, df = 1458, p-value = 3.803e-12
## alternative hypothesis: true correlation is not equal to 0
## 80 percent confidence interval:
## 0.1477356 0.2126767
## sample estimates:
## cor
## 0.1804028#GarageArea & GrLivArea
cor.test(train$GarageArea,train$GrLivArea,method = "pearson",conf.level = 0.80)##
## Pearson's product-moment correlation
##
## data: train$GarageArea and train$GrLivArea
## t = 20.276, df = 1458, p-value < 2.2e-16
## alternative hypothesis: true correlation is not equal to 0
## 80 percent confidence interval:
## 0.4423993 0.4947713
## sample estimates:
## cor
## 0.4689975With all three p-values at less than 0.05, we can reject the null hypothesis. We are confident that the correlation between the three variables are not zeroes. It is safe to say that we are 80% confident that the correlation between GrLivArea & LotArea is between 0.2315997 & 0.2940809, GarageArea & LotArea is between 0.1477356 & 0.2126767 and GarageArea & GrLivArea is between 0.1477356 & 0.2126767.
Linear Algebra and Correlation
Invert your correlation matrix from above. (This is known as the precision matrix and contains variance inflation factors on the diagonal.) Multiply the correlation matrix by the precision matrix, and then multiply the precision matrix by the correlation matrix. Conduct LU decomposition on the matrix.
Invert your correlation matrix from above.
precision_matrix<-solve(cormatrix)
round(precision_matrix,4)## train.LotArea train.GrLivArea train.GarageArea
## train.LotArea 1.0792 -0.2470 -0.0789
## train.GrLivArea -0.2470 1.3385 -0.5832
## train.GarageArea -0.0789 -0.5832 1.2877Multiply the matrix
#Correlation matrix by Precision matrix
corr_by_prec <- cormatrix%*%precision_matrix
round(corr_by_prec, 4)## train.LotArea train.GrLivArea train.GarageArea
## train.LotArea 1 0 0
## train.GrLivArea 0 1 0
## train.GarageArea 0 0 1#precision matrix by the correlation matrix
prec_by_corr <- precision_matrix%*%cormatrix
round(prec_by_corr, 4)## train.LotArea train.GrLivArea train.GarageArea
## train.LotArea 1 0 0
## train.GrLivArea 0 1 0
## train.GarageArea 0 0 1Conduct LU Decomposition
library(matrixcalc)
lu.decomposition(precision_matrix)## $L
## [,1] [,2] [,3]
## [1,] 1.00000000 0.0000000 0
## [2,] -0.22884393 1.0000000 0
## [3,] -0.07307553 -0.4689975 1
##
## $U
## [,1] [,2] [,3]
## [1,] 1.079209 -0.2469705 -0.07886378
## [2,] 0.000000 1.2819833 -0.60124693
## [3,] 0.000000 0.0000000 1.00000000Calculus-Based Probability & Statistics
Select a variable in the Kaggle.com training dataset that is skewed to the right, shift it so that the minimum value is absolutely above zero if necessary. Then load the MASS package and run fitdistr to fit an exponential probability density function.
mass_fit <- train$TotalBsmtSF
min(mass_fit)## [1] 0Fit an exponential probability density function
library(MASS)fit <- fitdistr(mass_fit, "exponential")
fit## rate
## 9.456896e-04
## (2.474983e-05)Find the optimal value of \(\lambda\) for this distribution, and then take 1000 samples from this exponential distribution using this value (e.g., rexp(1000,\(\lambda\))).
lambda<-fit$estimate
sim<- rexp(1000,lambda)
lambda## rate
## 0.0009456896plot and compare
hist(sim,breaks = 100)
hist(mass_fit, breaks = 100)
library(ggplot2)
sim.df <- data.frame(length = sim)
mass_fit.df <- data.frame(length = mass_fit)
sim.df$from <- 'sim'
mass_fit.df$from <- 'Mass_Fit'
both.df <- rbind(sim.df,mass_fit.df)
ggplot(both.df, aes(length, fill = from)) + geom_density(alpha = 0.2)
cumulative distribution function (CDF)
Using the exponential pdf, find the 5th and 95th percentiles using the cumulative distribution function (CDF).
quantile(sim, probs=c(0.05, 0.95)) ## 5% 95%
## 55.16854 3200.41083Also generate a 95% confidence interval from the empirical data, assuming normality.
mean(mass_fit)## [1] 1057.429normal<-rnorm(length(mass_fit),mean(mass_fit),sd(mass_fit))
hist(normal)
provide the empirical 5th percentile and 95th percentile of the data.
quantile(normal, probs=c(0.05, 0.95))## 5% 95%
## 343.4975 1762.2931normal.df <- data.frame(length = normal)
normal.df$from <- 'normal'
all.df <- rbind(both.df,normal.df)
ggplot(all.df, aes(length, fill = from)) + geom_density(alpha = 0.2)
Modeling
Build some type of multiple regression model and submit your model to the competition board. Provide your complete model summary and results with analysis. Report your Kaggle.com user name and score.
image
me(Id = test$Id, SalePrice = y_hat) summary(su