SVasquez_finalExam
Silverio J. Vasquez
May 20, 2018
R Markdown
This is an R Markdown document. Markdown is a simple formatting syntax for authoring HTML, PDF, and MS Word documents. For more details on using R Markdown see http://rmarkdown.rstudio.com.
When you click the Knit button a document will be generated that includes both content as well as the output of any embedded R code chunks within the document. You can embed an R code chunk like this:
train <- read.csv('train.csv',header=TRUE)
summary(train)## Id MSSubClass MSZoning LotFrontage
## Min. : 1.0 Min. : 20.0 C (all): 10 Min. : 21.00
## 1st Qu.: 365.8 1st Qu.: 20.0 FV : 65 1st Qu.: 59.00
## Median : 730.5 Median : 50.0 RH : 16 Median : 69.00
## Mean : 730.5 Mean : 56.9 RL :1151 Mean : 70.05
## 3rd Qu.:1095.2 3rd Qu.: 70.0 RM : 218 3rd Qu.: 80.00
## Max. :1460.0 Max. :190.0 Max. :313.00
## NA's :259
## LotArea Street Alley LotShape LandContour
## Min. : 1300 Grvl: 6 Grvl: 50 IR1:484 Bnk: 63
## 1st Qu.: 7554 Pave:1454 Pave: 41 IR2: 41 HLS: 50
## Median : 9478 NA's:1369 IR3: 10 Low: 36
## Mean : 10517 Reg:925 Lvl:1311
## 3rd Qu.: 11602
## Max. :215245
##
## Utilities LotConfig LandSlope Neighborhood Condition1
## AllPub:1459 Corner : 263 Gtl:1382 NAmes :225 Norm :1260
## NoSeWa: 1 CulDSac: 94 Mod: 65 CollgCr:150 Feedr : 81
## FR2 : 47 Sev: 13 OldTown:113 Artery : 48
## FR3 : 4 Edwards:100 RRAn : 26
## Inside :1052 Somerst: 86 PosN : 19
## Gilbert: 79 RRAe : 11
## (Other):707 (Other): 15
## Condition2 BldgType HouseStyle OverallQual
## Norm :1445 1Fam :1220 1Story :726 Min. : 1.000
## Feedr : 6 2fmCon: 31 2Story :445 1st Qu.: 5.000
## Artery : 2 Duplex: 52 1.5Fin :154 Median : 6.000
## PosN : 2 Twnhs : 43 SLvl : 65 Mean : 6.099
## RRNn : 2 TwnhsE: 114 SFoyer : 37 3rd Qu.: 7.000
## PosA : 1 1.5Unf : 14 Max. :10.000
## (Other): 2 (Other): 19
## OverallCond YearBuilt YearRemodAdd RoofStyle
## Min. :1.000 Min. :1872 Min. :1950 Flat : 13
## 1st Qu.:5.000 1st Qu.:1954 1st Qu.:1967 Gable :1141
## Median :5.000 Median :1973 Median :1994 Gambrel: 11
## Mean :5.575 Mean :1971 Mean :1985 Hip : 286
## 3rd Qu.:6.000 3rd Qu.:2000 3rd Qu.:2004 Mansard: 7
## Max. :9.000 Max. :2010 Max. :2010 Shed : 2
##
## RoofMatl Exterior1st Exterior2nd MasVnrType MasVnrArea
## CompShg:1434 VinylSd:515 VinylSd:504 BrkCmn : 15 Min. : 0.0
## Tar&Grv: 11 HdBoard:222 MetalSd:214 BrkFace:445 1st Qu.: 0.0
## WdShngl: 6 MetalSd:220 HdBoard:207 None :864 Median : 0.0
## WdShake: 5 Wd Sdng:206 Wd Sdng:197 Stone :128 Mean : 103.7
## ClyTile: 1 Plywood:108 Plywood:142 NA's : 8 3rd Qu.: 166.0
## Membran: 1 CemntBd: 61 CmentBd: 60 Max. :1600.0
## (Other): 2 (Other):128 (Other):136 NA's :8
## ExterQual ExterCond Foundation BsmtQual BsmtCond BsmtExposure
## Ex: 52 Ex: 3 BrkTil:146 Ex :121 Fa : 45 Av :221
## Fa: 14 Fa: 28 CBlock:634 Fa : 35 Gd : 65 Gd :134
## Gd:488 Gd: 146 PConc :647 Gd :618 Po : 2 Mn :114
## TA:906 Po: 1 Slab : 24 TA :649 TA :1311 No :953
## TA:1282 Stone : 6 NA's: 37 NA's: 37 NA's: 38
## Wood : 3
##
## BsmtFinType1 BsmtFinSF1 BsmtFinType2 BsmtFinSF2
## ALQ :220 Min. : 0.0 ALQ : 19 Min. : 0.00
## BLQ :148 1st Qu.: 0.0 BLQ : 33 1st Qu.: 0.00
## GLQ :418 Median : 383.5 GLQ : 14 Median : 0.00
## LwQ : 74 Mean : 443.6 LwQ : 46 Mean : 46.55
## Rec :133 3rd Qu.: 712.2 Rec : 54 3rd Qu.: 0.00
## Unf :430 Max. :5644.0 Unf :1256 Max. :1474.00
## NA's: 37 NA's: 38
## BsmtUnfSF TotalBsmtSF Heating HeatingQC CentralAir
## Min. : 0.0 Min. : 0.0 Floor: 1 Ex:741 N: 95
## 1st Qu.: 223.0 1st Qu.: 795.8 GasA :1428 Fa: 49 Y:1365
## Median : 477.5 Median : 991.5 GasW : 18 Gd:241
## Mean : 567.2 Mean :1057.4 Grav : 7 Po: 1
## 3rd Qu.: 808.0 3rd Qu.:1298.2 OthW : 2 TA:428
## Max. :2336.0 Max. :6110.0 Wall : 4
##
## Electrical X1stFlrSF X2ndFlrSF LowQualFinSF
## FuseA: 94 Min. : 334 Min. : 0 Min. : 0.000
## FuseF: 27 1st Qu.: 882 1st Qu.: 0 1st Qu.: 0.000
## FuseP: 3 Median :1087 Median : 0 Median : 0.000
## Mix : 1 Mean :1163 Mean : 347 Mean : 5.845
## SBrkr:1334 3rd Qu.:1391 3rd Qu.: 728 3rd Qu.: 0.000
## NA's : 1 Max. :4692 Max. :2065 Max. :572.000
##
## GrLivArea BsmtFullBath BsmtHalfBath FullBath
## Min. : 334 Min. :0.0000 Min. :0.00000 Min. :0.000
## 1st Qu.:1130 1st Qu.:0.0000 1st Qu.:0.00000 1st Qu.:1.000
## Median :1464 Median :0.0000 Median :0.00000 Median :2.000
## Mean :1515 Mean :0.4253 Mean :0.05753 Mean :1.565
## 3rd Qu.:1777 3rd Qu.:1.0000 3rd Qu.:0.00000 3rd Qu.:2.000
## Max. :5642 Max. :3.0000 Max. :2.00000 Max. :3.000
##
## HalfBath BedroomAbvGr KitchenAbvGr KitchenQual
## Min. :0.0000 Min. :0.000 Min. :0.000 Ex:100
## 1st Qu.:0.0000 1st Qu.:2.000 1st Qu.:1.000 Fa: 39
## Median :0.0000 Median :3.000 Median :1.000 Gd:586
## Mean :0.3829 Mean :2.866 Mean :1.047 TA:735
## 3rd Qu.:1.0000 3rd Qu.:3.000 3rd Qu.:1.000
## Max. :2.0000 Max. :8.000 Max. :3.000
##
## TotRmsAbvGrd Functional Fireplaces FireplaceQu GarageType
## Min. : 2.000 Maj1: 14 Min. :0.000 Ex : 24 2Types : 6
## 1st Qu.: 5.000 Maj2: 5 1st Qu.:0.000 Fa : 33 Attchd :870
## Median : 6.000 Min1: 31 Median :1.000 Gd :380 Basment: 19
## Mean : 6.518 Min2: 34 Mean :0.613 Po : 20 BuiltIn: 88
## 3rd Qu.: 7.000 Mod : 15 3rd Qu.:1.000 TA :313 CarPort: 9
## Max. :14.000 Sev : 1 Max. :3.000 NA's:690 Detchd :387
## Typ :1360 NA's : 81
## GarageYrBlt GarageFinish GarageCars GarageArea GarageQual
## Min. :1900 Fin :352 Min. :0.000 Min. : 0.0 Ex : 3
## 1st Qu.:1961 RFn :422 1st Qu.:1.000 1st Qu.: 334.5 Fa : 48
## Median :1980 Unf :605 Median :2.000 Median : 480.0 Gd : 14
## Mean :1979 NA's: 81 Mean :1.767 Mean : 473.0 Po : 3
## 3rd Qu.:2002 3rd Qu.:2.000 3rd Qu.: 576.0 TA :1311
## Max. :2010 Max. :4.000 Max. :1418.0 NA's: 81
## NA's :81
## GarageCond PavedDrive WoodDeckSF OpenPorchSF EnclosedPorch
## Ex : 2 N: 90 Min. : 0.00 Min. : 0.00 Min. : 0.00
## Fa : 35 P: 30 1st Qu.: 0.00 1st Qu.: 0.00 1st Qu.: 0.00
## Gd : 9 Y:1340 Median : 0.00 Median : 25.00 Median : 0.00
## Po : 7 Mean : 94.24 Mean : 46.66 Mean : 21.95
## TA :1326 3rd Qu.:168.00 3rd Qu.: 68.00 3rd Qu.: 0.00
## NA's: 81 Max. :857.00 Max. :547.00 Max. :552.00
##
## X3SsnPorch ScreenPorch PoolArea PoolQC
## Min. : 0.00 Min. : 0.00 Min. : 0.000 Ex : 2
## 1st Qu.: 0.00 1st Qu.: 0.00 1st Qu.: 0.000 Fa : 2
## Median : 0.00 Median : 0.00 Median : 0.000 Gd : 3
## Mean : 3.41 Mean : 15.06 Mean : 2.759 NA's:1453
## 3rd Qu.: 0.00 3rd Qu.: 0.00 3rd Qu.: 0.000
## Max. :508.00 Max. :480.00 Max. :738.000
##
## Fence MiscFeature MiscVal MoSold
## GdPrv: 59 Gar2: 2 Min. : 0.00 Min. : 1.000
## GdWo : 54 Othr: 2 1st Qu.: 0.00 1st Qu.: 5.000
## MnPrv: 157 Shed: 49 Median : 0.00 Median : 6.000
## MnWw : 11 TenC: 1 Mean : 43.49 Mean : 6.322
## NA's :1179 NA's:1406 3rd Qu.: 0.00 3rd Qu.: 8.000
## Max. :15500.00 Max. :12.000
##
## YrSold SaleType SaleCondition SalePrice
## Min. :2006 WD :1267 Abnorml: 101 Min. : 34900
## 1st Qu.:2007 New : 122 AdjLand: 4 1st Qu.:129975
## Median :2008 COD : 43 Alloca : 12 Median :163000
## Mean :2008 ConLD : 9 Family : 20 Mean :180921
## 3rd Qu.:2009 ConLI : 5 Normal :1198 3rd Qu.:214000
## Max. :2010 ConLw : 5 Partial: 125 Max. :755000
## (Other): 9x <- train$X1stFlrSF
skew(x)## [1] 1.373929summary(x)## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 334 882 1087 1163 1391 4692hist(x, probability = TRUE, main="Histogram of 1st Floor Square Footage")
y <- train$SalePrice
skew(y)## [1] 1.879009summary(y)## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 34900 129975 163000 180921 214000 755000hist(y, probability = TRUE, main="Histogram of Sales Price")
Probability
A is the conditional probability that 1st floor sq. footage is in the top quartile for all the observations where sales price is also in the top quartile, respectively.
B is the joint probability of both variables being their respective top quartile.
C is the conditional probability that 1st floor sq. footage is NOT in the top quartile for all the observations where sales price is in the top quartile, respectively.
pxgtx = quantile(x, 0.25)
pygty = quantile(y, 0.25)
dat <- cbind.data.frame(x,y)
a <- nrow(subset(dat, (x > pxgtx & y > pygty))) / nrow(subset(dat, y > pygty))
b <- nrow(subset(dat, (x > pxgtx & y > pygty))) / nrow(dat)
c <- nrow(subset(dat, (x < pxgtx & y > pygty))) / nrow(subset(dat, y > pygty))
print(paste0('P(X>x | Y>y)) = ', a))## [1] "P(X>x | Y>y)) = 0.827397260273973"print(paste0('P(X>x , Y>y)) = ', b))## [1] "P(X>x , Y>y)) = 0.620547945205479"print(paste0('P(X<x | Y>y)) = ', c))## [1] "P(X<x | Y>y)) = 0.170776255707763"Table of Counts
mat11 <-nrow(subset(dat, (x <= pxgtx & y <= pygty)))
mat12 <- nrow(subset(dat, (x <= pxgtx & y > pygty)))
mat1r <- mat11 + mat12
mat21 <- nrow(subset(dat, (x > pxgtx & y <= pygty)))
mat22 <- nrow(subset(dat, (x > pxgtx & y > pygty)))
mat2r <- mat21 + mat22
mat1c <- mat11 + mat21
mat2c <- mat12 + mat22
tot <- mat1c + mat2c
tot2 <- mat1r + mat2r
tot_check <- identical(tot,tot2)
tbl <- cbind.data.frame(c(mat11, mat21, mat1c), c(mat12, mat22, mat2c), c(mat1r, mat2r, tot2))
print(tbl)## c(mat11, mat21, mat1c) c(mat12, mat22, mat2c) c(mat1r, mat2r, tot2)
## 1 179 189 368
## 2 186 906 1092
## 3 365 1095 1460No, splitting the data in this manner doesn’t make them independent. The Chi-square test below tests this formally.
The Chi-square test’s p-value is significant, so we reject the null of independence.
prob_A <- 1092/1460
prob_B <- 1095/1460
prob_AB <- 906/1460
print('Does P(AB) = P(A) x P(B)?')## [1] "Does P(AB) = P(A) x P(B)?"print(identical(prob_A*prob_B, prob_AB))## [1] FALSEchisq.test(dat$x,dat$y)## Warning in chisq.test(dat$x, dat$y): Chi-squared approximation may be
## incorrect##
## Pearson's Chi-squared test
##
## data: dat$x and dat$y
## X-squared = 521930, df = 497820, p-value < 2.2e-16Descriptive and Inferential Statistics
summary(train)## Id MSSubClass MSZoning LotFrontage
## Min. : 1.0 Min. : 20.0 C (all): 10 Min. : 21.00
## 1st Qu.: 365.8 1st Qu.: 20.0 FV : 65 1st Qu.: 59.00
## Median : 730.5 Median : 50.0 RH : 16 Median : 69.00
## Mean : 730.5 Mean : 56.9 RL :1151 Mean : 70.05
## 3rd Qu.:1095.2 3rd Qu.: 70.0 RM : 218 3rd Qu.: 80.00
## Max. :1460.0 Max. :190.0 Max. :313.00
## NA's :259
## LotArea Street Alley LotShape LandContour
## Min. : 1300 Grvl: 6 Grvl: 50 IR1:484 Bnk: 63
## 1st Qu.: 7554 Pave:1454 Pave: 41 IR2: 41 HLS: 50
## Median : 9478 NA's:1369 IR3: 10 Low: 36
## Mean : 10517 Reg:925 Lvl:1311
## 3rd Qu.: 11602
## Max. :215245
##
## Utilities LotConfig LandSlope Neighborhood Condition1
## AllPub:1459 Corner : 263 Gtl:1382 NAmes :225 Norm :1260
## NoSeWa: 1 CulDSac: 94 Mod: 65 CollgCr:150 Feedr : 81
## FR2 : 47 Sev: 13 OldTown:113 Artery : 48
## FR3 : 4 Edwards:100 RRAn : 26
## Inside :1052 Somerst: 86 PosN : 19
## Gilbert: 79 RRAe : 11
## (Other):707 (Other): 15
## Condition2 BldgType HouseStyle OverallQual
## Norm :1445 1Fam :1220 1Story :726 Min. : 1.000
## Feedr : 6 2fmCon: 31 2Story :445 1st Qu.: 5.000
## Artery : 2 Duplex: 52 1.5Fin :154 Median : 6.000
## PosN : 2 Twnhs : 43 SLvl : 65 Mean : 6.099
## RRNn : 2 TwnhsE: 114 SFoyer : 37 3rd Qu.: 7.000
## PosA : 1 1.5Unf : 14 Max. :10.000
## (Other): 2 (Other): 19
## OverallCond YearBuilt YearRemodAdd RoofStyle
## Min. :1.000 Min. :1872 Min. :1950 Flat : 13
## 1st Qu.:5.000 1st Qu.:1954 1st Qu.:1967 Gable :1141
## Median :5.000 Median :1973 Median :1994 Gambrel: 11
## Mean :5.575 Mean :1971 Mean :1985 Hip : 286
## 3rd Qu.:6.000 3rd Qu.:2000 3rd Qu.:2004 Mansard: 7
## Max. :9.000 Max. :2010 Max. :2010 Shed : 2
##
## RoofMatl Exterior1st Exterior2nd MasVnrType MasVnrArea
## CompShg:1434 VinylSd:515 VinylSd:504 BrkCmn : 15 Min. : 0.0
## Tar&Grv: 11 HdBoard:222 MetalSd:214 BrkFace:445 1st Qu.: 0.0
## WdShngl: 6 MetalSd:220 HdBoard:207 None :864 Median : 0.0
## WdShake: 5 Wd Sdng:206 Wd Sdng:197 Stone :128 Mean : 103.7
## ClyTile: 1 Plywood:108 Plywood:142 NA's : 8 3rd Qu.: 166.0
## Membran: 1 CemntBd: 61 CmentBd: 60 Max. :1600.0
## (Other): 2 (Other):128 (Other):136 NA's :8
## ExterQual ExterCond Foundation BsmtQual BsmtCond BsmtExposure
## Ex: 52 Ex: 3 BrkTil:146 Ex :121 Fa : 45 Av :221
## Fa: 14 Fa: 28 CBlock:634 Fa : 35 Gd : 65 Gd :134
## Gd:488 Gd: 146 PConc :647 Gd :618 Po : 2 Mn :114
## TA:906 Po: 1 Slab : 24 TA :649 TA :1311 No :953
## TA:1282 Stone : 6 NA's: 37 NA's: 37 NA's: 38
## Wood : 3
##
## BsmtFinType1 BsmtFinSF1 BsmtFinType2 BsmtFinSF2
## ALQ :220 Min. : 0.0 ALQ : 19 Min. : 0.00
## BLQ :148 1st Qu.: 0.0 BLQ : 33 1st Qu.: 0.00
## GLQ :418 Median : 383.5 GLQ : 14 Median : 0.00
## LwQ : 74 Mean : 443.6 LwQ : 46 Mean : 46.55
## Rec :133 3rd Qu.: 712.2 Rec : 54 3rd Qu.: 0.00
## Unf :430 Max. :5644.0 Unf :1256 Max. :1474.00
## NA's: 37 NA's: 38
## BsmtUnfSF TotalBsmtSF Heating HeatingQC CentralAir
## Min. : 0.0 Min. : 0.0 Floor: 1 Ex:741 N: 95
## 1st Qu.: 223.0 1st Qu.: 795.8 GasA :1428 Fa: 49 Y:1365
## Median : 477.5 Median : 991.5 GasW : 18 Gd:241
## Mean : 567.2 Mean :1057.4 Grav : 7 Po: 1
## 3rd Qu.: 808.0 3rd Qu.:1298.2 OthW : 2 TA:428
## Max. :2336.0 Max. :6110.0 Wall : 4
##
## Electrical X1stFlrSF X2ndFlrSF LowQualFinSF
## FuseA: 94 Min. : 334 Min. : 0 Min. : 0.000
## FuseF: 27 1st Qu.: 882 1st Qu.: 0 1st Qu.: 0.000
## FuseP: 3 Median :1087 Median : 0 Median : 0.000
## Mix : 1 Mean :1163 Mean : 347 Mean : 5.845
## SBrkr:1334 3rd Qu.:1391 3rd Qu.: 728 3rd Qu.: 0.000
## NA's : 1 Max. :4692 Max. :2065 Max. :572.000
##
## GrLivArea BsmtFullBath BsmtHalfBath FullBath
## Min. : 334 Min. :0.0000 Min. :0.00000 Min. :0.000
## 1st Qu.:1130 1st Qu.:0.0000 1st Qu.:0.00000 1st Qu.:1.000
## Median :1464 Median :0.0000 Median :0.00000 Median :2.000
## Mean :1515 Mean :0.4253 Mean :0.05753 Mean :1.565
## 3rd Qu.:1777 3rd Qu.:1.0000 3rd Qu.:0.00000 3rd Qu.:2.000
## Max. :5642 Max. :3.0000 Max. :2.00000 Max. :3.000
##
## HalfBath BedroomAbvGr KitchenAbvGr KitchenQual
## Min. :0.0000 Min. :0.000 Min. :0.000 Ex:100
## 1st Qu.:0.0000 1st Qu.:2.000 1st Qu.:1.000 Fa: 39
## Median :0.0000 Median :3.000 Median :1.000 Gd:586
## Mean :0.3829 Mean :2.866 Mean :1.047 TA:735
## 3rd Qu.:1.0000 3rd Qu.:3.000 3rd Qu.:1.000
## Max. :2.0000 Max. :8.000 Max. :3.000
##
## TotRmsAbvGrd Functional Fireplaces FireplaceQu GarageType
## Min. : 2.000 Maj1: 14 Min. :0.000 Ex : 24 2Types : 6
## 1st Qu.: 5.000 Maj2: 5 1st Qu.:0.000 Fa : 33 Attchd :870
## Median : 6.000 Min1: 31 Median :1.000 Gd :380 Basment: 19
## Mean : 6.518 Min2: 34 Mean :0.613 Po : 20 BuiltIn: 88
## 3rd Qu.: 7.000 Mod : 15 3rd Qu.:1.000 TA :313 CarPort: 9
## Max. :14.000 Sev : 1 Max. :3.000 NA's:690 Detchd :387
## Typ :1360 NA's : 81
## GarageYrBlt GarageFinish GarageCars GarageArea GarageQual
## Min. :1900 Fin :352 Min. :0.000 Min. : 0.0 Ex : 3
## 1st Qu.:1961 RFn :422 1st Qu.:1.000 1st Qu.: 334.5 Fa : 48
## Median :1980 Unf :605 Median :2.000 Median : 480.0 Gd : 14
## Mean :1979 NA's: 81 Mean :1.767 Mean : 473.0 Po : 3
## 3rd Qu.:2002 3rd Qu.:2.000 3rd Qu.: 576.0 TA :1311
## Max. :2010 Max. :4.000 Max. :1418.0 NA's: 81
## NA's :81
## GarageCond PavedDrive WoodDeckSF OpenPorchSF EnclosedPorch
## Ex : 2 N: 90 Min. : 0.00 Min. : 0.00 Min. : 0.00
## Fa : 35 P: 30 1st Qu.: 0.00 1st Qu.: 0.00 1st Qu.: 0.00
## Gd : 9 Y:1340 Median : 0.00 Median : 25.00 Median : 0.00
## Po : 7 Mean : 94.24 Mean : 46.66 Mean : 21.95
## TA :1326 3rd Qu.:168.00 3rd Qu.: 68.00 3rd Qu.: 0.00
## NA's: 81 Max. :857.00 Max. :547.00 Max. :552.00
##
## X3SsnPorch ScreenPorch PoolArea PoolQC
## Min. : 0.00 Min. : 0.00 Min. : 0.000 Ex : 2
## 1st Qu.: 0.00 1st Qu.: 0.00 1st Qu.: 0.000 Fa : 2
## Median : 0.00 Median : 0.00 Median : 0.000 Gd : 3
## Mean : 3.41 Mean : 15.06 Mean : 2.759 NA's:1453
## 3rd Qu.: 0.00 3rd Qu.: 0.00 3rd Qu.: 0.000
## Max. :508.00 Max. :480.00 Max. :738.000
##
## Fence MiscFeature MiscVal MoSold
## GdPrv: 59 Gar2: 2 Min. : 0.00 Min. : 1.000
## GdWo : 54 Othr: 2 1st Qu.: 0.00 1st Qu.: 5.000
## MnPrv: 157 Shed: 49 Median : 0.00 Median : 6.000
## MnWw : 11 TenC: 1 Mean : 43.49 Mean : 6.322
## NA's :1179 NA's:1406 3rd Qu.: 0.00 3rd Qu.: 8.000
## Max. :15500.00 Max. :12.000
##
## YrSold SaleType SaleCondition SalePrice
## Min. :2006 WD :1267 Abnorml: 101 Min. : 34900
## 1st Qu.:2007 New : 122 AdjLand: 4 1st Qu.:129975
## Median :2008 COD : 43 Alloca : 12 Median :163000
## Mean :2008 ConLD : 9 Family : 20 Mean :180921
## 3rd Qu.:2009 ConLI : 5 Normal :1198 3rd Qu.:214000
## Max. :2010 ConLw : 5 Partial: 125 Max. :755000
## (Other): 9plot(y~x, main="Scatterplot of 1st Floor Square Footage and Sales Price",
xlab="1st Floor Square Footage", ylab="Sales Price")
cor(train[sapply(train, is.numeric)])## Id MSSubClass LotFrontage LotArea
## Id 1.0000000000 0.011156478 NA -0.033225519
## MSSubClass 0.0111564782 1.000000000 NA -0.139781082
## LotFrontage NA NA 1 NA
## LotArea -0.0332255186 -0.139781082 NA 1.000000000
## OverallQual -0.0283647539 0.032627708 NA 0.105805742
## OverallCond 0.0126089248 -0.059315817 NA -0.005636270
## YearBuilt -0.0127127154 0.027850137 NA 0.014227652
## YearRemodAdd -0.0219976419 0.040581045 NA 0.013788427
## MasVnrArea NA NA NA NA
## BsmtFinSF1 -0.0050240490 -0.069835749 NA 0.214103131
## BsmtFinSF2 -0.0059676720 -0.065648579 NA 0.111169745
## BsmtUnfSF -0.0079397034 -0.140759481 NA -0.002618360
## TotalBsmtSF -0.0154145661 -0.238518409 NA 0.260833135
## X1stFlrSF 0.0104960410 -0.251758352 NA 0.299474579
## X2ndFlrSF 0.0055898489 0.307885721 NA 0.050985948
## LowQualFinSF -0.0442299581 0.046473756 NA 0.004778970
## GrLivArea 0.0082727577 0.074853180 NA 0.263116167
## BsmtFullBath 0.0022885556 0.003491026 NA 0.158154531
## BsmtHalfBath -0.0201547452 -0.002332535 NA 0.048045571
## FullBath 0.0055874529 0.131608222 NA 0.126030627
## HalfBath 0.0067838113 0.177354389 NA 0.014259469
## BedroomAbvGr 0.0377185542 -0.023438028 NA 0.119689908
## KitchenAbvGr 0.0029512364 0.281721040 NA -0.017783871
## TotRmsAbvGrd 0.0272387244 0.040380065 NA 0.190014778
## Fireplaces -0.0197716324 -0.045569340 NA 0.271364010
## GarageYrBlt NA NA NA NA
## GarageCars 0.0165696771 -0.040109793 NA 0.154870740
## GarageArea 0.0176337785 -0.098671543 NA 0.180402755
## WoodDeckSF -0.0296431972 -0.012579358 NA 0.171697687
## OpenPorchSF -0.0004769113 -0.006100121 NA 0.084773809
## EnclosedPorch 0.0028892179 -0.012036622 NA -0.018339734
## X3SsnPorch -0.0466347889 -0.043824549 NA 0.020422830
## ScreenPorch 0.0013302086 -0.026030177 NA 0.043160378
## PoolArea 0.0570439048 0.008282708 NA 0.077672392
## MiscVal -0.0062424048 -0.007683291 NA 0.038067692
## MoSold 0.0211721766 -0.013584643 NA 0.001204988
## YrSold 0.0007117940 -0.021407038 NA -0.014261407
## SalePrice -0.0219167194 -0.084284135 NA 0.263843354
## OverallQual OverallCond YearBuilt YearRemodAdd
## Id -0.02836475 0.012608925 -0.012712715 -0.021997642
## MSSubClass 0.03262771 -0.059315817 0.027850137 0.040581045
## LotFrontage NA NA NA NA
## LotArea 0.10580574 -0.005636270 0.014227652 0.013788427
## OverallQual 1.00000000 -0.091932343 0.572322769 0.550683924
## OverallCond -0.09193234 1.000000000 -0.375983196 0.073741498
## YearBuilt 0.57232277 -0.375983196 1.000000000 0.592854976
## YearRemodAdd 0.55068392 0.073741498 0.592854976 1.000000000
## MasVnrArea NA NA NA NA
## BsmtFinSF1 0.23966597 -0.046230856 0.249503197 0.128450547
## BsmtFinSF2 -0.05911869 0.040229170 -0.049106831 -0.067758514
## BsmtUnfSF 0.30815893 -0.136840570 0.149040392 0.181133087
## TotalBsmtSF 0.53780850 -0.171097515 0.391452002 0.291065583
## X1stFlrSF 0.47622383 -0.144202784 0.281985859 0.240379268
## X2ndFlrSF 0.29549288 0.028942116 0.010307660 0.140023779
## LowQualFinSF -0.03042928 0.025494320 -0.183784344 -0.062419100
## GrLivArea 0.59300743 -0.079685865 0.199009714 0.287388520
## BsmtFullBath 0.11109779 -0.054941515 0.187598550 0.119469879
## BsmtHalfBath -0.04015016 0.117820915 -0.038161806 -0.012337032
## FullBath 0.55059971 -0.194149489 0.468270787 0.439046484
## HalfBath 0.27345810 -0.060769327 0.242655910 0.183330612
## BedroomAbvGr 0.10167636 0.012980060 -0.070651217 -0.040580928
## KitchenAbvGr -0.18388223 -0.087000855 -0.174800246 -0.149597521
## TotRmsAbvGrd 0.42745234 -0.057583166 0.095589128 0.191739816
## Fireplaces 0.39676504 -0.023819978 0.147716399 0.112581318
## GarageYrBlt NA NA NA NA
## GarageCars 0.60067072 -0.185757511 0.537850092 0.420622155
## GarageArea 0.56202176 -0.151521371 0.478953820 0.371599809
## WoodDeckSF 0.23892339 -0.003333699 0.224880142 0.205725920
## OpenPorchSF 0.30881882 -0.032588814 0.188685840 0.226297633
## EnclosedPorch -0.11393686 0.070356184 -0.387267783 -0.193919147
## X3SsnPorch 0.03037057 0.025503660 0.031354513 0.045285810
## ScreenPorch 0.06488636 0.054810529 -0.050364435 -0.038740011
## PoolArea 0.06516584 -0.001984942 0.004949728 0.005829372
## MiscVal -0.03140621 0.068776806 -0.034383139 -0.010286249
## MoSold 0.07081517 -0.003510839 0.012398471 0.021490002
## YrSold -0.02734671 0.043949746 -0.013617680 0.035743247
## SalePrice 0.79098160 -0.077855894 0.522897333 0.507100967
## MasVnrArea BsmtFinSF1 BsmtFinSF2 BsmtUnfSF
## Id NA -0.005024049 -0.005967672 -0.007939703
## MSSubClass NA -0.069835749 -0.065648579 -0.140759481
## LotFrontage NA NA NA NA
## LotArea NA 0.214103131 0.111169745 -0.002618360
## OverallQual NA 0.239665966 -0.059118693 0.308158927
## OverallCond NA -0.046230856 0.040229170 -0.136840570
## YearBuilt NA 0.249503197 -0.049106831 0.149040392
## YearRemodAdd NA 0.128450547 -0.067758514 0.181133087
## MasVnrArea 1 NA NA NA
## BsmtFinSF1 NA 1.000000000 -0.050117400 -0.495251469
## BsmtFinSF2 NA -0.050117400 1.000000000 -0.209294492
## BsmtUnfSF NA -0.495251469 -0.209294492 1.000000000
## TotalBsmtSF NA 0.522396052 0.104809538 0.415359605
## X1stFlrSF NA 0.445862656 0.097117448 0.317987438
## X2ndFlrSF NA -0.137078986 -0.099260316 0.004469092
## LowQualFinSF NA -0.064502597 0.014806998 0.028166688
## GrLivArea NA 0.208171130 -0.009639892 0.240257268
## BsmtFullBath NA 0.649211754 0.158678061 -0.422900477
## BsmtHalfBath NA 0.067418478 0.070948134 -0.095804288
## FullBath NA 0.058543137 -0.076443862 0.288886055
## HalfBath NA 0.004262424 -0.032147837 -0.041117530
## BedroomAbvGr NA -0.107354677 -0.015728114 0.166643317
## KitchenAbvGr NA -0.081006851 -0.040751236 0.030085868
## TotRmsAbvGrd NA 0.044315624 -0.035226548 0.250647061
## Fireplaces NA 0.260010920 0.046920709 0.051574882
## GarageYrBlt NA NA NA NA
## GarageCars NA 0.224053522 -0.038263513 0.214175190
## GarageArea NA 0.296970385 -0.018226592 0.183302698
## WoodDeckSF NA 0.204306145 0.067898326 -0.005316424
## OpenPorchSF NA 0.111760613 0.003092562 0.129005415
## EnclosedPorch NA -0.102303306 0.036543339 -0.002537855
## X3SsnPorch NA 0.026450506 -0.029993398 0.020764006
## ScreenPorch NA 0.062020623 0.088871251 -0.012579273
## PoolArea NA 0.140491286 0.041709055 -0.035092241
## MiscVal NA 0.003571473 0.004939781 -0.023836645
## MoSold NA -0.015726948 -0.015210738 0.034888443
## YrSold NA 0.014358922 0.031705637 -0.041258195
## SalePrice NA 0.386419806 -0.011378121 0.214479106
## TotalBsmtSF X1stFlrSF X2ndFlrSF LowQualFinSF
## Id -0.0154145661 0.010496041 0.005589849 -0.0442299581
## MSSubClass -0.2385184093 -0.251758352 0.307885721 0.0464737559
## LotFrontage NA NA NA NA
## LotArea 0.2608331345 0.299474579 0.050985948 0.0047789699
## OverallQual 0.5378084986 0.476223829 0.295492879 -0.0304292840
## OverallCond -0.1710975146 -0.144202784 0.028942116 0.0254943199
## YearBuilt 0.3914520021 0.281985859 0.010307660 -0.1837843444
## YearRemodAdd 0.2910655826 0.240379268 0.140023779 -0.0624191001
## MasVnrArea NA NA NA NA
## BsmtFinSF1 0.5223960520 0.445862656 -0.137078986 -0.0645025969
## BsmtFinSF2 0.1048095376 0.097117448 -0.099260316 0.0148069979
## BsmtUnfSF 0.4153596052 0.317987438 0.004469092 0.0281666881
## TotalBsmtSF 1.0000000000 0.819529975 -0.174511950 -0.0332453873
## X1stFlrSF 0.8195299750 1.000000000 -0.202646181 -0.0142406727
## X2ndFlrSF -0.1745119501 -0.202646181 1.000000000 0.0633529501
## LowQualFinSF -0.0332453873 -0.014240673 0.063352950 1.0000000000
## GrLivArea 0.4548682025 0.566023969 0.687501064 0.1346828130
## BsmtFullBath 0.3073505537 0.244671104 -0.169493952 -0.0471434219
## BsmtHalfBath -0.0003145818 0.001955654 -0.023854784 -0.0058415048
## FullBath 0.3237224136 0.380637495 0.421377983 -0.0007095096
## HalfBath -0.0488037386 -0.119915909 0.609707300 -0.0270800493
## BedroomAbvGr 0.0504499555 0.127400749 0.502900613 0.1056065685
## KitchenAbvGr -0.0689006426 0.068100588 0.059305753 0.0075217443
## TotRmsAbvGrd 0.2855725637 0.409515979 0.616422635 0.1311847760
## Fireplaces 0.3395193239 0.410531085 0.194560892 -0.0212721434
## GarageYrBlt NA NA NA NA
## GarageCars 0.4345848343 0.439316808 0.183925583 -0.0944795202
## GarageArea 0.4866654638 0.489781654 0.138346959 -0.0676014132
## WoodDeckSF 0.2320186091 0.235458623 0.092165418 -0.0254436480
## OpenPorchSF 0.2472637463 0.211671225 0.208026063 0.0182510391
## EnclosedPorch -0.0954777367 -0.065291701 0.061988691 0.0610812378
## X3SsnPorch 0.0373837273 0.056104374 -0.024357648 -0.0042956104
## ScreenPorch 0.0844889859 0.088758073 0.040606448 0.0267994130
## PoolArea 0.1260531321 0.131524976 0.081486878 0.0621573723
## MiscVal -0.0184789224 -0.021095719 0.016196875 -0.0037928708
## MoSold 0.0131961786 0.031371560 0.035164427 -0.0221739606
## YrSold -0.0149686480 -0.013603771 -0.028699914 -0.0289208798
## SalePrice 0.6135805516 0.605852185 0.319333803 -0.0256061300
## GrLivArea BsmtFullBath BsmtHalfBath FullBath
## Id 0.008272758 0.0022885556 -0.0201547452 0.0055874529
## MSSubClass 0.074853180 0.0034910258 -0.0023325346 0.1316082224
## LotFrontage NA NA NA NA
## LotArea 0.263116167 0.1581545311 0.0480455709 0.1260306265
## OverallQual 0.593007430 0.1110977861 -0.0401501577 0.5505997094
## OverallCond -0.079685865 -0.0549415154 0.1178209151 -0.1941494887
## YearBuilt 0.199009714 0.1875985500 -0.0381618057 0.4682707872
## YearRemodAdd 0.287388520 0.1194698791 -0.0123370321 0.4390464839
## MasVnrArea NA NA NA NA
## BsmtFinSF1 0.208171130 0.6492117536 0.0674184779 0.0585431369
## BsmtFinSF2 -0.009639892 0.1586780608 0.0709481337 -0.0764438620
## BsmtUnfSF 0.240257268 -0.4229004774 -0.0958042882 0.2888860555
## TotalBsmtSF 0.454868203 0.3073505537 -0.0003145818 0.3237224136
## X1stFlrSF 0.566023969 0.2446711042 0.0019556536 0.3806374950
## X2ndFlrSF 0.687501064 -0.1694939517 -0.0238547839 0.4213779829
## LowQualFinSF 0.134682813 -0.0471434219 -0.0058415048 -0.0007095096
## GrLivArea 1.000000000 0.0348360495 -0.0189184832 0.6300116463
## BsmtFullBath 0.034836050 1.0000000000 -0.1478709605 -0.0645120486
## BsmtHalfBath -0.018918483 -0.1478709605 1.0000000000 -0.0545358120
## FullBath 0.630011646 -0.0645120486 -0.0545358120 1.0000000000
## HalfBath 0.415771636 -0.0309049591 -0.0123399001 0.1363805887
## BedroomAbvGr 0.521269511 -0.1506728092 0.0465188484 0.3632519830
## KitchenAbvGr 0.100063165 -0.0415025464 -0.0379443502 0.1331152142
## TotRmsAbvGrd 0.825489374 -0.0532752361 -0.0238363413 0.5547842535
## Fireplaces 0.461679134 0.1379277084 0.0289755866 0.2436705031
## GarageYrBlt NA NA NA NA
## GarageCars 0.467247419 0.1318812244 -0.0208910590 0.4696720433
## GarageArea 0.468997477 0.1791894804 -0.0245355796 0.4056562085
## WoodDeckSF 0.247432821 0.1753151901 0.0401612233 0.1877032138
## OpenPorchSF 0.330223962 0.0673414614 -0.0253237579 0.2599774255
## EnclosedPorch 0.009113210 -0.0499106491 -0.0085553339 -0.1150929635
## X3SsnPorch 0.020643190 -0.0001060915 0.0351136309 0.0353530166
## ScreenPorch 0.101510396 0.0231477258 0.0321214072 -0.0081060933
## PoolArea 0.170205336 0.0676155562 0.0200246298 0.0496038256
## MiscVal -0.002415640 -0.0230470249 -0.0073665245 -0.0142898450
## MoSold 0.050239681 -0.0253608943 0.0328727052 0.0558721290
## YrSold -0.036525820 0.0670491377 -0.0465238818 -0.0196688407
## SalePrice 0.708624478 0.2271222331 -0.0168441543 0.5606637627
## HalfBath BedroomAbvGr KitchenAbvGr TotRmsAbvGrd
## Id 0.006783811 0.037718554 0.002951236 0.027238724
## MSSubClass 0.177354389 -0.023438028 0.281721040 0.040380065
## LotFrontage NA NA NA NA
## LotArea 0.014259469 0.119689908 -0.017783871 0.190014778
## OverallQual 0.273458099 0.101676356 -0.183882235 0.427452343
## OverallCond -0.060769327 0.012980060 -0.087000855 -0.057583166
## YearBuilt 0.242655910 -0.070651217 -0.174800246 0.095589128
## YearRemodAdd 0.183330612 -0.040580928 -0.149597521 0.191739816
## MasVnrArea NA NA NA NA
## BsmtFinSF1 0.004262424 -0.107354677 -0.081006851 0.044315624
## BsmtFinSF2 -0.032147837 -0.015728114 -0.040751236 -0.035226548
## BsmtUnfSF -0.041117530 0.166643317 0.030085868 0.250647061
## TotalBsmtSF -0.048803739 0.050449956 -0.068900643 0.285572564
## X1stFlrSF -0.119915909 0.127400749 0.068100588 0.409515979
## X2ndFlrSF 0.609707300 0.502900613 0.059305753 0.616422635
## LowQualFinSF -0.027080049 0.105606569 0.007521744 0.131184776
## GrLivArea 0.415771636 0.521269511 0.100063165 0.825489374
## BsmtFullBath -0.030904959 -0.150672809 -0.041502546 -0.053275236
## BsmtHalfBath -0.012339900 0.046518848 -0.037944350 -0.023836341
## FullBath 0.136380589 0.363251983 0.133115214 0.554784254
## HalfBath 1.000000000 0.226651484 -0.068262549 0.343414858
## BedroomAbvGr 0.226651484 1.000000000 0.198596758 0.676619936
## KitchenAbvGr -0.068262549 0.198596758 1.000000000 0.256045409
## TotRmsAbvGrd 0.343414858 0.676619936 0.256045409 1.000000000
## Fireplaces 0.203648508 0.107569681 -0.123936235 0.326114480
## GarageYrBlt NA NA NA NA
## GarageCars 0.219178152 0.086106438 -0.050633892 0.362288571
## GarageArea 0.163549364 0.065252530 -0.064433047 0.337822121
## WoodDeckSF 0.108080303 0.046853773 -0.090130273 0.165983884
## OpenPorchSF 0.199740148 0.093809572 -0.070090610 0.234191588
## EnclosedPorch -0.095316526 0.041570435 0.037312385 0.004151299
## X3SsnPorch -0.004972488 -0.024477796 -0.024600359 -0.006683241
## ScreenPorch 0.072425845 0.044299691 -0.051613366 0.059382600
## PoolArea 0.022381498 0.070702584 -0.014525116 0.083757350
## MiscVal 0.001290145 0.007766972 0.062340724 0.024762884
## MoSold -0.009049888 0.046543860 0.026588907 0.036907077
## YrSold -0.010268669 -0.036013893 0.031687207 -0.034516354
## SalePrice 0.284107676 0.168213154 -0.135907371 0.533723156
## Fireplaces GarageYrBlt GarageCars GarageArea
## Id -0.019771632 NA 0.01656968 0.01763378
## MSSubClass -0.045569340 NA -0.04010979 -0.09867154
## LotFrontage NA NA NA NA
## LotArea 0.271364010 NA 0.15487074 0.18040276
## OverallQual 0.396765038 NA 0.60067072 0.56202176
## OverallCond -0.023819978 NA -0.18575751 -0.15152137
## YearBuilt 0.147716399 NA 0.53785009 0.47895382
## YearRemodAdd 0.112581318 NA 0.42062215 0.37159981
## MasVnrArea NA NA NA NA
## BsmtFinSF1 0.260010920 NA 0.22405352 0.29697039
## BsmtFinSF2 0.046920709 NA -0.03826351 -0.01822659
## BsmtUnfSF 0.051574882 NA 0.21417519 0.18330270
## TotalBsmtSF 0.339519324 NA 0.43458483 0.48666546
## X1stFlrSF 0.410531085 NA 0.43931681 0.48978165
## X2ndFlrSF 0.194560892 NA 0.18392558 0.13834696
## LowQualFinSF -0.021272143 NA -0.09447952 -0.06760141
## GrLivArea 0.461679134 NA 0.46724742 0.46899748
## BsmtFullBath 0.137927708 NA 0.13188122 0.17918948
## BsmtHalfBath 0.028975587 NA -0.02089106 -0.02453558
## FullBath 0.243670503 NA 0.46967204 0.40565621
## HalfBath 0.203648508 NA 0.21917815 0.16354936
## BedroomAbvGr 0.107569681 NA 0.08610644 0.06525253
## KitchenAbvGr -0.123936235 NA -0.05063389 -0.06443305
## TotRmsAbvGrd 0.326114480 NA 0.36228857 0.33782212
## Fireplaces 1.000000000 NA 0.30078877 0.26914124
## GarageYrBlt NA 1 NA NA
## GarageCars 0.300788766 NA 1.00000000 0.88247541
## GarageArea 0.269141238 NA 0.88247541 1.00000000
## WoodDeckSF 0.200018796 NA 0.22634214 0.22466631
## OpenPorchSF 0.169405327 NA 0.21356945 0.24143467
## EnclosedPorch -0.024821869 NA -0.15143416 -0.12177672
## X3SsnPorch 0.011257239 NA 0.03576529 0.03508670
## ScreenPorch 0.184530270 NA 0.05049379 0.05141176
## PoolArea 0.095073522 NA 0.02093353 0.06104727
## MiscVal 0.001408605 NA -0.04308013 -0.02739991
## MoSold 0.046357102 NA 0.04052173 0.02797380
## YrSold -0.024095565 NA -0.03911690 -0.02737794
## SalePrice 0.466928837 NA 0.64040920 0.62343144
## WoodDeckSF OpenPorchSF EnclosedPorch X3SsnPorch
## Id -0.029643197 -0.0004769113 0.002889218 -0.0466347889
## MSSubClass -0.012579358 -0.0061001212 -0.012036622 -0.0438245492
## LotFrontage NA NA NA NA
## LotArea 0.171697687 0.0847738088 -0.018339734 0.0204228296
## OverallQual 0.238923392 0.3088188234 -0.113936859 0.0303705671
## OverallCond -0.003333699 -0.0325888135 0.070356184 0.0255036600
## YearBuilt 0.224880142 0.1886858400 -0.387267783 0.0313545131
## YearRemodAdd 0.205725920 0.2262976327 -0.193919147 0.0452858098
## MasVnrArea NA NA NA NA
## BsmtFinSF1 0.204306145 0.1117606134 -0.102303306 0.0264505062
## BsmtFinSF2 0.067898326 0.0030925622 0.036543339 -0.0299933980
## BsmtUnfSF -0.005316424 0.1290054146 -0.002537855 0.0207640057
## TotalBsmtSF 0.232018609 0.2472637463 -0.095477737 0.0373837273
## X1stFlrSF 0.235458623 0.2116712255 -0.065291701 0.0561043745
## X2ndFlrSF 0.092165418 0.2080260632 0.061988691 -0.0243576484
## LowQualFinSF -0.025443648 0.0182510391 0.061081238 -0.0042956104
## GrLivArea 0.247432821 0.3302239617 0.009113210 0.0206431897
## BsmtFullBath 0.175315190 0.0673414614 -0.049910649 -0.0001060915
## BsmtHalfBath 0.040161223 -0.0253237579 -0.008555334 0.0351136309
## FullBath 0.187703214 0.2599774255 -0.115092963 0.0353530166
## HalfBath 0.108080303 0.1997401475 -0.095316526 -0.0049724884
## BedroomAbvGr 0.046853773 0.0938095716 0.041570435 -0.0244777964
## KitchenAbvGr -0.090130273 -0.0700906099 0.037312385 -0.0246003587
## TotRmsAbvGrd 0.165983884 0.2341915878 0.004151299 -0.0066832410
## Fireplaces 0.200018796 0.1694053271 -0.024821869 0.0112572390
## GarageYrBlt NA NA NA NA
## GarageCars 0.226342138 0.2135694456 -0.151434160 0.0357652851
## GarageArea 0.224666307 0.2414346721 -0.121776720 0.0350867002
## WoodDeckSF 1.000000000 0.0586606086 -0.125988888 -0.0327706336
## OpenPorchSF 0.058660609 1.0000000000 -0.093079318 -0.0058424993
## EnclosedPorch -0.125988888 -0.0930793175 1.000000000 -0.0373052828
## X3SsnPorch -0.032770634 -0.0058424993 -0.037305283 1.0000000000
## ScreenPorch -0.074181351 0.0743039439 -0.082864245 -0.0314358470
## PoolArea 0.073378207 0.0607621115 0.054202562 -0.0079915489
## MiscVal -0.009551228 -0.0185837390 0.018360600 0.0003539653
## MoSold 0.021011044 0.0712548848 -0.028887266 0.0294737952
## YrSold 0.022270451 -0.0576193601 -0.009915937 0.0186449254
## SalePrice 0.324413445 0.3158562271 -0.128577958 0.0445836653
## ScreenPorch PoolArea MiscVal MoSold
## Id 0.001330209 0.057043905 -0.0062424048 0.021172177
## MSSubClass -0.026030177 0.008282708 -0.0076832913 -0.013584643
## LotFrontage NA NA NA NA
## LotArea 0.043160378 0.077672392 0.0380676920 0.001204988
## OverallQual 0.064886360 0.065165844 -0.0314062105 0.070815172
## OverallCond 0.054810529 -0.001984942 0.0687768061 -0.003510839
## YearBuilt -0.050364435 0.004949728 -0.0343831387 0.012398471
## YearRemodAdd -0.038740011 0.005829372 -0.0102862488 0.021490002
## MasVnrArea NA NA NA NA
## BsmtFinSF1 0.062020623 0.140491286 0.0035714735 -0.015726948
## BsmtFinSF2 0.088871251 0.041709055 0.0049397812 -0.015210738
## BsmtUnfSF -0.012579273 -0.035092241 -0.0238366451 0.034888443
## TotalBsmtSF 0.084488986 0.126053132 -0.0184789224 0.013196179
## X1stFlrSF 0.088758073 0.131524976 -0.0210957195 0.031371560
## X2ndFlrSF 0.040606448 0.081486878 0.0161968746 0.035164427
## LowQualFinSF 0.026799413 0.062157372 -0.0037928708 -0.022173961
## GrLivArea 0.101510396 0.170205336 -0.0024156396 0.050239681
## BsmtFullBath 0.023147726 0.067615556 -0.0230470249 -0.025360894
## BsmtHalfBath 0.032121407 0.020024630 -0.0073665245 0.032872705
## FullBath -0.008106093 0.049603826 -0.0142898450 0.055872129
## HalfBath 0.072425845 0.022381498 0.0012901448 -0.009049888
## BedroomAbvGr 0.044299691 0.070702584 0.0077669720 0.046543860
## KitchenAbvGr -0.051613366 -0.014525116 0.0623407240 0.026588907
## TotRmsAbvGrd 0.059382600 0.083757350 0.0247628842 0.036907077
## Fireplaces 0.184530270 0.095073522 0.0014086054 0.046357102
## GarageYrBlt NA NA NA NA
## GarageCars 0.050493792 0.020933531 -0.0430801281 0.040521730
## GarageArea 0.051411762 0.061047272 -0.0273999144 0.027973800
## WoodDeckSF -0.074181351 0.073378207 -0.0095512282 0.021011044
## OpenPorchSF 0.074303944 0.060762111 -0.0185837390 0.071254885
## EnclosedPorch -0.082864245 0.054202562 0.0183606001 -0.028887266
## X3SsnPorch -0.031435847 -0.007991549 0.0003539653 0.029473795
## ScreenPorch 1.000000000 0.051307395 0.0319457608 0.023216992
## PoolArea 0.051307395 1.000000000 0.0296686509 -0.033736640
## MiscVal 0.031945761 0.029668651 1.0000000000 -0.006494550
## MoSold 0.023216992 -0.033736640 -0.0064945502 1.000000000
## YrSold 0.010694106 -0.059688932 0.0049062625 -0.145721413
## SalePrice 0.111446571 0.092403549 -0.0211895796 0.046432245
## YrSold SalePrice
## Id 0.000711794 -0.02191672
## MSSubClass -0.021407038 -0.08428414
## LotFrontage NA NA
## LotArea -0.014261407 0.26384335
## OverallQual -0.027346708 0.79098160
## OverallCond 0.043949746 -0.07785589
## YearBuilt -0.013617680 0.52289733
## YearRemodAdd 0.035743247 0.50710097
## MasVnrArea NA NA
## BsmtFinSF1 0.014358922 0.38641981
## BsmtFinSF2 0.031705637 -0.01137812
## BsmtUnfSF -0.041258195 0.21447911
## TotalBsmtSF -0.014968648 0.61358055
## X1stFlrSF -0.013603771 0.60585218
## X2ndFlrSF -0.028699914 0.31933380
## LowQualFinSF -0.028920880 -0.02560613
## GrLivArea -0.036525820 0.70862448
## BsmtFullBath 0.067049138 0.22712223
## BsmtHalfBath -0.046523882 -0.01684415
## FullBath -0.019668841 0.56066376
## HalfBath -0.010268669 0.28410768
## BedroomAbvGr -0.036013893 0.16821315
## KitchenAbvGr 0.031687207 -0.13590737
## TotRmsAbvGrd -0.034516354 0.53372316
## Fireplaces -0.024095565 0.46692884
## GarageYrBlt NA NA
## GarageCars -0.039116904 0.64040920
## GarageArea -0.027377940 0.62343144
## WoodDeckSF 0.022270451 0.32441344
## OpenPorchSF -0.057619360 0.31585623
## EnclosedPorch -0.009915937 -0.12857796
## X3SsnPorch 0.018644925 0.04458367
## ScreenPorch 0.010694106 0.11144657
## PoolArea -0.059688932 0.09240355
## MiscVal 0.004906262 -0.02118958
## MoSold -0.145721413 0.04643225
## YrSold 1.000000000 -0.02892259
## SalePrice -0.028922585 1.00000000str(train)## 'data.frame': 1460 obs. of 81 variables:
## $ Id : int 1 2 3 4 5 6 7 8 9 10 ...
## $ MSSubClass : int 60 20 60 70 60 50 20 60 50 190 ...
## $ MSZoning : Factor w/ 5 levels "C (all)","FV",..: 4 4 4 4 4 4 4 4 5 4 ...
## $ LotFrontage : int 65 80 68 60 84 85 75 NA 51 50 ...
## $ LotArea : int 8450 9600 11250 9550 14260 14115 10084 10382 6120 7420 ...
## $ Street : Factor w/ 2 levels "Grvl","Pave": 2 2 2 2 2 2 2 2 2 2 ...
## $ Alley : Factor w/ 2 levels "Grvl","Pave": NA NA NA NA NA NA NA NA NA NA ...
## $ LotShape : Factor w/ 4 levels "IR1","IR2","IR3",..: 4 4 1 1 1 1 4 1 4 4 ...
## $ LandContour : Factor w/ 4 levels "Bnk","HLS","Low",..: 4 4 4 4 4 4 4 4 4 4 ...
## $ Utilities : Factor w/ 2 levels "AllPub","NoSeWa": 1 1 1 1 1 1 1 1 1 1 ...
## $ LotConfig : Factor w/ 5 levels "Corner","CulDSac",..: 5 3 5 1 3 5 5 1 5 1 ...
## $ LandSlope : Factor w/ 3 levels "Gtl","Mod","Sev": 1 1 1 1 1 1 1 1 1 1 ...
## $ Neighborhood : Factor w/ 25 levels "Blmngtn","Blueste",..: 6 25 6 7 14 12 21 17 18 4 ...
## $ Condition1 : Factor w/ 9 levels "Artery","Feedr",..: 3 2 3 3 3 3 3 5 1 1 ...
## $ Condition2 : Factor w/ 8 levels "Artery","Feedr",..: 3 3 3 3 3 3 3 3 3 1 ...
## $ BldgType : Factor w/ 5 levels "1Fam","2fmCon",..: 1 1 1 1 1 1 1 1 1 2 ...
## $ HouseStyle : Factor w/ 8 levels "1.5Fin","1.5Unf",..: 6 3 6 6 6 1 3 6 1 2 ...
## $ OverallQual : int 7 6 7 7 8 5 8 7 7 5 ...
## $ OverallCond : int 5 8 5 5 5 5 5 6 5 6 ...
## $ YearBuilt : int 2003 1976 2001 1915 2000 1993 2004 1973 1931 1939 ...
## $ YearRemodAdd : int 2003 1976 2002 1970 2000 1995 2005 1973 1950 1950 ...
## $ RoofStyle : Factor w/ 6 levels "Flat","Gable",..: 2 2 2 2 2 2 2 2 2 2 ...
## $ RoofMatl : Factor w/ 8 levels "ClyTile","CompShg",..: 2 2 2 2 2 2 2 2 2 2 ...
## $ Exterior1st : Factor w/ 15 levels "AsbShng","AsphShn",..: 13 9 13 14 13 13 13 7 4 9 ...
## $ Exterior2nd : Factor w/ 16 levels "AsbShng","AsphShn",..: 14 9 14 16 14 14 14 7 16 9 ...
## $ MasVnrType : Factor w/ 4 levels "BrkCmn","BrkFace",..: 2 3 2 3 2 3 4 4 3 3 ...
## $ MasVnrArea : int 196 0 162 0 350 0 186 240 0 0 ...
## $ ExterQual : Factor w/ 4 levels "Ex","Fa","Gd",..: 3 4 3 4 3 4 3 4 4 4 ...
## $ ExterCond : Factor w/ 5 levels "Ex","Fa","Gd",..: 5 5 5 5 5 5 5 5 5 5 ...
## $ Foundation : Factor w/ 6 levels "BrkTil","CBlock",..: 3 2 3 1 3 6 3 2 1 1 ...
## $ BsmtQual : Factor w/ 4 levels "Ex","Fa","Gd",..: 3 3 3 4 3 3 1 3 4 4 ...
## $ BsmtCond : Factor w/ 4 levels "Fa","Gd","Po",..: 4 4 4 2 4 4 4 4 4 4 ...
## $ BsmtExposure : Factor w/ 4 levels "Av","Gd","Mn",..: 4 2 3 4 1 4 1 3 4 4 ...
## $ BsmtFinType1 : Factor w/ 6 levels "ALQ","BLQ","GLQ",..: 3 1 3 1 3 3 3 1 6 3 ...
## $ BsmtFinSF1 : int 706 978 486 216 655 732 1369 859 0 851 ...
## $ BsmtFinType2 : Factor w/ 6 levels "ALQ","BLQ","GLQ",..: 6 6 6 6 6 6 6 2 6 6 ...
## $ BsmtFinSF2 : int 0 0 0 0 0 0 0 32 0 0 ...
## $ BsmtUnfSF : int 150 284 434 540 490 64 317 216 952 140 ...
## $ TotalBsmtSF : int 856 1262 920 756 1145 796 1686 1107 952 991 ...
## $ Heating : Factor w/ 6 levels "Floor","GasA",..: 2 2 2 2 2 2 2 2 2 2 ...
## $ HeatingQC : Factor w/ 5 levels "Ex","Fa","Gd",..: 1 1 1 3 1 1 1 1 3 1 ...
## $ CentralAir : Factor w/ 2 levels "N","Y": 2 2 2 2 2 2 2 2 2 2 ...
## $ Electrical : Factor w/ 5 levels "FuseA","FuseF",..: 5 5 5 5 5 5 5 5 2 5 ...
## $ X1stFlrSF : int 856 1262 920 961 1145 796 1694 1107 1022 1077 ...
## $ X2ndFlrSF : int 854 0 866 756 1053 566 0 983 752 0 ...
## $ LowQualFinSF : int 0 0 0 0 0 0 0 0 0 0 ...
## $ GrLivArea : int 1710 1262 1786 1717 2198 1362 1694 2090 1774 1077 ...
## $ BsmtFullBath : int 1 0 1 1 1 1 1 1 0 1 ...
## $ BsmtHalfBath : int 0 1 0 0 0 0 0 0 0 0 ...
## $ FullBath : int 2 2 2 1 2 1 2 2 2 1 ...
## $ HalfBath : int 1 0 1 0 1 1 0 1 0 0 ...
## $ BedroomAbvGr : int 3 3 3 3 4 1 3 3 2 2 ...
## $ KitchenAbvGr : int 1 1 1 1 1 1 1 1 2 2 ...
## $ KitchenQual : Factor w/ 4 levels "Ex","Fa","Gd",..: 3 4 3 3 3 4 3 4 4 4 ...
## $ TotRmsAbvGrd : int 8 6 6 7 9 5 7 7 8 5 ...
## $ Functional : Factor w/ 7 levels "Maj1","Maj2",..: 7 7 7 7 7 7 7 7 3 7 ...
## $ Fireplaces : int 0 1 1 1 1 0 1 2 2 2 ...
## $ FireplaceQu : Factor w/ 5 levels "Ex","Fa","Gd",..: NA 5 5 3 5 NA 3 5 5 5 ...
## $ GarageType : Factor w/ 6 levels "2Types","Attchd",..: 2 2 2 6 2 2 2 2 6 2 ...
## $ GarageYrBlt : int 2003 1976 2001 1998 2000 1993 2004 1973 1931 1939 ...
## $ GarageFinish : Factor w/ 3 levels "Fin","RFn","Unf": 2 2 2 3 2 3 2 2 3 2 ...
## $ GarageCars : int 2 2 2 3 3 2 2 2 2 1 ...
## $ GarageArea : int 548 460 608 642 836 480 636 484 468 205 ...
## $ GarageQual : Factor w/ 5 levels "Ex","Fa","Gd",..: 5 5 5 5 5 5 5 5 2 3 ...
## $ GarageCond : Factor w/ 5 levels "Ex","Fa","Gd",..: 5 5 5 5 5 5 5 5 5 5 ...
## $ PavedDrive : Factor w/ 3 levels "N","P","Y": 3 3 3 3 3 3 3 3 3 3 ...
## $ WoodDeckSF : int 0 298 0 0 192 40 255 235 90 0 ...
## $ OpenPorchSF : int 61 0 42 35 84 30 57 204 0 4 ...
## $ EnclosedPorch: int 0 0 0 272 0 0 0 228 205 0 ...
## $ X3SsnPorch : int 0 0 0 0 0 320 0 0 0 0 ...
## $ ScreenPorch : int 0 0 0 0 0 0 0 0 0 0 ...
## $ PoolArea : int 0 0 0 0 0 0 0 0 0 0 ...
## $ PoolQC : Factor w/ 3 levels "Ex","Fa","Gd": NA NA NA NA NA NA NA NA NA NA ...
## $ Fence : Factor w/ 4 levels "GdPrv","GdWo",..: NA NA NA NA NA 3 NA NA NA NA ...
## $ MiscFeature : Factor w/ 4 levels "Gar2","Othr",..: NA NA NA NA NA 3 NA 3 NA NA ...
## $ MiscVal : int 0 0 0 0 0 700 0 350 0 0 ...
## $ MoSold : int 2 5 9 2 12 10 8 11 4 1 ...
## $ YrSold : int 2008 2007 2008 2006 2008 2009 2007 2009 2008 2008 ...
## $ SaleType : Factor w/ 9 levels "COD","Con","ConLD",..: 9 9 9 9 9 9 9 9 9 9 ...
## $ SaleCondition: Factor w/ 6 levels "Abnorml","AdjLand",..: 5 5 5 1 5 5 5 5 1 5 ...
## $ SalePrice : int 208500 181500 223500 140000 250000 143000 307000 200000 129900 118000 ...Correlations
Test correlations across 3 different variables.
The p-values for all pairwise correlation tests show that all the below correlations are statistical significant than zero, at the 92% confidence level. The correlations are also fairly high, which means that one should be careful when including all 3 variables into a forecasting model (i.e., multicollinearity).
cols <- c('SalePrice','GrLivArea','X1stFlrSF')
cr <- train[cols]
cor_tbl <- cor(cr)
cor.test(cr$SalePrice,cr$GrLivArea, conf.level = 0.92)##
## Pearson's product-moment correlation
##
## data: cr$SalePrice and cr$GrLivArea
## t = 38.348, df = 1458, p-value < 2.2e-16
## alternative hypothesis: true correlation is not equal to 0
## 92 percent confidence interval:
## 0.6850407 0.7307245
## sample estimates:
## cor
## 0.7086245cor.test(cr$SalePrice,cr$X1stFlrSF, conf.level = 0.92) ##
## Pearson's product-moment correlation
##
## data: cr$SalePrice and cr$X1stFlrSF
## t = 29.078, df = 1458, p-value < 2.2e-16
## alternative hypothesis: true correlation is not equal to 0
## 92 percent confidence interval:
## 0.5760142 0.6340778
## sample estimates:
## cor
## 0.6058522cor.test(cr$GrLivArea,cr$X1stFlrSF, conf.level = 0.92)##
## Pearson's product-moment correlation
##
## data: cr$GrLivArea and cr$X1stFlrSF
## t = 26.217, df = 1458, p-value < 2.2e-16
## alternative hypothesis: true correlation is not equal to 0
## 92 percent confidence interval:
## 0.5340458 0.5963849
## sample estimates:
## cor
## 0.566024If we assume independence, then the formula is 1-(1-alpha)^(number of test) reference - http://www.biostat.umn.edu/~cavanr/asfChap4-1.pdf
There’s cause for concern given FWER is roughly 22%, but the correlation tests are very significant.
FWER = 1 - (1-0.08)**3Linear Algebra and Correlation
precision_matrix <- solve(cor_tbl)
# Multiply precision matrix by correlation matrix
print(precision_matrix %*% cor_tbl)## SalePrice GrLivArea X1stFlrSF
## SalePrice 1.000000e+00 2.775558e-16 4.440892e-16
## GrLivArea -2.775558e-16 1.000000e+00 -5.551115e-17
## X1stFlrSF 0.000000e+00 0.000000e+00 1.000000e+00# Multiply correlation matrix by precision matrix
print(cor_tbl %*% precision_matrix)## SalePrice GrLivArea X1stFlrSF
## SalePrice 1.000000e+00 -5.551115e-17 0.000000e+00
## GrLivArea 0.000000e+00 1.000000e+00 1.110223e-16
## X1stFlrSF 2.220446e-16 5.551115e-17 1.000000e+00# LU decomposition on the precision matrix
lu(precision_matrix)## $L
## SalePrice GrLivArea X1stFlrSF
## SalePrice 1.0000000 0.000000 0
## GrLivArea -0.5380938 1.000000 0
## X1stFlrSF -0.3012782 -0.566024 1
##
## $U
## SalePrice GrLivArea X1stFlrSF
## SalePrice 2.292718 -1.233697 -0.6907460
## GrLivArea 0.000000 1.471417 -0.8328574
## X1stFlrSF 0.000000 0.000000 1.0000000Calculus
The minimum for my x variable is higher than 0; thus, no shifting is needed.
The distribution of the exponentially simulated data (see below) captures more of the observations in the fat tails of the empirical distribution. This also helps explain the larger right skew in the simulated data.
# Minimum is above 0, i.e., no shifting needed
print(min(x))## [1] 334# new distribution
x_exp <- fitdistr(x, densfun = "exponential")
lam <- x_exp$estimate
new_x <- rexp(1000,lam)
hist(new_x, main = '1000 Simulation Histogram')
# CDF at 5% and 95%
qexp(0.05,lam,lower.tail = TRUE,log.p = FALSE)## [1] 59.63495qexp(0.95,lam,lower.tail = TRUE,log.p = FALSE)## [1] 3482.918# empirical distribution
hist(x, main="Histogram of 1st Floor Square Footage - Empirical")
# empirical 5% and 95%
quantile(x,c(0.05,0.95))## 5% 95%
## 672.95 1831.25Modeling
I attempted to fit a lasso regression so that unnecessary variables are dropped. Additionally, the data was cleaned to remove features that had too many NAs. I failed at scaling because I didn’t know how to unscale the data. After an initial OLS model was ran, I did some arbitrary feature selection based on significant t-stats for the “ols_final” model.
kaggle u/n: Silverio Vasquez Submission score = 0.86421
na_count <-sapply(train, function(y) sum(length(which(is.na(y)))))
ids_train <- train$Id
drop_cols <- c('PoolQC','Fence','MiscFeature','Alley','FireplaceQu','Id', 'GrLivArea','ElectricalMix','TotalBsmtSF','Exterior2ndCBlock')
train_clean <- train[,!(names(train) %in% drop_cols)]
# train_new <- train_clean %>% mutate_if(is.numeric, scale)
df <- train_clean[, sapply(train_clean, function(col) length(unique(col))) > 2]
ols <- lm(SalePrice ~ ., data=df)
summary(ols)##
## Call:
## lm(formula = SalePrice ~ ., data = df)
##
## Residuals:
## Min 1Q Median 3Q Max
## -183188 -9628 172 9024 183188
##
## Coefficients: (2 not defined because of singularities)
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -1.356e+06 1.331e+06 -1.019 0.308589
## MSSubClass -4.126e+01 1.251e+02 -0.330 0.741636
## MSZoningFV 4.694e+04 1.430e+04 3.282 0.001070 **
## MSZoningRH 3.666e+04 1.534e+04 2.389 0.017086 *
## MSZoningRL 3.553e+04 1.257e+04 2.826 0.004824 **
## MSZoningRM 3.292e+04 1.162e+04 2.832 0.004733 **
## LotFrontage 9.044e+01 5.787e+01 1.563 0.118427
## LotArea 7.921e-01 1.571e-01 5.043 5.58e-07 ***
## LotShapeIR2 6.205e+03 5.874e+03 1.056 0.291083
## LotShapeIR3 9.666e+03 1.301e+04 0.743 0.457862
## LotShapeReg 2.754e+03 2.116e+03 1.302 0.193398
## LandContourHLS 1.118e+04 6.654e+03 1.681 0.093154 .
## LandContourLow -1.921e+04 9.967e+03 -1.927 0.054266 .
## LandContourLvl 6.716e+03 4.974e+03 1.350 0.177336
## LotConfigCulDSac 1.467e+04 5.185e+03 2.829 0.004779 **
## LotConfigFR2 -1.054e+04 5.531e+03 -1.906 0.057031 .
## LotConfigFR3 -1.238e+04 1.373e+04 -0.901 0.367595
## LotConfigInside -2.046e+02 2.329e+03 -0.088 0.929999
## LandSlopeMod 6.789e+03 5.252e+03 1.293 0.196459
## LandSlopeSev -3.993e+04 1.649e+04 -2.422 0.015627 *
## NeighborhoodBlueste 1.174e+03 2.120e+04 0.055 0.955853
## NeighborhoodBrDale 8.029e+03 1.319e+04 0.609 0.542982
## NeighborhoodBrkSide 1.323e+03 1.200e+04 0.110 0.912198
## NeighborhoodClearCr -8.938e+03 1.245e+04 -0.718 0.472935
## NeighborhoodCollgCr -6.808e+03 8.776e+03 -0.776 0.438123
## NeighborhoodCrawfor 1.540e+04 1.048e+04 1.470 0.141933
## NeighborhoodEdwards -1.802e+04 9.733e+03 -1.852 0.064420 .
## NeighborhoodGilbert -1.052e+04 9.445e+03 -1.113 0.265807
## NeighborhoodIDOTRR 2.006e+03 1.362e+04 0.147 0.882902
## NeighborhoodMeadowV -2.605e+03 1.410e+04 -0.185 0.853465
## NeighborhoodMitchel -1.568e+04 1.021e+04 -1.537 0.124769
## NeighborhoodNAmes -1.160e+04 9.437e+03 -1.230 0.219122
## NeighborhoodNoRidge 2.824e+04 1.021e+04 2.766 0.005786 **
## NeighborhoodNPkVill 1.133e+04 2.019e+04 0.561 0.574792
## NeighborhoodNridgHt 1.934e+04 8.873e+03 2.179 0.029574 *
## NeighborhoodNWAmes -1.548e+04 9.953e+03 -1.555 0.120210
## NeighborhoodOldTown -6.981e+03 1.197e+04 -0.583 0.560027
## NeighborhoodSawyer -2.935e+03 9.967e+03 -0.294 0.768496
## NeighborhoodSawyerW 1.301e+03 9.512e+03 0.137 0.891202
## NeighborhoodSomerst -1.504e+02 1.055e+04 -0.014 0.988622
## NeighborhoodStoneBr 4.517e+04 1.004e+04 4.501 7.67e-06 ***
## NeighborhoodSWISU -1.439e+03 1.186e+04 -0.121 0.903419
## NeighborhoodTimber -1.079e+04 9.732e+03 -1.108 0.268003
## NeighborhoodVeenker 8.274e+03 1.361e+04 0.608 0.543259
## Condition1Feedr 3.918e+03 6.441e+03 0.608 0.543165
## Condition1Norm 1.359e+04 5.086e+03 2.672 0.007678 **
## Condition1PosA 7.795e+03 1.724e+04 0.452 0.651248
## Condition1PosN 3.065e+03 1.163e+04 0.263 0.792289
## Condition1RRAe -1.254e+04 1.192e+04 -1.052 0.293004
## Condition1RRAn 9.471e+03 8.153e+03 1.162 0.245711
## Condition1RRNe 1.322e+04 2.564e+04 0.516 0.606331
## Condition1RRNn -5.735e+03 1.647e+04 -0.348 0.727792
## Condition2Feedr -4.538e+03 2.761e+04 -0.164 0.869481
## Condition2Norm -8.784e+02 2.373e+04 -0.037 0.970483
## Condition2PosA 4.608e+04 4.754e+04 0.969 0.332636
## Condition2PosN -2.283e+05 3.222e+04 -7.086 2.83e-12 ***
## Condition2RRNn 7.862e+03 3.063e+04 0.257 0.797517
## BldgType2fmCon -1.974e+03 1.861e+04 -0.106 0.915553
## BldgTypeDuplex -1.446e+04 1.078e+04 -1.341 0.180202
## BldgTypeTwnhs -1.983e+04 1.444e+04 -1.374 0.169809
## BldgTypeTwnhsE -1.570e+04 1.346e+04 -1.166 0.243766
## HouseStyle1.5Unf 1.260e+04 1.058e+04 1.191 0.233922
## HouseStyle1Story 1.278e+04 5.935e+03 2.153 0.031623 *
## HouseStyle2.5Fin -2.620e+04 1.678e+04 -1.561 0.118793
## HouseStyle2.5Unf -1.389e+04 1.253e+04 -1.109 0.267915
## HouseStyle2Story -4.853e+03 4.604e+03 -1.054 0.292151
## HouseStyleSFoyer 9.658e+03 8.760e+03 1.103 0.270531
## HouseStyleSLvl 6.996e+03 7.732e+03 0.905 0.365806
## OverallQual 7.146e+03 1.327e+03 5.383 9.39e-08 ***
## OverallCond 6.747e+03 1.164e+03 5.797 9.44e-09 ***
## YearBuilt 3.096e+02 1.034e+02 2.994 0.002827 **
## YearRemodAdd 7.699e+01 7.325e+01 1.051 0.293492
## RoofStyleGable 4.098e+04 3.098e+04 1.323 0.186236
## RoofStyleGambrel 4.526e+04 3.246e+04 1.394 0.163566
## RoofStyleHip 3.943e+04 3.102e+04 1.271 0.204093
## RoofStyleMansard 5.452e+04 3.467e+04 1.572 0.116218
## RoofMatlCompShg 6.946e+05 3.892e+04 17.847 < 2e-16 ***
## RoofMatlMembran 8.202e+05 6.031e+04 13.599 < 2e-16 ***
## RoofMatlRoll 6.912e+05 4.786e+04 14.442 < 2e-16 ***
## RoofMatlTar&Grv 7.087e+05 4.702e+04 15.072 < 2e-16 ***
## RoofMatlWdShake 6.784e+05 4.738e+04 14.319 < 2e-16 ***
## RoofMatlWdShngl 7.530e+05 4.009e+04 18.783 < 2e-16 ***
## Exterior1stBrkComm -5.520e+04 3.979e+04 -1.387 0.165712
## Exterior1stBrkFace 9.706e+02 1.567e+04 0.062 0.950636
## Exterior1stCBlock 3.879e+03 3.159e+04 0.123 0.902288
## Exterior1stCemntBd -2.495e+04 2.519e+04 -0.990 0.322349
## Exterior1stHdBoard -2.103e+04 1.580e+04 -1.331 0.183568
## Exterior1stImStucc -6.152e+04 3.060e+04 -2.010 0.044705 *
## Exterior1stMetalSd -7.580e+03 1.831e+04 -0.414 0.679022
## Exterior1stPlywood -2.578e+04 1.580e+04 -1.632 0.103012
## Exterior1stStone -6.674e+03 4.179e+04 -0.160 0.873140
## Exterior1stStucco -1.596e+04 1.760e+04 -0.907 0.364681
## Exterior1stVinylSd -1.584e+04 1.592e+04 -0.995 0.320179
## Exterior1stWd Sdng -1.526e+04 1.524e+04 -1.001 0.316865
## Exterior1stWdShing -1.362e+04 1.621e+04 -0.840 0.400949
## Exterior2ndAsphShn 2.003e+04 2.634e+04 0.760 0.447283
## Exterior2ndBrk Cmn 2.624e+04 2.620e+04 1.001 0.316888
## Exterior2ndBrkFace 8.250e+03 1.626e+04 0.507 0.611954
## Exterior2ndCBlock NA NA NA NA
## Exterior2ndCmentBd 2.617e+04 2.467e+04 1.061 0.289149
## Exterior2ndHdBoard 1.646e+04 1.527e+04 1.078 0.281257
## Exterior2ndImStucc 4.236e+04 1.698e+04 2.494 0.012801 *
## Exterior2ndMetalSd 1.088e+04 1.769e+04 0.615 0.538853
## Exterior2ndOther -1.524e+04 3.015e+04 -0.505 0.613424
## Exterior2ndPlywood 1.580e+04 1.476e+04 1.071 0.284578
## Exterior2ndStone -6.083e+02 3.027e+04 -0.020 0.983974
## Exterior2ndStucco 1.444e+04 1.682e+04 0.859 0.390767
## Exterior2ndVinylSd 1.619e+04 1.527e+04 1.060 0.289240
## Exterior2ndWd Sdng 1.690e+04 1.454e+04 1.162 0.245377
## Exterior2ndWd Shng 9.835e+03 1.506e+04 0.653 0.513953
## MasVnrTypeBrkFace 8.912e+03 9.333e+03 0.955 0.339856
## MasVnrTypeNone 1.377e+04 9.326e+03 1.476 0.140207
## MasVnrTypeStone 1.679e+04 9.630e+03 1.744 0.081539 .
## MasVnrArea 1.992e+01 6.849e+00 2.908 0.003730 **
## ExterQualFa -2.658e+04 1.549e+04 -1.716 0.086489 .
## ExterQualGd -2.220e+04 5.688e+03 -3.904 0.000102 ***
## ExterQualTA -2.256e+04 6.422e+03 -3.513 0.000466 ***
## ExterCondFa 1.185e+04 2.850e+04 0.416 0.677561
## ExterCondGd 3.000e+03 2.729e+04 0.110 0.912476
## ExterCondTA 6.393e+03 2.726e+04 0.235 0.814618
## FoundationCBlock 5.298e+03 4.047e+03 1.309 0.190886
## FoundationPConc 5.112e+03 4.344e+03 1.177 0.239511
## FoundationStone 1.111e+04 1.267e+04 0.877 0.380752
## FoundationWood -4.193e+04 2.016e+04 -2.079 0.037891 *
## BsmtQualFa -9.648e+03 7.495e+03 -1.287 0.198350
## BsmtQualGd -1.744e+04 3.843e+03 -4.538 6.46e-06 ***
## BsmtQualTA -1.477e+04 4.989e+03 -2.961 0.003153 **
## BsmtCondGd 7.946e+01 6.494e+03 0.012 0.990241
## BsmtCondPo 2.395e+04 4.366e+04 0.549 0.583430
## BsmtCondTA 1.813e+03 5.237e+03 0.346 0.729269
## BsmtExposureGd 1.260e+04 3.784e+03 3.331 0.000903 ***
## BsmtExposureMn -5.810e+03 3.695e+03 -1.573 0.116164
## BsmtExposureNo -6.923e+03 2.672e+03 -2.591 0.009725 **
## BsmtFinType1BLQ 1.338e+03 3.568e+03 0.375 0.707653
## BsmtFinType1GLQ 7.137e+03 3.215e+03 2.220 0.026700 *
## BsmtFinType1LwQ -4.149e+03 4.749e+03 -0.874 0.382592
## BsmtFinType1Rec 3.669e+02 3.733e+03 0.098 0.921731
## BsmtFinType1Unf 2.551e+03 3.655e+03 0.698 0.485378
## BsmtFinSF1 4.513e+01 7.182e+00 6.284 5.18e-10 ***
## BsmtFinType2BLQ -7.605e+03 1.007e+04 -0.755 0.450253
## BsmtFinType2GLQ -2.993e+03 1.173e+04 -0.255 0.798680
## BsmtFinType2LwQ -1.105e+04 9.758e+03 -1.132 0.257868
## BsmtFinType2Rec -9.400e+03 9.293e+03 -1.012 0.312048
## BsmtFinType2Unf -5.571e+03 9.999e+03 -0.557 0.577556
## BsmtFinSF2 3.681e+01 1.182e+01 3.114 0.001904 **
## BsmtUnfSF 2.575e+01 6.759e+00 3.810 0.000149 ***
## HeatingGasW -1.004e+03 8.491e+03 -0.118 0.905874
## HeatingGrav 1.353e+04 2.471e+04 0.548 0.584074
## HeatingOthW -1.364e+04 3.037e+04 -0.449 0.653387
## HeatingQCFa -1.088e+02 6.282e+03 -0.017 0.986188
## HeatingQCGd -3.449e+03 2.611e+03 -1.321 0.186793
## HeatingQCPo -2.511e+03 3.054e+04 -0.082 0.934494
## HeatingQCTA -2.787e+03 2.651e+03 -1.051 0.293367
## ElectricalFuseF -7.746e+02 8.427e+03 -0.092 0.926779
## ElectricalFuseP -3.823e+03 2.682e+04 -0.143 0.886691
## ElectricalMix NA NA NA NA
## ElectricalSBrkr 7.608e+02 3.785e+03 0.201 0.840753
## X1stFlrSF 4.312e+01 7.666e+00 5.625 2.50e-08 ***
## X2ndFlrSF 7.055e+01 7.120e+00 9.909 < 2e-16 ***
## LowQualFinSF 1.209e+01 2.797e+01 0.432 0.665639
## BsmtFullBath 1.807e+02 2.480e+03 0.073 0.941935
## BsmtHalfBath -1.661e+03 3.784e+03 -0.439 0.660745
## FullBath 4.864e+03 2.912e+03 1.670 0.095274 .
## HalfBath 2.466e+03 2.685e+03 0.918 0.358622
## BedroomAbvGr -3.577e+03 1.802e+03 -1.985 0.047410 *
## KitchenAbvGr -1.465e+04 8.575e+03 -1.708 0.087950 .
## KitchenQualFa -1.937e+04 8.014e+03 -2.417 0.015855 *
## KitchenQualGd -2.328e+04 4.100e+03 -5.680 1.83e-08 ***
## KitchenQualTA -2.022e+04 4.773e+03 -4.235 2.52e-05 ***
## TotRmsAbvGrd 1.021e+03 1.195e+03 0.855 0.392781
## FunctionalMaj2 -7.474e+03 1.903e+04 -0.393 0.694627
## FunctionalMin1 2.605e+03 1.072e+04 0.243 0.808032
## FunctionalMin2 2.960e+02 1.063e+04 0.028 0.977797
## FunctionalMod -3.798e+03 1.371e+04 -0.277 0.781796
## FunctionalTyp 1.181e+04 9.072e+03 1.302 0.193322
## Fireplaces 3.145e+03 1.724e+03 1.824 0.068512 .
## GarageTypeAttchd 1.869e+04 1.316e+04 1.420 0.156063
## GarageTypeBasment 2.551e+04 1.551e+04 1.645 0.100332
## GarageTypeBuiltIn 1.684e+04 1.387e+04 1.214 0.225015
## GarageTypeCarPort 3.207e+04 1.866e+04 1.719 0.085973 .
## GarageTypeDetchd 2.290e+04 1.313e+04 1.745 0.081386 .
## GarageYrBlt 4.174e+01 7.595e+01 0.550 0.582703
## GarageFinishRFn -2.022e+03 2.433e+03 -0.831 0.406258
## GarageFinishUnf -3.387e+02 3.047e+03 -0.111 0.911511
## GarageCars 4.454e+03 2.778e+03 1.604 0.109176
## GarageArea 1.180e+01 9.581e+00 1.232 0.218263
## GarageQualFa -1.063e+05 3.459e+04 -3.074 0.002177 **
## GarageQualGd -9.632e+04 3.559e+04 -2.706 0.006942 **
## GarageQualPo -1.365e+05 4.723e+04 -2.890 0.003949 **
## GarageQualTA -1.019e+05 3.419e+04 -2.979 0.002970 **
## GarageCondFa 1.032e+05 3.922e+04 2.632 0.008642 **
## GarageCondGd 1.058e+05 4.188e+04 2.526 0.011727 *
## GarageCondPo 1.112e+05 4.269e+04 2.604 0.009357 **
## GarageCondTA 1.041e+05 3.881e+04 2.684 0.007419 **
## PavedDriveP -2.392e+03 7.493e+03 -0.319 0.749594
## PavedDriveY 2.501e+02 4.988e+03 0.050 0.960027
## WoodDeckSF 1.084e+01 7.546e+00 1.437 0.151176
## OpenPorchSF -7.947e+00 1.527e+01 -0.520 0.602920
## EnclosedPorch -5.582e+00 1.606e+01 -0.348 0.728246
## X3SsnPorch 4.264e+01 2.699e+01 1.580 0.114568
## ScreenPorch 2.181e+01 1.529e+01 1.426 0.154121
## PoolArea 7.628e+01 2.290e+01 3.331 0.000901 ***
## MiscVal -1.983e+00 5.239e+00 -0.378 0.705177
## MoSold -3.787e+02 3.097e+02 -1.223 0.221807
## YrSold -1.526e+02 6.554e+02 -0.233 0.815944
## SaleTypeCon 2.056e+04 1.952e+04 1.054 0.292324
## SaleTypeConLD 1.473e+04 1.480e+04 0.995 0.319932
## SaleTypeConLI 2.275e+03 1.660e+04 0.137 0.890999
## SaleTypeConLw 1.427e+03 1.471e+04 0.097 0.922720
## SaleTypeCWD 1.148e+04 1.430e+04 0.802 0.422493
## SaleTypeNew 1.157e+04 1.817e+04 0.636 0.524653
## SaleTypeOth 2.277e+04 2.536e+04 0.898 0.369649
## SaleTypeWD -2.567e+03 5.325e+03 -0.482 0.629929
## SaleConditionAdjLand 2.992e+04 2.761e+04 1.084 0.278849
## SaleConditionAlloca 2.972e+03 1.306e+04 0.228 0.820036
## SaleConditionFamily -2.249e+03 7.192e+03 -0.313 0.754577
## SaleConditionNormal 5.082e+03 3.740e+03 1.359 0.174495
## SaleConditionPartial 5.472e+03 1.748e+04 0.313 0.754375
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 24250 on 878 degrees of freedom
## (366 observations deleted due to missingness)
## Multiple R-squared: 0.9317, Adjusted R-squared: 0.915
## F-statistic: 55.73 on 215 and 878 DF, p-value: < 2.2e-16keep_cols <- c('SalePrice','OverallQual','Condition2PosN','NeighborhoodStoneBr','LotArea','RoofMatlMembran','BsmtUnfSF','X1stFlrSF')
new_df <- df[,(names(df) %in% keep_cols)]
ols_final <- lm(SalePrice ~., data=new_df)
summary(ols_final)##
## Call:
## lm(formula = SalePrice ~ ., data = new_df)
##
## Residuals:
## Min 1Q Median 3Q Max
## -421625 -23864 -2215 19383 346697
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -1.219e+05 5.207e+03 -23.408 < 2e-16 ***
## LotArea 8.674e-01 1.182e-01 7.340 3.53e-13 ***
## OverallQual 3.857e+04 9.366e+02 41.178 < 2e-16 ***
## BsmtUnfSF -1.455e+01 2.732e+00 -5.325 1.17e-07 ***
## X1stFlrSF 5.737e+01 3.522e+00 16.289 < 2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 42720 on 1455 degrees of freedom
## Multiple R-squared: 0.7116, Adjusted R-squared: 0.7108
## F-statistic: 897.7 on 4 and 1455 DF, p-value: < 2.2e-16# residuals centered around 0, which is good
hist(ols_final$residuals)
# residuals don't show serial correlation or heteroskedasticity
plot(ols_final$residuals)
# qqplot confirms the above
qqnorm(ols_final$residuals)
test <- read.csv('test.csv',header=TRUE)
ids_test <- test$Id
test_clean <- test[,!(names(test) %in% drop_cols)]
# test_new <- test_clean %>% mutate_if(is.numeric, scale)
test_df <- test_clean[,(names(test_clean) %in% keep_cols)]
pred <- predict(ols_final,test_df)
# pred_uns <- unscale(pred,test_clean)
kaggle <- cbind(test$Id,pred)
colnames(kaggle) <- c('Id','SalePrice')
# file to submit
write.csv(kaggle, file="SJV_Kaggle.csv")