DATA 605 Final Project
Dhananjay Kumar
5/18/2017
# Import Libraries
library(MASS)
library(forecast)## Warning: package 'forecast' was built under R version 3.3.3Load data set
train<-read.csv("train.csv")1.Probability
Pick one of the quantitative independent variables from the training data set (train.csv) , and define that variable as X. Pick SalePrice as the dependent variable, and define it as Y for the next analysis.
dim(train)## [1] 1460 81In our analysis we will take LotArea as the independent variable X.
Calculate as a minimum the below probabilities a through c. Assume the small letter “x” is estimated as the 4th quartile of the X variable, and the small letter “y” is estimated as the 2d quartile of the Y variable. Interpret the meaning of all probabilities
x <- quantile(train$LotArea)["75%"]
y <- quantile(train$SalePrice)["50%"]a.P(X>x | Y>y)
This represents the probability that value of X is above the 3rd Quartile given that Y is above the 2nd Quartile. In other words this represents the probability that the LotArea is above the 3rd Quartile given that the Sale Price is above the 2nd Quartile.
total<-nrow(train)
xg_yg<-nrow(subset(train,LotArea>x & SalePrice>y))
yg<-nrow(subset(train,SalePrice>y))
p1<-xg_yg/total
p2<-yg/total
a<-p1/p2
a## [1] 0.3791209b. P(X>x, Y>y)
This represents the probability that value of X is above the 3rd Quartile and Y is above 2nd Quartile. In other words this represents the probability that the LotArea is above the 3rd Quartile and the Sale Price is above the 2nd Quartile.
xg<-nrow(subset(train,LotArea>x))
yg<-nrow(subset(train,SalePrice>y))
p3 <-xg/total
p4<-yg/total
b<-p3*p4
b## [1] 0.1246575c.P(Xy)
This represents the probability that value of X is below 3rd Quartile given that Y is below 2nd Quartile. In other words this represents the probability that the LotArea is below 3rd Quartile given that the Sale Price is also below the 2nd Quartile.
xl<-nrow(subset(train,LotArea<x))
yl<-nrow(subset(train,SalePrice<y))
p3 <-xl/total
p4<-yl/total
c<-p3*p4
c## [1] 0.3739726Does splitting the training data in this fashion make them independent? In other words, does P(X|Y)=P(X)P(Y))? Check mathematically, and then evaluate by running a Chi Square test for association. You might have to research this.
# Check P(A|B) = P(A).P(B)
A<-xg
B<-yg
p6=p1/p4
p7=p3*p4
check<-(p6==p7)
check## [1] FALSEChi-sq test for Independence
H0: SalesPrice and LotArea are independent Ha: SalesPrice and LotArea are not independent
tbl = table(train$LotArea, train$SalePrice)
chisq.test(tbl)## Warning in chisq.test(tbl): Chi-squared approximation may be incorrect##
## Pearson's Chi-squared test
##
## data: tbl
## X-squared = 735090, df = 709660, p-value < 2.2e-16p-value < 2.2e-16, since p value <0.05 we reject the null hypothesis.
2.Descriptive and Inferential Statistics
Provide univariate descriptive statistics and appropriate plots for both variables. Provide a scatterplot of X and Y. Transform both variables simultaneously using Box-Cox transformations. You might have to research this. Using the transformed variables, run a correlation analysis and interpret. Test the hypothesis that the correlation between these variables is 0 and provide a 99% confidence interval. Discuss the meaning of your analysis
Summary
summary(train$SalePrice)## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 34900 130000 163000 180900 214000 755000summary(train$LotArea)## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 1300 7554 9478 10520 11600 215200Scatter Plot
plot(train$SalePrice,train$LotArea)
Histogram
hist(train$SalePrice,xlab= "Sales Price", main = "Houses Sales Price")
hist(train$LotArea,xlab= "Lot Area", main = "Houses Lot Area")
Box Cox transforamtion on Sales Price and LotArea
lambda1 <-BoxCox.lambda(train$SalePrice)
trans.SalesPrice<-BoxCox(train$SalePrice,lambda1)
hist(trans.SalesPrice)
lambda2 <-BoxCox.lambda(train$LotArea)
trans.LotArea<-BoxCox(train$LotArea,lambda2)
hist(trans.LotArea)
Correlation Anaylsis
Using the transformed variables, run a correlation analysis and interpret. Test the hypothesis that the correlation between these variables is 0 and provide a 99% confidence interval. Discuss the meaning of your analysis.
# Correlation matrix
tab<- cbind(trans.SalesPrice,trans.LotArea)
mat<-cor(tab)
mat## trans.SalesPrice trans.LotArea
## trans.SalesPrice 1.0000000 0.3893308
## trans.LotArea 0.3893308 1.0000000Test the hypothesis
Cor-relation matrix shows there is a positive cor-relation between sales price and Lot area.
cor.test(trans.SalesPrice,trans.LotArea, method = "pearson" , conf.level = 0.99)##
## Pearson's product-moment correlation
##
## data: trans.SalesPrice and trans.LotArea
## t = 16.14, df = 1458, p-value < 2.2e-16
## alternative hypothesis: true correlation is not equal to 0
## 99 percent confidence interval:
## 0.3306244 0.4450358
## sample estimates:
## cor
## 0.3893308The correlation test suggests that there is between transformed values of SalePrice and LotArea. 99 % confidence interval: 0.3306244 0.4450358
3.Linear Algebra and Correlation
Invert your correlation matrix. (This is known as the precision matrix and contains variance inflation factors on the diagonal.) Multiply the correlation matrix by the precision matrix, and then multiply the precision matrix by the correlation matrix.
# Invert correlation matrix (precision matrix)
inv<-solve(mat)
inv## trans.SalesPrice trans.LotArea
## trans.SalesPrice 1.1786593 -0.4588883
## trans.LotArea -0.4588883 1.1786593# Multiply the correlation matrix by the precision matrix
matrix1 <-mat %*% inv
matrix1## trans.SalesPrice trans.LotArea
## trans.SalesPrice 1.000000e+00 0
## trans.LotArea 5.551115e-17 1# Then multiply the precision matrix by the correlation matrix
matrix2<- inv %*% mat
matrix2## trans.SalesPrice trans.LotArea
## trans.SalesPrice 1.000000e+00 0
## trans.LotArea 5.551115e-17 14.Calculus-Based Probability & Statistics
For your non-transformed independent variable, location shift it so that the minimum value is above zero.
# shift Independent variable (LotArea) such that min value is > 0
min_Lot <- min(train$LotArea,na.rm = TRUE)
min_Lot## [1] 1300The minimum value is already greater than 0 so we need not do any shift.
Density function
Load the MASS package and run fitdistr to fit a density function of your choice. Find the optimal value of the parameters for this distribution.
df <- fitdistr(train$LotArea, 'exponential')
estimate <- df$estimateTake 1000 samples from this distribution (e.g., rexp(1000, l) for an exponential). Plot a histogram and compare it with a histogram of your non-transformed original variable
Lot_sa<- rexp(1000, estimate)
hist(Lot_sa)
# Histogram of non-trnasformed LotArea
hist(train$LotArea)
comparing the histograms we see the data is still positively skewed as in the original dataset, but with the estimations, it is more spread out.
5.Modeling
Build some type of regression model and submit your model to the competition board.
summary(train)## Id MSSubClass MSZoning LotFrontage
## Min. : 1.0 Min. : 20.0 C (all): 10 Min. : 21.00
## 1st Qu.: 365.8 1st Qu.: 20.0 FV : 65 1st Qu.: 59.00
## Median : 730.5 Median : 50.0 RH : 16 Median : 69.00
## Mean : 730.5 Mean : 56.9 RL :1151 Mean : 70.05
## 3rd Qu.:1095.2 3rd Qu.: 70.0 RM : 218 3rd Qu.: 80.00
## Max. :1460.0 Max. :190.0 Max. :313.00
## NA's :259
## LotArea Street Alley LotShape LandContour
## Min. : 1300 Grvl: 6 Grvl: 50 IR1:484 Bnk: 63
## 1st Qu.: 7554 Pave:1454 Pave: 41 IR2: 41 HLS: 50
## Median : 9478 NA's:1369 IR3: 10 Low: 36
## Mean : 10517 Reg:925 Lvl:1311
## 3rd Qu.: 11602
## Max. :215245
##
## Utilities LotConfig LandSlope Neighborhood Condition1
## AllPub:1459 Corner : 263 Gtl:1382 NAmes :225 Norm :1260
## NoSeWa: 1 CulDSac: 94 Mod: 65 CollgCr:150 Feedr : 81
## FR2 : 47 Sev: 13 OldTown:113 Artery : 48
## FR3 : 4 Edwards:100 RRAn : 26
## Inside :1052 Somerst: 86 PosN : 19
## Gilbert: 79 RRAe : 11
## (Other):707 (Other): 15
## Condition2 BldgType HouseStyle OverallQual
## Norm :1445 1Fam :1220 1Story :726 Min. : 1.000
## Feedr : 6 2fmCon: 31 2Story :445 1st Qu.: 5.000
## Artery : 2 Duplex: 52 1.5Fin :154 Median : 6.000
## PosN : 2 Twnhs : 43 SLvl : 65 Mean : 6.099
## RRNn : 2 TwnhsE: 114 SFoyer : 37 3rd Qu.: 7.000
## PosA : 1 1.5Unf : 14 Max. :10.000
## (Other): 2 (Other): 19
## OverallCond YearBuilt YearRemodAdd RoofStyle
## Min. :1.000 Min. :1872 Min. :1950 Flat : 13
## 1st Qu.:5.000 1st Qu.:1954 1st Qu.:1967 Gable :1141
## Median :5.000 Median :1973 Median :1994 Gambrel: 11
## Mean :5.575 Mean :1971 Mean :1985 Hip : 286
## 3rd Qu.:6.000 3rd Qu.:2000 3rd Qu.:2004 Mansard: 7
## Max. :9.000 Max. :2010 Max. :2010 Shed : 2
##
## RoofMatl Exterior1st Exterior2nd MasVnrType MasVnrArea
## CompShg:1434 VinylSd:515 VinylSd:504 BrkCmn : 15 Min. : 0.0
## Tar&Grv: 11 HdBoard:222 MetalSd:214 BrkFace:445 1st Qu.: 0.0
## WdShngl: 6 MetalSd:220 HdBoard:207 None :864 Median : 0.0
## WdShake: 5 Wd Sdng:206 Wd Sdng:197 Stone :128 Mean : 103.7
## ClyTile: 1 Plywood:108 Plywood:142 NA's : 8 3rd Qu.: 166.0
## Membran: 1 CemntBd: 61 CmentBd: 60 Max. :1600.0
## (Other): 2 (Other):128 (Other):136 NA's :8
## ExterQual ExterCond Foundation BsmtQual BsmtCond BsmtExposure
## Ex: 52 Ex: 3 BrkTil:146 Ex :121 Fa : 45 Av :221
## Fa: 14 Fa: 28 CBlock:634 Fa : 35 Gd : 65 Gd :134
## Gd:488 Gd: 146 PConc :647 Gd :618 Po : 2 Mn :114
## TA:906 Po: 1 Slab : 24 TA :649 TA :1311 No :953
## TA:1282 Stone : 6 NA's: 37 NA's: 37 NA's: 38
## Wood : 3
##
## BsmtFinType1 BsmtFinSF1 BsmtFinType2 BsmtFinSF2
## ALQ :220 Min. : 0.0 ALQ : 19 Min. : 0.00
## BLQ :148 1st Qu.: 0.0 BLQ : 33 1st Qu.: 0.00
## GLQ :418 Median : 383.5 GLQ : 14 Median : 0.00
## LwQ : 74 Mean : 443.6 LwQ : 46 Mean : 46.55
## Rec :133 3rd Qu.: 712.2 Rec : 54 3rd Qu.: 0.00
## Unf :430 Max. :5644.0 Unf :1256 Max. :1474.00
## NA's: 37 NA's: 38
## BsmtUnfSF TotalBsmtSF Heating HeatingQC CentralAir
## Min. : 0.0 Min. : 0.0 Floor: 1 Ex:741 N: 95
## 1st Qu.: 223.0 1st Qu.: 795.8 GasA :1428 Fa: 49 Y:1365
## Median : 477.5 Median : 991.5 GasW : 18 Gd:241
## Mean : 567.2 Mean :1057.4 Grav : 7 Po: 1
## 3rd Qu.: 808.0 3rd Qu.:1298.2 OthW : 2 TA:428
## Max. :2336.0 Max. :6110.0 Wall : 4
##
## Electrical X1stFlrSF X2ndFlrSF LowQualFinSF
## FuseA: 94 Min. : 334 Min. : 0 Min. : 0.000
## FuseF: 27 1st Qu.: 882 1st Qu.: 0 1st Qu.: 0.000
## FuseP: 3 Median :1087 Median : 0 Median : 0.000
## Mix : 1 Mean :1163 Mean : 347 Mean : 5.845
## SBrkr:1334 3rd Qu.:1391 3rd Qu.: 728 3rd Qu.: 0.000
## NA's : 1 Max. :4692 Max. :2065 Max. :572.000
##
## GrLivArea BsmtFullBath BsmtHalfBath FullBath
## Min. : 334 Min. :0.0000 Min. :0.00000 Min. :0.000
## 1st Qu.:1130 1st Qu.:0.0000 1st Qu.:0.00000 1st Qu.:1.000
## Median :1464 Median :0.0000 Median :0.00000 Median :2.000
## Mean :1515 Mean :0.4253 Mean :0.05753 Mean :1.565
## 3rd Qu.:1777 3rd Qu.:1.0000 3rd Qu.:0.00000 3rd Qu.:2.000
## Max. :5642 Max. :3.0000 Max. :2.00000 Max. :3.000
##
## HalfBath BedroomAbvGr KitchenAbvGr KitchenQual
## Min. :0.0000 Min. :0.000 Min. :0.000 Ex:100
## 1st Qu.:0.0000 1st Qu.:2.000 1st Qu.:1.000 Fa: 39
## Median :0.0000 Median :3.000 Median :1.000 Gd:586
## Mean :0.3829 Mean :2.866 Mean :1.047 TA:735
## 3rd Qu.:1.0000 3rd Qu.:3.000 3rd Qu.:1.000
## Max. :2.0000 Max. :8.000 Max. :3.000
##
## TotRmsAbvGrd Functional Fireplaces FireplaceQu GarageType
## Min. : 2.000 Maj1: 14 Min. :0.000 Ex : 24 2Types : 6
## 1st Qu.: 5.000 Maj2: 5 1st Qu.:0.000 Fa : 33 Attchd :870
## Median : 6.000 Min1: 31 Median :1.000 Gd :380 Basment: 19
## Mean : 6.518 Min2: 34 Mean :0.613 Po : 20 BuiltIn: 88
## 3rd Qu.: 7.000 Mod : 15 3rd Qu.:1.000 TA :313 CarPort: 9
## Max. :14.000 Sev : 1 Max. :3.000 NA's:690 Detchd :387
## Typ :1360 NA's : 81
## GarageYrBlt GarageFinish GarageCars GarageArea GarageQual
## Min. :1900 Fin :352 Min. :0.000 Min. : 0.0 Ex : 3
## 1st Qu.:1961 RFn :422 1st Qu.:1.000 1st Qu.: 334.5 Fa : 48
## Median :1980 Unf :605 Median :2.000 Median : 480.0 Gd : 14
## Mean :1979 NA's: 81 Mean :1.767 Mean : 473.0 Po : 3
## 3rd Qu.:2002 3rd Qu.:2.000 3rd Qu.: 576.0 TA :1311
## Max. :2010 Max. :4.000 Max. :1418.0 NA's: 81
## NA's :81
## GarageCond PavedDrive WoodDeckSF OpenPorchSF EnclosedPorch
## Ex : 2 N: 90 Min. : 0.00 Min. : 0.00 Min. : 0.00
## Fa : 35 P: 30 1st Qu.: 0.00 1st Qu.: 0.00 1st Qu.: 0.00
## Gd : 9 Y:1340 Median : 0.00 Median : 25.00 Median : 0.00
## Po : 7 Mean : 94.24 Mean : 46.66 Mean : 21.95
## TA :1326 3rd Qu.:168.00 3rd Qu.: 68.00 3rd Qu.: 0.00
## NA's: 81 Max. :857.00 Max. :547.00 Max. :552.00
##
## X3SsnPorch ScreenPorch PoolArea PoolQC
## Min. : 0.00 Min. : 0.00 Min. : 0.000 Ex : 2
## 1st Qu.: 0.00 1st Qu.: 0.00 1st Qu.: 0.000 Fa : 2
## Median : 0.00 Median : 0.00 Median : 0.000 Gd : 3
## Mean : 3.41 Mean : 15.06 Mean : 2.759 NA's:1453
## 3rd Qu.: 0.00 3rd Qu.: 0.00 3rd Qu.: 0.000
## Max. :508.00 Max. :480.00 Max. :738.000
##
## Fence MiscFeature MiscVal MoSold
## GdPrv: 59 Gar2: 2 Min. : 0.00 Min. : 1.000
## GdWo : 54 Othr: 2 1st Qu.: 0.00 1st Qu.: 5.000
## MnPrv: 157 Shed: 49 Median : 0.00 Median : 6.000
## MnWw : 11 TenC: 1 Mean : 43.49 Mean : 6.322
## NA's :1179 NA's:1406 3rd Qu.: 0.00 3rd Qu.: 8.000
## Max. :15500.00 Max. :12.000
##
## YrSold SaleType SaleCondition SalePrice
## Min. :2006 WD :1267 Abnorml: 101 Min. : 34900
## 1st Qu.:2007 New : 122 AdjLand: 4 1st Qu.:129975
## Median :2008 COD : 43 Alloca : 12 Median :163000
## Mean :2008 ConLD : 9 Family : 20 Mean :180921
## 3rd Qu.:2009 ConLI : 5 Normal :1198 3rd Qu.:214000
## Max. :2010 ConLw : 5 Partial: 125 Max. :755000
## (Other): 9# removing featues having large number of NA's
drops <- c("Street","Alley","Utilities","LandSlope","BsmtFinSF2","Heating",
"LowQualFinSF","BsmtFullBath","BsmtHalfBath","GarageYrBlt","EnclosedPorch",
"3SsnPorch","ScreenPorch","PoolArea","PoolQC","Fence","MiscFeature",
"MiscVal","YrSold","SaleType","SaleCondition","GarageQual","GarageCond",
"LotFrontage","GarageType","GarageFinish","FireplaceQu","YearRemodAdd")
new_train<-train[ , !(names(train) %in% drops)]
new_train<-na.omit(new_train)
dim(new_train)## [1] 1412 54str(new_train)## 'data.frame': 1412 obs. of 54 variables:
## $ Id : int 1 2 3 4 5 6 7 8 9 10 ...
## $ MSSubClass : int 60 20 60 70 60 50 20 60 50 190 ...
## $ MSZoning : Factor w/ 5 levels "C (all)","FV",..: 4 4 4 4 4 4 4 4 5 4 ...
## $ LotArea : int 8450 9600 11250 9550 14260 14115 10084 10382 6120 7420 ...
## $ LotShape : Factor w/ 4 levels "IR1","IR2","IR3",..: 4 4 1 1 1 1 4 1 4 4 ...
## $ LandContour : Factor w/ 4 levels "Bnk","HLS","Low",..: 4 4 4 4 4 4 4 4 4 4 ...
## $ LotConfig : Factor w/ 5 levels "Corner","CulDSac",..: 5 3 5 1 3 5 5 1 5 1 ...
## $ Neighborhood: Factor w/ 25 levels "Blmngtn","Blueste",..: 6 25 6 7 14 12 21 17 18 4 ...
## $ Condition1 : Factor w/ 9 levels "Artery","Feedr",..: 3 2 3 3 3 3 3 5 1 1 ...
## $ Condition2 : Factor w/ 8 levels "Artery","Feedr",..: 3 3 3 3 3 3 3 3 3 1 ...
## $ BldgType : Factor w/ 5 levels "1Fam","2fmCon",..: 1 1 1 1 1 1 1 1 1 2 ...
## $ HouseStyle : Factor w/ 8 levels "1.5Fin","1.5Unf",..: 6 3 6 6 6 1 3 6 1 2 ...
## $ OverallQual : int 7 6 7 7 8 5 8 7 7 5 ...
## $ OverallCond : int 5 8 5 5 5 5 5 6 5 6 ...
## $ YearBuilt : int 2003 1976 2001 1915 2000 1993 2004 1973 1931 1939 ...
## $ RoofStyle : Factor w/ 6 levels "Flat","Gable",..: 2 2 2 2 2 2 2 2 2 2 ...
## $ RoofMatl : Factor w/ 8 levels "ClyTile","CompShg",..: 2 2 2 2 2 2 2 2 2 2 ...
## $ Exterior1st : Factor w/ 15 levels "AsbShng","AsphShn",..: 13 9 13 14 13 13 13 7 4 9 ...
## $ Exterior2nd : Factor w/ 16 levels "AsbShng","AsphShn",..: 14 9 14 16 14 14 14 7 16 9 ...
## $ MasVnrType : Factor w/ 4 levels "BrkCmn","BrkFace",..: 2 3 2 3 2 3 4 4 3 3 ...
## $ MasVnrArea : int 196 0 162 0 350 0 186 240 0 0 ...
## $ ExterQual : Factor w/ 4 levels "Ex","Fa","Gd",..: 3 4 3 4 3 4 3 4 4 4 ...
## $ ExterCond : Factor w/ 5 levels "Ex","Fa","Gd",..: 5 5 5 5 5 5 5 5 5 5 ...
## $ Foundation : Factor w/ 6 levels "BrkTil","CBlock",..: 3 2 3 1 3 6 3 2 1 1 ...
## $ BsmtQual : Factor w/ 4 levels "Ex","Fa","Gd",..: 3 3 3 4 3 3 1 3 4 4 ...
## $ BsmtCond : Factor w/ 4 levels "Fa","Gd","Po",..: 4 4 4 2 4 4 4 4 4 4 ...
## $ BsmtExposure: Factor w/ 4 levels "Av","Gd","Mn",..: 4 2 3 4 1 4 1 3 4 4 ...
## $ BsmtFinType1: Factor w/ 6 levels "ALQ","BLQ","GLQ",..: 3 1 3 1 3 3 3 1 6 3 ...
## $ BsmtFinSF1 : int 706 978 486 216 655 732 1369 859 0 851 ...
## $ BsmtFinType2: Factor w/ 6 levels "ALQ","BLQ","GLQ",..: 6 6 6 6 6 6 6 2 6 6 ...
## $ BsmtUnfSF : int 150 284 434 540 490 64 317 216 952 140 ...
## $ TotalBsmtSF : int 856 1262 920 756 1145 796 1686 1107 952 991 ...
## $ HeatingQC : Factor w/ 5 levels "Ex","Fa","Gd",..: 1 1 1 3 1 1 1 1 3 1 ...
## $ CentralAir : Factor w/ 2 levels "N","Y": 2 2 2 2 2 2 2 2 2 2 ...
## $ Electrical : Factor w/ 5 levels "FuseA","FuseF",..: 5 5 5 5 5 5 5 5 2 5 ...
## $ X1stFlrSF : int 856 1262 920 961 1145 796 1694 1107 1022 1077 ...
## $ X2ndFlrSF : int 854 0 866 756 1053 566 0 983 752 0 ...
## $ GrLivArea : int 1710 1262 1786 1717 2198 1362 1694 2090 1774 1077 ...
## $ FullBath : int 2 2 2 1 2 1 2 2 2 1 ...
## $ HalfBath : int 1 0 1 0 1 1 0 1 0 0 ...
## $ BedroomAbvGr: int 3 3 3 3 4 1 3 3 2 2 ...
## $ KitchenAbvGr: int 1 1 1 1 1 1 1 1 2 2 ...
## $ KitchenQual : Factor w/ 4 levels "Ex","Fa","Gd",..: 3 4 3 3 3 4 3 4 4 4 ...
## $ TotRmsAbvGrd: int 8 6 6 7 9 5 7 7 8 5 ...
## $ Functional : Factor w/ 7 levels "Maj1","Maj2",..: 7 7 7 7 7 7 7 7 3 7 ...
## $ Fireplaces : int 0 1 1 1 1 0 1 2 2 2 ...
## $ GarageCars : int 2 2 2 3 3 2 2 2 2 1 ...
## $ GarageArea : int 548 460 608 642 836 480 636 484 468 205 ...
## $ PavedDrive : Factor w/ 3 levels "N","P","Y": 3 3 3 3 3 3 3 3 3 3 ...
## $ WoodDeckSF : int 0 298 0 0 192 40 255 235 90 0 ...
## $ OpenPorchSF : int 61 0 42 35 84 30 57 204 0 4 ...
## $ X3SsnPorch : int 0 0 0 0 0 320 0 0 0 0 ...
## $ MoSold : int 2 5 9 2 12 10 8 11 4 1 ...
## $ SalePrice : int 208500 181500 223500 140000 250000 143000 307000 200000 129900 118000 ...
## - attr(*, "na.action")=Class 'omit' Named int [1:48] 18 40 91 103 157 183 235 260 333 343 ...
## .. ..- attr(*, "names")= chr [1:48] "18" "40" "91" "103" ...Fit the multilinear regression model
attach(new_train)drops1 <- c("Street","Alley","Utilities","LandSlope","BsmtFinSF2","Heating",
"LowQualFinSF","BsmtFullBath","BsmtHalfBath","GarageYrBlt","EnclosedPorch",
"3SsnPorch","ScreenPorch","PoolArea","PoolQC","Fence","MiscFeature",
"MiscVal","YrSold","SaleType","SaleCondition","GarageQual","GarageCond",
"LotFrontage","GarageType","GarageFinish","FireplaceQu","YearRemodAdd",
"Exterior1st","BsmtFinType1","RoofStyle","Exterior2nd", "BsmtFinType2","HeatingQC","OpenPorchSF","Foundation","Electrical","WoodDeckSF", "X3SsnPorch","CentralAir","PavedDrive","MoSold","LotShape","GarageCars","HalfBath", "ExterCond","Id")
new_train<-train[ , !(names(train) %in% drops1)]
f11<-lm(SalePrice ~ .,data =new_train)
summary(f11)##
## Call:
## lm(formula = SalePrice ~ ., data = new_train)
##
## Residuals:
## Min 1Q Median 3Q Max
## -186873 -10425 376 9916 186873
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -1.419e+06 1.302e+05 -10.904 < 2e-16 ***
## MSSubClass -1.035e+02 8.325e+01 -1.243 0.213908
## MSZoningFV 3.609e+04 1.182e+04 3.053 0.002313 **
## MSZoningRH 2.342e+04 1.182e+04 1.982 0.047711 *
## MSZoningRL 2.762e+04 9.905e+03 2.788 0.005380 **
## MSZoningRM 2.567e+04 9.228e+03 2.782 0.005481 **
## LotArea 4.762e-01 8.235e-02 5.782 9.22e-09 ***
## LandContourHLS 1.182e+04 5.131e+03 2.304 0.021368 *
## LandContourLow -5.152e+03 6.328e+03 -0.814 0.415703
## LandContourLvl 6.075e+03 3.583e+03 1.695 0.090270 .
## LotConfigCulDSac 8.004e+03 3.173e+03 2.522 0.011774 *
## LotConfigFR2 -6.811e+03 4.042e+03 -1.685 0.092244 .
## LotConfigFR3 -1.306e+04 1.298e+04 -1.006 0.314537
## LotConfigInside -8.632e+02 1.759e+03 -0.491 0.623727
## NeighborhoodBlueste -4.460e+03 1.897e+04 -0.235 0.814201
## NeighborhoodBrDale -3.593e+02 1.074e+04 -0.033 0.973310
## NeighborhoodBrkSide -3.881e+03 9.246e+03 -0.420 0.674716
## NeighborhoodClearCr -1.924e+04 9.051e+03 -2.126 0.033725 *
## NeighborhoodCollgCr -1.163e+04 7.207e+03 -1.614 0.106869
## NeighborhoodCrawfor 9.586e+03 8.392e+03 1.142 0.253520
## NeighborhoodEdwards -1.893e+04 7.952e+03 -2.381 0.017418 *
## NeighborhoodGilbert -1.307e+04 7.660e+03 -1.706 0.088287 .
## NeighborhoodIDOTRR -8.972e+03 1.044e+04 -0.859 0.390515
## NeighborhoodMeadowV -7.556e+03 1.021e+04 -0.740 0.459222
## NeighborhoodMitchel -2.298e+04 8.106e+03 -2.835 0.004658 **
## NeighborhoodNAmes -1.917e+04 7.677e+03 -2.497 0.012651 *
## NeighborhoodNoRidge 2.296e+04 8.339e+03 2.754 0.005971 **
## NeighborhoodNPkVill -4.407e+02 1.068e+04 -0.041 0.967083
## NeighborhoodNridgHt 1.433e+04 7.440e+03 1.925 0.054387 .
## NeighborhoodNWAmes -2.419e+04 7.809e+03 -3.098 0.001990 **
## NeighborhoodOldTown -1.660e+04 9.486e+03 -1.750 0.080415 .
## NeighborhoodSawyer -1.413e+04 8.096e+03 -1.745 0.081163 .
## NeighborhoodSawyerW -9.892e+03 7.718e+03 -1.282 0.200189
## NeighborhoodSomerst -1.029e+03 9.033e+03 -0.114 0.909283
## NeighborhoodStoneBr 3.073e+04 8.208e+03 3.744 0.000189 ***
## NeighborhoodSWISU -1.060e+04 9.627e+03 -1.101 0.270945
## NeighborhoodTimber -1.940e+04 8.104e+03 -2.394 0.016819 *
## NeighborhoodVeenker -3.624e+03 1.016e+04 -0.357 0.721364
## Condition1Feedr 2.812e+03 4.985e+03 0.564 0.572739
## Condition1Norm 8.961e+03 4.091e+03 2.190 0.028680 *
## Condition1PosA 6.449e+03 9.915e+03 0.650 0.515506
## Condition1PosN 7.295e+03 7.340e+03 0.994 0.320455
## Condition1RRAe -1.498e+04 9.083e+03 -1.649 0.099427 .
## Condition1RRAn 8.218e+03 6.794e+03 1.210 0.226650
## Condition1RRNe -4.593e+03 1.794e+04 -0.256 0.797996
## Condition1RRNn -1.306e+03 1.254e+04 -0.104 0.917049
## Condition2Feedr -2.483e+02 2.232e+04 -0.011 0.991127
## Condition2Norm -2.568e+02 1.912e+04 -0.013 0.989288
## Condition2PosA 3.784e+04 3.273e+04 1.156 0.247849
## Condition2PosN -2.284e+05 2.690e+04 -8.491 < 2e-16 ***
## Condition2RRAe -3.466e+04 3.170e+04 -1.094 0.274348
## Condition2RRAn 2.017e+03 3.127e+04 0.065 0.948575
## Condition2RRNn 8.366e+02 2.636e+04 0.032 0.974681
## BldgType2fmCon 7.087e+03 1.237e+04 0.573 0.566925
## BldgTypeDuplex -9.197e+03 7.073e+03 -1.300 0.193719
## BldgTypeTwnhs -1.764e+04 9.927e+03 -1.777 0.075756 .
## BldgTypeTwnhsE -1.113e+04 8.985e+03 -1.239 0.215536
## HouseStyle1.5Unf 1.491e+04 7.432e+03 2.006 0.045069 *
## HouseStyle1Story 1.080e+04 4.187e+03 2.580 0.009993 **
## HouseStyle2.5Fin -1.491e+04 1.178e+04 -1.266 0.205837
## HouseStyle2.5Unf -8.926e+03 8.662e+03 -1.030 0.303007
## HouseStyle2Story -3.556e+03 3.370e+03 -1.055 0.291446
## HouseStyleSFoyer 9.696e+03 6.386e+03 1.518 0.129181
## HouseStyleSLvl 1.049e+04 5.349e+03 1.962 0.049953 *
## OverallQual 6.968e+03 9.908e+02 7.033 3.26e-12 ***
## OverallCond 6.129e+03 7.387e+02 8.297 2.64e-16 ***
## YearBuilt 4.015e+02 6.017e+01 6.673 3.70e-11 ***
## RoofMatlCompShg 6.229e+05 2.960e+04 21.044 < 2e-16 ***
## RoofMatlMembran 6.678e+05 4.061e+04 16.444 < 2e-16 ***
## RoofMatlMetal 6.341e+05 3.914e+04 16.202 < 2e-16 ***
## RoofMatlRoll 6.138e+05 3.860e+04 15.901 < 2e-16 ***
## RoofMatlTar&Grv 6.165e+05 3.111e+04 19.815 < 2e-16 ***
## RoofMatlWdShake 6.204e+05 3.188e+04 19.458 < 2e-16 ***
## RoofMatlWdShngl 6.901e+05 3.069e+04 22.489 < 2e-16 ***
## MasVnrTypeBrkFace 6.741e+03 6.833e+03 0.987 0.324035
## MasVnrTypeNone 1.260e+04 6.896e+03 1.828 0.067823 .
## MasVnrTypeStone 1.380e+04 7.233e+03 1.908 0.056624 .
## MasVnrArea 1.841e+01 5.798e+00 3.175 0.001532 **
## ExterQualFa -1.820e+04 1.004e+04 -1.813 0.070119 .
## ExterQualGd -2.241e+04 4.814e+03 -4.656 3.56e-06 ***
## ExterQualTA -2.590e+04 5.301e+03 -4.885 1.16e-06 ***
## BsmtQualFa -1.864e+04 6.229e+03 -2.992 0.002820 **
## BsmtQualGd -2.374e+04 3.280e+03 -7.237 7.78e-13 ***
## BsmtQualTA -2.239e+04 4.003e+03 -5.594 2.71e-08 ***
## BsmtCondGd 4.511e+02 5.184e+03 0.087 0.930669
## BsmtCondPo 1.389e+04 1.988e+04 0.699 0.484863
## BsmtCondTA 4.355e+03 4.060e+03 1.073 0.283623
## BsmtExposureGd 1.413e+04 3.007e+03 4.701 2.87e-06 ***
## BsmtExposureMn -4.020e+03 3.058e+03 -1.315 0.188792
## BsmtExposureNo -7.245e+03 2.190e+03 -3.308 0.000964 ***
## BsmtFinSF1 1.183e+01 4.501e+00 2.628 0.008702 **
## BsmtUnfSF -6.002e+00 4.435e+00 -1.353 0.176210
## TotalBsmtSF 2.635e+01 5.925e+00 4.447 9.45e-06 ***
## X1stFlrSF 1.339e+01 1.833e+01 0.730 0.465255
## X2ndFlrSF 3.531e+01 1.756e+01 2.011 0.044550 *
## GrLivArea 3.837e+01 1.791e+01 2.143 0.032330 *
## FullBath 3.114e+03 2.018e+03 1.543 0.123065
## BedroomAbvGr -4.883e+03 1.358e+03 -3.594 0.000337 ***
## KitchenAbvGr -1.298e+04 5.505e+03 -2.358 0.018515 *
## KitchenQualFa -2.526e+04 5.983e+03 -4.222 2.59e-05 ***
## KitchenQualGd -2.479e+04 3.457e+03 -7.170 1.25e-12 ***
## KitchenQualTA -2.580e+04 3.853e+03 -6.696 3.19e-11 ***
## TotRmsAbvGrd 1.874e+03 9.536e+02 1.965 0.049571 *
## FunctionalMaj2 -7.375e+03 1.351e+04 -0.546 0.585203
## FunctionalMin1 1.303e+03 8.734e+03 0.149 0.881411
## FunctionalMin2 4.301e+03 8.642e+03 0.498 0.618837
## FunctionalMod -7.091e+02 1.060e+04 -0.067 0.946669
## FunctionalSev -5.400e+04 2.764e+04 -1.954 0.050937 .
## FunctionalTyp 1.528e+04 7.494e+03 2.038 0.041709 *
## Fireplaces 3.121e+03 1.316e+03 2.371 0.017861 *
## GarageArea 2.145e+01 4.364e+00 4.914 1.01e-06 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 23950 on 1303 degrees of freedom
## (46 observations deleted due to missingness)
## Multiple R-squared: 0.9158, Adjusted R-squared: 0.9087
## F-statistic: 128.8 on 110 and 1303 DF, p-value: < 2.2e-16Analysis of Variance Table
anova(f11)## Analysis of Variance Table
##
## Response: SalePrice
## Df Sum Sq Mean Sq F value Pr(>F)
## MSSubClass 1 6.1248e+10 6.1248e+10 106.7616 < 2.2e-16 ***
## MSZoning 4 9.4363e+11 2.3591e+11 411.2084 < 2.2e-16 ***
## LotArea 1 4.2262e+11 4.2262e+11 736.6632 < 2.2e-16 ***
## LandContour 3 1.4999e+11 4.9995e+10 87.1462 < 2.2e-16 ***
## LotConfig 4 5.7037e+10 1.4259e+10 24.8553 < 2.2e-16 ***
## Neighborhood 24 3.5803e+12 1.4918e+11 260.0290 < 2.2e-16 ***
## Condition1 8 5.8743e+10 7.3428e+09 12.7992 < 2.2e-16 ***
## Condition2 7 4.4010e+10 6.2872e+09 10.9591 1.663e-13 ***
## BldgType 4 2.3412e+11 5.8530e+10 102.0227 < 2.2e-16 ***
## HouseStyle 7 1.1692e+11 1.6702e+10 29.1137 < 2.2e-16 ***
## OverallQual 1 1.0849e+12 1.0849e+12 1891.1184 < 2.2e-16 ***
## OverallCond 1 6.7382e+09 6.7382e+09 11.7453 0.0006290 ***
## YearBuilt 1 5.8141e+10 5.8141e+10 101.3442 < 2.2e-16 ***
## RoofMatl 7 9.9274e+10 1.4182e+10 24.7206 < 2.2e-16 ***
## MasVnrType 3 3.4936e+10 1.1645e+10 20.2989 7.373e-13 ***
## MasVnrArea 1 9.7848e+10 9.7848e+10 170.5585 < 2.2e-16 ***
## ExterQual 3 1.3479e+11 4.4928e+10 78.3143 < 2.2e-16 ***
## BsmtQual 3 1.1199e+11 3.7331e+10 65.0708 < 2.2e-16 ***
## BsmtCond 3 1.4355e+09 4.7851e+08 0.8341 0.4751423
## BsmtExposure 3 8.8566e+10 2.9522e+10 51.4596 < 2.2e-16 ***
## BsmtFinSF1 1 1.3810e+11 1.3810e+11 240.7265 < 2.2e-16 ***
## BsmtUnfSF 1 1.2944e+11 1.2944e+11 225.6197 < 2.2e-16 ***
## TotalBsmtSF 1 9.9565e+10 9.9565e+10 173.5504 < 2.2e-16 ***
## X1stFlrSF 1 1.1941e+11 1.1941e+11 208.1433 < 2.2e-16 ***
## X2ndFlrSF 1 1.7398e+11 1.7398e+11 303.2687 < 2.2e-16 ***
## GrLivArea 1 2.9766e+09 2.9766e+09 5.1886 0.0228975 *
## FullBath 1 7.3793e+06 7.3793e+06 0.0129 0.9097194
## BedroomAbvGr 1 8.6148e+09 8.6148e+09 15.0164 0.0001119 ***
## KitchenAbvGr 1 2.7553e+09 2.7553e+09 4.8028 0.0285900 *
## KitchenQual 3 3.2562e+10 1.0854e+10 18.9197 5.167e-12 ***
## TotRmsAbvGrd 1 2.8602e+09 2.8602e+09 4.9855 0.0257297 *
## Functional 6 1.5328e+10 2.5547e+09 4.4531 0.0001805 ***
## Fireplaces 1 2.7777e+09 2.7777e+09 4.8418 0.0279528 *
## GarageArea 1 1.3852e+10 1.3852e+10 24.1457 1.007e-06 ***
## Residuals 1303 7.4752e+11 5.7369e+08
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1plot(f11$residuals)
Analysis
We used multi linear regression model to predict the house prices. We employed Stepwise Backward elimination technique to get the significant features. As per the above summary, we got an R-squared value of 0.9087 which means our model is able to explain about 90% variability of the response data. As seen, most of the explanatory variables are significant (p-value < 0.05).
test <- read.csv(file="test.csv",head=TRUE,sep=",")
new_test<-test[ , !(names(test) %in% drops1)]
#new_test<-test[,'Id']
result_data <- predict(f11, new_test)
result_data<-cbind(test$Id,result_data)
colnames(result_data) <- c("Id","SalePrice")
#View(result_data)result_data<-data.frame(result_data)
for (i in 1:nrow(result_data)){
if (is.na(result_data[i,2]))
result_data[i,2]<-mean(result_data[,2], na.rm = TRUE)
}write.csv(result_data,"final.csv")Kaggle Score
My Kaggle user name is dhnanjay and my score is 0.20060
