DS624_HW4_JagdishChhabria
Jagdish Chhabria
10/3/2021
3.1. The UC Irvine Machine Learning Repository contains a data set related to glass identification. The data consist of 214 glass samples labeled as one of seven class categories. There are nine predictors, including the refractive index and percentages of eight elements: Na, Mg, Al, Si, K, Ca, Ba, and Fe.
## 'data.frame': 214 obs. of 10 variables:
## $ RI : num 1.52 1.52 1.52 1.52 1.52 ...
## $ Na : num 13.6 13.9 13.5 13.2 13.3 ...
## $ Mg : num 4.49 3.6 3.55 3.69 3.62 3.61 3.6 3.61 3.58 3.6 ...
## $ Al : num 1.1 1.36 1.54 1.29 1.24 1.62 1.14 1.05 1.37 1.36 ...
## $ Si : num 71.8 72.7 73 72.6 73.1 ...
## $ K : num 0.06 0.48 0.39 0.57 0.55 0.64 0.58 0.57 0.56 0.57 ...
## $ Ca : num 8.75 7.83 7.78 8.22 8.07 8.07 8.17 8.24 8.3 8.4 ...
## $ Ba : num 0 0 0 0 0 0 0 0 0 0 ...
## $ Fe : num 0 0 0 0 0 0.26 0 0 0 0.11 ...
## $ Type: Factor w/ 6 levels "1","2","3","5",..: 1 1 1 1 1 1 1 1 1 1 ...## RI Na Mg Al
## Min. :1.511 Min. :10.73 Min. :0.000 Min. :0.290
## 1st Qu.:1.517 1st Qu.:12.91 1st Qu.:2.115 1st Qu.:1.190
## Median :1.518 Median :13.30 Median :3.480 Median :1.360
## Mean :1.518 Mean :13.41 Mean :2.685 Mean :1.445
## 3rd Qu.:1.519 3rd Qu.:13.82 3rd Qu.:3.600 3rd Qu.:1.630
## Max. :1.534 Max. :17.38 Max. :4.490 Max. :3.500
## Si K Ca Ba
## Min. :69.81 Min. :0.0000 Min. : 5.430 Min. :0.000
## 1st Qu.:72.28 1st Qu.:0.1225 1st Qu.: 8.240 1st Qu.:0.000
## Median :72.79 Median :0.5550 Median : 8.600 Median :0.000
## Mean :72.65 Mean :0.4971 Mean : 8.957 Mean :0.175
## 3rd Qu.:73.09 3rd Qu.:0.6100 3rd Qu.: 9.172 3rd Qu.:0.000
## Max. :75.41 Max. :6.2100 Max. :16.190 Max. :3.150
## Fe Type
## Min. :0.00000 1:70
## 1st Qu.:0.00000 2:76
## Median :0.00000 3:17
## Mean :0.05701 5:13
## 3rd Qu.:0.10000 6: 9
## Max. :0.51000 7:29## Glass
##
## 10 Variables 214 Observations
## --------------------------------------------------------------------------------
## RI
## n missing distinct Info Mean Gmd .05 .10
## 214 0 178 1 1.518 0.003054 1.515 1.516
## .25 .50 .75 .90 .95
## 1.517 1.518 1.519 1.522 1.524
##
## lowest : 1.51115 1.51131 1.51215 1.51299 1.51316
## highest: 1.52725 1.52739 1.52777 1.53125 1.53393
## --------------------------------------------------------------------------------
## Na
## n missing distinct Info Mean Gmd .05 .10
## 214 0 142 1 13.41 0.8685 12.41 12.68
## .25 .50 .75 .90 .95
## 12.91 13.30 13.83 14.40 14.85
##
## lowest : 10.73 11.02 11.03 11.23 11.45, highest: 14.99 15.01 15.15 15.79 17.38
## --------------------------------------------------------------------------------
## Mg
## n missing distinct Info Mean Gmd .05 .10
## 214 0 94 0.992 2.685 1.417 0.000 0.000
## .25 .50 .75 .90 .95
## 2.115 3.480 3.600 3.757 3.850
##
## lowest : 0.00 0.33 0.78 1.01 1.35, highest: 3.90 3.93 3.97 3.98 4.49
## --------------------------------------------------------------------------------
## Al
## n missing distinct Info Mean Gmd .05 .10
## 214 0 118 1 1.445 0.5311 0.696 0.870
## .25 .50 .75 .90 .95
## 1.190 1.360 1.630 2.074 2.394
##
## lowest : 0.29 0.34 0.47 0.51 0.56, highest: 2.79 2.88 3.02 3.04 3.50
## --------------------------------------------------------------------------------
## Si
## n missing distinct Info Mean Gmd .05 .10
## 214 0 133 1 72.65 0.8047 71.31 71.77
## .25 .50 .75 .90 .95
## 72.28 72.79 73.09 73.30 73.52
##
## lowest : 69.81 69.89 70.16 70.26 70.43, highest: 73.88 74.45 74.55 75.18 75.41
## --------------------------------------------------------------------------------
## K
## n missing distinct Info Mean Gmd .05 .10
## 214 0 65 0.996 0.4971 0.4385 0.0000 0.0000
## .25 .50 .75 .90 .95
## 0.1225 0.5550 0.6100 0.6800 0.7600
##
## lowest : 0.00 0.02 0.03 0.04 0.05, highest: 1.46 1.68 1.76 2.70 6.21
## --------------------------------------------------------------------------------
## Ca
## n missing distinct Info Mean Gmd .05 .10
## 214 0 143 1 8.957 1.334 7.812 7.970
## .25 .50 .75 .90 .95
## 8.240 8.600 9.172 10.443 11.561
##
## lowest : 5.43 5.79 5.87 6.47 6.65, highest: 13.44 14.40 14.68 14.96 16.19
## --------------------------------------------------------------------------------
## Ba
## n missing distinct Info Mean Gmd .05 .10
## 214 0 34 0.444 0.175 0.3157 0.00 0.00
## .25 .50 .75 .90 .95
## 0.00 0.00 0.00 0.64 1.57
##
## lowest : 0.00 0.06 0.09 0.11 0.14, highest: 1.68 1.71 2.20 2.88 3.15
## --------------------------------------------------------------------------------
## Fe
## n missing distinct Info Mean Gmd .05 .10
## 214 0 32 0.695 0.05701 0.08794 0.000 0.000
## .25 .50 .75 .90 .95
## 0.000 0.000 0.100 0.220 0.267
##
## lowest : 0.00 0.01 0.03 0.05 0.06, highest: 0.32 0.34 0.35 0.37 0.51
## --------------------------------------------------------------------------------
## Type
## n missing distinct
## 214 0 6
##
## lowest : 1 2 3 5 6, highest: 2 3 5 6 7
##
## Value 1 2 3 5 6 7
## Frequency 70 76 17 13 9 29
## Proportion 0.327 0.355 0.079 0.061 0.042 0.136
## --------------------------------------------------------------------------------From the above, we can see that none of the variables have any missing values.
###(a) Using visualizations, explore the predictor variables to understand their distributions as well as the relationships between predictors.
From the above, we can see that while some predictors such as Al and Ri are close to normal, most such as Fe and Ba are skewed, while Mg seems to be saddle-shaped.
Let’s plot the histograms using another function.
Let’s now check the correlations between the different predictors in this dataset.
From the above, we can see that Ri and Ca are strongly positively correlated, while Si and Ri are reasonably negatively correlated. All other pair-wise correlations fall in between these 2 extremes.
Let’s take a closer look at these 2 sets of correlations: Ri and CA, and Si and Ri.
###(b) Do there appear to be any outliers in the data? Are any predictors skewed?
From the box plots above, we can see that most of the predictors such as Ba, Ca, Al, CA, Fe and RI have outliers. Let’s dig deeper into which predictors have the most outliers.
| x | |
|---|---|
| Ba | 0.1775701 |
| Ca | 0.1214953 |
| Al | 0.0841121 |
| RI | 0.0794393 |
| Si | 0.0560748 |
| Fe | 0.0560748 |
| Na | 0.0327103 |
| K | 0.0327103 |
| Mg | 0.0000000 |
The above shows the % of values that are considered outliers for each of the predictors. Ba and Ca have the highest % of outliers at 17.75 and 12.14% of the number of values respectively.
We now check the skewness for each predictor.
| x | |
|---|---|
| K | 6.4600889 |
| Ba | 3.3686800 |
| Ca | 2.0184463 |
| Fe | 1.7298107 |
| RI | 1.6027151 |
| Al | 0.8946104 |
| Na | 0.4478343 |
| Si | -0.7202392 |
| Mg | -1.1364523 |
Skewness is a measure of symmetry. Negative skewness indicates that the mean of the data values is less than the median, and the data distribution is left-skewed. Positive skewness would indicates that the mean of the data values is more than the median, and the data distribution is right-skewed. Skewness values close to zero indicate symmetry in data distribution i.e. that the mean and median are close in value which is a characteristic of the normal distribution. From the above data, we can see that Na is more symmetrical, while K is highly right-skewed, followed y Ba. Mg is left-skewed. In general, most of the predictors are quite skewed.
(c) Are there any relevant transformations of one or more predictors that might improve the classification model?
BoxCox transformation or log transformation might improve the classification model. Removing outliers before doing those transformations may be the right approach for improving the classification model. Centering and scaling is another option that might improve the model performance.
## Created from 214 samples and 9 variables
##
## Pre-processing:
## - Box-Cox transformation (5)
## - centered (9)
## - ignored (0)
## - principal component signal extraction (9)
## - scaled (9)
##
## Lambda estimates for Box-Cox transformation:
## -2, -0.1, 0.5, 2, -1.1
## PCA needed 7 components to capture 95 percent of the varianceFrom the above, we can see that the BoxCox transformation was performed on 5 of the predictors, while all 9 predictors were centered and scaled, and used for the PCA feature extraction.
The above plot shows that the PCA components are much less skewed than the original predictors, so the transformation helped.
The above plot shows that none of the PCA components are correlated with each other.
3.2. The soybean data can also be found at the UC Irvine Machine Learning Repository. Data were collected to predict disease in 683 soybeans. The 35 predictors are mostly categorical and include information on the environmental conditions (e.g., temperature, precipitation) and plant conditions (e.g., left spots, mold growth). The outcome labels consist of 19 distinct classes.
The Soybean dataset is a data frame with 683 observations on 36 variables. There are 35 categorical attributes, all numerical and a nominal denoting the class. There are 19 values for the Class target variable. Of these 19, 4 classes have very little data. Of the 35 predictors, all are categorical variables - some nominal and some ordered. The value “dna” means does not apply. The values for attributes are encoded numerically, with the first value encoded as “0,” the second as “1,” and so forth.
## Class date plant.stand precip temp
## brown-spot : 92 5 :149 0 :354 0 : 74 0 : 80
## alternarialeaf-spot: 91 4 :131 1 :293 1 :112 1 :374
## frog-eye-leaf-spot : 91 3 :118 NA's: 36 2 :459 2 :199
## phytophthora-rot : 88 2 : 93 NA's: 38 NA's: 30
## anthracnose : 44 6 : 90
## brown-stem-rot : 44 (Other):101
## (Other) :233 NA's : 1
## hail crop.hist area.dam sever seed.tmt germ plant.growth
## 0 :435 0 : 65 0 :123 0 :195 0 :305 0 :165 0 :441
## 1 :127 1 :165 1 :227 1 :322 1 :222 1 :213 1 :226
## NA's:121 2 :219 2 :145 2 : 45 2 : 35 2 :193 NA's: 16
## 3 :218 3 :187 NA's:121 NA's:121 NA's:112
## NA's: 16 NA's: 1
##
##
## leaves leaf.halo leaf.marg leaf.size leaf.shread leaf.malf leaf.mild
## 0: 77 0 :221 0 :357 0 : 51 0 :487 0 :554 0 :535
## 1:606 1 : 36 1 : 21 1 :327 1 : 96 1 : 45 1 : 20
## 2 :342 2 :221 2 :221 NA's:100 NA's: 84 2 : 20
## NA's: 84 NA's: 84 NA's: 84 NA's:108
##
##
##
## stem lodging stem.cankers canker.lesion fruiting.bodies ext.decay
## 0 :296 0 :520 0 :379 0 :320 0 :473 0 :497
## 1 :371 1 : 42 1 : 39 1 : 83 1 :104 1 :135
## NA's: 16 NA's:121 2 : 36 2 :177 NA's:106 2 : 13
## 3 :191 3 : 65 NA's: 38
## NA's: 38 NA's: 38
##
##
## mycelium int.discolor sclerotia fruit.pods fruit.spots seed
## 0 :639 0 :581 0 :625 0 :407 0 :345 0 :476
## 1 : 6 1 : 44 1 : 20 1 :130 1 : 75 1 :115
## NA's: 38 2 : 20 NA's: 38 2 : 14 2 : 57 NA's: 92
## NA's: 38 3 : 48 4 :100
## NA's: 84 NA's:106
##
##
## mold.growth seed.discolor seed.size shriveling roots
## 0 :524 0 :513 0 :532 0 :539 0 :551
## 1 : 67 1 : 64 1 : 59 1 : 38 1 : 86
## NA's: 92 NA's:106 NA's: 92 NA's:106 2 : 15
## NA's: 31
##
##
## From the above, we can see that all the predictors have missing values.
## 'data.frame': 683 obs. of 36 variables:
## $ Class : Factor w/ 19 levels "2-4-d-injury",..: 11 11 11 11 11 11 11 11 11 11 ...
## $ date : Factor w/ 7 levels "0","1","2","3",..: 7 5 4 4 7 6 6 5 7 5 ...
## $ plant.stand : Ord.factor w/ 2 levels "0"<"1": 1 1 1 1 1 1 1 1 1 1 ...
## $ precip : Ord.factor w/ 3 levels "0"<"1"<"2": 3 3 3 3 3 3 3 3 3 3 ...
## $ temp : Ord.factor w/ 3 levels "0"<"1"<"2": 2 2 2 2 2 2 2 2 2 2 ...
## $ hail : Factor w/ 2 levels "0","1": 1 1 1 1 1 1 1 2 1 1 ...
## $ crop.hist : Factor w/ 4 levels "0","1","2","3": 2 3 2 2 3 4 3 2 4 3 ...
## $ area.dam : Factor w/ 4 levels "0","1","2","3": 2 1 1 1 1 1 1 1 1 1 ...
## $ sever : Factor w/ 3 levels "0","1","2": 2 3 3 3 2 2 2 2 2 3 ...
## $ seed.tmt : Factor w/ 3 levels "0","1","2": 1 2 2 1 1 1 2 1 2 1 ...
## $ germ : Ord.factor w/ 3 levels "0"<"1"<"2": 1 2 3 2 3 2 1 3 2 3 ...
## $ plant.growth : Factor w/ 2 levels "0","1": 2 2 2 2 2 2 2 2 2 2 ...
## $ leaves : Factor w/ 2 levels "0","1": 2 2 2 2 2 2 2 2 2 2 ...
## $ leaf.halo : Factor w/ 3 levels "0","1","2": 1 1 1 1 1 1 1 1 1 1 ...
## $ leaf.marg : Factor w/ 3 levels "0","1","2": 3 3 3 3 3 3 3 3 3 3 ...
## $ leaf.size : Ord.factor w/ 3 levels "0"<"1"<"2": 3 3 3 3 3 3 3 3 3 3 ...
## $ leaf.shread : Factor w/ 2 levels "0","1": 1 1 1 1 1 1 1 1 1 1 ...
## $ leaf.malf : Factor w/ 2 levels "0","1": 1 1 1 1 1 1 1 1 1 1 ...
## $ leaf.mild : Factor w/ 3 levels "0","1","2": 1 1 1 1 1 1 1 1 1 1 ...
## $ stem : Factor w/ 2 levels "0","1": 2 2 2 2 2 2 2 2 2 2 ...
## $ lodging : Factor w/ 2 levels "0","1": 2 1 1 1 1 1 2 1 1 1 ...
## $ stem.cankers : Factor w/ 4 levels "0","1","2","3": 4 4 4 4 4 4 4 4 4 4 ...
## $ canker.lesion : Factor w/ 4 levels "0","1","2","3": 2 2 1 1 2 1 2 2 2 2 ...
## $ fruiting.bodies: Factor w/ 2 levels "0","1": 2 2 2 2 2 2 2 2 2 2 ...
## $ ext.decay : Factor w/ 3 levels "0","1","2": 2 2 2 2 2 2 2 2 2 2 ...
## $ mycelium : Factor w/ 2 levels "0","1": 1 1 1 1 1 1 1 1 1 1 ...
## $ int.discolor : Factor w/ 3 levels "0","1","2": 1 1 1 1 1 1 1 1 1 1 ...
## $ sclerotia : Factor w/ 2 levels "0","1": 1 1 1 1 1 1 1 1 1 1 ...
## $ fruit.pods : Factor w/ 4 levels "0","1","2","3": 1 1 1 1 1 1 1 1 1 1 ...
## $ fruit.spots : Factor w/ 4 levels "0","1","2","4": 4 4 4 4 4 4 4 4 4 4 ...
## $ seed : Factor w/ 2 levels "0","1": 1 1 1 1 1 1 1 1 1 1 ...
## $ mold.growth : Factor w/ 2 levels "0","1": 1 1 1 1 1 1 1 1 1 1 ...
## $ seed.discolor : Factor w/ 2 levels "0","1": 1 1 1 1 1 1 1 1 1 1 ...
## $ seed.size : Factor w/ 2 levels "0","1": 1 1 1 1 1 1 1 1 1 1 ...
## $ shriveling : Factor w/ 2 levels "0","1": 1 1 1 1 1 1 1 1 1 1 ...
## $ roots : Factor w/ 3 levels "0","1","2": 1 1 1 1 1 1 1 1 1 1 ...(a) Investigate the frequency distributions for the categorical predictors. Are any of the distributions degenerate in the ways discussed earlier in this chapter?
##
## 0 1
## 532 59Degenerate predictors are defined as those that are singled-valued i.e. variance = 0, or those where the the frequency of the most frequent value is at least 20 times the frequency of the next-most frequent value. In this dataset, we use the nearZeroVar function.
## [1] 19 26 28From the above, we can see that there are 3 degenerate predictors with near zero variance i.e. leaf.mild, mycelium and sclerotia.
## [1] "leaf.mild" "mycelium" "sclerotia"###(b) Roughly 18% of the data are missing. Are there particular predictors that are more likely to be missing? Is the pattern of missing data related to the classes?
| x | |
|---|---|
| hail | 121 |
| sever | 121 |
| seed.tmt | 121 |
| lodging | 121 |
| germ | 112 |
| leaf.mild | 108 |
| fruiting.bodies | 106 |
| fruit.spots | 106 |
| seed.discolor | 106 |
| shriveling | 106 |
| leaf.shread | 100 |
| seed | 92 |
| mold.growth | 92 |
| seed.size | 92 |
| leaf.halo | 84 |
| leaf.marg | 84 |
| leaf.size | 84 |
| leaf.malf | 84 |
| fruit.pods | 84 |
| precip | 38 |
| stem.cankers | 38 |
| canker.lesion | 38 |
| ext.decay | 38 |
| mycelium | 38 |
| int.discolor | 38 |
| sclerotia | 38 |
| plant.stand | 36 |
| roots | 31 |
| temp | 30 |
| crop.hist | 16 |
| plant.growth | 16 |
| stem | 16 |
| date | 1 |
| area.dam | 1 |
| leaves | 0 |
| x | |
|---|---|
| hail | 0.1771596 |
| sever | 0.1771596 |
| seed.tmt | 0.1771596 |
| lodging | 0.1771596 |
| germ | 0.1639824 |
| leaf.mild | 0.1581259 |
| fruiting.bodies | 0.1551977 |
| fruit.spots | 0.1551977 |
| seed.discolor | 0.1551977 |
| shriveling | 0.1551977 |
| leaf.shread | 0.1464129 |
| seed | 0.1346999 |
| mold.growth | 0.1346999 |
| seed.size | 0.1346999 |
| leaf.halo | 0.1229868 |
| leaf.marg | 0.1229868 |
| leaf.size | 0.1229868 |
| leaf.malf | 0.1229868 |
| fruit.pods | 0.1229868 |
| precip | 0.0556369 |
| stem.cankers | 0.0556369 |
| canker.lesion | 0.0556369 |
| ext.decay | 0.0556369 |
| mycelium | 0.0556369 |
| int.discolor | 0.0556369 |
| sclerotia | 0.0556369 |
| plant.stand | 0.0527086 |
| roots | 0.0453880 |
| temp | 0.0439239 |
| crop.hist | 0.0234261 |
| plant.growth | 0.0234261 |
| stem | 0.0234261 |
| date | 0.0014641 |
| area.dam | 0.0014641 |
| leaves | 0.0000000 |
The above table shows that about 18% of the values are missing for 4 of the predictors.
| leaves | date | area.dam | crop.hist | plant.growth | stem | temp | roots | plant.stand | precip | stem.cankers | canker.lesion | ext.decay | mycelium | int.discolor | sclerotia | leaf.halo | leaf.marg | leaf.size | leaf.malf | fruit.pods | seed | mold.growth | seed.size | leaf.shread | fruiting.bodies | fruit.spots | seed.discolor | shriveling | leaf.mild | germ | hail | sever | seed.tmt | lodging | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 562 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 |
| 13 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 13 |
| 55 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 19 |
| 8 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 20 |
| 9 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 11 |
| 6 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 13 |
| 14 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 24 |
| 15 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 28 |
| 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 30 |
| 0 | 1 | 1 | 16 | 16 | 16 | 30 | 31 | 36 | 38 | 38 | 38 | 38 | 38 | 38 | 38 | 84 | 84 | 84 | 84 | 84 | 92 | 92 | 92 | 100 | 106 | 106 | 106 | 106 | 108 | 112 | 121 | 121 | 121 | 121 | 2337 |
The above shows that 562 records have no missing values. The remaining rows can be studied further to analyze any patterns in missing date. For example,it seems that the following 3 predictors have missing values in conjunction: seed, mold.growth and seed.size - this may be of interest to a biologist who can use domain knowledge to better understand why these 3 predictors always have missing values in conjunction with each other.
##
## Variables sorted by number of missings:
## Variable Count
## hail 0.177159590
## sever 0.177159590
## seed.tmt 0.177159590
## lodging 0.177159590
## germ 0.163982430
## leaf.mild 0.158125915
## fruiting.bodies 0.155197657
## fruit.spots 0.155197657
## seed.discolor 0.155197657
## shriveling 0.155197657
## leaf.shread 0.146412884
## seed 0.134699854
## mold.growth 0.134699854
## seed.size 0.134699854
## leaf.halo 0.122986823
## leaf.marg 0.122986823
## leaf.size 0.122986823
## leaf.malf 0.122986823
## fruit.pods 0.122986823
## precip 0.055636896
## stem.cankers 0.055636896
## canker.lesion 0.055636896
## ext.decay 0.055636896
## mycelium 0.055636896
## int.discolor 0.055636896
## sclerotia 0.055636896
## plant.stand 0.052708638
## roots 0.045387994
## temp 0.043923865
## crop.hist 0.023426061
## plant.growth 0.023426061
## stem 0.023426061
## date 0.001464129
## area.dam 0.001464129
## leaves 0.000000000The above plot re-iterates that this soybean dataset has about 82% completeness. It shows the combinations of predictors that are typically missing together or not at all. There is not a single predictor that is missing on its own - it is always groups of variables that are missing together. This may have something to do with how the data was collected and be of interest to a domain expert.
###(c) Develop a strategy for handling missing data, either by eliminating predictors or imputation.
The approach I would follow is to eliminate the 3 degenerate predictors followed by impuration of missing values for the remaining predictors.
The MICE package provides different methods for imputing missing values such as Predictive Mean Matching and CART.
| x | |
|---|---|
| Class | 0 |
| date | 0 |
| plant.stand | 0 |
| precip | 0 |
| temp | 0 |
| hail | 0 |
| crop.hist | 0 |
| area.dam | 0 |
| sever | 0 |
| seed.tmt | 0 |
| germ | 0 |
| plant.growth | 0 |
| leaves | 0 |
| leaf.halo | 0 |
| leaf.marg | 0 |
| leaf.size | 0 |
| leaf.shread | 0 |
| leaf.malf | 0 |
| stem | 0 |
| lodging | 0 |
| stem.cankers | 0 |
| canker.lesion | 0 |
| fruiting.bodies | 0 |
| ext.decay | 0 |
| int.discolor | 0 |
| fruit.pods | 0 |
| fruit.spots | 0 |
| seed | 0 |
| mold.growth | 0 |
| seed.discolor | 0 |
| seed.size | 0 |
| shriveling | 0 |
| roots | 0 |
In order to compare the different imputation methods, one would need to check the impact of the imputation on the variance of the predictor, as well as it’s impact on the target variable.