a)

Using visualizations, explore the predictor variables to understand their distributions as well as the relationships between predictors.

b)

Do there appear to be any outliers in the data? Are any predictors skewed?

From the above graphs, we can deduct that there seems to be some outliers present in the data, also as previously presented, some predictors are moderately skewed.

c)

Are there any relevant transformations of one or more predictors that might improve the classification model?

I believe that some relevant transformations might improve the classification model.

For this approach, I will administer a series of transformations to multiple data sets, the caret class preProcess has the ability to transform, center, scale, or impute values, as well as apply the spatial sign transformation and feature extraction. The function calculates the required quantities for the transformation. After calling the preProcess function, the predict method applies the results to a set of data, Kuhn, M. & Johnson, K (2018).

trans <- preProcess(segData, method = c("BoxCox", "center", "scale", "pca"))

## Created from 214 samples and 9 variables
## 
## Pre-processing:
##   - Box-Cox transformation (5)
##   - centered (9)
##   - ignored (0)
##   - principal component signal extraction (9)
##   - scaled (9)
## 
## Lambda estimates for Box-Cox transformation:
## -2, -0.1, 0.5, 2, -1.1
## PCA needed 7 components to capture 95 percent of the variance

The primary advantage of PCA, and the reason that it has retained its popularity as a data reduction method, is that it creates components that are uncorrelated.

The subsequent PCs are derived such that these linear combinations capture the most remaining variability while also being uncorrelated with all previous PCs.

Let’s see what transformation was performed on each component.

## $BoxCox
## [1] "RI" "Na" "Al" "Si" "Ca"
## 
## $center
## [1] "RI" "Na" "Mg" "Al" "Si" "K"  "Ca" "Ba" "Fe"
## 
## $scale
## [1] "RI" "Na" "Mg" "Al" "Si" "K"  "Ca" "Ba" "Fe"
## 
## $pca
## [1] "RI" "Na" "Mg" "Al" "Si" "K"  "Ca" "Ba" "Fe"
## 
## $ignore
## character(0)

Let’s visualize the results after the transformations.

transformed <- predict(trans, segData)

Now, let’s visualize the underlying correlation relationships between Principal Component predictors. For this, I will make use of the corrplot function from the corrplot library. The idea: they should be low correlated.

In effect, after the above transformations, the correlations in between the PCs are very small (near zero).

a)

Investigate the frequency distributions for the categorical predictors. Are any of the distributions degenerate in the ways discussed earlier in this chapter?

Let’s see what ?Soybean express. The below results are given with the help() function from R.

Description

There are 19 classes, only the first 15 of which have been used in prior work. The folklore seems to be that the last four classes are unjustified by the data since they have so few examples. There are 35 categorical attributes, some nominal and some ordered. The value “dna” means does not apply. The values for attributes are encoded numerically, with the first value encoded as “0,” the second as “1,” and so forth.

Usage

data(Soybean)

Format

A data frame with 683 observations on 36 variables. There are 35 categorical attributes, all numerical and a nominal denoting the class.

[,1]    Class   the 19 classes
[,2]    date    apr(0),may(1),june(2),july(3),aug(4),sept(5),oct(6).
[,3]    plant.stand normal(0),lt-normal(1).
[,4]    precip  lt-norm(0),norm(1),gt-norm(2).
[,5]    temp    lt-norm(0),norm(1),gt-norm(2).
[,6]    hail    yes(0),no(1).
[,7]    crop.hist   dif-lst-yr(0),s-l-y(1),s-l-2-y(2), s-l-7-y(3).
[,8]    area.dam    scatter(0),low-area(1),upper-ar(2),whole-field(3).
[,9]    sever   minor(0),pot-severe(1),severe(2).
[,10]   seed.tmt    none(0),fungicide(1),other(2).
[,11]   germ    90-100%(0),80-89%(1),lt-80%(2).
[,12]   plant.growth    norm(0),abnorm(1).
[,13]   leaves  norm(0),abnorm(1).
[,14]   leaf.halo   absent(0),yellow-halos(1),no-yellow-halos(2).
[,15]   leaf.marg   w-s-marg(0),no-w-s-marg(1),dna(2).
[,16]   leaf.size   lt-1/8(0),gt-1/8(1),dna(2).
[,17]   leaf.shread absent(0),present(1).
[,18]   leaf.malf   absent(0),present(1).
[,19]   leaf.mild   absent(0),upper-surf(1),lower-surf(2).
[,20]   stem    norm(0),abnorm(1).
[,21]   lodging yes(0),no(1).
[,22]   stem.cankers    absent(0),below-soil(1),above-s(2),ab-sec-nde(3).
[,23]   canker.lesion   dna(0),brown(1),dk-brown-blk(2),tan(3).
[,24]   fruiting.bodies absent(0),present(1).
[,25]   ext.decay   absent(0),firm-and-dry(1),watery(2).
[,26]   mycelium    absent(0),present(1).
[,27]   int.discolor    none(0),brown(1),black(2).
[,28]   sclerotia   absent(0),present(1).
[,29]   fruit.pods  norm(0),diseased(1),few-present(2),dna(3).
[,30]   fruit.spots absent(0),col(1),br-w/blk-speck(2),distort(3),dna(4).
[,31]   seed    norm(0),abnorm(1).
[,32]   mold.growth absent(0),present(1).
[,33]   seed.discolor   absent(0),present(1).
[,34]   seed.size   norm(0),lt-norm(1).
[,35]   shriveling  absent(0),present(1).
[,36]   roots   norm(0),rotted(1),galls-cysts(2).

Preamble: A rule of thumb for detecting near-zero variance predictors is:

• The fraction of unique values over the sample size is low (say 10 %).
• The ratio of the frequency of the most prevalent value to the frequency of the second most prevalent value is large (say around 20).

If both of these criteria are true and the model in question is susceptible to this type of predictor, it may be advantageous to remove the variable from the model.

Now, from the above graphs, we can quickly identify some possible distributions that fit the above rule of thumb, some of the possible distributions are: leaf.malf, leaf.mild, lodging, mycelium, int.discolor and sclerotia.

Let’s read some of the calculations for those distributions

leaf.malf

From the above, we can find a ratio of 554 / 45 = 12, which is a good value in order to keep this data set.

leaf.mild

From the above, we can find a ratio of 535 / 20 = 27, which is NOT a good value in order to keep this data set.

lodging

From the above, we can find a ratio of 520 / 42 = 12, which is a good value in order to keep this data set.

mycelium

From the above, we can find a ratio of 639 / 6 = 106, which is NOT a good value in order to keep this data set.

int.discolor

From the above, we can find a ratio of 581 / 44 = 13, which is a good value in order to keep this data set.

sclerotia

From the above, we can find a ratio of 625 / 20 = 31, which is NOT a good value in order to keep this data set.

As reported above, in effect, there are some distributions that are degenerate.

b)

Roughly 18% of the data are missing. Are there particular predictors that are more likely to be missing? Is the pattern of missing data related to the classes?

In order to answer, I would like to have a visualization of the missing data. For this purpose, I will make use of the function vis_miss from the library naniar.

As we can notice, it seems that diverse blocks of Observations present missing data. The real amount of missing data is about 9.5%; but the reference made on the beginning is for the case in which complete cases are considered; that is, about 562 observations are considered complete cases with no missing data; thus out of the 683 observations, represent about 82 % of complete data, which means 18% is missing from the complete cases.

Also, from the above visualization we can quickly identify that some predictors are more susceptible to present missing data compare to others.

Let’s examine, if the classes have some sort of relationship compared to the missing data.

First, let’s examine the observations that have missing data, that is, there are 121 records with missing values on them.

Now, let’s see the frequency of classes in the missing data.

Let’s read the respective frequency distribution values:

From the above table, we can confirm that in effect, the pattern of missing data related to the classes.

c)

Develop a strategy for handling missing data, either by eliminating predictors or imputation.

From a), we can already conclude that is better to eliminate some of the predictors since the information held on them produces a very low variance and is better just to remove them. Those are: leaf.mild, mycelium and sclerotia.

Now, since the classes some of the classes represent only observations with missing data, it would not be wise to remove those observations, otherwise we run the risk of not predicting this class as all, since we will not have any meaningful way to predict.

Let’s see the respective frequency distributions for the missing data.

Now that we have visual representations, is important to note that informative missingness can induce significant bias in the model, hence, from my perspective, is safe to continue as follows: