Data preparation for Q1

For the Data preparation for K-means Clustering algorithm the following data preparation steps were made and the reasons are provided below

• Remove Initial: Character value that identifies patient with which will provide no bearing on the model output

• Removal totalized features: Remove features that are summations of other features to reduce redundancy and noise in the dataset

• Removal Psych_meds.: Removed due to large amount of missing data (>60% observations missing)

• Imputation of missing data with KNN: the remaining data was imputed with K-nearestneighbors (KNN) as a way to fill in missing gaps. alternative methods include median, mean, or bag imputations but it was determed that KNN provides the best results with minimal effect on computational effort.

• Numeric to Factor Conversions: Several variables were with low distribution were converted into classes based on their categorical nature displayed in the histograms presented in the Data Exploration section. This conversion was made to all variables in the dataset except for Age, ADHD.Total, and MD.Total.

• Dummifying Variables: Newly transformed categorical variables were binarized into 0/1. This is particularly important for k-means because k-means will not be able to distinguish the eucliiand distances properly between classes that span more than 2 categories. For example a feature with categories 1,2,3 will not properly be computed in k-means because 1,3 will measure greater distances than 1,2, thus binarizing each of these categories such that for example 3 would be its own column with 0/1 for presence/absence is absolutely necessary.

• Normalization: Again, due to the euclidian nature of k-means clustering, features need to be normalized such that the distances they are centered and scaled the mean is 0 and the Stdev is 1, this scales all the data to allow kmeans to appropriately place centroids and observations at appropriate distances.

• Colinearity test: Colinearity was tested and it was determined that there was not sufficient colinearity between any variables such that they needed to be removed for this reason alone.

• Removed low-variance features: Removing any extremely low-variance data that will not provide useful data to the model and will only contribute to noise. At first glance, From Data Exploration section Stimulants seems like a low-variance variable with majority of categories recorded at 0. This will be confirmed statistically with tidymodels. Based on the model adjustment below, there were many features that were too sparse to provide valuable information to the model that including but not limited to: Race_X3, Race_X6, ADHD_Q5_X5, Alcohol_X0.6 and more. The total amount of features used in model after removing sparse parameters went from 238 to 147 The model recipe is shown below.

Q1_ADHD<-ADHD_data %>% recipe(~.) %>% 
  step_rm(Initial, Psych_meds.) %>% 
  step_rm(contains("total")) %>% 
  step_impute_knn(all_predictors()) %>% 
  step_mutate_at(-Age, fn = ~ as.factor(.)) %>% 
  step_dummy(all_nominal(), one_hot = T) %>% 
  step_normalize(all_predictors()) %>%
  step_nzv(all_predictors()) %>% 
  step_corr(all_predictors()) %>% 
  prep() #%>% 

Q1_ADHD 

## Data Recipe
## 
## Inputs:
## 
##       role #variables
##  predictor         54
## 
## Training data contained 175 data points and 120 incomplete rows. 
## 
## Operations:
## 
## Variables removed Initial, Psych_meds. [trained]
## Variables removed ADHD_Total, MD_TOTAL [trained]
## K-nearest neighbor imputation for Sex, Race, ADHD_Q1, ADHD_Q2, ADHD_Q3, ... [trained]
## Variable mutation for Sex, Race, ADHD_Q1, ADHD_Q2, ... [trained]
## Dummy variables from Sex, Race, ADHD_Q1, ADHD_Q2, ADHD_Q3, ADHD_Q4, ... [trained]
## Centering and scaling for Age, Sex_X1, Sex_X2, Race_X1, Race_X2, ... [trained]
## Sparse, unbalanced variable filter removed Race_X3, Race_X6, ... [trained]
## Correlation filter removed Race_X2, Sex_X1, MD_Q1a_X0, ... [trained]

After applying all those data transformations to the ADHD data, the following table was produced with 147 variables and 175 observations of those variables.

Q1_ADHD<- Q1_ADHD %>%  bake(ADHD_data)

Q1_ADHD %>% kable() %>% 
  scroll_box(height = "200px", width = "800px")