Machine Learning - Learning Lab 2 Badge

As a reminder, to earn a badge for each lab, you are required to respond to a set of prompts for two parts:

In Part I, you will reflect on your understanding of key concepts and begin to think about potential next steps for your own study.
In Part II, you will create a simple data product in R that demonstrates your ability to apply an analytic technique introduced in this learning lab.

Part I: Reflect and Plan

Part A:

Like we considered after LL1, how good was the machine learning model we developed in the case study? Stepping back, how successful is this as a predictive model of students’ success in the class using data collected through roughly the first one month of the class? How might this model be used in practice?

Hard for me to say, I’m not sure how to judge the size of RMSE.

Would you be comfortable using this? What if you read about someone using such a model as a reviewer of research. Please add your thoughts and reflections following the bullet point below.

I would need to look further into how to judge in a similar context whether or not the predictive accuracy was good for such a context. I would also not use it for any kind of interpretation given that this method is not to be used that way and as a reviewer I would react negatively to anyone trying to use this model in that way.

How might the model be improved? Share any ideas you have at this time below:

Additional features. Assuming we can’t add new variables out of whole cloth, my first thought would be to look at adding interaction effects between the variables in the model. I would also want to make sure that all of the preprocessing is optimized for the variables that we do have since I am new to the way that the recipes package handles conducting these steps.

Part B: Again, use the institutional library (e.g. NCSU Library), Google Scholar or search engine to locate a research article, presentation, or resource that applies machine learning to an educational context aligned with your research interests. More specifically, locate a machine learning study that involve making predictions – and, ideally, one that involved in some way engineering features from data.

Provide an APA citation for your selected study.
- Bhutto, E. S., Siddiqui, I. F., Arain, Q. A., & Anwar, M. (2020, February). Predicting students’ academic performance through supervised machine learning. In 2020 International Conference on Information Science and Communication Technology (ICISCT) (pp. 1-6). IEEE.
What research questions were the authors of this study trying to address and why did they consider these questions important?
- How can machine learning predict students academic performance using features drawn from a LMS, including student fixed information as well as LMS behavior related features?
What were the results of these analyses?
- They predicted student performance based on three categories based on final scores: bad, average, and good. They were able to do this with 79% accuracy with a support vector machine and 73% accuracy with logistic regression.

Part II: Data Product

For the data product, you are asked to investigate and add to our recipe a feature engineering step we did not carry out.

Run the code below through the step in which you write down the recipe.

library(tidyverse)
library(here)
library(tidymodels)

d <- read_csv("data/online-sci-data-joined.csv")

data_with_grades <-  read_csv("data/data-to-model-no-gradebook.csv")

set.seed(20220712)

train_test_split <- initial_split(d, prop = .80)

data_train <- training(train_test_split)

kfcv <- vfold_cv(data_train, v = 20) # this differentiates this from what we did before

Here’s where you can add a new feature engineering step. For the sake of this badge, choose from among those options here: https://recipes.tidymodels.org/reference/index.html. You can see more - if helpful - here: https://www.tmwr.org/recipes.html

my_rec <- recipe(final_grade ~ ., data = data_train) %>% 
    step_normalize(all_numeric_predictors()) %>% # standardizes numeric variables
    step_nzv(all_predictors()) %>% # remove predictors with a "near-zero variance"
    step_dummy(all_nominal_predictors()) %>%  # dummy code all factor variables
    step_novel(all_nominal_predictors()) %>% # add a musing label for factors
    #step_interact(all_numeric_predictors()) %>%
    step_impute_knn(all_predictors()) # impute missing data for all predictor variables

my_rec <- my_rec %>% 
    update_role(student_id, new_role = "id variable") %>%
    update_role(course_id, new_role = "id variable")

my_rec

## Recipe
## 
## Inputs:
## 
##         role #variables
##  id variable          2
##      outcome          1
##    predictor         15
## 
## Operations:
## 
## Centering and scaling for all_numeric_predictors()
## Sparse, unbalanced variable filter on all_predictors()
## Dummy variables from all_nominal_predictors()
## Novel factor level assignment for all_nominal_predictors()
## K-nearest neighbor imputation for all_predictors()

Run the remaining steps.

my_mod <-
    linear_reg() %>% 
    set_engine("glm") %>%
    set_mode("regression")

my_wf <-
    workflow() %>%
    add_model(my_mod) %>% 
    add_recipe(my_rec)

fitted_model_resamples <- fit_resamples(my_wf, resamples = kfcv,
                              control = control_grid(save_pred = TRUE))#,
                              #extract = extract_preprocessor()) # this allows us to inspect the predictions


fitted_model_resamples %>% collect_metrics()

## # A tibble: 2 × 6
##   .metric .estimator  mean     n std_err .config             
##   <chr>   <chr>      <dbl> <int>   <dbl> <chr>               
## 1 rmse    standard   8.98     20  0.561  Preprocessor1_Model1
## 2 rsq     standard   0.818    20  0.0213 Preprocessor1_Model1

fitted_model_resamples %>% 
    unnest(.metrics) %>% 
    filter(.metric == "rmse") # we also get another metric, the ROC; we focus just on accuracy for now

## # A tibble: 20 × 8
##    splits           id     .metric .estimator .estimate .config         .notes  
##    <list>           <chr>  <chr>   <chr>          <dbl> <chr>           <list>  
##  1 <split [414/22]> Fold01 rmse    standard       10.7  Preprocessor1_… <tibble>
##  2 <split [414/22]> Fold02 rmse    standard        9.38 Preprocessor1_… <tibble>
##  3 <split [414/22]> Fold03 rmse    standard        7.99 Preprocessor1_… <tibble>
##  4 <split [414/22]> Fold04 rmse    standard        5.25 Preprocessor1_… <tibble>
##  5 <split [414/22]> Fold05 rmse    standard        7.68 Preprocessor1_… <tibble>
##  6 <split [414/22]> Fold06 rmse    standard       10.1  Preprocessor1_… <tibble>
##  7 <split [414/22]> Fold07 rmse    standard        8.74 Preprocessor1_… <tibble>
##  8 <split [414/22]> Fold08 rmse    standard        5.41 Preprocessor1_… <tibble>
##  9 <split [414/22]> Fold09 rmse    standard        6.48 Preprocessor1_… <tibble>
## 10 <split [414/22]> Fold10 rmse    standard        6.72 Preprocessor1_… <tibble>
## 11 <split [414/22]> Fold11 rmse    standard        7.29 Preprocessor1_… <tibble>
## 12 <split [414/22]> Fold12 rmse    standard        7.96 Preprocessor1_… <tibble>
## 13 <split [414/22]> Fold13 rmse    standard       11.6  Preprocessor1_… <tibble>
## 14 <split [414/22]> Fold14 rmse    standard       11.4  Preprocessor1_… <tibble>
## 15 <split [414/22]> Fold15 rmse    standard        6.60 Preprocessor1_… <tibble>
## 16 <split [414/22]> Fold16 rmse    standard        8.91 Preprocessor1_… <tibble>
## 17 <split [415/21]> Fold17 rmse    standard        8.46 Preprocessor1_… <tibble>
## 18 <split [415/21]> Fold18 rmse    standard       11.5  Preprocessor1_… <tibble>
## 19 <split [415/21]> Fold19 rmse    standard       13.5  Preprocessor1_… <tibble>
## 20 <split [415/21]> Fold20 rmse    standard       14.0  Preprocessor1_… <tibble>
## # … with 1 more variable: .predictions <list>

Did that feature engineering make any difference compared to the mean predictive accuracy you found in the case study? Add a few notes below:

Barely, the predictive accuracy was slightly better than in the case study, 9.1% mean RMSE vs 8.98% in this case.

Knit & Submit

Congratulations, you’ve completed your Prediction badge! Complete the following steps to submit your work for review:

Change the name of the author: in the YAML header at the very top of this document to your name. As noted in Reproducible Research in R, The YAML header controls the style and feel for knitted document but doesn’t actually display in the final output.
Click the yarn icon above to “knit” your data product to a HTML file that will be saved in your R Project folder.
Commit your changes in GitHub Desktop and push them to your online GitHub repository.
Publish your HTML page the web using one of the following publishing methods:
- Publish on RPubs by clicking the “Publish” button located in the Viewer Pane when you knit your document. Note, you will need to quickly create a RPubs account.
- Publishing on GitHub using either GitHub Pages or the HTML previewer.
Post a new discussion on GitHub to our ML badges forum. In your post, include a link to your published web page and a short reflection highlighting one thing you learned from this lab and one thing you’d like to explore further.

Machine Learning - Learning Lab 2 Badge

Lexi Lishinski

July 14, 2022

Part I: Reflect and Plan

Part II: Data Product

Knit & Submit