Machine Learning - Learning Lab 2 Badge
Meina Zhu
July 14, 2022
As a reminder, to earn a badge for each lab, you are required to respond to a set of prompts for two parts:
In Part I, you will reflect on your understanding of key concepts and begin to think about potential next steps for your own study.
In Part II, you will create a simple data product in R that demonstrates your ability to apply an analytic technique introduced in this learning lab.
Part I: Reflect and Plan
Part A:
- Like we considered after LL1, how good was the machine learning model we developed in the case study? Stepping back, how successful is this as a predictive model of students’ success in the class using data collected through roughly the first one month of the class? How might this model be used in practice?
- It was good.
- Would you be comfortable using this? What if you read about someone using such a model as a reviewer of research. Please add your thoughts and reflections following the bullet point below.
- Yes. I am. It has pretty good prediction.
- How might the model be improved? Share any ideas you have at this time below:
- I am not sure. Maybe it could be improved using different model.
Part B: Again, use the institutional library (e.g. NCSU Library), Google Scholar or search engine to locate a research article, presentation, or resource that applies machine learning to an educational context aligned with your research interests. More specifically, locate a machine learning study that involve making predictions – and, ideally, one that involved in some way engineering features from data.
Provide an APA citation for your selected study.
- Hussain, M., Zhu, W., Zhang, W., Abidi, S. M. R., & Ali, S. (2019). Using machine learning to predict student difficulties from learning session data. Artificial Intelligence Review, 52(1), 381-407.
What research questions were the authors of this study trying to address and why did they consider these questions important?
• To identify the most appropriate machine learning algorithms for predicting the difficulty an individual student would have in the next session of a digital design course based on prior session activities and the current session.
• To investigate which machine learning algorithms used in the current study are appropriate for predicting student difficulty in the next session of digital design course while using the fewest features.
What were the results of these analyses?
- The results show that ANNs and SVMs achieve higher accuracy than do other algorithms. ANNs and SVMs can easily be integrated into the TEL system; thus, instructors are expected to report improved student's performance during the subsequent session.
Part II: Data Product
For the data product, you are asked to investigate and add to our recipe a feature engineering step we did not carry out.
Run the code below through the step in which you write down the recipe.
library(tidyverse)## ── Attaching packages ─────────────────────────────────────── tidyverse 1.3.1 ──## ✔ ggplot2 3.3.5 ✔ purrr 0.3.4
## ✔ tibble 3.1.7 ✔ dplyr 1.0.9
## ✔ tidyr 1.2.0 ✔ stringr 1.4.0
## ✔ readr 2.1.2 ✔ forcats 0.5.1## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag() masks stats::lag()library(here)## here() starts at /Users/meinazhu/Documents/GitHub/machine-learninglibrary(tidymodels)## ── Attaching packages ────────────────────────────────────── tidymodels 0.2.0 ──## ✔ broom 0.7.12 ✔ rsample 1.0.0
## ✔ dials 1.0.0 ✔ tune 1.0.0
## ✔ infer 1.0.2 ✔ workflows 1.0.0
## ✔ modeldata 1.0.0 ✔ workflowsets 0.2.1
## ✔ parsnip 1.0.0 ✔ yardstick 1.0.0
## ✔ recipes 1.0.1## ── Conflicts ───────────────────────────────────────── tidymodels_conflicts() ──
## ✖ scales::discard() masks purrr::discard()
## ✖ dplyr::filter() masks stats::filter()
## ✖ recipes::fixed() masks stringr::fixed()
## ✖ dplyr::lag() masks stats::lag()
## ✖ yardstick::spec() masks readr::spec()
## ✖ recipes::step() masks stats::step()
## • Dig deeper into tidy modeling with R at https://www.tmwr.orgd <- read_csv("data/online-sci-data-joined.csv")## Rows: 10920 Columns: 25## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## chr (11): course_id, gender, enrollment_reason, enrollment_status, subject,...
## dbl (12): student_id, int, uv, percomp, tv, sum_discussion_posts, sum_n_wor...
## lgl (1): status
## time (1): last_access_date
##
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.data_with_final_grade <- read_csv("data/data-to-model-no-gradebook.csv")## Rows: 546 Columns: 16
## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## chr (7): course_id, gender, enrollment_reason, enrollment_status, subject, s...
## dbl (9): student_id, final_grade, time_spent, int, uv, percomp, tv, sum_disc...
##
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.data_with_final_grade <- data_with_final_grade %>%
select(student_id, course_id, final_grade)
d <- left_join(d, data_with_final_grade, by= c("student_id", "course_id"))
d <- d %>% distinct(student_id, course_id, .keep_all = TRUE)
set.seed(20220712)
train_test_split <- initial_split(d, prop = .80)
data_train <- training(train_test_split)
kfcv <- vfold_cv(data_train, v = 20) # this differentiates this from what we did beforeHere’s where you can add a new feature engineering step. For the sake of this badge, choose from among those options here: https://recipes.tidymodels.org/reference/index.html. You can see more - if helpful - here: https://www.tmwr.org/recipes.html
my_rec <- recipe(final_grade ~
int + uv + tv + student_id + course_id + sum_discussion_posts + sum_n_words, data = data_train) %>%
update_role(student_id,course_id,new_role = "ID variable") %>%
step_normalize(all_numeric_predictors()) %>% # standardizes numeric variables
step_nzv(all_predictors()) %>% # remove predictors with a "near-zero variance"
step_novel(all_nominal_predictors()) %>% # add a musing label for factors
step_dummy(all_nominal_predictors()) %>% # dummy code all factor variables
step_impute_knn(all_predictors()) # impute missing data for all predictor variablesRun the remaining steps.
my_mod <-
linear_reg() %>%
set_engine("lm") %>%
set_mode("regression")
my_wf <-
workflow() %>%
add_model(my_mod) %>%
add_recipe(my_rec)
fitted_model <- fit_resamples(my_wf, resamples = kfcv,
control = control_grid(save_pred = TRUE)) # this allows us to inspect the predictions
fitted_model %>%
unnest(.metrics) %>%
filter(.metric == "rmse") # we also get another metric, the ROC; we focus just on accuracy for now## # A tibble: 20 × 8
## splits id .metric .estimator .estimate .config .notes
## <list> <chr> <chr> <chr> <dbl> <chr> <list>
## 1 <split [414/22]> Fold01 rmse standard 24.8 Preprocessor1_… <tibble>
## 2 <split [414/22]> Fold02 rmse standard 22.6 Preprocessor1_… <tibble>
## 3 <split [414/22]> Fold03 rmse standard 17.8 Preprocessor1_… <tibble>
## 4 <split [414/22]> Fold04 rmse standard 20.7 Preprocessor1_… <tibble>
## 5 <split [414/22]> Fold05 rmse standard 13.3 Preprocessor1_… <tibble>
## 6 <split [414/22]> Fold06 rmse standard 22.0 Preprocessor1_… <tibble>
## 7 <split [414/22]> Fold07 rmse standard 15.0 Preprocessor1_… <tibble>
## 8 <split [414/22]> Fold08 rmse standard 20.1 Preprocessor1_… <tibble>
## 9 <split [414/22]> Fold09 rmse standard 20.1 Preprocessor1_… <tibble>
## 10 <split [414/22]> Fold10 rmse standard 26.7 Preprocessor1_… <tibble>
## 11 <split [414/22]> Fold11 rmse standard 18.8 Preprocessor1_… <tibble>
## 12 <split [414/22]> Fold12 rmse standard 19.1 Preprocessor1_… <tibble>
## 13 <split [414/22]> Fold13 rmse standard 26.2 Preprocessor1_… <tibble>
## 14 <split [414/22]> Fold14 rmse standard 19.5 Preprocessor1_… <tibble>
## 15 <split [414/22]> Fold15 rmse standard 12.8 Preprocessor1_… <tibble>
## 16 <split [414/22]> Fold16 rmse standard 16.7 Preprocessor1_… <tibble>
## 17 <split [415/21]> Fold17 rmse standard 17.8 Preprocessor1_… <tibble>
## 18 <split [415/21]> Fold18 rmse standard 24.2 Preprocessor1_… <tibble>
## 19 <split [415/21]> Fold19 rmse standard 22.4 Preprocessor1_… <tibble>
## 20 <split [415/21]> Fold20 rmse standard 30.2 Preprocessor1_… <tibble>
## # … with 1 more variable: .predictions <list>fitted_model %>%
collect_metrics()## # A tibble: 2 × 6
## .metric .estimator mean n std_err .config
## <chr> <chr> <dbl> <int> <dbl> <chr>
## 1 rmse standard 20.5 20 1.01 Preprocessor1_Model1
## 2 rsq standard 0.119 20 0.0256 Preprocessor1_Model1Did that feature engineering make any difference compared to the mean predictive accuracy you found in the case study? Add a few notes below:
Knit & Submit
Congratulations, you’ve completed your Prediction badge! Complete the following steps to submit your work for review:
Change the name of the
author:in the YAML header at the very top of this document to your name. As noted in Reproducible Research in R, The YAML header controls the style and feel for knitted document but doesn’t actually display in the final output.Click the yarn icon above to “knit” your data product to a HTML file that will be saved in your R Project folder.
Commit your changes in GitHub Desktop and push them to your online GitHub repository.
Publish your HTML page the web using one of the following publishing methods:
Publish on RPubs by clicking the “Publish” button located in the Viewer Pane when you knit your document. Note, you will need to quickly create a RPubs account.
Publishing on GitHub using either GitHub Pages or the HTML previewer.
Post a new discussion on GitHub to our ML badges forum. In your post, include a link to your published web page and a short reflection highlighting one thing you learned from this lab and one thing you’d like to explore further.