Machine Learning - Lab 2 Badge

In this lab you will respond to a set of prompts for two parts.

Part I: Data Product

For the data product, you will interpret a different type of model – a model in a regression mode.

So far, we have specified and interpreted a classification model: one predicting a dichotomous outcome (i.e., whether students pass a course). In many cases, however, we are interested in predicting a continuous outcome (e.g., students’ number of points in a course or their score on a final exam).

While many parts of the machine learning process are the same for a regression machine learning model, one key part that is relevant to this lab is different: their interpretation. The confusion matrix we created to parse the predictive strength of our classification model does not pertain to regression machine learning models. Different metrics are used. For this lab, you will specify and interpret a regression machine learning model.

The requirements are as follows:

Change your outcome to students’ final exam performance (note: check the data dictionary for a pointer!).
Using the same data (and testing and training data sets), recipe, and workflow as you used in the case study, change the mode of your model from classification to regression and change the engine from a glm to an lm model.
Interpret your regression machine learning model in terms of three regression machine learning model metrics: MAE, MSE, and RMSE. Read about these metrics here. Similar to how we interpreted the classification machine learning metrics, focus on the substantive meaning of these statistics.

Please use the code chunk below for your code:

library(tidyverse)

## ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
## ✔ dplyr     1.1.4     ✔ readr     2.1.5
## ✔ forcats   1.0.0     ✔ stringr   1.5.1
## ✔ ggplot2   3.5.0     ✔ tibble    3.2.1
## ✔ lubridate 1.9.3     ✔ tidyr     1.3.1
## ✔ purrr     1.0.2     
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag()    masks stats::lag()
## ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors

library(tidymodels)

## ── Attaching packages ────────────────────────────────────── tidymodels 1.2.0 ──
## ✔ broom        1.0.5      ✔ rsample      1.2.1 
## ✔ dials        1.2.1      ✔ tune         1.2.0 
## ✔ infer        1.0.7      ✔ workflows    1.1.4 
## ✔ modeldata    1.3.0      ✔ workflowsets 1.1.0 
## ✔ parsnip      1.2.1      ✔ yardstick    1.3.1 
## ✔ recipes      1.0.10     
## ── Conflicts ───────────────────────────────────────── tidymodels_conflicts() ──
## ✖ scales::discard() masks purrr::discard()
## ✖ dplyr::filter()   masks stats::filter()
## ✖ recipes::fixed()  masks stringr::fixed()
## ✖ dplyr::lag()      masks stats::lag()
## ✖ yardstick::spec() masks readr::spec()
## ✖ recipes::step()   masks stats::step()
## • Use suppressPackageStartupMessages() to eliminate package startup messages

library(janitor)

## 
## Attaching package: 'janitor'
## 
## The following objects are masked from 'package:stats':
## 
##     chisq.test, fisher.test

assessments <- read_csv("data/oulad-assessments.csv")

## Rows: 173912 Columns: 10
## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## chr (3): code_module, code_presentation, assessment_type
## dbl (7): id_assessment, id_student, date_submitted, is_banked, score, date, ...
## 
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

code_module_dates <- assessments %>% 
    group_by(code_module, code_presentation) %>% 
    summarize(quantile_cutoff_date = quantile(date, probs = .25, na.rm = TRUE))

## `summarise()` has grouped output by 'code_module'. You can override using the
## `.groups` argument.

assessments_joined <- left_join(assessments, code_module_dates)

## Joining with `by = join_by(code_module, code_presentation)`

assessments_filtered <- assessments_joined %>% 
    filter(date < quantile_cutoff_date) # filter the data so only assignments before the cutoff date are included

assessments_summarized <- assessments_filtered %>% 
    mutate(weighted_score = score * weight) %>% # create a new variable that accounts for the "weight" (comparable to points) given each assignment
    group_by(id_student) %>% 
    summarize(mean_weighted_score = mean(weighted_score)) 

students <- read.csv("data/oulad-students.csv")

students <- students %>% 
    mutate(pass = ifelse(final_result == "Pass", 1, 0)) %>% # creates a dummy code
    mutate(pass = as.factor(pass)) # makes the variable a factor, helping later steps

students <- students %>% 
    mutate(imd_band = factor(imd_band, levels = c("0-10%",
                                                  "10-20%",
                                                  "20-30%",
                                                  "30-40%",
                                                  "40-50%",
                                                  "50-60%",
                                                  "60-70%",
                                                  "70-80%",
                                                  "80-90%",
                                                  "90-100%"))) %>% # this creates a factor with ordered levels
    mutate(imd_band = as.integer(imd_band)) # this changes the levels into integers based on the order of the factor levels

students_and_assessments <- left_join(assessments_summarized, students)

## Joining with `by = join_by(id_student)`

set.seed(20230712)

students_and_assessments <- students_and_assessments %>% 
    drop_na(mean_weighted_score)

train_test_split <- initial_split(students_and_assessments, prop = .50, strata = "pass")
data_train <- training(train_test_split)
data_test <- testing(train_test_split)


my_rec <- recipe(mean_weighted_score ~ disability +
                     date_registration + 
                     gender +
                     code_module +
                     pass, 
                 data = data_train) %>% 
    step_dummy(disability) %>% 
    step_dummy(gender) %>%  
    step_dummy(code_module)


# specify model
my_mod <-
    linear_reg() %>% 
    set_engine("lm") %>% # generalized linear model
    set_mode("regression") # since we are predicting a dichotomous outcome, specify classification; for a number, specify regression

# specify workflow
my_wf <-
    workflow() %>% # create a workflow
    add_model(my_mod) %>% # add the model we wrote above
    add_recipe(my_rec) # add our recipe we wrote above

fitted_model <- fit(my_wf, data = data_train)

class_metrics <- metric_set(mae, rmse)

final_fit <- last_fit(my_wf, train_test_split, metrics = class_metrics)

collect_predictions(final_fit)

## # A tibble: 12,318 × 5
##    .pred id                .row mean_weighted_score .config             
##    <dbl> <chr>            <int>               <dbl> <chr>               
##  1  699. train/test split     1                600  Preprocessor1_Model1
##  2  474. train/test split     3                698. Preprocessor1_Model1
##  3  717. train/test split     4                780  Preprocessor1_Model1
##  4  515. train/test split     5                734. Preprocessor1_Model1
##  5  312. train/test split     6                156  Preprocessor1_Model1
##  6  502. train/test split     7               1035  Preprocessor1_Model1
##  7  703. train/test split    11                410  Preprocessor1_Model1
##  8  557. train/test split    12                688. Preprocessor1_Model1
##  9  488. train/test split    14                400  Preprocessor1_Model1
## 10  669. train/test split    17                870  Preprocessor1_Model1
## # ℹ 12,308 more rows

collect_metrics(final_fit)

## # A tibble: 2 × 4
##   .metric .estimator .estimate .config             
##   <chr>   <chr>          <dbl> <chr>               
## 1 mae     standard        197. Preprocessor1_Model1
## 2 rmse    standard        256. Preprocessor1_Model1

Please add your interpretations here:

MAE: 197.4 - this means it would be about 197 points off from the weighted score, which is significant enough to warrant further fixes. Its low enough that it is able to get something out of the data and when the mean scores are higher such as the 1000 scores then the model doesn’t do too badly in that sense overall.
MSE: Since RMSE is 256.3 and its the squareroot of MSE, using 256.33^2 gives 65,705 which should be the MSE
RMSE: 256.3 - I’m not sure what to make of this one as this seems exceedingly high for the RMSE value. from what I’ve been reading and looking at the value you would want is the lower the better and even though this data set works with scoring from 0-1000 this still seems like its quite a lot.

Part II: Reflect and Plan

What is an example of an outcome related to your research interests that could be modeled using a classification machine learning model?

The first example I could think of is using classification to find if a student passes a class or hits the average of the country for their final scores. Doing this would give me an idea of whether or not a student is hitting where they’re at and if certain counties are able to having higher averages when its wealthier than if its poorer.

What is an example of an outcome related to your research interests that could be modeled using a regression machine learning model?

Regression is what I used before when talking in my discussion board about how if I want to use wealth as a reason for education going up more than thats exactly how I would use it with regression. Finding out that in counties with higher wealth and the more it goes up results in higher education scores.

Look back to the study you identified for the first machine learning lab badge activity. Was the outcome one that is modeled using a classification or a regression machine learning model? Identify which mode(s) the authors of that paper used and briefly discuss the appropriateness of their decision.

My study that I used was using a regression model in order to figure out based upon a number of characteristics such as family income, parents educational attainment, household composition, race, ethnicity, etc. in order to put that into a model that will calculate the likelihood that a person with average characteristics completes at least a two year degree. They also later on split up the data into different wealth quartiles to further emphasize the difference in wealth at specific levels. This makes sense as for my same study I also believed using regression in order to try and find a continuous variable would work much better overall at determining whether you saw an increase in one affecting an increase in the other.

Knit and Publish

Complete the following steps to knit and publish your work:

First, change the name of the author: in the YAML header at the very top of this document to your name. The YAML header controls the style and feel for knitted document but doesn’t actually display in the final output.
Next, click the knit button in the toolbar above to “knit” your R Markdown document to a HTML file that will be saved in your R Project folder. You should see a formatted webpage appear in your Viewer tab in the lower right pan or in a new browser window. Let’s us know if you run into any issues with knitting.
Finally, publish your webpage on RPubs by clicking the “Publish” button located in the Viewer Pane after you knit your document. See screenshot below.

Have fun!

Machine Learning - Lab 2 Badge

James Frye

April 12, 2024

Part I: Data Product

Part II: Reflect and Plan

Knit and Publish