Machine Learning - Lab 1 Independent Assignment
James Frye
April 2, 2024
The lab provides space to work with data and to reflect on how the concepts and techniques introduced in each lab might apply to your own research.
To earn a badge for each lab, you are required to respond to a set of prompts for two parts:
In Part I, you will extend our model by adding another variable.
In Part II, you will reflect on your understanding of key concepts and begin to think about potential next steps for your own study.
Part I: Extending our model
In this part of the badge activity, please add another variable – a variable for the number of days before the start of the module students registered. This variable will be a third predictor. By adding it, you’ll be able to examine how much more accurate your model is (if at al, as this variable might not have great predictive power). Note that this variable is a number and so no pre-processing is necessary.
In doing so, please move all of your code needed to run the analysis over from your case study file here. This is essential for your analysis to be reproducible. You may wish to break your code into multiple chunks based on the overall purpose of the code in the chunk (e.g., loading packages and data, wrangling data, and each of the machine learning steps).
library(tidyverse)## Warning: package 'tidyverse' was built under R version 4.3.3## Warning: package 'ggplot2' was built under R version 4.3.3## Warning: package 'tibble' was built under R version 4.3.3## Warning: package 'tidyr' was built under R version 4.3.3## Warning: package 'readr' was built under R version 4.3.3## Warning: package 'purrr' was built under R version 4.3.3## Warning: package 'dplyr' was built under R version 4.3.3## Warning: package 'stringr' was built under R version 4.3.3## Warning: package 'forcats' was built under R version 4.3.3## Warning: package 'lubridate' was built under R version 4.3.3## ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
## ✔ dplyr 1.1.4 ✔ readr 2.1.5
## ✔ forcats 1.0.0 ✔ stringr 1.5.1
## ✔ ggplot2 3.5.0 ✔ tibble 3.2.1
## ✔ lubridate 1.9.3 ✔ tidyr 1.3.1
## ✔ purrr 1.0.2
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag() masks stats::lag()
## ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errorslibrary(tidymodels)## Warning: package 'tidymodels' was built under R version 4.3.3## ── Attaching packages ────────────────────────────────────── tidymodels 1.2.0 ──
## ✔ broom 1.0.5 ✔ rsample 1.2.1
## ✔ dials 1.2.1 ✔ tune 1.2.0
## ✔ infer 1.0.7 ✔ workflows 1.1.4
## ✔ modeldata 1.3.0 ✔ workflowsets 1.1.0
## ✔ parsnip 1.2.1 ✔ yardstick 1.3.1
## ✔ recipes 1.0.10## Warning: package 'broom' was built under R version 4.3.3## Warning: package 'dials' was built under R version 4.3.3## Warning: package 'scales' was built under R version 4.3.3## Warning: package 'infer' was built under R version 4.3.3## Warning: package 'modeldata' was built under R version 4.3.3## Warning: package 'parsnip' was built under R version 4.3.3## Warning: package 'recipes' was built under R version 4.3.3## Warning: package 'rsample' was built under R version 4.3.3## Warning: package 'tune' was built under R version 4.3.3## Warning: package 'workflows' was built under R version 4.3.3## Warning: package 'workflowsets' was built under R version 4.3.3## Warning: package 'yardstick' was built under R version 4.3.3## ── Conflicts ───────────────────────────────────────── tidymodels_conflicts() ──
## ✖ scales::discard() masks purrr::discard()
## ✖ dplyr::filter() masks stats::filter()
## ✖ recipes::fixed() masks stringr::fixed()
## ✖ dplyr::lag() masks stats::lag()
## ✖ yardstick::spec() masks readr::spec()
## ✖ recipes::step() masks stats::step()
## • Search for functions across packages at https://www.tidymodels.org/find/library(janitor)## Warning: package 'janitor' was built under R version 4.3.3##
## Attaching package: 'janitor'
##
## The following objects are masked from 'package:stats':
##
## chisq.test, fisher.teststudents <- read_csv("data/oulad-students.csv")## Rows: 32593 Columns: 15
## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## chr (9): code_module, code_presentation, gender, region, highest_education, ...
## dbl (6): id_student, num_of_prev_attempts, studied_credits, module_presentat...
##
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.glimpse(students)## Rows: 32,593
## Columns: 15
## $ code_module <chr> "AAA", "AAA", "AAA", "AAA", "AAA", "AAA", "…
## $ code_presentation <chr> "2013J", "2013J", "2013J", "2013J", "2013J"…
## $ id_student <dbl> 11391, 28400, 30268, 31604, 32885, 38053, 4…
## $ gender <chr> "M", "F", "F", "F", "F", "M", "M", "F", "F"…
## $ region <chr> "East Anglian Region", "Scotland", "North W…
## $ highest_education <chr> "HE Qualification", "HE Qualification", "A …
## $ imd_band <chr> "90-100%", "20-30%", "30-40%", "50-60%", "5…
## $ age_band <chr> "55<=", "35-55", "35-55", "35-55", "0-35", …
## $ num_of_prev_attempts <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0…
## $ studied_credits <dbl> 240, 60, 60, 60, 60, 60, 60, 120, 90, 60, 6…
## $ disability <chr> "N", "N", "Y", "N", "N", "N", "N", "N", "N"…
## $ final_result <chr> "Pass", "Pass", "Withdrawn", "Pass", "Pass"…
## $ module_presentation_length <dbl> 268, 268, 268, 268, 268, 268, 268, 268, 268…
## $ date_registration <dbl> -159, -53, -92, -52, -176, -110, -67, -29, …
## $ date_unregistration <dbl> NA, NA, 12, NA, NA, NA, NA, NA, NA, NA, NA,…students <- students %>%
mutate(pass = ifelse(final_result == "Pass", 1, 0)) %>% # creates a new variable named "pass" and a dummy code of 1 if value of final_result equals "pass" and 0 if not
mutate(pass = as.factor(pass)) # makes the variable a factor, helping later steps
students <- students %>%
mutate(disability = as.factor(disability))
students <- students %>%
mutate(imd_band = factor(imd_band, levels = c("0-10%",
"10-20%",
"20-30%",
"30-40%",
"40-50%",
"50-60%",
"60-70%",
"70-80%",
"80-90%",
"90-100%"))) %>% # this creates a factor with ordered levels
mutate(imd_band = as.integer(imd_band)) # this changes the levels into integers based on the order of the factor levels
students## # A tibble: 32,593 × 16
## code_module code_presentation id_student gender region highest_education
## <chr> <chr> <dbl> <chr> <chr> <chr>
## 1 AAA 2013J 11391 M East Angli… HE Qualification
## 2 AAA 2013J 28400 F Scotland HE Qualification
## 3 AAA 2013J 30268 F North West… A Level or Equiv…
## 4 AAA 2013J 31604 F South East… A Level or Equiv…
## 5 AAA 2013J 32885 F West Midla… Lower Than A Lev…
## 6 AAA 2013J 38053 M Wales A Level or Equiv…
## 7 AAA 2013J 45462 M Scotland HE Qualification
## 8 AAA 2013J 45642 F North West… A Level or Equiv…
## 9 AAA 2013J 52130 F East Angli… A Level or Equiv…
## 10 AAA 2013J 53025 M North Regi… Post Graduate Qu…
## # ℹ 32,583 more rows
## # ℹ 10 more variables: imd_band <int>, age_band <chr>,
## # num_of_prev_attempts <dbl>, studied_credits <dbl>, disability <fct>,
## # final_result <chr>, module_presentation_length <dbl>,
## # date_registration <dbl>, date_unregistration <dbl>, pass <fct>set.seed(20230712)
train_test_split <- initial_split(students, prop = .80)
data_train <- training(train_test_split)
data_test <- testing(train_test_split)
my_rec <- recipe(pass ~ disability + imd_band + date_registration, data = data_train)
my_rec## ## ── Recipe ──────────────────────────────────────────────────────────────────────## ## ── Inputs## Number of variables by role## outcome: 1
## predictor: 3# specify model
my_mod <-
logistic_reg()
my_mod <-
logistic_reg() %>%
set_engine("glm") %>% # generalized linear model
set_mode("classification") # since we are predicting a dichotomous outcome, specify classification; for a number, specify regression
my_mod## Logistic Regression Model Specification (classification)
##
## Computational engine: glmmy_wf <-
workflow() %>% # create a workflow
add_model(my_mod) %>% # add the model we wrote above
add_recipe(my_rec) # add our recipe we wrote above
fitted_model <- fit(my_wf, data = data_train)
fitted_model## ══ Workflow [trained] ══════════════════════════════════════════════════════════
## Preprocessor: Recipe
## Model: logistic_reg()
##
## ── Preprocessor ────────────────────────────────────────────────────────────────
## 0 Recipe Steps
##
## ── Model ───────────────────────────────────────────────────────────────────────
##
## Call: stats::glm(formula = ..y ~ ., family = stats::binomial, data = data)
##
## Coefficients:
## (Intercept) disabilityY imd_band date_registration
## -0.667029 -0.280013 0.059134 0.001643
##
## Degrees of Freedom: 22371 Total (i.e. Null); 22368 Residual
## (3702 observations deleted due to missingness)
## Null Deviance: 29800
## Residual Deviance: 29580 AIC: 29590final_fit <- last_fit(my_mod, my_rec, train_test_split)
# collect test split predictions
final_fit %>%
collect_predictions()## # A tibble: 6,519 × 7
## .pred_class .pred_0 .pred_1 id .row pass .config
## <fct> <dbl> <dbl> <chr> <int> <fct> <chr>
## 1 0 0.640 0.360 train/test split 2 1 Preprocessor1_Model1
## 2 0 0.598 0.402 train/test split 4 1 Preprocessor1_Model1
## 3 0 0.632 0.368 train/test split 7 1 Preprocessor1_Model1
## 4 <NA> NA NA train/test split 10 1 Preprocessor1_Model1
## 5 0 0.620 0.380 train/test split 16 0 Preprocessor1_Model1
## 6 <NA> NA NA train/test split 18 1 Preprocessor1_Model1
## 7 0 0.617 0.383 train/test split 21 1 Preprocessor1_Model1
## 8 0 0.591 0.409 train/test split 24 1 Preprocessor1_Model1
## 9 0 0.537 0.463 train/test split 33 1 Preprocessor1_Model1
## 10 <NA> NA NA train/test split 35 1 Preprocessor1_Model1
## # ℹ 6,509 more rowsfinal_fit %>%
collect_predictions() %>% # see test set predictions
select(.pred_class, pass) %>% # just to make the output easier to view
mutate(correct = .pred_class == pass) # create a new variable, correct, telling us when the model was and was not correct## # A tibble: 6,519 × 3
## .pred_class pass correct
## <fct> <fct> <lgl>
## 1 0 1 FALSE
## 2 0 1 FALSE
## 3 0 1 FALSE
## 4 <NA> 1 NA
## 5 0 0 TRUE
## 6 <NA> 1 NA
## 7 0 1 FALSE
## 8 0 1 FALSE
## 9 0 1 FALSE
## 10 <NA> 1 NA
## # ℹ 6,509 more rowsfinal_fit %>%
collect_predictions() %>% # see test set predictions
select(.pred_class, pass) %>% # just to make the output easier to view
mutate(correct = .pred_class == pass) %>% # create a new variable, correct, telling us when the model was and was not correct
tabyl(correct)## correct n percent valid_percent
## FALSE 2071 0.3176868 0.3728844
## TRUE 3483 0.5342844 0.6271156
## NA 965 0.1480288 NAI kept getting an error no matter what I tried to do and so I used my_wf instead of the fitted model as the workaround to get accuracy
Previous results: 63% (0.627451)
New results: 63% (0.6271156)
How does the accuracy of this new model compare? Add a few reflections below:
Seems as though the new variable slightly decreases the accuracy of the model overall and has little predictive power as well.
Part II: Reflect and Plan
Part A: Please refer back to Breiman’s (2001) article for these three questions.
- Can you summarize the primary difference between the two cultures of statistical modeling that Breiman outlines in his paper?
Breiman says multiple times throughout the paper that data models or stochastic models have the issue of being accurate while being simple and easily interpretable. This makes them appeal to a wide variety of audiences, but can also be misleading depending on the data shown. As said by Breiman “But when a model is fit to data to draw quantitative conclusions:
• The conclusions are about the model’s mechanism, and not about nature’s mechanism.
It follows that:
• If the model is a poor emulation of nature, the conclusions may be wrong.” (Breiman, 3)
Even when the model is checked using tests such as the goodness-of-fit model which displays a 5% significance level it can still not fit the data. It can prove its accuracy in some ways, but only in extreme cases will the model not work, which means its suspect to be inaccurate depending on the data used and how the person used it. (Breiman, 4)
In contrast Breiman says algorithmic models such as neural nets, decision trees, and vector machines predict accuracy much stronger than models do, with no consideration for interpretability. As he says “Using complex predictors may be unpleasant, but the soundest path is to go for predictive accuracy first, then try to understand why.” (Breiman, 9) This is the primary reason he prefers algorithmic models over stochastic models as the accuracy is more important than the interpretability which can be figured out later.
- How has the advent of big data and machine learning affected or reinforced Breiman’s argument since the article was published?
Using AI as an example it’s very clear that having algorithmic models that can have high accuracy are extremely important in the building of AI and machine learning in general. Having something that can be interpretable can be useful in terms of understanding how it works, but for AI it mostly needs to understand the accuracy of the model itself to build off an idea. Having a self-driving car that can’t identify signs due to inaccuracy or being unable to understand routes from a GPS would get someone killed. Knowing how a model works in this case is much less important than having data to train the AI to be accurate using an algorithmic model as its brain so it can process the information that you don’t need to.
This is becoming normal as well with how AI is used when copying art pieces and voices. While you won’t understand the entire data set and the model that’s being interpreted the AI uses an algorithm to find pieces that standout and patterns. The output and how well received it is becomes its accuracy which is worked out from thousands to millions of pieces of data to find similarities in its algorithm that we could miss doing it manually.
- Breiman emphasized the importance of predictive accuracy over understanding why a method works. To what extent do you agree or disagree with this stance?
I’d have to say I agree wholeheartedly in that fact considering the advances made recently with the use of algorithmic models. People work on models that work specifically for data in order to have a high degree of accuracy to prove ideas. Being able to explain this and interpret it is getting easier and easier with the use of machine learning and AI meaning having something be inaccurate for the sake of interpretability is becoming less useful as time goes on.
Using the example from above of AI in art and voices this seems to be the case for a lot of algorithms produced by AI in order to train in patterns and recognition. Having a high degree of accuracy based on the output is much more important than it is in understanding exactly how the data itself is being interpreted, which you can find out later anyway. If you did it the other way around with data models AI would take forever to reach what it has already accomplished.
Part B:
- How good was the machine learning model you developed in the badge activity? What if you read about someone using such a model as a reviewer of research? Please add your thoughts and reflections following the bullet point below.
- Using the numbers and accuracy from the model it says that it is at a 63% accuracy rating. I’d say that 63% is too low in order to predict something and you would need more predictors or a better model in general in order to use it. This could have way too many errors to go off of it to be any use at all and would take everything the reviewer said with a grain of salt.
- How might the model be improved? Share any ideas you have at this time below:
- The model would need more data sets that could buff up the model more, or it would need better predictor variables with predictive power that correlates. Other ideas is using a different model in general like what we’ve been learning in class from classification such as the support vector machine or the k-nearest neighbor. These may not be fully useful, but they could provide another chance to find the strongest accuracy if it fits the data well enough. Better feature engineering would also help achieve better results as you could search for better predictive variables.
Part C: Use the institutional library (e.g. NU Library), Google Scholar or search engine to locate a research article, presentation, or resource that applies machine learning to an educational context aligned with your research interests. More specifically, locate a machine learning study that involves making predictions.
Provide an APA citation for your selected study.
- Braga, B., McKernan, S. M., Ratcliffe, C., & Baum, S. (2017). Wealth inequality is a barrier to education and social mobility. Washington, DC: Urban Institute.
What research questions were the authors of this study trying to address and why did they consider these questions important?
“Using data from the longitudinal Panel Study of Income Dynamics, we explore the relationship between family wealth and children’s higher education achievement and upward mobility.” (Braga, McKernan, Ratcliffe, Baum, 1) To be more specific they looked at people age 18-25 to see if they were able to get a college degree of any kind in this range.
They thought this was an important question because people with a low income or wealth in general become less likely to continue to educate themselves and get a 4 yr college degree. Knowing this they think it is important that resources are given to less fortunate people so they can dig themselves out of the poverty they face by getting educated and in turn find better paying jobs otherwise they’ll be left behind as the requirements for jobs increases.
What were the results of these analyses?
“Family wealth is strongly associated with both higher educational attainment and upward educational mobility, suggesting that family wealth is an important factor in promoting greater educational achievement. “ (Braga, McKernan, Ratcliffe, Baum, 5)
Taking info of the statistics they obtained from the study also showed that people from high-wealth families (roughly $223,500) are more than one and half times more likely to complete at least a two year degree than low-wealth families (wealth below $2000).
If your parents did not graduate from college then with the same wealth values they’re twice as likely to go and get a degree compared to low-wealth families.
Knit and Publish
Complete the following steps to knit and publish your work:
First, change the name of the
author:in the YAML header at the very top of this document to your name. The YAML header controls the style and feel for knitted document but doesn’t actually display in the final output.Next, click the knit button in the toolbar above to “knit” your R Markdown document to a HTML file that will be saved in your R Project folder. You should see a formatted webpage appear in your Viewer tab in the lower right pan or in a new browser window. Let your instructor know if you run into any issues with knitting.
Finally, publish your webpage on Rpubs by clicking the “Publish” button located in the Viewer Pane after you knit your document.
Your First Machine Learning Badge
Congratulations, you’ve completed your first badge activity! To receive credit for this assignment and earn your first official Lab Badge, submit the link on Blackboard and share with your instructor.
Once your instructor has checked your link, you will be provided a physical version of the badge below!