Presentation information

• Link to video that goes with this presentation: https://youtu.be/aql1u3Wi8Dg

• Link to these slides: https://rpubs.com/AnshulKumar/ClassifyStudents1 (not case sensitive)

• Most of the R code used to make this document can be found here: https://rpubs.com/anshulkumar/EducAnalytics1

• Pressing keys on your keyboard, you can modify how you view this presentation:

BE SURE TO PAUSE THE VIDEO ANY TIME YOU FEEL LIKE IT.

Sorting mechanism example – Colander filter for spaghetti

Learning analytics with students

This presentation uses an example from:

• P. Cortez and A. Silva. Using Data Mining to Predict Secondary School Student Performance. In A. Brito and J. Teixeira Eds., Proceedings of 5th FUture BUsiness TEChnology Conference (FUBUTEC 2008) pp. 5-12, Porto, Portugal, April, 2008, EUROSIS, ISBN 978-9077381-39-7. http://www3.dsi.uminho.pt/pcortez/student.pdf.

• Download the data from: Student Performance Data Set. UCI Machine Learning Repository. Center for Machine Learning and Intelligent Systems. https://archive.ics.uci.edu/ml/datasets/Student+Performance. Download the file student.zip, then open the file student-por.csv to see the data.

• Also available at kaggle.com: https://www.kaggle.com/larsen0966/student-performance-data-set

Full analytics process – Roadmap

Order: green, black, red

Data summary and procedure

Here’s what we do and don’t know for each cohort:

LAST YEAR’s data is complete. THIS YEAR’s data is incomplete.

Our procedure:

1. Look for patterns in LAST YEAR’s data.

2. Predict final exam results for THIS YEAR’s students, using only their demographic information and mid-term grades.

3. Provide extra support to at-risk students in THIS YEAR’s cohort.

How can we achieve our goal?

• We need a sorting mechanism that we can “pour” our students into.

• The sorting mechanism will tell us who is predicted to fail the course (at risk students) and who is predicted to pass.

What is a decision tree?

Feature (variable) importance in decision tree

Test accuracy with the 162 students we left out before

162 students from LAST YEAR were not used to create the decision tree model. We know their final grades.

Model testing procedure:

1. Pretend that these students were not part of LAST YEAR’s cohort.

2. Ask computer to predict their final grades.

3. But wait! We actually know their final grades.

4. Compare true final grades to predicted final grades.

5. If the predictions were close enough, use this model to make predictions for THIS YEAR’s students.

Confusion matrix – Predicted results

Out of the 162 students used to test the model:

• 24 people who actually failed were correctly predicted to fail
• 12 people who actually passed were incorrectly predicted to fail
• 5 people who actually failed incorrectly were predicted to pass
• 121 people who actually passed were correctly predicted to pass

Pass/fail cutoff for predictions: 10

Success metrics – cutoff = 10

.

\[\text{accuracy} = \frac{\text{correct predictions}}{\text{total predictions attempted}} = \frac{24+121}{162} = .895\]

.

\[\text{students to remediate} = 24+12 = 36\] .

\[\text{students who fell through the cracks} = 5\] .

Pass/fail cutoff for predictions: 10

PAUSE TO REVIEW IF YOU WANT

Success metrics – cutoff = 11.5 (increased)

.

\[\text{accuracy} = \frac{\text{correct predictions}}{\text{total predictions attempted}} = \frac{29+77}{162} = .654\]

.

\[\text{students to remediate} = 29+56 = 85\] .

\[\text{students who fell through the cracks} = 0\] .

Pass/fail cutoff for predictions: 11.5

PAUSE TO REVIEW IF YOU WANT

Side-by-side comparison

Which model is best?

• Depends on your needs and resources

• Maybe use multiple models
  • Warn people at risk in cutoff = 11.5 model
  • Give a lot of attention to those at risk in cutoff = 10 model.

• Be sure to combine with other sources of information

Full analytics process – What did we just do?

Order: green, black, red

A new project at MGHIHP

• The PA and HPEd programs at MGHIHP are currently working together to apply this to a project aimed at improving student outcomes in the PA program.

• Goal: Increase students’ chances of passing the PANCE (Physician Assistant National Certifying Exam) in cohorts graduating in 2020 and 2021.

• Data: All data from TBL-based curriculum for cohorts graduating 2017–2019.

• TBL = team based learning. Data is collected on student progress very frequently.

A few technical terms – Optional, pause and read

• Independent variables: These are all of the data that we are using to make a prediction. In our case, these are all of the data other than G3.

• Dependent variable: This what you are trying to predict, the outcome you care about. In our case, this is the final exam grade, G3.

• Machine learning: Machine learning is a group of analysis techniques that help us do predictive analytics. Machine learning and predictive analytics are subsets of artificial intelligence and statistical analysis.

• There are many many machine learning methods that were not demonstrated in this presentation. They include: logistic regression, k-nearest neighbors, random forest, Naive Bayes, support vector machines. All use different strategies to divide up the data and make predictions. In all cases, we still train and test a predictive model before using it to make new predictions.

Notes

• Assistance for this presentation came from a number of people in the HPEd and PA programs at MGHIHP.

• This presentation was created for the students of the MS and PhD programs in HPEd (health professions education) at MGHIHP.

Discussion – Pause to read and discuss later

Here are some questions to consider:

• How can predictive/learning analytics help you in your own work as an educator or at your organization/institution? What predictions would be useful for you to make?

• How can you leverage data that your institution already collects (or that it is well-positioned to collect) using predictive analytic methods?

• What would be the benefits and detriments of incorporating predictive analytics into your institution’s practices and processes?

• Could predictive analysis complement any already-ongoing initiatives at your institution?

• What would the ethical implications be of using predictive analytics at your institution? Would it cause unfair discrimination against particular learners? Would it help level the playing field for all learners?