RMarkUs

library(RMarkUs)

Introduction

The minimum requirements of RMarkUs to auto grade students’ problem sets consist of 3 files: a test file in .R specifying what variables will be evaluated, a solution file in .R or .qmd/.rmd and a problem set file in .R or .qmd/.rmd.

Solution File Set-up

A solution file should contain all questions and answers stored as variables. For instance, a solution .Rmd may contain a YAML header, a question and answers stored as variables:

title: "Problem Set 1"
author: ""
output: pdf_document

Question: What are the first 2 numbers in positive integers?

first_num <- 1
second_num <- 2

Problem Set File

Usually, a problem set file is the same as the solution file except that the real values of variables in the problem set file will be replaced by NULL.

first_num <- NULL
second_num <- NULL

The problem set file will be shared with students so that they can complete and submit to MarkUs.

Test Script

After loading the necessary package including RMarkUs, one should set the work directory to the source file test_script.R if instructor wants to test it locally.

Load instructor and student solutions

instructor_environment <- load_solutions("path/to/solution.Rmd") 
student_environment <- load_student_submission(file="path/to/problemset.Rmd", file_MarkUs = "problem-set-on-MarkUs.Rmd")

Instructors have the option to check if the environments are properly loaded by calling without affecting the grading i.e., this function does not affect students’ grades and should be mainly used by instructors to debug:

test_environment_loaded(instructor_environment)
test_environment_loaded(student_environment)

  1. If the variable is not NULL, the message is: “The environment is loaded successfully.”

  2. If the variable is NULL, it means the solution or submission if not loaded properly, usually due to 2 things:

    1. It is possible that the working directory is not set properly. If this is the case, setting it to the source file (assuming all files are in the same folder in a local setting) will solve the problem.

    2. The error message will remind instructors: “The environment is not loaded properly. Please check if .Rmd file name in load environment function matches the actual .Rmd file name.”

Check Student Number

We also allow additional checks for student numbers (unique identifiers for students) by calling function:

test_student_number(varname="STUDENT_NUMBER", student_environment=student_environment, instructor_environment=instructor_environment)

In the solution file, instructor needs to create a variable:

STUDENT_NUMBER <- RMarkUs::get_student_number()

In the problem set file distributed to students, instructor will ask students to provide their own student number (numeric):

STUDENT_NUMBER <- 130 # replace 130 by your real student number

This variable (student number) will be compared on MarkUs automatically.

Reduce Plagiarism by Random Seed

Instructors can ask students to set seed using the student number variable and then run simulations based on the seed. Then each student will provide unique answers. When running on MarkUs, the student responses will be compared against solution based on the same seed to check for correctness.

Getting Started with Automatic Grading Functions

This guide will introduce the purpose and usage of testLinearModel(), testDataFrame(), testVector()and testScalar().

1. testLinearModel()

The testLinearModel() function is designed to validate a student’s linear model against an instructor’s expected model. This function can check the presence, correctness, and specific characteristics of a linear model. Though theoretically speaking two differently models can have the same degrees of freedom (n-p) and the same amount of residuals, we believe practically it would be hard (if impossible) to construct such models without good understandings of these concepts. For the purpose of grading, as long as two models have the same degrees of freedom (n-p) and residuals (and AIC if specified), then we have strong reasons to believe they are on the right track.

P.S. If malicious users intentionally design a model that passes the function, it will show that they are experienced in linear models, which deserves credits.

Function Arguments:

Example Usage:

testLinearModel("model_variable")

2. testDataFrame()

The testDataFrame() function checks if a student’s data frame meets the instructor’s expectations. It can validate the presence, correctness, size, class, attributes, and column order of the data frame.

Function Arguments:

Example Usage:

testDataFrame("student_df")

3. testVector()

The testVector() function validates vectors submitted by students, checking for properties like presence, length, datatype, order, and correctness.

Function Arguments:

Example Usage:

testVector("student_vector")

4. testScalar()

The testScalar() function checks scalar values submitted by students against expected values provided by the instructor. This function is particularly useful for validating simple data types such as numeric values, characters, or logical values.

Function Arguments:

Example Usage:

testScalar("student_scalar")