ACAD-1441 Digital Portfolio
Digital Teaching and Technology Learning in HE
Nadya Belenky
2023-06-20
Let’s start with the basics: “What is this format even?”
This is an R Markdown document. Markdown is a simple formatting syntax for authoring HTML, PDF, and MS Word documents. For more details on using R Markdown see http://rmarkdown.rstudio.com.
The last two sentences is the standard definition that comes with a basic R markdown file. Seemed good enough to explain it to the peer group, so I left it. This is the format that I’ll be doing my ACAD-1441 Digital Portfolio in, nothing too flash or too complicated (I hope). For all intents, this is an html page with a few bells and whistles (like, one whistle). But now you’ve got me wondering if R Markdown integrates with Moodle, how next level would that be (Update: Unclear, implementation looks rocky).
Maybe more important: “WHY is this format happening to me? I’m just an innocent peer reviewer/assessor!”
This is a throwback to last term, but using R markdown has been a learning goal of mine for years. I’ve been inching closer (downloading R Studio, taking a R course) but I’ve never made the time for R Markdown specifically so it just got added to a pile of goals that we’ll call Nadya’s One-Day-Maybe list.
In the end, ACAD-1441 did this to itself, planting all kinds of pivotal “propositions” in my mind. For example:
-Does technology add value? Or are we just doing something else because it’s novel?
-How can we use Virtual Learning Environments (VLEs) better? Or at least, how do we prevent them from becoming glorified dropbox folder that never get used or opened and are just a tick box on the endless checklist. God, life’s relentless.
Now, remember back in Term 1 when we learned about meta-cognition? Thinking about how you learn best, about how thinking about learning makes you better at learning. Thinking about how you like to learn. Me, myself, I like a bit verve, a bit of quirk. I like the flexibility to do a lot of different things, and I appreciate open source software, like R. But most important of all: Microsoft can kick rocks.
So that’ll be why I’m shaping this portfolio in R Markdown.
My Ed Tech Focal Point: Introduction to Statistics (STAT-1034)
Let’s start over. Hi, my name is Nadya Belenky. I’m a senior lecturer at the University of Greenwich. I have a tactile keyboard that make every day at work sound like I’m going to war with imaginary ants. I’m an epidemiologist by training, but more of a health systems/health services researchers. What that mans for the University is that I teach mostly research methods, statistics, and epidemiology modules. It’s come to my attention that these are all modules that students experience stress and anxiety over.
I just taught STAT-1034 in Term 2, 2023. So far so good. The evaluations (FRESH OFF THE PRESS at the time that the formative assessment is due [April 24, 2023]) came back looking very strong, their mock exam scores give me a sense that most of the students will pass - it looks like the work that I put into that module paid off, right?
So why am I still thinking about it? On paper, there’s nothing else to do for this module except kick back and make sure they let me lead it again next academic year. But in practice, it’s the pebble in my shoe.
Everything went fine, so what went wrong?
Statistics is at that delightful intersection, where the things that you have to learn are abstract, but then the way you have to apply them is very concrete. Take, example, the Central Limit Theorem (CLT). Yikes! But then, the thing that you get from CLT is this idea of a normal distribution, which is where we get our ideas about confidence intervals, which is the whole reason we can calculate confidence intervals (by hand, or, more likely, by computer, in R!).
The abstract-concrete continuum means that the module (and, by extension, the students) has to be shifting back and forth between theory and application, which takes time and effort for both instructors and students. As an instructor, in-class applied work is time-consuming. For students, tackling the balance can be frustrating because they are either missing the math to build the theory on, or because they can’t do the math because they’re missing the theory. And once you start missing either one or the other, that problem compounds pretty quickly.
I should probably also mention, because it’s going to become relevant in Section 6, that I was giving them take-home exercises every week - well, almost every week. I wasn’t marking them or anything (who has the time), but they did get some kind of at-home work, and then the following week, I’d release the solutions. I asked them to upload what their answers, but that was mainly for accountability, I didn’t have even vaguely close to enough to mark 130 students’ take-home exercises every week.
Speaking of marking though, I did add a mini mock-exam at mid-term, which I did mark. And that alone took about 100 years, so it’s safe to say that I knew I wouldn’t have time for any more bonus marking.
Punchline: All of that is to say that the mock exams went over well with students, and that the feedback they received on those exams was formative and useful for their progression in the class. However, my experience with the module supports that it is not possible to add additional mock exam questions and mark them.
Where should the technology be going with this module?
At first I was thinking, boom, summer project: every week gets its own Moodle quiz, I’ll specify the solutions set and some kind of helpful long-form explanation text, then it’s marked in real-time as students complete the quiz (thanks Moodle, for taking on my marking work for me) and students get insta-feedback without me having to lift a finger. Plus I’ll have some kind of trackable log for how many of them are doing the work and which questions they’re failing at. Big impact, for a relatively small (summer) lift, right?
But this module has me dreaming bigger. I’m all over MIT’s open courseware as part of my Continuing Professional Development, especially in the arena of educational technology, and now MIT is telling me that ADAPTIVE learning is the way of the future.
Adaptive Learning and Pedagogic Theory
The basics: adaptive learning tries to automatically cutomise the learning experience to the learner, by dynamically adapting the content to input by the learner.(Somyürek 2015) When you bring technology-based systems into play, we’re talking about providing students with immediate feedback, without using up any ad-hoc, in-person resources. The very thing that we identified earlier as being in short supply during the typical academic term.
There are a handful of the principles from ACAD-1441 that I’d like to tie-in into this idea/project: The PIC-RAT Model (Kimmons, Graham, and West 2020), and Diana Laurillard’s six learning types (Laurillard 2013)
When I originally inherited this module, there was one mock exam, at the end of the module. Problem is, if you get to the end of module and fail the mock exam, that’s still helpful information for you to have. However, it’s not quite as useful as, say, getting that warning sign earlier on in the term, when it’s easier to course-correct. As I later learned in my exploration of adaptive testing, this type of testing (both the mock exam and the actual exam) can be referred to as “fixed linear testing”.(Philip Chalmers 2016)
The PIC-RAT Model
The PIC-RAT model works on the assumption that learning happens better when technology influences traditional practice by shifting students into the creative and interactive segments of the matrix.(Kimmons, Graham, and West 2020)
Basically, applying PIC to the students’ relationship to the the statistics module (and the educational technology therein), we apply the PIC-RAT matrix by asking two questions:
- What is the technology use’s effect on practice?
-Replacement
-Amplification
-Transformation
- What are students doing with the technology?
-Passive
-Interacting
-Creating
If we take a broader view of the adaptive quizzing and take it as an extension of the VLE, we replace the traditional word document weekly exercises (turnitin assignment) with adaptive quiz and amplify the static exercises by increasing interactivity and individualized learning. As a consequence, we take something that students are interacting with mainly passively (the VLE, through watching recorded lectures) and amp up the interactivity.
As students become more comfortable with the material, the burden of adding to the question bank can be distributed among the student body (finally make the large, 120-person module work to the lecturer’s advantage) by involving the students in creating a single question to add to the adaptive quiz question bank.
Let’s shift gears to learning types. The six being: Acquisition, Investigation, Practice, Discussion, Collaboration, and Production. This may seem obvious, but acquisition and practice are closely tied in statistics learning. They’re doing plenty of acquisition work in the current version of the module: through the live lectures, and then also through the lecture recordings. Let’s save Practice for last for now.
Investigation, where students go hunting for their own knowledge, say, at the library, is tricky for statistics. For one, I’ve learned that students reach for the go-to of the internet (youtube for introductory statistics, random webpages explaining statistical concepts), there is a ton of statistics content out there for introductory statistics. The enduring problem with this approach, however, is that approach and terminology generally vary slightly across all of these resources, even though the core concepts being tackled remain the same. This may seem like a small point, but for a student being introduced to statistics for the first time, these discrepancies can loom large and impede the learning process. Investigation ideally is focused on the library and on an internally consistent introductory statistics book, but, like I said, the library, even the e-books, tend to be the second choice after the general internet for Year 1 students.
Discussion: The characteristics of this class, primarily size, make this another difficult sell for this particular module. Although peer-to-peer learning (through discussion) can be exceptionally valuable for statistics content, it has to be built on a solid (or at least solidifying) knowledge base.
All out of order now, but let’s skip ahead to Production: Initially, I was thinking there would be space for some productive learning at the end of the adaptive learning process, where students submit an individual question for consideration to the question bank. Still a good idea, connecting students to the adaptive learning cycle, off-loading some of the implementation time costs to 100+ students. However, following the learning types, I wanted to add on a bonus idea, which is that individual production of additional quiz questions might work fine, but given the numbers, it might be more productive to have students *collaborate** on producing 2-3 higher quality questions. That would give them the opportunity to work in groups and to do some goal-focused peer teaching, clarifying concepts for each other that their questions are based on. Teams of students producing higher quality questions also cuts down on the screening time required on the instructors end to assess the incoming questions.
Adaptive Learning in Practice
Obviously, I’m not the first person to come up with this idea, or to even ask whether it’s possible within Moodle. Long story short, after a bit of probing, it turns out that Moodle offers an adaptive quiz plug-in. * still writing, something like Spoiler Alert, someone else has already had this idea and has already developed it for Moodle.That someone else is Middlebury College, who published the code on Github (thanks Middlebury), but it’s not being updated by them anymore. What that means is that if I ever wanted to tinker with it myself, I have that option, but for now, we’re going file that Github code under “Rabbit Holes” and pursue a more mainstream option, which is trying to get the existing plug-in to work for me and STAT-1034.
Basically, moving from this: 
to this:
An adaptive quiz!
What does Adaptive Quiz look like
As we already know, adaptive quizes allow instructors to create quizzes that adapt to students’ knowledge, as students interact with the quiz. What that means is that every time students answer quiz questions, the subsequent quiz question will be adjusted to be either more or less difficult, based on the student’s previous answer (Figure 1).(Wainer et al. 2000)
Moodle’s Adaptive Quiz feature requires instructors to put together several components in order for it to function adaptively, making it a little more of a time investment compared to Moodle’s traditional quiz feature.
Figure 1
How to implement Moodle’s Adaptive Quiz Feature at the University
Got in touch with Greenwich’s Academic Technology team, because the adaptive quiz feature isn’t standard on Greenwich’s Moodle because it’s a little esoteric and user-developed. Consequently, you need an administrative account to add those kinds of non-standard features (because once you add a tool to Greenwich’s Moodle, it gets added to everyone’s Moodle options).
Nowe we’re getting into the weeds, but for any of you who are curious about how the process of requesting a user-developed plug-in to Greenwich’s Moodle options, you can read on. Everyone else, go ahead and skip to the next section. Basically because it’s a university-wide decision (seems a little wild that it can’t be implemented module-by-module? But that’s definitely filed under “Rabbit Hole”), it has to be “sandboxed” first. What that means is that the educational technology team sets up a fake module moodle, where they can test the feature. The testing process is looking for obvious things like, whether the plug-in still works, whether the plug-in will gel with the most recent Moodle update, that kind of thing. That brings us up to date in the process. Educational technology is testing the plug-in for mass implementation. In the meantime, I’ll probably need to be working on a Plan B, because adaptive quizzing would be valuable to a lot of modules at Greenwich. Plan B is still TBD.
Late-breaking update on Plan B: A little more information on R’s role in adaptive testing
Turns out rabbit holes are irresistible, and also because it’s been a few months now since the educational technology team took my request for adding Adaptive Quiz to Moodle and I haven’t gotten anything enthusiastic back. I suspect having a solid Plan B will actually be necessary. It’s hard to get universities to implement university-wide change. Especially when only a single module (that we know of!) is pushing for it.
It turns out, R does include a few different R packages that tackle computerized adaptive testing (CAT), the R package catR (Magis and Raı̂che 2012) being one of the most popular, but so far it those R packages are more geared towards simulating CAT, rather than having a CAT that students can use. Why would we need to simulate CAT, you ask? Not to get too into the statistical weeds about it, but it turns out that the CAT-ish question of “what happens if we assume a classroom of 100 students who start out with X level of baseline knowledge?” Basically, it turns into a whole laboratory of statistical simulations. Not very classroom friendly. Not to mention that none of the dominant CAT packages in R include a visual interface, so basically they’re for statistics only, not for class. Critically though, those simulation packages do allow you to get at an educational/causal inference question surrounding how much testing you need to do until you can say with X% certainty that the student’s result/score was not by chance. Let’s put a pin in that for future.
How would I test whether the addition of adaptive quizzes was a value-add to the module?
In a never-ending quest to avoid implemented educational tech that no one uses and which, as a consequence, isn’t helpful to anyone. This is where we start thinking about Lauren Anstey and Gavan Watson’s rubric for evaluating e-learning tools in higher education,(Anstey and Watson 2018) focusing primarily on functionality and accessibility, because the rest of the rubric falls under being embedded into the VLE.
Functionality or, “does the tool serve its intended purpose well?”
It’s a tricky module, ranging in size for 40 to 130, depending on the year. Adaptive testing has the advantage here that it is inherently adaptable to Scale, both scaling up and scaling down. Adaptive testing should also be, at the very least least, no more difficult to use than the current version of take-home exercises. It’s intuitive in the sense that it presents as a standard multiple choice quiz, where all the statistical workings and adaptiveness are happening under the surface, out of view of the student. Further, I would expect adaptive testing to be more functional in terms of ease of use based on being able to start students at whatever level is literally easy enough for them to complete successfully. Where this approach fails in terms of functionality is in terms of available tech support, of which, for Moodle adaptive quiz, at least, there is no current support available. Adaptive Quiz also doesn’t perform well in the area of hypermediality, in the strict sense of giving students multiple forms of media or improving their control of their own engagement. It does, however give students a lesson organization that is non-linear and non-fixed, because it adapts to the individual learner.
Accessibility or, providing a “flexible, adaptable curriculum […] to support multiple learning approaches and engagement” and a user’s technology access are both integral to successful adoption of educational technologies.
In this case, user-focused participation is the primary strange of adaptive testing, because they address the diverse knowledge starting points of the range of students participating in an introductory statistics module. Because the activity would be Moodle-based, in other words, integrated within the current dominant Learning Management System (LMS), there is no additional required equipment or additional downloads to participate in adaptive quizzing, in addition to no additional cost of use. Though at this point, it’s not clear that the adaptive quiz activity will function consistently as Moodle continues to update. It seems unlikely that the mobile version of adaptive quiz will be as easy to use as the desktop version, and offline access suffers from the same accessibility issues as the VLE in general.
Reflection
Wait, so what’s the connection between R Markdown and this idea about adaptive quizzes? I guess the unifying theme that I’m seeing in all of it, and a point that I think we’ve been critically circling in ACAD-1441, is that (premise) interactive things (R Markdown, adaptive quizzes) are good - . They make learning fun and self-directed. But none of this is gamified. A little critical element of that premise about interactivity and self-directedness, is that neither of these things (R markdown, adaptive quizzes) are fun in the same way that playing Zelda is fun. Zelda is a literal fun game. Statistics/Statistical programming both have steeper hills to climb. I think what I’m landing on is that maybe neither one of those things needs to be gamified to the point where they become Mario Cart, but I think a little nudge into the interactivity zone (weekly quizzes) combined with the individualized nature of an adaptive algorithm could fuel a flow state,(Csikszentmihalyi, Abuhamdeh, and Nakamura 2005) in every student, every week. Imagine: a statistics class where no one feels foolish or like they’re out of step when they begin the module.
Alright, some critical treatment of the negatives. Adaptives quizzes are focused on self-directed learning, they’re sort of driven by individualism, and they don’t play a role (that I can see?) in academic community building.(McLaughlin 2019) Those online communities have been linked to student satisfaction,(Liu et al. 2007) and putting additional emphasis on individualized learning through these adaptive testing may contribute to a sense of isolation, potentially already prevalent in a Year 1 statistics module.
They are a lot of upfront work. Making it easier for students to blaze through question sets means the pre-term work of creating sufficiently large sets of questions, across content areas, and at varying levels of difficulty. What’s more it requires assessing and tagging each question with its difficulty level.
Some quick napkin math: At approximately 18 content areas, requiring, say, 3 difficulty levels per content area, and then 5 questions per level (within each content area), that’s, at minimum, 270 questions. If development takes at least 10 minutes, we’re talking about 45 hours for the development of the question pool, without setting up the actual adaptive quiz (quizzes?). Long-term, this can be mitigated a little bit, but folded students into the creative process of coming up with new questions for the questions bank, but in the short-term, the start-up costs seem inevitable.
Where does this leave us?
The summer is long enough to produce a question bank that can support an adaptive quiz. Probably also long enough to chase down the possibility of folding in the Adaptive Quiz activity into Moodle with the Educational Technology team. Probably not long enough to learn enough about R Markdown visual interfaces to combine the R CAT simulation packages with some kind of visual interface and then pull the newly developed question bank into that interface.
In the immediate, I do know a lot more about R markdown now, which can’t hurt. Also a lot more about computerized testing, as well as computerized adaptive testing. One of the unexpected advantages about the format of this digital portfolio is that it revolves on iterative change, and small updates over time - it seems like a good place to track the rest of the exploration process.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++