Introduction

Doing a qualitative analysis in education research provides a wealth of data and a never ending stream of work. In the age of big data, it is possible to acquire large swaths of text, but to analyze it by hand would require more people and more time than we usually have. How can text analysis aid in the analysis of qualitative data, specifically interviews?

Data sources and target audience

In 2023, I conducted two interviews with statistics professors to understand how they felt about the role of technology in teaching. The purpose of these interviews was to give a deeper look at what statistics professors have to consider when wanting to implement technology in the classroom, when they might, why or why not, and how they view technology. Thus, the analysis was originally meant for those interested in statistics education and understanding the role of technology in teaching.

This new analysis is meant for the same audience, but includes those interested in automating qualitative research in statistics education. Since qualitative methods are the most common in understanding teacher feelings and perceptions, text analytics could help future researchers deal with the heavy load of analyzing interview data.

Wrangling the data

Text preprocessing

First, I download all the libraries I would need.

library(dplyr)
library(readr)
library(stringr)
library(tidytext)
library(tidyr)
library(igraph)
library(SnowballC)
library(topicmodels)
library(ldatuning)
library(ggplot2)
library(ggraph)

The data comes in two separate files. Neither file is tab or value delimited, so I had to read in each line at a time. It didn’t come with a column name, so I added one called “Text”.

interview_1 <- as.data.frame(readLines("Person_1.txt")) 
colnames(interview_1) <- "Text"

There was some extra text that didn’t need to be there, including timestamps and names. I removed those. Note that the name “Sheryl King” is not the real name of the person I interviewed.

interview_1_cleaned <- interview_1  %>% filter(Text != "WEBVTT", Text != "", !str_detect(Text,"[0-9]"),
                        !str_detect(Text,"Matthew Ferrell:")) %>% 
  mutate(Text=gsub("Sheryl King:",replacement="",x=Text))

interview_1_cleaned$Interview_1 <- "Interview 1"

I performed these same steps with the second interview. Finally, I combined them together.

interview_data <- bind_rows(interview_1_cleaned, interview_2_cleaned)
colnames(interview_data) <- c("Text","Interview")
head(interview_data, n=1)
##                                                                                              Text
## 1  very strong accent. And you would just be hysterical on how some of this stuff got translated.
##     Interview
## 1 Interview 1

Tokenizing

Using the tidytext r package, I tokenized the interviews into words. I also included a few extra stop words that needed to be removed that wouldn’t contribute to relevant themes.

extra_stop_words <- data.frame(word=c("yeah","lot","gonna","ii","iii","bit","husband",
                                      "breast"))

unigram_interview <- interview_data %>% unnest_tokens(input=Text, output = word, token = "words") %>%
  anti_join(stop_words) %>% anti_join(extra_stop_words) 

head(unigram_interview, n=5)
##     Interview       word
## 1 Interview 1     strong
## 2 Interview 1     accent
## 3 Interview 1 hysterical
## 4 Interview 1      stuff
## 5 Interview 1 translated

Creating two mode graph data

Part of the analysis involved looking at the words each interviewee used graphically before performing topic modeling. This required me to count my data and then transform it using bipartite_mapping.

unigram_interview2 <- unigram_interview %>% 
  count(Interview,word, sort=TRUE) %>% filter(n >=4) %>% select(Interview,word)

unigram_graph <- graph_from_data_frame(unigram_interview2)
V(unigram_graph)$type <- bipartite_mapping(unigram_graph)$type

Creating document term matrix

In preparation for LDA, I transformed my tokenized data into a document term matrix.

interview_dtm <- unigram_interview %>%
  count(Interview, word) %>%
  cast_dtm(Interview, word, n)

Analysis

I applied two techniques to analyze the data: text network analysis and topic modeling using LDA.

Text Network Analysis

The first interview is shown on the left, the second on the right. Here we can see that the similarities they share center around their teaching: how do we help students, what do we talk about, what are they thinking about? However, the first interview (left) talked about a variety of statistical topics ranging from distributions, software, and sampling. The second (right) focused on their online work, teaching programming, and building community in the online sphere.

Topic Modeling using LDA

The first step was to find the optimal number of topics, or k. Based on the plot below, it looks like k=10 is the best, since after the number of topics hovers around the same Griffiths value.

After this, I ran LDA, and took a look at the top 5 terms for each topic.

##      Topic 1         Topic 2    Topic 3      Topic 4     Topic 5     
## [1,] "distribution"  "people"   "move"       "education" "thinking"  
## [2,] "bar"           "online"   "experience" "time"      "level"     
## [3,] "private"       "stuff"    "understand" "calculus"  "statistics"
## [4,] "business"      "person"   "feel"       "files"     "fairly"    
## [5,] "distributions" "students" "questions"  "plane"     "capability"
##      Topic 6      Topic 7       Topic 8   Topic 9     Topic 10  
## [1,] "students"   "school"      "time"    "kids"      "class"   
## [2,] "sampling"   "data"        "office"  "schools"   "courses" 
## [3,] "talking"    "liberal"     "meeting" "web"       "hard"    
## [4,] "department" "programming" "bring"   "basic"     "question"
## [5,] "math"       "certificate" "money"   "wonderful" "justin"

The topics look fairly random at first, with some clearly focused on interview 2 (topic 2 with online and stuff, and topic 7 talking about programming and data) and interview 1 (topic 6 with sampling and the department).To get a better view of the top terms, I created a graph using the beta values, which represents the probability of the word being in a particular topic.

None of the terms have high probabilities of being in particular topics, though each topic has pretty different distributions of words. In order to understand these values better, it will be helpful to see how much of each topic is in each interview.

Gamma represents the proportion of each topic in each interview. For example, almost 45% of interview 2 is composed of words from topic 2. This aligns with what I said earlier about this topic focusing on what instructor 2 does. It appears that only topic 3 shares overlap. This has top terms such as “move”, “understand”, “feel”, “experience” and “questions”, seeming to reflect a sort of affectual part of teaching. Both teachers contained good mixes of this. However, for the other topics, where one is high, it’s low for the other instructor. It looks like the LDA, even with 10 topics, managed to confine them to each interviewee, performing more of a “within persons” thematic analysis.

For interview 1, topics 6,1, and 5 are the top three. Topic 6 centered around the first interviewee’s concern with teaching sampling distributions to the students in class. Topic 1 doesn’t seem to have a good pattern. Topic 5 appears to be about thinking about students and their statistics capability. This lines up with the introductory class this instructor teaches.

For interview 2, topics 2, 7, and 10 are the top 3. Topic 2, like I’ve mentioned before, has to do with the online nature of this instructor’s teaching and the people who take these classes. Topic 7 is a little more muddled, with mentions of data and programming but also liberal. Topic 10 is also teaching centered, with concerns about the classes, courses, and questions asked, though with a random student being mentioned.

Communication

Key insights

The number one insight I found from the text network was that though each instructor focused on and cared about teaching, they had to focus on and care about different aspects of it since their circumstances and who they teach were very different. Even the statistical topics they discussed were different. This shows that two people may be much more different than similar, even within the same university or department.

One insight from the topic modeling that was surprising to me is that the topic the two most overlapped on had to do with feeling and experiencing things in the classroom. Teaching is much more than preparing and delivering materials, but involves heartfelt effort and at times anxiety for the welfare of those you teach.

Limitations

When a person reads something, they bring with them their experiences, their cultural background and understanding of human norms. When the computer reads something, it has the words. Compared to when I did this analysis myself, the text network and topic modeling woefully underperformed and gave scant insights. Even with ten topics, LDA performed a within persons analysis. It makes sense: based on the words alone, the two teachers didn’t share much in common. It was only through looking beyond the words themselves and seeing the meaning behind them that I could make comparisons.

One interesting issue is ethical. While coding, I noticed that several pieces of private information (names in this case) appeared frequently. I realized that any analysis that uses interviews and uses the words themselves specifically for the analysis might include things the speakers didn’t want shared. It’s important when performing these analyses to recognize this risk and look out for information leaks.

Future avenues

Using large language models is much better suited to performing qualitative analyses than the simpler methods that I showed in this case study. LLM’s have trained on enormous corpi. They have learned more nuanced meanings and how those meanings change based on context. They can summarize text and answer questions about user supplied text. Thus, it would be interesting to see how a human and a LLM could work together to jointly analyze qualitative data.