Mentor Program Survey Data Analysis (Zimmerman)

Data

mentoring <- read_csv(
  "Univ Entp Ctr Mentoring Data 12-18-24.csv",
  col_types = cols(.default = col_character()),
  name_repair = "unique"
)

#Remove the Qualtrics question-text and ImportId rows
mentoring <- mentoring %>%
  slice(-(1:2))

#Convert columns to numeric/date/etc. where appropriate
mentoring <- mentoring %>%
  type_convert()

#Remove the preview/test response
mentoring <- mentoring %>%
  filter(DistributionChannel != "preview")

usable_obs <- mentoring %>%
  summarise(
    `Q39 - Satisfaction` = sum(!is.na(Q39_1)),
    `Q26 - Mentor Training` = sum(!is.na(Q26)),
    `Q12a - Mentor Screening` = sum(!is.na(Q12...29)),
    `Q6 - Mentoring Type` = sum(!is.na(Q6)),
    `Q27 - Mentee Training` = sum(!is.na(Q27)),
    `Q17 - Number of Mentors` = sum(!is.na(Q17)),
    `Q19 - Number of Mentees` = sum(!is.na(Q19)),
    `Q32 - Meeting Requirements` = sum(!is.na(Q32))
  ) %>%
  pivot_longer(
    everything(),
    names_to = "Variable",
    values_to = "Usable Observations"
  )

usable_obs

# A tibble: 8 × 2
  Variable                   `Usable Observations`
  <chr>                                      <int>
1 Q39 - Satisfaction                            26
2 Q26 - Mentor Training                         27
3 Q12a - Mentor Screening                       27
4 Q6 - Mentoring Type                           40
5 Q27 - Mentee Training                         27
6 Q17 - Number of Mentors                       27
7 Q19 - Number of Mentees                       26
8 Q32 - Meeting Requirements                    27

Just showing that for most of the variables we actually care about, we only have around 26–27 usable responses. So even though there were 55 usable survey responses overall, we don’t really have 55 observations available for most of the analysis.

That makes regression harder, especially if we try to include several variables at once. With such a small sample, adding more predictors can make the results unstable and harder to trust. Because of that, I think regression with one predictor at a time, or maybe two when there is a good reason for it, makes more sense.

Even then, we would still need to check the usual regression assumptions before relying on the results.

I know you really wanted to work with Q17 and Q19. My understanding was that the goal was to see what characteristics are associated with programs that have higher mentor or mentee counts. The issue is that the data for these two variables is pretty messy. We have very low and very high values, ranges, and responses reported per month, per semester, etc., as you can see below. Because of that, any analysis using Q17 or Q19 would be harder to clean and interpret, and the results would be less straightforward.

head(mentoring$Q19[!is.na(mentoring$Q19)])

[1] "10-20 at any one time"                 
[2] "45"                                    
[3] "Varies, but on average 20 per semester"
[4] "15-20"                                 
[5] "90"                                    
[6] "varies"                                

head(mentoring$Q17[!is.na(mentoring$Q19)])

[1] "35"                                                                                                                                                                                           
[2] "13"                                                                                                                                                                                           
[3] "30"                                                                                                                                                                                           
[4] "2-3"                                                                                                                                                                                          
[5] "100+"                                                                                                                                                                                         
[6] "We have a list of 300 interested mentors. Sometimes we try to make an individual match. At certain times during the year, we put out a mass call for mentors and usually get 25-50 responses."

It’s also important to have a defined set of things we actually want to test instead of just running tests on everything and keeping whatever comes out significant. If there really is no relationship, each test run at the .05 level still has a 5% chance of showing a significant result just by chance. So the more tests we run, the more likely we are to eventually find something that looks significant even if it really isn’t. Because of that, we need to adjust for the number of tests we run, which makes it harder for a result to be considered statistically significant.

For the main analysis, I think Q39 makes the most sense to focus on. It is the cleanest outcome, directly measures satisfaction with the success of the mentoring program, and we have enough usable responses to compare groups. Based on the research question, the literature, and the variables that make the most sense to compare, I want to test whether satisfaction differs between centers that do and do not have mentor training/orientation (Q26), systematic mentor screening (Q12a), group mentoring (Q6), and mentee training/orientation (Q27).

Instead of running a separate regression for each one, which would require more assumptions, I am using the Wilcoxon rank-sum test. This is a nonparametric test that basically compares the satisfaction scores between the two groups and tests whether there is enough of a difference to be considered statistically significant.

Since we are doing four main tests, I am using a Bonferroni adjustment to account for the increased chance of a false positive. Instead of comparing each p-value to .05, we divide .05 by 4, giving us a cutoff of .0125. My analysis is shown below. For the rest of the variables, I would mainly use descriptive statistics rather than continuing to run more and more significance tests.

Main Analysis (Q39)

Q39 vs Q26 (Mentor training/orientation)

h1 <- mentoring %>% select("Q26", "Q39_1") %>% filter(!is.na(Q26), !is.na(Q39_1)) 
dim(h1)

h1 %>% group_by(Q26) %>% summarise("Avg_Sat_Score" = mean(Q39_1))

# A tibble: 2 × 2
    Q26 Avg_Sat_Score
  <dbl>         <dbl>
1     1          6.06
2     2          4.6 

h1 %>% count(Q26)

# A tibble: 2 × 2
    Q26     n
  <dbl> <int>
1     1    16
2     2    10

h1$Q26 <- as.factor(h1$Q26)

model <- lm(Q39_1 ~ Q26, data = h1)
summary(model)

Call:
lm(formula = Q39_1 ~ Q26, data = h1)

Residuals:
    Min      1Q  Median      3Q     Max 
-2.6000 -0.0625 -0.0625  0.4000  2.4000 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)   6.0625     0.2857  21.222  < 2e-16 ***
Q262         -1.4625     0.4606  -3.175  0.00408 ** 
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 1.143 on 24 degrees of freedom
Multiple R-squared:  0.2958,    Adjusted R-squared:  0.2664 
F-statistic: 10.08 on 1 and 24 DF,  p-value: 0.00408

wilcox_test(Q39_1 ~ Q26,
            data = h1,
            distribution = "exact")

    Exact Wilcoxon-Mann-Whitney Test

data:  Q39_1 by Q26 (1, 2)
Z = 2.825, p-value = 0.002416
alternative hypothesis: true mu is not equal to 0

Since 0.002416 < .0125, the exact Wilcoxon rank-sum test showed a statistically significant difference in Q39 satisfaction between centers that trained/oriented mentors and centers that did not. Centers that trained mentors had higher satisfaction scores.

In the best-practices section of the white paper, I believe you said something along the lines of mentors should be carefully selected, trained, and monitored. And training helps create more consistent mentoring experiences and “improves the likelihood of successful mentoring relationships.” So this can help support that.

“Centers that trained/oriented mentors reported higher satisfaction scores on average (6.06) than centers that did not (4.60). This difference was statistically significant using the Bonferroni-adjusted significance level (exact Wilcoxon rank-sum test, p = .0024; adjusted α = .0125).”lastl

Q39 vs Q12a (Systematic Mentor Screening)

mentoring %>% select(Q12...29, ResponseId ) %>% filter(ResponseId %in% c("R_1L1NCLAo9DccZHy", "R_6kByFGcXtrDD9gM", "R_7g9Z78s3btx4dAl"))

h2 <- mentoring %>% select("Q12...29", "Q39_1") %>% filter(!is.na(Q12...29), !is.na(Q39_1)) 
dim(h2)

h2 %>% group_by(Q12...29) %>% summarise("Avg_Sat_Score" = mean(Q39_1))

# A tibble: 2 × 2
  Q12...29 Avg_Sat_Score
     <dbl>         <dbl>
1        1          6.08
2        2          4.92

h2 %>% count(Q12...29)

# A tibble: 2 × 2
  Q12...29     n
     <dbl> <int>
1        1    13
2        2    13

h2$Q12...29 <- as.factor(h2$Q12...29)

wilcox_test(Q39_1 ~ Q12...29,
            data = h2,
            distribution = "exact")

    Exact Wilcoxon-Mann-Whitney Test

data:  Q39_1 by Q12...29 (1, 2)
Z = 2.3445, p-value = 0.01752
alternative hypothesis: true mu is not equal to 0

Using the Bonferroni-adjusted significance level of .0125, the difference was not statistically significant (p = .0175). However, there was still a noticeable difference in the observed satisfaction scores. Centers that systematically screened mentors had an average satisfaction score of 6.08 compared with 4.92 for centers that did not. So while there is not enough statistical evidence to call the result significant after the adjustment, the pattern is still worth noting, especially because it lines up with the literature.

I believe you said that mentor selection is an important factor, and you specifically mention systematic identification and screening of mentors as a recommended practice. So even though this result is not statistically significant after the Bonferroni adjustment, the higher satisfaction scores among centers that systematically screen mentors still line up with and can help support that point.

Q39 vs Q6 (Group vs no group mentoring)

1 = Traditional mentoring 2 = Peer mentoring 3 = Group mentoring 4 = Subject matter mentoring 5 = Other

mentoring %>% count(Q6)

h3 <- mentoring %>% select("Q6", "Q39_1") %>% filter(!is.na(Q6), !is.na(Q39_1)) 
dim(h3)

h3$Q6 <- as.character(h3$Q6)

h3$group_mentoring <- ifelse(
  grepl("3", h3$Q6), 1, 2
)

h3 %>% group_by(group_mentoring) %>% summarise("Avg_Sat_Score" = mean(Q39_1))

# A tibble: 2 × 2
  group_mentoring Avg_Sat_Score
            <dbl>         <dbl>
1               1          6.07
2               2          4.83

h3 %>% count(group_mentoring)

# A tibble: 2 × 2
  group_mentoring     n
            <dbl> <int>
1               1    14
2               2    12

h3$group_mentoring <- as.factor(h3$group_mentoring)

wilcox_test(Q39_1 ~ group_mentoring,
            data = h3,
            distribution = "exact")

    Exact Wilcoxon-Mann-Whitney Test

data:  Q39_1 by group_mentoring (1, 2)
Z = 2.2704, p-value = 0.02
alternative hypothesis: true mu is not equal to 0

Using the Bonferroni-adjusted significance level of .0125, this result is not statistically significant (p = .0200). However, centers that offered group mentoring still had higher satisfaction scores than centers that did not, so there is still a noticeable difference in the sample.

In the white paper, you discuss group mentoring as a model that combines some of the benefits of traditional and peer mentoring, and you also mention literature that recommends group or multiple mentoring in higher education entrepreneurship programs. So while this result does not meet the adjusted significance cutoff, it still goes in the same direction as the literature and can help support that discussion.

“Centers that offered group mentoring reported higher satisfaction scores on average (6.07) than centers that did not offer group mentoring (4.83). However, the difference did not meet the Bonferroni-adjusted significance level (exact Wilcoxon rank-sum test, p = .020; adjusted α = .0125).”

Q39 vs Q27 (mentee training/orientation)

h4 <- mentoring %>% select("Q27", "Q39_1") %>% filter(!is.na(Q27), !is.na(Q39_1)) 
dim(h4)

h4 %>% group_by(Q27) %>% summarise("Avg_Sat_Score" = mean(Q39_1))

# A tibble: 2 × 2
    Q27 Avg_Sat_Score
  <dbl>         <dbl>
1     1          6   
2     2          4.82

h4 %>% count(Q27)

# A tibble: 2 × 2
    Q27     n
  <dbl> <int>
1     1    15
2     2    11

h4$Q27 <- as.factor(h4$Q27)

wilcox_test(Q39_1 ~ Q27,
            data = h4,
            distribution = "exact")

    Exact Wilcoxon-Mann-Whitney Test

data:  Q39_1 by Q27 (1, 2)
Z = 2.1819, p-value = 0.02779
alternative hypothesis: true mu is not equal to 0

“Centers that trained/oriented mentees reported higher satisfaction scores on average (6.00) than centers that did not (4.82). However, the difference did not meet the Bonferroni-adjusted significance level (exact Wilcoxon rank-sum test, p = .0278; adjusted α = .0125).”

This also lines up with the white paper, where you say that both mentors and mentees benefit from selection, training, and monitoring, and you specifically note that mentees also need training. So even though this result is not statistically significant after the Bonferroni adjustment, the pattern in the data still supports that idea.

Descriptive Comparisons

I also looked descriptively at software/platform use and whether the Center was endowed. I did not include these as part of the four formal hypothesis tests above because I wanted to limit the number of significance tests being run.

For both comparisons, the average satisfaction scores happened to be the same. Centers that used mentoring software/platforms had an average satisfaction score of 5.00 compared with 5.87 for centers that did not use software. Similarly, endowed Centers had an average satisfaction score of 5.00 compared with 5.87 for Centers that were not endowed.

These two weren’t as tied to the literature as the others, but I saw potential.

Q39 vs Q29 (software vs no software)

h5 <- mentoring %>% select("Q29", "Q39_1") %>% filter(!is.na(Q29), !is.na(Q39_1)) 
dim(h5)

h5$software <- ifelse(
  grepl(9, h5$Q29), 2, 1
)

h5 %>% count(software)

# A tibble: 2 × 2
  software     n
     <dbl> <int>
1        1    11
2        2    15

h5 %>% group_by(software) %>% summarise("Avg_Sat_Score" = mean(Q39_1))

# A tibble: 2 × 2
  software Avg_Sat_Score
     <dbl>         <dbl>
1        1          5   
2        2          5.87

Q39 vs Q40 (endowed)

h6 <- mentoring %>% select("Q40", "Q39_1") %>% filter(!is.na(Q40), !is.na(Q39_1)) 
dim(h6)

h6 %>% count(Q40)

# A tibble: 2 × 2
    Q40     n
  <dbl> <int>
1     1    11
2     2    15

h6 %>% group_by(Q40) %>% summarise("Avg_Sat_Score" = mean(Q39_1))

# A tibble: 2 × 2
    Q40 Avg_Sat_Score
  <dbl>         <dbl>
1     1          5   
2     2          5.87

Secondary Analysis (Q17, Q19 Analysis)

Cleaning Data

h7 <- mentoring %>% filter(!is.na(Q19))

h7 %>%
  select(ResponseId, Q19)

# A tibble: 26 × 2
   ResponseId        Q19                                   
   <chr>             <chr>                                 
 1 R_6kByFGcXtrDD9gM 10-20 at any one time                 
 2 R_3DSuvNsKvRLzbDC 45                                    
 3 R_69I3YvqhQwelWAV Varies, but on average 20 per semester
 4 R_3keU47VEwpCTqvv 15-20                                 
 5 R_1eQpcV7IaXVatlA 90                                    
 6 R_6fTXSDC9MbASfEM varies                                
 7 R_1ihoyM0M3Z2lPG1 40+                                   
 8 R_3PTsrty8cQTVCGi 24 teams (75 students)                
 9 R_1uR7lvRTgWp1lQw 20                                    
10 R_5ZGNn4d91ZltHTb 68                                    
# ℹ 16 more rows

I decided to drop centers that gave responses like “varies,” “per month,” or “annually” because those answers were not directly comparable to the other mentee counts. For responses reported as a number with a “+”, I used the stated number as a conservative estimate (for example, 100+ became 100). For ranges such as 10–20, I used the midpoint of the range. This leaves us with 21 observations.

# dropping ids with the "a month" stuff
h7 <- h7 %>%
  filter(!ResponseId %in% c(
    "R_69I3YvqhQwelWAV",
    "R_6fTXSDC9MbASfEM",
    "R_1gZhRNgdQkOPqj7",
    "R_3rpjXtCtopGgouZ",
    "R_1uR7uWwaLikf5cJ"
  ))

h7 <- h7 %>%
  mutate(
    Q19 = case_when(
      #10-20 at any one time -> midpoint = 15
      ResponseId == "R_6kByFGcXtrDD9gM" ~ "15",
      
      #15-20 -> midpoint = 18 (17.5 is weird)
      ResponseId == "R_3keU47VEwpCTqvv" ~ "18",
      
      #40+ -> use 40
      ResponseId == "R_1ihoyM0M3Z2lPG1" ~ "40",
      
      #24 teams (75 students) -> use 75 mentees
      ResponseId == "R_3PTsrty8cQTVCGi" ~ "75",
      
      #100+ -> use 100
      ResponseId == "R_62X1veV8FqyOn7u" ~ "100",
      
      #30+ -> use 30
      ResponseId == "R_6oYkV2dub1z07M5" ~ "30",
      
      TRUE ~ Q19
    )
  )

h7 <- h7 %>% mutate(Q19 = as.numeric(Q19))

h7 %>%
  select(ResponseId, Q19)

# A tibble: 21 × 2
   ResponseId          Q19
   <chr>             <dbl>
 1 R_6kByFGcXtrDD9gM    15
 2 R_3DSuvNsKvRLzbDC    45
 3 R_3keU47VEwpCTqvv    18
 4 R_1eQpcV7IaXVatlA    90
 5 R_1ihoyM0M3Z2lPG1    40
 6 R_3PTsrty8cQTVCGi    75
 7 R_1uR7lvRTgWp1lQw    20
 8 R_5ZGNn4d91ZltHTb    68
 9 R_5qdJti6oWwo85y6    30
10 R_6fJQhpX4tsywd21    25
# ℹ 11 more rows

h10 <- mentoring %>% filter(!is.na(Q17))

h10 <- h10 %>% filter(ResponseId != "R_6fTXSDC9MbASfEM")

h10 <- h10 %>%
  mutate(
    Q17 = case_when(
      
      #2-3 -> midpoint
      ResponseId == "R_3keU47VEwpCTqvv" ~ "2.5",
      
      #100+ -> conservative lower bound
      ResponseId == "R_1eQpcV7IaXVatlA" ~ "100",
      
      #stable of >100 -> conservative lower bound
      ResponseId == "R_1ihoyM0M3Z2lPG1" ~ "100",
      
      #75 currently formally signed up; ~300 total list
      #use the 75 currently signed up
      ResponseId == "R_3PTsrty8cQTVCGi" ~ "75",
      
      #Active 57; 260 on the roll
      #use active mentors
      ResponseId == "R_5ZGNn4d91ZltHTb" ~ "57",
      
      #30 people -> 30
      ResponseId == "R_1gZhRNgdQkOPqj7" ~ "30",
      
      #80-100 -> midpoint
      ResponseId == "R_3rpjXtCtopGgouZ" ~ "90",
      
      #20-25 -> midpoint
      ResponseId == "R_1wslaqFQKrvPQLG" ~ "22.5",
      
      #25+ -> conservative lower bound
      ResponseId == "R_6oYkV2dub1z07M5" ~ "25",
      
      TRUE ~ Q17
    )
  )

h10 <- h10 %>% mutate(Q17 = as.numeric(Q17))

Q19 vs Q32 (Require/Specify Mentor Meetings)

h8 <- h7 %>% select(Q19, Q32)

h8 %>% count(Q32)

# A tibble: 2 × 2
    Q32     n
  <dbl> <int>
1     1    14
2     2     7

h8 %>% group_by(Q32) %>% summarise("Median Number of Mentees" = median(Q19))

# A tibble: 2 × 2
    Q32 `Median Number of Mentees`
  <dbl>                      <dbl>
1     1                       42.5
2     2                       18  

Programs that require or specify mentor-mentee meetings had a higher median number of mentees (42.5) than programs that did not (18). This suggests that more structured programs tend to have more mentees.

h8$Q32 <- as.factor(h8$Q32)

wilcox_test(Q19 ~ Q32,
            data = h8,
            distribution = "exact")

    Exact Wilcoxon-Mann-Whitney Test

data:  Q19 by Q32 (1, 2)
Z = 2.3562, p-value = 0.01666
alternative hypothesis: true mu is not equal to 0

h11 <- h10 %>% select(Q17, Q32)

h11 %>% group_by(Q32) %>% summarise("Median Number of Mentors" = median(Q17))

# A tibble: 2 × 2
    Q32 `Median Number of Mentors`
  <dbl>                      <dbl>
1     1                         50
2     2                         27

h11$Q32 <- as.factor(h11$Q32)

wilcox_test(Q17 ~ Q32,
            data = h11,
            distribution = "exact")

    Exact Wilcoxon-Mann-Whitney Test

data:  Q17 by Q32 (1, 2)
Z = 1.669, p-value = 0.09861
alternative hypothesis: true mu is not equal to 0

Q19 vs Q9 (Formal vs Informal)

h9 <- h7 %>% select(Q19, Q9)

h9 %>% count(Q9)

# A tibble: 2 × 2
     Q9     n
  <dbl> <int>
1     1    16
2     2     5

h9 %>% group_by(Q9) %>% summarise("Median Number of Mentees" = median(Q19))

# A tibble: 2 × 2
     Q9 `Median Number of Mentees`
  <dbl>                      <dbl>
1     1                         35
2     2                         15

h9$Q9 <- as.factor(h9$Q9)

wilcox_test(Q19 ~ Q9,
            data = h9,
            distribution = "exact")

    Exact Wilcoxon-Mann-Whitney Test

data:  Q19 by Q9 (1, 2)
Z = 1.8627, p-value = 0.06369
alternative hypothesis: true mu is not equal to 0

Programs that were formal had a higher median number of mentees (35) than informal programs (15). This suggests that more structured, formal mentoring programs tend to have more mentees.

h12 <- h10 %>% select(Q17, Q9)

h12 %>% group_by(Q9) %>% summarise("Median Number of Mentors" = median(Q17))

# A tibble: 2 × 2
     Q9 `Median Number of Mentors`
  <dbl>                      <dbl>
1     1                       42.5
2     2                       23.8

h12$Q9 <- as.factor(h12$Q9)

wilcox_test(Q17 ~ Q9,
            data = h12,
            distribution = "exact")

    Exact Wilcoxon-Mann-Whitney Test

data:  Q17 by Q9 (1, 2)
Z = 1.5845, p-value = 0.1186
alternative hypothesis: true mu is not equal to 0

Requested Additional Analysis

Q39 vs Q16 (Satisfaction vs Require identifying goals)

h16 <- mentoring %>% select("Q16", "Q39_1") %>% filter(!is.na(Q16), !is.na(Q39_1)) 
dim(h2)

h16 %>% group_by(Q16) %>% summarise("Avg_Sat_Score" = mean(Q39_1))

# A tibble: 2 × 2
    Q16 Avg_Sat_Score
  <dbl>         <dbl>
1     1          5.57
2     2          5.42

h16 %>% count(Q16)

# A tibble: 2 × 2
    Q16     n
  <dbl> <int>
1     1    14
2     2    12

h16$Q16 <- as.factor(h16$Q16)

wilcox_test(Q39_1 ~ Q16,
            data = h16,
            distribution = "exact")

    Exact Wilcoxon-Mann-Whitney Test

data:  Q39_1 by Q16 (1, 2)
Z = 0.35138, p-value = 0.7478
alternative hypothesis: true mu is not equal to 0

Q19 vs Q16 (Mentee Count vs Require identifying goals)

h17 <- h7 %>% select("Q16", "Q19") %>% filter(!is.na(Q16), !is.na(Q19)) 
dim(h17)

h17 %>% group_by(Q16) %>% summarise("Median Number of Mentees" = median(Q19))

# A tibble: 2 × 2
    Q16 `Median Number of Mentees`
  <dbl>                      <dbl>
1     1                         35
2     2                         25

h17$Q16 <- as.factor(h17$Q16)

wilcox_test(Q19 ~ Q16,
            data = h17,
            distribution = "exact")

    Exact Wilcoxon-Mann-Whitney Test

data:  Q19 by Q16 (1, 2)
Z = 0.91782, p-value = 0.3758
alternative hypothesis: true mu is not equal to 0

Q17 vs Q16 (Require identifying goals)

h18 <- h10 %>% select("Q16", "Q17") %>% filter(!is.na(Q16), !is.na(Q17)) 
dim(h18)

h18 %>% group_by(Q16) %>% summarise("Median Number of Mentors" = median(Q17))

# A tibble: 2 × 2
    Q16 `Median Number of Mentors`
  <dbl>                      <dbl>
1     1                         45
2     2                         27

h18$Q16 <- as.factor(h18$Q16)

wilcox_test(Q17 ~ Q16,
            data = h18,
            distribution = "exact")

    Exact Wilcoxon-Mann-Whitney Test

data:  Q17 by Q16 (1, 2)
Z = 1.2619, p-value = 0.2158
alternative hypothesis: true mu is not equal to 0

Q39 vs Q32 (Satisfaction vs Requiring meetings)

h19 <- mentoring %>% select("Q32", "Q39_1") %>% filter(!is.na(Q32), !is.na(Q39_1)) 
dim(h19)

h19 %>% group_by(Q32) %>% summarise("Avg_Sat_Score" = mean(Q39_1))

# A tibble: 2 × 2
    Q32 Avg_Sat_Score
  <dbl>         <dbl>
1     1           5.5
2     2           5.5

h19 %>% count(Q32)

# A tibble: 2 × 2
    Q32     n
  <dbl> <int>
1     1    18
2     2     8

h19$Q32 <- as.factor(h19$Q32)

wilcox_test(Q39_1 ~ Q32,
            data = h19,
            distribution = "exact")

    Exact Wilcoxon-Mann-Whitney Test

data:  Q39_1 by Q32 (1, 2)
Z = 0.35033, p-value = 0.7451
alternative hypothesis: true mu is not equal to 0

Q39 vs Q22 (Satisfaction vs Mentor’s Financial Involvement)

mentoring %>% group_by(Q22) %>% summarise(count = n())

# A tibble: 6 × 2
    Q22 count
  <dbl> <int>
1     1     2
2     2    14
3     3     2
4     4     8
5     5     1
6    NA    28

h20 <- mentoring %>%
  mutate(
    Q22_B = case_when(
      Q22 %in% c(1, 2) ~ 2,
      Q22 %in% c(3, 4) ~ 1,
      TRUE ~ NA_real_
    )
  )

h21 <- h20 %>% select("Q22_B", "Q39_1") %>% filter(!is.na(Q22_B), !is.na(Q39_1)) 
dim(h21)

[1] 25  2

h21 %>% group_by(Q22_B) %>% summarise("Avg_Sat_Score" = mean(Q39_1))

# A tibble: 2 × 2
  Q22_B Avg_Sat_Score
  <dbl>         <dbl>
1     1          5.4 
2     2          5.47

h21 %>% count(Q22_B)

# A tibble: 2 × 2
  Q22_B     n
  <dbl> <int>
1     1    10
2     2    15

h21$Q22_B <- as.factor(h21$Q22_B)

wilcox_test(Q39_1 ~ Q22_B,
            data = h21,
            distribution = "exact")

    Exact Wilcoxon-Mann-Whitney Test

data:  Q39_1 by Q22_B (1, 2)
Z = 0, p-value = 1
alternative hypothesis: true mu is not equal to 0

Q39 vs Q28 (Satisfaction vs Protocol to Remove Mentor)

mentoring %>% group_by(Q28) %>% summarise(count = n())

# A tibble: 6 × 2
    Q28 count
  <dbl> <int>
1     1     4
2     3     2
3     4    13
4    12     7
5    13     1
6    NA    28

h22 <- mentoring %>%
  mutate(
    Q28_B = case_when(
      Q28 %in% c("1", "2", "3", "12", "13") ~ 1,
      Q28 == "4" ~ 2,
      TRUE ~ NA_real_
    )
  )

h22 %>% group_by(Q28_B) %>% summarise(count = n())

# A tibble: 3 × 2
  Q28_B count
  <dbl> <int>
1     1    14
2     2    13
3    NA    28

h22 <- h22 %>% select("Q28_B", "Q39_1") %>% filter(!is.na(Q28_B), !is.na(Q39_1)) 
dim(h22)

[1] 26  2

h22 %>% group_by(Q28_B) %>% summarise("Avg_Sat_Score" = mean(Q39_1))

# A tibble: 2 × 2
  Q28_B Avg_Sat_Score
  <dbl>         <dbl>
1     1          5.62
2     2          5.38

h22 %>% count(Q28_B)

# A tibble: 2 × 2
  Q28_B     n
  <dbl> <int>
1     1    13
2     2    13

h22$Q28_B <- as.factor(h22$Q28_B)

wilcox_test(Q39_1 ~ Q28_B,
            data = h22,
            distribution = "exact")

    Exact Wilcoxon-Mann-Whitney Test

data:  Q39_1 by Q28_B (1, 2)
Z = 0.48508, p-value = 0.6313
alternative hypothesis: true mu is not equal to 0

Q39 vs Q37 (Is your mentoring program related to one of your Center’s goals)

h23 <- mentoring %>% select("Q37", "Q39_1") %>% filter(!is.na(Q37), !is.na(Q39_1)) 
dim(h23)

[1] 26  2

h23 %>% group_by(Q37) %>% summarise("Avg_Sat_Score" = mean(Q39_1))

# A tibble: 2 × 2
    Q37 Avg_Sat_Score
  <dbl>         <dbl>
1     1          5.55
2     2          5.33

h23 %>% count(Q37)

# A tibble: 2 × 2
    Q37     n
  <dbl> <int>
1     1    20
2     2     6

h23$Q37 <- as.factor(h23$Q37)

wilcox_test(Q39_1 ~ Q37,
            data = h23,
            distribution = "exact")

    Exact Wilcoxon-Mann-Whitney Test

data:  Q39_1 by Q37 (1, 2)
Z = 0.92745, p-value = 0.3959
alternative hypothesis: true mu is not equal to 0

Q39 vs Q23 (Online only meetings vs not)

mentoring %>% group_by(Q23) %>% summarise(count = n())

# A tibble: 7 × 2
    Q23 count
  <dbl> <int>
1     1     1
2     3     2
3    13     6
4   123     4
5   134     4
6  1234    10
7    NA    28

It appears that 25 centers offer in-person meetings, while only 2 do not. This makes Q23 too unbalanced for a meaningful comparison.

Q19 vs Q29 (Mentee count vs Software use)

h24 <- h7 %>% select("Q29", "Q19") %>% filter(!is.na(Q29), !is.na(Q19)) 
dim(h24)

h24$software <- ifelse(
  grepl(9, h24$Q29), 2, 1
)

h24 %>% count(software)

# A tibble: 2 × 2
  software     n
     <dbl> <int>
1        1     7
2        2    14

h24 %>% group_by(software) %>% summarise("Median Number of Mentees" = median(Q19))

# A tibble: 2 × 2
  software `Median Number of Mentees`
     <dbl>                      <dbl>
1        1                       30  
2        2                       27.5

h24$software <- as.factor(h24$software)

wilcox_test(Q19 ~ software,
            data = h24,
            distribution = "exact")

    Exact Wilcoxon-Mann-Whitney Test

data:  Q19 by software (1, 2)
Z = 0.4488, p-value = 0.675
alternative hypothesis: true mu is not equal to 0

Q17 vs Q29 (Mentor count vs Software use)

h25 <- h10 %>% select("Q29", "Q17") %>% filter(!is.na(Q29), !is.na(Q17)) 
dim(h25)

h25$software <- ifelse(
  grepl(9, h25$Q29), 2, 1
)

h25 %>% count(software)

# A tibble: 2 × 2
  software     n
     <dbl> <int>
1        1    11
2        2    15

h25 %>% group_by(software) %>% summarise("Median Number of Mentors" = median(Q17))

# A tibble: 2 × 2
  software `Median Number of Mentors`
     <dbl>                      <dbl>
1        1                         40
2        2                         25

h25$software <- as.factor(h25$software)

wilcox_test(Q17 ~ software,
            data = h25,
            distribution = "exact")

    Exact Wilcoxon-Mann-Whitney Test

data:  Q17 by software (1, 2)
Z = 1.819, p-value = 0.07047
alternative hypothesis: true mu is not equal to 0

Q39 vs Q31 (Satisfaction vs Majority Mentees Undergrad)

h31 <- mentoring %>%
  mutate(
    Q31_total = rowSums(across(Q31_1:Q31_8)),
    Q31_B = case_when(
      Q31_total == 100 & Q31_1 > 50 ~ 1,
      Q31_total == 100 & Q31_1 <= 50 ~ 0,
      TRUE ~ NA_real_
    )
  )

# Check group counts
h31 %>%
  filter(!is.na(Q39_1)) %>%
  count(Q31_B)

# A tibble: 3 × 2
  Q31_B     n
  <dbl> <int>
1     0    10
2     1    15
3    NA     1

h31 %>% group_by(Q31_B) %>% summarise("Avg_Sat_Score" = mean(Q39_1))

# A tibble: 3 × 2
  Q31_B Avg_Sat_Score
  <dbl>         <dbl>
1     0          5.9 
2     1          5.27
3    NA         NA   

# Compare satisfaction
wilcox.test(Q39_1 ~ Q31_B, data = h31)

    Wilcoxon rank sum test with continuity correction

data:  Q39_1 by Q31_B
W = 91.5, p-value = 0.3493
alternative hypothesis: true location shift is not equal to 0

Q39 vs Q9 (Formal vs informal centers)

h30 <- mentoring %>% select("Q9", "Q39_1") %>% filter(!is.na(Q9), !is.na(Q39_1)) 
dim(h30)

h30 %>% group_by(Q9) %>% summarise("Avg_Sat_Score" = mean(Q39_1))

# A tibble: 2 × 2
     Q9 Avg_Sat_Score
  <dbl>         <dbl>
1     1          5.90
2     2          3.8 

h30 %>% count(Q9)

# A tibble: 2 × 2
     Q9     n
  <dbl> <int>
1     1    21
2     2     5

h30$Q9 <- as.factor(h30$Q9)

wilcox_test(Q39_1 ~ Q9,
            data = h30,
            distribution = "exact")

    Exact Wilcoxon-Mann-Whitney Test

data:  Q39_1 by Q9 (1, 2)
Z = 2.5642, p-value = 0.005853
alternative hypothesis: true mu is not equal to 0

Holm Adjusted P-values

raw_p <- c(
  Q39_Q26  = 0.002416,
  Q39_Q9 = 0.005853,
  Q39_Q12a = 0.01752,
  Q39_Q6   = 0.02000,
  Q39_Q27  = 0.02779,
  Q19_Q32  = 0.01666,
  Q17_Q32  = 0.09861,
  Q19_Q9   = 0.06369,
  Q17_Q9   = 0.11860,
  Q39_Q16  = 0.74780,
  Q19_Q16  = 0.37580,
  Q17_Q16  = 0.21580,
  Q39_Q32  = 0.74510,
  Q39_Q22  = 1.00000,
  Q39_Q28  = 0.63130,
  Q39_Q37  = 0.39590,
  Q19_Q29  = 0.67500,
  Q17_Q29  = 0.07047,
  Q39_Q31  = 0.34930
)

results <- data.frame(
  Comparison = names(raw_p),
  Raw_P = unname(raw_p),
  Holm_P = unname(p.adjust(raw_p, method = "holm"))
)

results$Significant_Holm <- results$Holm_P < 0.05

results

   Comparison    Raw_P   Holm_P Significant_Holm
1     Q39_Q26 0.002416 0.045904             TRUE
2      Q39_Q9 0.005853 0.105354            FALSE
3    Q39_Q12a 0.017520 0.283220            FALSE
4      Q39_Q6 0.020000 0.300000            FALSE
5     Q39_Q27 0.027790 0.389060            FALSE
6     Q19_Q32 0.016660 0.283220            FALSE
7     Q17_Q32 0.098610 1.000000            FALSE
8      Q19_Q9 0.063690 0.827970            FALSE
9      Q17_Q9 0.118600 1.000000            FALSE
10    Q39_Q16 0.747800 1.000000            FALSE
11    Q19_Q16 0.375800 1.000000            FALSE
12    Q17_Q16 0.215800 1.000000            FALSE
13    Q39_Q32 0.745100 1.000000            FALSE
14    Q39_Q22 1.000000 1.000000            FALSE
15    Q39_Q28 0.631300 1.000000            FALSE
16    Q39_Q37 0.395900 1.000000            FALSE
17    Q19_Q29 0.675000 1.000000            FALSE
18    Q17_Q29 0.070470 0.845640            FALSE
19    Q39_Q31 0.349300 1.000000            FALSE

Building Models

models <- mentoring %>%
  mutate(
    mentor_training = case_when(
      Q26 == 1 ~ 1,
      Q26 == 2 ~ 0,
      TRUE ~ NA_real_
    ),
    formal_program = case_when(
      Q9 == 1 ~ 1,
      Q9 == 2 ~ 0,
      TRUE ~ NA_real_
    ),
    mentor_screening = case_when(
      Q12...29 == 1 ~ 1,
      Q12...29 == 2 ~ 0,
      TRUE ~ NA_real_
    ),
    group_mentoring = case_when(
      is.na(Q6) ~ NA_real_,
      grepl("3", as.character(Q6)) ~ 1,
      TRUE ~ 0
    )
  )

Model 1: Mentor training + formal program → Satisfaction

m1 <- models %>%
  select(Q39_1, mentor_training, formal_program) %>%
  filter(complete.cases(.))

m1 %>% count(mentor_training, formal_program)

# A tibble: 4 × 3
  mentor_training formal_program     n
            <dbl>          <dbl> <int>
1               0              0     3
2               0              1     7
3               1              0     2
4               1              1    14

model1 <- lm(
  Q39_1 ~ mentor_training + formal_program,
  data = m1
)

summary(model1)

Call:
lm(formula = Q39_1 ~ mentor_training + formal_program, data = m1)

Residuals:
    Min      1Q  Median      3Q     Max 
-1.3409 -0.2873 -0.2873  0.7127  1.8604 

Coefficients:
                Estimate Std. Error t value Pr(>|t|)    
(Intercept)       3.3409     0.4319   7.735 7.61e-08 ***
mentor_training   1.1477     0.3741   3.068 0.005442 ** 
formal_program    1.7987     0.4618   3.895 0.000729 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 0.9061 on 23 degrees of freedom
Multiple R-squared:  0.5757,    Adjusted R-squared:  0.5388 
F-statistic:  15.6 on 2 and 23 DF,  p-value: 5.227e-05

Diagnostics

diagnostics1 <- m1 %>%
  mutate(
    residual = residuals(model1),
    predicted = fitted(model1),
    group = interaction(mentor_training, formal_program)
  )

boxplot(residual ~ group,
        data = diagnostics1,
        main = "Residuals by Group",
        xlab = "Mentor Training / Formal Program",
        ylab = "Residuals")

stripchart(residual ~ group,
           data = diagnostics1,
           vertical = TRUE,
           method = "jitter",
           add = TRUE,
           pch = 16)

abline(h = 0, col = "red", lty = 2)

car::leveneTest(residual ~ group,
                data = diagnostics1,
                center = median)

Levene's Test for Homogeneity of Variance (center = median)
      Df F value Pr(>F)
group  3  0.8695 0.4718
      22               

shapiro.test(residuals(model1))

    Shapiro-Wilk normality test

data:  residuals(model1)
W = 0.89828, p-value = 0.01433

qqnorm(residuals(model1))
qqline(residuals(model1), col = "red")

# Cook's distance plot
plot(cooks.distance(model1),
     type = "h",
     main = "Cook's Distance - Model 1",
     xlab = "Observation",
     ylab = "Cook's Distance")

# Reference cutoff
abline(h = 4 / nobs(model1), col = "red", lty = 2)

which(cooks.distance(model1) > 4 / nobs(model1))

 2  9 14 15 23 
 2  9 14 15 23 

Model 2: Mentor screening + group mentoring → Satisfaction (ignore)

m2 <- models %>%
  select(Q39_1, mentor_screening, group_mentoring) %>%
  filter(complete.cases(.))

m2 %>% count(mentor_screening, group_mentoring)

# A tibble: 4 × 3
  mentor_screening group_mentoring     n
             <dbl>           <dbl> <int>
1                0               0     8
2                0               1     5
3                1               0     4
4                1               1     9

model2 <- lm(
  Q39_1 ~ mentor_screening + group_mentoring,
  data = m2
)

summary(model2)

Call:
lm(formula = Q39_1 ~ mentor_screening + group_mentoring, data = m2)

Residuals:
    Min      1Q  Median      3Q     Max 
-2.5486 -0.3765  0.4514  0.5673  1.5972 

Coefficients:
                 Estimate Std. Error t value Pr(>|t|)    
(Intercept)        4.5486     0.3670  12.394 1.16e-11 ***
mentor_screening   0.8543     0.4733   1.805   0.0842 .  
group_mentoring    0.9737     0.4747   2.051   0.0518 .  
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 1.148 on 23 degrees of freedom
Multiple R-squared:  0.319, Adjusted R-squared:  0.2598 
F-statistic: 5.387 on 2 and 23 DF,  p-value: 0.01205

lmtest::coeftest(model2, vcov. = sandwich::vcovHC(model2, type = "HC3"))

t test of coefficients:

                 Estimate Std. Error t value  Pr(>|t|)    
(Intercept)       4.54858    0.48929  9.2962 2.974e-09 ***
mentor_screening  0.85425    0.51602  1.6555   0.11141    
group_mentoring   0.97368    0.54166  1.7976   0.08538 .  
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Diagnostics 2

diagnostics2 <- m2 %>%
  mutate(
    residual = residuals(model2),
    predicted = fitted(model2),
    group = interaction(mentor_screening, group_mentoring)
  )

boxplot(residual ~ group,
        data = diagnostics2,
        main = "Residuals by Group",
        xlab = "Mentor Screening / Group Mentoring",
        ylab = "Residuals")

stripchart(residual ~ group,
           data = diagnostics2,
           vertical = TRUE,
           method = "jitter",
           add = TRUE,
           pch = 16)

abline(h = 0, col = "red", lty = 2)

car::leveneTest(residual ~ group,
                data = diagnostics2,
                center = median)

Levene's Test for Homogeneity of Variance (center = median)
      Df F value Pr(>F)
group  3  1.0817 0.3774
      22               

shapiro.test(residuals(model2))

    Shapiro-Wilk normality test

data:  residuals(model2)
W = 0.90717, p-value = 0.02272

qqnorm(residuals(model2))
qqline(residuals(model2), col = "red")

# Cook's distance plot
plot(cooks.distance(model2),
     type = "h",
     main = "Cook's Distance - Model 2",
     xlab = "Observation",
     ylab = "Cook's Distance")

# Reference cutoff
abline(h = 4 / nobs(model2), col = "red", lty = 2)

which(cooks.distance(model2) > 4 / nobs(model2))

14 15 
14 15 

Model 3: Required meetings + formal program → Number of mentees

m3 <- h7 %>%
  mutate(
    required_meetings = case_when(
      Q32 == 1 ~ 1,
      Q32 == 2 ~ 0,
      TRUE ~ NA_real_
    ),
    formal_program = case_when(
      Q9 == 1 ~ 1,
      Q9 == 2 ~ 0,
      TRUE ~ NA_real_
    )
  ) %>%
  select(Q19, required_meetings, formal_program) %>%
  filter(complete.cases(.))

m3 %>% count(required_meetings, formal_program)

# A tibble: 4 × 3
  required_meetings formal_program     n
              <dbl>          <dbl> <int>
1                 0              0     2
2                 0              1     5
3                 1              0     3
4                 1              1    11

model3 <- lm(
  log(Q19) ~ required_meetings + formal_program,
  data = m3
)

summary(model3)

Call:
lm(formula = log(Q19) ~ required_meetings + formal_program, data = m3)

Residuals:
    Min      1Q  Median      3Q     Max 
-1.4292 -0.4923  0.1902  0.4632  0.7681 

Coefficients:
                  Estimate Std. Error t value Pr(>|t|)    
(Intercept)         2.3561     0.3404   6.921 1.81e-06 ***
required_meetings   0.6825     0.3000   2.275   0.0354 *  
formal_program      0.8549     0.3321   2.575   0.0191 *  
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 0.6461 on 18 degrees of freedom
Multiple R-squared:  0.4158,    Adjusted R-squared:  0.3509 
F-statistic: 6.406 on 2 and 18 DF,  p-value: 0.007926

Diagnostics 3

diagnostics3 <- m3 %>%
  mutate(
    residual = residuals(model3),
    predicted = fitted(model3),
    group = interaction(required_meetings, formal_program)
  )

boxplot(residual ~ group,
        data = diagnostics3,
        main = "Residuals by Group",
        xlab = "Required Meetings / Formal Program",
        ylab = "Residuals")

stripchart(residual ~ group,
           data = diagnostics3,
           vertical = TRUE,
           method = "jitter",
           add = TRUE,
           pch = 16)

abline(h = 0, col = "red", lty = 2)

car::leveneTest(residual ~ group,
                data = diagnostics3,
                center = median)

Levene's Test for Homogeneity of Variance (center = median)
      Df F value Pr(>F)
group  3  0.7576 0.5332
      17               

shapiro.test(residuals(model3))

    Shapiro-Wilk normality test

data:  residuals(model3)
W = 0.93028, p-value = 0.1394

qqnorm(residuals(model3))
qqline(residuals(model3), col = "red")

# Cook's distance plot
plot(cooks.distance(model3),
     type = "h",
     main = "Cook's Distance - Model 3",
     xlab = "Observation",
     ylab = "Cook's Distance")

# Reference cutoff
abline(h = 4 / nobs(model3), col = "red", lty = 2)

which(cooks.distance(model3) > 4 / nobs(model3))

12 
12