LACI HR Grader Analytics Project
Kiran Krishnamurthi
8/27/2021
## [1] "Timestamp"
## [2] "Name"
## [3] "Candidate Name"
## [4] "Are you..."
## [5] "Question 1: LACI projects require members to make logical and innovative decisions. Tell us a time of when you disagreed with a group and the steps you took to justify your decision."
## [6] "2. On your written app you mentioned you were passionate about [INSERT NONPROFIT]. What about their mission resonates with you and why?"
## [7] "3. What do you think are some of the social barriers the nonprofit you mentioned faces in serving their target population? Please note that social barriers are different from business challenges."
## [8] "4. In a client interview, what would you ask Reach for the Stars to gauge if they have the capacity to grow and why?"
## [9] "5. As mentioned earlier, Reach for the Stars has a special focus on communities with low-income students and students of color. What data would you need to figure out where the need might be greatest for Reach for the Stars’ services and how would you obtain it?"
## [10] "What else are they planning on being involved in this semester? (Only ask if they were red-flagged)"
## [11] "What were this candidate’s pros/strengths?"
## [12] "What are your biggest qualms or hesitations about the candidate (if any)?"
## [13] "Should this candidate move onto second rounds?"
## [14] "Describe WHY you feel this way, focusing on the connections drawn throughout the each individual question and details they do or do not give. Include specific details and examples from the interview."
## [15] "If there were any technical difficulties during the interview, please describe what happened."## # A tibble: 0 x 12
## # … with 12 variables: Name <chr>, Q1_AVG <dbl>, Q2_AVG <dbl>, Q3_AVG <dbl>,
## # Q4_AVG <dbl>, Q5_AVG <dbl>, Q1_Diff <dbl>, Q2_Diff <dbl>, Q3_Diff <dbl>,
## # Q4_Diff <dbl>, Q5_Diff <dbl>, Over_Under <dbl>## [1] "Your name"
## [2] "Interviewee name"
## [3] "Are you...?"
## [4] "Question #1: \"LACI projects are often dynamic and require members to adapt quickly to changes. Can you tell us about a time you encountered a challenge? How did you adapt and what did you learn? How would you apply these lessons in LACI?”"
## [5] "Question #2: You mentioned in your application that you're particularly passionate about __________ (insert cause from #2 on written app). How did you first discover this issue and what have you learned about it? How have your thoughts about the issue developed since?"
## [6] "Question #3 [follow up]: With unlimited time and resources, what steps would you take to address this issue?"
## [7] "Question #4 (CASE): [1] \"Before you give this organization any recommendations on how to manage their current challenges, what information would you want to know about the organization or situation as a whole?\" (Feel free to take a minute to think about it) [2] “Given this new information, how might it change the way you’re thinking about the nonprofit’s current challenges? What additional information would you want to collect, and what considerations would you now have to make. If this would not change the course of your research, please explain your reasoning.\""
## [8] "What were this candidate's pros/strengths?"
## [9] "What were this candidate's cons/weaknesses?"
## [10] "Should this candidate move onto second rounds?"
## [11] "Explain WHY you feel this way, focusing on the connections drawn throughout the each individual question and details they do or do not give. Include specific details and examples from the interview."## # A tibble: 0 x 10
## # … with 10 variables: Name <chr>, Q1_AVG <dbl>, Q2_AVG <dbl>, Q3_AVG <dbl>,
## # Q4_AVG <dbl>, Q1_Diff <dbl>, Q2_Diff <dbl>, Q3_Diff <dbl>, Q4_Diff <dbl>,
## # Over_Under <dbl>## [1] "Your Name"
## [2] "Interviewee Name"
## [3] "Are you...?"
## [4] "Question #1: LACI members are tasked with many assignments and roles that require time management and prioritization. Describe a time where you felt overwhelmed by various roles or assignments. How did you overcome these challenges and adapt to the demand?"
## [5] "Question #2: You mentioned in your application you’re particularly passionate about __________ (insert cause from #2 on written app). What do you think are some of the biggest barriers to solving this problem? With unlimited time and resources, what steps would you take to overcome these barriers?"
## [6] "Question #3: What do you think is the biggest challenge facing LA or your home community? What are the long term impacts of this challenge?"
## [7] "Question #4 (CASE QUESTION): [1] What type of information would you need to know BEFORE thinking about possible recommendations? Consider info about the client itself, the industry, and general business practices. (Feel free to take a minute to think about it) [2] Of the information you would like to know, what would you prioritize in your research? Pick 3-5 of the types of information you stated previously."
## [8] "What else are they planning on being involved in this semester? (Only ask if they were red-flagged)"
## [9] "What were this candidate’s pros/strengths?"
## [10] "What are your biggest qualms or hesitations about the candidate (if any)?"
## [11] "Should this candidate move onto second rounds?"
## [12] "Describe WHY you feel this way, focusing on the connections drawn throughout the each individual question and details they do or do not give. Include specific details and examples from the interview."## # A tibble: 0 x 10
## # … with 10 variables: Name <chr>, Q1_AVG <dbl>, Q2_AVG <dbl>, Q3_AVG <dbl>,
## # Q4_AVG <dbl>, Q1_Diff <dbl>, Q2_Diff <dbl>, Q3_Diff <dbl>, Q4_Diff <dbl>,
## # Over_Under <dbl>## [1] "Timestamp"
## [2] "Your Name"
## [3] "Interviewee Name"
## [4] "Question #1: Tell us about a time in an organization or job in which you had to work on a team to accomplish a goal or task. What role did you play in the team, and how did you contribute? How does this reflect, more generally, your personal approach to teamwork?"
## [5] "Question #2: Describe a time when you took initiative and did something that wasn't asked of you. What did you do? Why did you feel compelled to do it?"
## [6] "Question #3: On your written application, you mentioned being particularly passionate about __________. Can you talk about your personal connection or interest in this issue? Feel free to talk about your own past or current involvement, connection to those affected, or other factors that have catalyzed your interest."
## [7] "Question #4: If you had unlimited time and resources, how would you approach crafting or innovating a solution to this social issue? Feel free to think big but be specific about your intended plan of action."
## [8] "Question #5 (CASE QUESTION): If you were going to meet with your client for the first time to talk about the project, (1) what kinds of questions would you ask them (let them answer, then ask next part)? Once you have spoken with your client, what kinds of research would you begin to conduct to help form some recommendations for them?"
## [9] "What else are they planning on being involved in this semester? (Only ask if they were red-flagged)"
## [10] "Was this person clearly passionate about social impact?"
## [11] "Describe WHY you selected the above answer, citing their demonstrated understanding or experience from the interview."
## [12] "What were this candidate’s pros/strengths?"
## [13] "What are your biggest qualms or hesitations about the candidate (if any)?"
## [14] "Should this candidate move onto second rounds?"
## [15] "Describe WHY you feel this way, focusing on how well they communicated, addressed difficult questions, and spoke about teamwork experiences. Include specific details and examples from the interview."## # A tibble: 0 x 12
## # … with 12 variables: Name <chr>, Q1_AVG <dbl>, Q2_AVG <dbl>, Q3_AVG <dbl>,
## # Q4_AVG <dbl>, Q5_AVG <dbl>, Q1_Diff <dbl>, Q2_Diff <dbl>, Q3_Diff <dbl>,
## # Q4_Diff <dbl>, Q5_Diff <dbl>, Over_Under <dbl>
LACI Grader Analytics over the Past 5 Recruitment Cycles
Last year HR set out to help standardize our grading efforts. Evaluating candidates is a qualitative process and leaves lots of room for subjectivity and bias in the way we might judge certain candidates. Sure, these sound like problems but it’s important to recognize that these issues are fundamental to the nature of how we run our recruitment. We’re looking for qualities and characteristecs in potential LACI members, and without a standardized algorithm to detect what we’re looking for the process is left in the hands of our proctors. There are ways to help ensure consistency in our qualitative grading, such as creating a rubric and defining specific criteria we’re looking for. Nonetheless, the problem that still exists is that we’re mapping a numeric scale onto something that’s purely qualitative and at the end of the day relies on the proctor to make a decision how they feel the candidates ansewr best aligns with the scoring.
Our standardization effort from last year was 2-fold. First, we realized that some questions will have higher average scores than others. Ideally, the average score for all questions is a 3, which would create a perfect bell curve of responses. Nonetheless, some questions will have averages closer to 4 and some closer to 2. The problem that arises here is that if a candidate scores a 4 on a question that’s got a higher average, their performance is equally weighted as someone who might score a 4 on a question with a lower average. In this situation, scoring a 4 on a question with a lower average is the more impressive response given that the majority of candidates had responses of lower scores, but without differentiating distributions in scoring across questions we’re left without a way to distinguish this performance. Thus, last year HR began evaluating candidate scoring on the basis of the Z scores for each question.
Secondly, last year HR also created the “fake applicant” grading as part of calibrations. For this, we wrote out answers to the written and interview questions which we felt provided a variety of quality in their responses. Calibrations attendees were then told to score the responses based upon how they felt the answers aligned with the rubric to give us an equitable distribution of scoring. The importance of the fake applicant is that it was possibly the only time ALL recruitment participants would’ve scored the same answers, which allowed us at HR to get a sense of how LACI members grade while controlling the response quality variable.
Evaluation of our Spring 2021 Efforts
These are the density functions for each question from our written application grading where we paired proctors based upon their grading tendencies. The Q1, Q2, and Q3 scores used in this are the average score across the two proctors for each candidate. As you can see, the density functions above aren’t perfect bell curves. Questions 1 and 2 were more traditional LACI questions (one personal reflection and one Why LACI? question), whereas question 3 was our diversity question. The results aren’t perfect for any of the three questions but in this instance question 3 is furthest from a normal distribution.
The plots above present density distributions of the four interview questions we asked in the Spring 2021 semester. The goal of these plots would be to reveal a normalized distribution as evidenced by a bell-curve. The reasoning for our goal is that we attempted to pair proctors together based upon their grading tendencies. Therefore, inflated scores in these scenarios should be brought back to the average thanks to a harsher counter perspective from a second proctor. These visualizations don’t determine anything statistically conclusive, but we can infer that our results weren’t as successful as we would’ve hoped given the distribution shapes not properly alligning with the average of 3 or being bell-shaped.
Fall_2021 = read_excel("Fall 2021 Written App Scorecard (Responses).xlsx")
Fall_2021_I = read_excel("LACI First Round Scorecard Fall 2021 (Responses).xlsx")
Fall_2021_I = Fall_2021_I[complete.cases(Fall_2021_I) == FALSE,]
colnames(Fall_2021_I)## [1] "Timestamp"
## [2] "Name"
## [3] "Candidate Name"
## [4] "Are you..."
## [5] "Q1 Score (Time Management)"
## [6] "Q2 Score (Teamwork)"
## [7] "Q3 Score (Social Impact)"
## [8] "Q4 Score (Mini Case Part 1)"
## [9] "Q5 Score (Mini Case Part 2)"
## [10] "What else are they planning on being involved in this semester? (Only ask if they were red-flagged)"
## [11] "What were this candidate’s pros/strengths?"
## [12] "What are your biggest qualms or hesitations about the candidate (if any)?"
## [13] "Should this candidate move onto second rounds?"
## [14] "Describe WHY you feel this way, focusing on the connections drawn throughout the each individual question and details they do or do not give. Include specific details and examples from the interview."
## [15] "If there were any difficulties or things to note outside of the above questions please mark them down here!"Fall_2021_I$Q1 = Fall_2021_I$`Q1 Score (Time Management)`
Fall_2021_I$Q2 = Fall_2021_I$`Q2 Score (Teamwork)`
Fall_2021_I$Q3 = Fall_2021_I$`Q3 Score (Social Impact)`
Fall_2021_I$Q4 = Fall_2021_I$`Q4 Score (Mini Case Part 1)`
Fall_2021_I$Q5 = Fall_2021_I$`Q5 Score (Mini Case Part 2)`
F2021_I_Aggregate = Fall_2021_I %>% summarise(Group_Q1_AVG = mean(Q1),
Group_Q2_AVG = mean(Q2),
Group_Q3_AVG = mean(Q3),
Group_Q4_AVG = mean(Q4),
Group_Q5_AVG = mean(Q5)) %>%
mutate(Name = "Group_Average") %>% relocate(Name, .before = Group_Q1_AVG)
F2021_I_Indiv = Fall_2021_I %>% group_by(Name) %>%
summarise(Q1_AVG = mean(Q1),
Q2_AVG = mean(Q2),
Q3_AVG = mean(Q3),
Q4_AVG = mean(Q4),
Q5_AVG = mean(Q5))
F2021_I_Indiv$Q1_Diff = F2021_I_Indiv$Q1_AVG - test_group$Group_Q1_AVG
F2021_I_Indiv$Q2_Diff = F2021_I_Indiv$Q2_AVG - test_group$Group_Q2_AVG
F2021_I_Indiv$Q3_Diff = F2021_I_Indiv$Q3_AVG - test_group$Group_Q3_AVG
F2021_I_Indiv$Q4_Diff = F2021_I_Indiv$Q4_AVG - test_interview_grouped$Group_Q4_AVG
F2021_I_Indiv$Q5_Diff = F2021_I_Indiv$Q5_AVG - F2021_I_Aggregate$Group_Q5_AVG
F2021_I_Indiv$Over_Under = F2021_I_Indiv$Q1_Diff + F2021_I_Indiv$Q2_Diff +
F2021_I_Indiv$Q3_Diff + F2021_I_Indiv$Q4_Diff + F2021_I_Indiv$Q5_Diff
F2021_I_Indiv %>% arrange(desc(Over_Under)) %>% view()