Replication Report for Study 2 by Galperin, Hahl, Stering, & Guo (2020, Administrative Science Quarterly)
Jiwon Byun (jwbyun@stanford.edu)
December 01, 2021
Introduction
Employers’ selection of job applicants is an important labor market proesses for both employers and individuals. For organizations, access to high-quality workers is a source of competitive advantage. For individuals, access to certain jobs at certain timing can have a large impact on one’s career trajectories. Job signaling is the most fundamental and important aspect of selection processes in labor market. Information asymmetries between employers and job applicants provide challenges for both ends.
Most studies have focused on job signaling and applicant capabilities when trying to understand selection processes in labor markets. On the other hand, increasing number of studies suggest that selection decisions are affected not only by perceptions of applicantss’ capability but also by perceptions of applicants’ commitment to the organization.
While previous literature investigated both the effect of perceptions of applicants’ capability and the effect of perceptions of applicants’ commitment on the likelihood of a candidate’s being selected for a job, the relationship between the two dimensions had been understudied. In an attempt to shed light on such a gap in the literature, Galperin, Hahl, Sterling, and Guo (Gelperin et al. henceforth) conducted a series of online field experiments and published an article in ASQ in the year of 2020. (For reference, see Links section in the below.)
Galperin et al. (2020) suggested that signals that influence perceptions of capability in labor markets may have discernable effects of the perceptions of a candidate’s commitment, which also affects the likelihood of the candidate’s being selected for the job. The paper included four studies, three of which were online field experiments. In this study, I plan to replicate Study 2.
Justification for Choice of Study
In the original paper, the authors conducted a direct test of the effect of applicants’ commitment and high-capability rejection. Study 2 was conducted to directly manipulated perceived commitment of an applicant to complement one of the other studies where they could not directly manipulate commitment. This is important because the main contribution of the study is suggested (and proved) mediation effect of commitment on likelihood of a job offer.
Anticipated Challenges
I anticipate two challenges while attempting to replicate the results. First, I may run into sample issue. This study is about job signals and hiring managers’ decision making in making job offers. Hence, it is important that I only have hiring managers (defined by someone with experiences as a hiring manager) in the sample. Since I have to rely on Prolifics to do the sampling, less appropriate sample might be acquired in the end. Similar issue applies to random assignment of participants. Second, survey design might be difficult. The main part of this replication study is manipulating commitment of a job applicant. Since commitment could mean different things both conceptually and practically, it is essential that I manipulate the main concepts elaborately. To overcome this challenge, I plan to reach out to original authors and try to acquire survey items that they used for the original study.
Methods
Description of the Steps Required to Replicate the Results
To replicate the results of Study 2, I plan to run the same online experiment on Prolific. Prior to running the main study, I plan to run two pilot studies. The first pilot study will involve non-naive participants, whereas the second pilot study will have naive participants on MTurk.
For the main study, participants will be recruited to fill out a “survey for hiring managers” through Prolific in October-November 2021.
The study will have the same structure as that of the original study: 2 by 2 between-subjects design. Participants will be randomly assigned to one of the four conditions. (2 capability conditions X 2 commitment conditions) They will be given the same stimuli as the original study.
I plan to conduct a manipulation check for both candidate’s perceived capability and perceived commitment by combining the questionnaires included in the survey.
With the data acquired from the survey experiment, I plan to conduct two t-tests to replicate the key results of Study 2. As an exploratory analysis, I plan to conduct a two-way ANOVA to examine the interaction effects.
Power Analysis
The original study had a sample size of 212. For the first t-test (neutral commitment condition), the statistics were: t = 3.66, p < 0.001. For the second t-test (high commitment condition), the results were: t = -3.34, p = 0.001.
Since I have two t-test for the study, I chose the t-test with smaller t value and used the study for the power analysis, which is the second t-test with t = -3.34. Then I did a calculation assuming a balanced cell size.
Results of the effect size calculation for the second t-test (calculated by using mean and s.d.) were as follows: Cohen's d = 0.6554, effect-size r = 0.3114. I conducted the effect size calculation using the t-value and degrees of freedom as well and obtained very similar results.
Based on the effect size, I conducted a power analysis. Results for the power analysis for 80%, 90%, 95% power (for the second t-test) are as follows:
- 80% power: 76
- 90% power: 100
- 95% power: 124
Since the original work did have two t-tests, I needed to take the results and multiply by 2 to get the results for power analysis for the whole study. Therefore, the final results for the power analysis were as follows:
- 80% power: 152
- 90% power: 200
- 95% power: 248
The sample size for the 80% power (based on the original effect of the second t-test) was 152. I could not have 152 participants due to some constraints (mainly financial.) The original study paid the participants high compensation ($6.00) for this cognitively engaging study, I want to provide a reasonable amount of compensation to each participant. Considering the approximate time needed to complete the study and the hourly pay, I decided to pay $2.00 each. My budget is limited at about $400.00 for the study, so I decided have a sample size of 150.
Planned Sample
The sample will consist of 150 participants recruited from Prolific. In order to participate in the study, individuals should have an experience hiring people for the organization because this study addresses job decision making processes of hiring managers in the labor market.
Materials
The survey experiment materials were coded in Qualtrics. Participants recuited on Prolific could click on the link to the survey on Qualtrics. Participants were randomly assigned to 2 capability conditions that involved two different job candidate resumes. Participants were also randomly assigned to 2 commitment conditions given by information from HR department. The procedure for the study is described below.
Procedure
"The procedure for Study 2 was very similar to that for Study 1. We used the same job candidate profiles as in Study 1, but instead of evaluating both candidates, participants viewed only one profile and were asked to evaluate only that candidate applying to a ‘corporate finance position in a medium-size health care company.’ To verify that participants understood the directions, we asked them a series of attention-check questions. If they answered incorrectly, the context was repeated to them as a reinforcement.
One additional difference between Study 1 and Study 2 was that in this experiment, participants were told that their job as ‘line manager’ meant they ‘should not concern themselves with negotiating the offer, but should only consider this candidate’s potential performance on the job.’ This was repeated before the participant was asked about his or her likelihood of giving the candidate an offer. We also asked participants to rate from 1 (low) to 7 (high) how concerned they were about the candidate accepting the job. There was no difference across conditions in this measure, and results presented below are robust to controlling for this measure. This indicates that participants behaved as intended and focused on post-hire commitment rather than concerns about the candidate’s acceptance of a job offer.
Participants were randomly assigned to view either the extremely highcapability candidate or the moderately high-capability candidate, who were presented in the same way as they were in Study 1. Next, participants were randomly assigned to one of two commitment-information conditions: neutral or high. All participants were told, ‘As part of the application process, the candidate completed an assessment delivered by your firm’s HR department,’ that the candidate scored a 75 on the assessment, and that ‘candidates that have scored at least 60 points have skills that match the requirements of the job.’ In the neutral-commitment condition, nothing was mentioned about the assessment’s relationship with commitment. Participants were told that ‘this assessment was able to predict the candidate’s ability to do the job in your firm.’ In contrast, participants randomly assigned to the high-commitment condition were told that ‘this assessment was able to predict the candidate’s likely commitment to your organization (e.g., likelihood to stay with your firm, be motivated and work well with others).’ This procedure resulted in a 2 (extremely high capability/moderately high capability) by 2 (high commitment/neutral commitment) between-subject design.
Participants were then asked about their likelihood of giving the candidate a job offer based on the same scale used in Study 1. This variable served as the dependent variable." (Galperin et al., 2020)
For this replication, the original work’s procedure (described in the quotation above) was followed precisely.
Manipulation Checks
"As a manipulation check after rating their likelihood to give the candidate an offer, participants were then asked to evaluate the candidate’s capability by rating him on a 7-point scale (1 = low and 7 = high) on four criteria related to capability: ‘‘how competent the candidate would be in his job’’; ‘‘how productive the candidate was in his previous job’’; ‘‘how skilled the candidate is’’; and ‘‘how much finance experience the candidate has.’’ We combined these four measures to create a measure for the candidate’s perceived capability (Cronbach’s alpha = .88). Participants randomly assigned to the extremely high-capability conditions (N = 106, mean = 5.86, S.D. = .82) rated the candidate higher in perceived capability (t = − 6.60, d.f. = 210, p < .001, two-tailed test) than those assigned to the moderately high-capability condition (N = 106, mean = 4.99, S.D. = 1.08).
Participants were also asked about perceived commitment after these ratings. They were asked to rate (1 = low, 7 = high) the candidate on the following four dimensions: ‘‘likely commitment to the organization,’’ ‘‘commitment to the industry,’’ ‘‘willingness to stay with the company,’’ and ‘‘willingness to work extra hours.’’ We added commitment to the industry to the measure (compared with the Study 1 scale) to address the change in industry. These ratings were averaged to create a perceived organizational commitment score (Cronbach’s alpha = .89). Participants randomly assigned to the highcommitment conditions (N = 105, mean = 5.44, S.D. = 1.16) rated the candidate higher in commitment (t = − 5.94, d.f. = 210, p < .001, two-tailed test) than those assigned to the neutral-commitment conditions (N = 107, mean = 4.51, S.D. = 1.57)." (Galpein et al., 2020)
For this replication, the original work’s procedure (described in the quotation above) for manipulation checks was followed precisely.
Analysis Plan
Confirmatory statistical analysis is going to involve two t-tests, following the methods used in the original work. I run a t-test for High-commitment conditions, and another t-test for Neutral-commitment conditions. By doing so, I can test differential effects of extremely high capability on likelihood of getting an offer for different commitment conditions.
I plan to conduct the same manipulation check analyses of those from original work. There are 4 measures for capability and 4 measures for commitment. As the original authors did, I will average the ratings for capability and commitment (separately for each manipulation) and created a perceived capability measure and a perceived commitment measure. Then I will examine Cronbach’s alpha value and conduct t-tests across different treatment groups.
Since the original work aims to explore the interactional effects of capability and commitment on a candidate’s likelihood of getting an offer, I plan to conduct a two-wawy ANOVA as an exploratogy analysis. Two-way ANOVA can be a good statitical model since I am trying to examine the interation effect of candidate’s capability and commitment (Capability X Commitment) on the likelihood of receiving an offer. By using the two-way factorial design, I can effectively examine the potential interaction effect between Capability and Commitment.
Pilot B and Sampling
While the first pilot (Pilot A) went smoothly, I had some issues with Pilot B. So I had to run Pilot B three times to identify the appropriate strategy to run the study on Prolific. Three pilots were conducted on Prolific between November 12, 2021 and November 14, 2021. All three pilots were conducted with sample size of N=4. The participants were recruited on Prolific. They had to have hiring experience to be eligible for the study. The original plan was to recruit participants internationally on Prolific. After the first Pilot B, I realized that this sampling strategy was not appropriate. Even with the language filter, (fluency in English) international participants were taking too long to complete the study. The median value for the duration of the stuay was approximately 24 minutes for a 10-minute study. Therefore I conducted the second Pilot B with different sampling strategy. I added another filter so that only residents of U.S. can participate in the study. I also deleted all the open-ended questions from the survey. After running the second pilot, I could see that the problem was more about participants’ country of residence than having open-ended questions in the survey. Hence, I ran the third Pilot B with the original survey design. The sampling strategy was the same as the second pilot (individuals with hiring experience who live in the U.S., fluent in English). The third pilot ran without any problem and I was able to collect data.
Differences from Original Study
The most important difference is that my study involves a different sample. First, the sample size will be different. The original work a sample size of N=212. My sample will have fewer participants than that of original paper. Since the study involves 2 x 2 design, such a difference in sample sizes may result in different statistical power. Furthermore, the nature of the sample may vary. While the original work recruited participants through Qualtrics panel, I will recruit participants through Prolific. Substantial differences may exist when filtering for the target participants. The most important aspect of the sample is that the sample should include only individuals who served as hiring managers. For the original study, the participants were Qualtrics-verified hiring managers who were U.S. based. All the participants could provide evidence of having served as a hiring manager. On the other hand, I do not have a means of verifying participants’ experience as hiring managers. I have to rely on self-report on Prolific. Hence, it is possible that the nature of my sample will be different from the original study. Another important difference is timing. This research studies hiring in the labor market. With hiring, timing is very important because there are a lot of time sensitivities. Although the original work was published in 2020, Study 2 was conducted in December 2016. This replication is conducted in November 2021. Since there is 5-year time difference, how people think about hiring could have changed substantially over those 5 years, especially since the outbreak of the COVID-19 pandemic. I acknowledge that any unforeseen issues are the sole responsiblitity of mine.
Methods Addendum
Actual Sample
Sample size: My final sample consists of N=137 participants. I originally planned to have a sample size of N=150. However, I could not recruit 150 participants due to the limited budget. I had to run multiple pilot studies as I encountered minor issues as I was conducting pilot B. As a result, I spent more than my original budget for the pilot studies. Therefore, I recruited as many participants as I could within my constrained budget, which resulted with a sample with 137 participants. (The original work had a sample of N=212.)
Demographics: The final sample consists of individuals who currently reside in the U.S. The original plan was to recruit internationally. However, Pilot B raised concerns. Job candidate’s qualification and evaluations depend on the nuiances in the labor market, which may be different across different regions. So I decided to limit the sample to residents of U.S. only, which follows the original work’s approach since the original sample had hiring managers who were U.S. based.
Data exclusions: Individuals who failed to check eligibility question were excluded from the final sample. As an eligibility check, the survey asked participants if they had any hiring experience in the beginning. Those who reported that they did not have any hiring experience were excluded from the study. Furthermore, individuals who entered the survey and dropped out voluntarily without completing the questionnaire were also excluded from the final sample.
Differences from Pre-Data Collection Methods Plan
None.
Results
Data Preparation
First, when the study is done, download data from Qualtrics using qualtRics package. (Other than qualtRics package, load other essential packages for data analysis and visualization, such as: tidyverse, knitr, ggplot2, psych, stats, and etc.)
Second, clean the data. Check for missing values and exclude observations with missing values. If needed, conduct data manipulation. Clean and manipulate data so that I have a clean tidy data.
Third, conduct descriptive analyses. Before running the main statistical models, run some descriptive analysis and show the descriptives as needed.
Fourth, conduct main statistical analysis. Present the findings in a table.
Fifth, do some data visualization so that the readers can view the results easily.
Lastly, if needed, run some additional analysis. (Check for homoscedasticity, conduct robustness checks, more explaratory analysis, etc.)
Load Relevant Libraries and Functions
library(qualtRics)
library(foreign)
library(ggplot2)
library(MASS)
library(Hmisc)## Loading required package: lattice## Loading required package: survival## Loading required package: Formula##
## Attaching package: 'Hmisc'## The following objects are masked from 'package:base':
##
## format.pval, unitslibrary(tidyverse)## ── Attaching packages ─────────────────────────────────────── tidyverse 1.3.1 ──## ✓ tibble 3.1.5 ✓ dplyr 1.0.7
## ✓ tidyr 1.1.4 ✓ stringr 1.4.0
## ✓ readr 2.0.2 ✓ forcats 0.5.1
## ✓ purrr 0.3.4## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## x dplyr::filter() masks stats::filter()
## x dplyr::lag() masks stats::lag()
## x dplyr::select() masks MASS::select()
## x dplyr::src() masks Hmisc::src()
## x dplyr::summarize() masks Hmisc::summarize()library(knitr)
library(gridExtra)##
## Attaching package: 'gridExtra'## The following object is masked from 'package:dplyr':
##
## combinelibrary(haven)
library(Rmisc)## Loading required package: plyr## ------------------------------------------------------------------------------## You have loaded plyr after dplyr - this is likely to cause problems.
## If you need functions from both plyr and dplyr, please load plyr first, then dplyr:
## library(plyr); library(dplyr)## ------------------------------------------------------------------------------##
## Attaching package: 'plyr'## The following objects are masked from 'package:dplyr':
##
## arrange, count, desc, failwith, id, mutate, rename, summarise,
## summarize## The following object is masked from 'package:purrr':
##
## compact## The following objects are masked from 'package:Hmisc':
##
## is.discrete, summarizelibrary(purrr)
library(broom)
library(rstatix)##
## Attaching package: 'rstatix'## The following objects are masked from 'package:plyr':
##
## desc, mutate## The following object is masked from 'package:MASS':
##
## select## The following object is masked from 'package:stats':
##
## filterlibrary(ggpubr)##
## Attaching package: 'ggpubr'## The following object is masked from 'package:plyr':
##
## mutatelibrary(reshape2)##
## Attaching package: 'reshape2'## The following object is masked from 'package:tidyr':
##
## smithslibrary(psych)##
## Attaching package: 'psych'## The following object is masked from 'package:Hmisc':
##
## describe## The following objects are masked from 'package:ggplot2':
##
## %+%, alpha
Import Data
# load survey
#data_fin_raw <- fetch_survey(surveyID = "SV_82M7ojjd1rLcGPA", force_request = T)# save data
#write_csv(data_fin_raw, "/Users/jbyun/Dropbox/2021 Fall/PSYCH 251/galperin2020/data/data_fin_raw.csv")# load data from csv
data_fin_raw <- read_csv("/Users/jbyun/Dropbox/2021 Fall/PSYCH 251/galperin2020/data/data_fin_raw.csv")## Warning: One or more parsing issues, see `problems()` for details## Rows: 143 Columns: 78## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## chr (41): Status, IPAddress, ResponseId, DistributionChannel, UserLanguage,...
## dbl (22): Progress, Duration (in seconds), LocationLatitude, LocationLongit...
## lgl (12): Finished, RecipientLastName, RecipientFirstName, RecipientEmail, ...
## dttm (3): StartDate, EndDate, RecordedDate##
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
Data Exclusion / Filtering
data_raw <- data_fin_raw# exclude participants who did not meet the eligibility requirement
# or those who might have been included by accident (i.e. myself) who did not go through Prolific
# 6 obs dropped
data <- data_raw %>%
filter(hiringexperience == "Yes") %>%
filter(!is.na(prolific_id))
describe(data)## Warning in FUN(newX[, i], ...): no non-missing arguments to min; returning Inf
## Warning in FUN(newX[, i], ...): no non-missing arguments to min; returning Inf
## Warning in FUN(newX[, i], ...): no non-missing arguments to min; returning Inf
## Warning in FUN(newX[, i], ...): no non-missing arguments to min; returning Inf
## Warning in FUN(newX[, i], ...): no non-missing arguments to min; returning Inf
## Warning in FUN(newX[, i], ...): no non-missing arguments to min; returning Inf
## Warning in FUN(newX[, i], ...): no non-missing arguments to min; returning Inf
## Warning in FUN(newX[, i], ...): no non-missing arguments to min; returning Inf
## Warning in FUN(newX[, i], ...): no non-missing arguments to min; returning Inf
## Warning in FUN(newX[, i], ...): no non-missing arguments to min; returning Inf
## Warning in FUN(newX[, i], ...): no non-missing arguments to min; returning Inf
## Warning in FUN(newX[, i], ...): no non-missing arguments to min; returning Inf
## Warning in FUN(newX[, i], ...): no non-missing arguments to min; returning Inf
## Warning in FUN(newX[, i], ...): no non-missing arguments to min; returning Inf
## Warning in FUN(newX[, i], ...): no non-missing arguments to min; returning Inf## Warning in FUN(newX[, i], ...): no non-missing arguments to max; returning -Inf
## Warning in FUN(newX[, i], ...): no non-missing arguments to max; returning -Inf
## Warning in FUN(newX[, i], ...): no non-missing arguments to max; returning -Inf
## Warning in FUN(newX[, i], ...): no non-missing arguments to max; returning -Inf
## Warning in FUN(newX[, i], ...): no non-missing arguments to max; returning -Inf
## Warning in FUN(newX[, i], ...): no non-missing arguments to max; returning -Inf
## Warning in FUN(newX[, i], ...): no non-missing arguments to max; returning -Inf
## Warning in FUN(newX[, i], ...): no non-missing arguments to max; returning -Inf
## Warning in FUN(newX[, i], ...): no non-missing arguments to max; returning -Inf
## Warning in FUN(newX[, i], ...): no non-missing arguments to max; returning -Inf
## Warning in FUN(newX[, i], ...): no non-missing arguments to max; returning -Inf
## Warning in FUN(newX[, i], ...): no non-missing arguments to max; returning -Inf
## Warning in FUN(newX[, i], ...): no non-missing arguments to max; returning -Inf
## Warning in FUN(newX[, i], ...): no non-missing arguments to max; returning -Inf
## Warning in FUN(newX[, i], ...): no non-missing arguments to max; returning -Inf## vars n mean sd median
## StartDate 1 137 NaN NA NA
## EndDate 2 137 NaN NA NA
## Status* 3 137 1.00 0.00 1.00
## IPAddress* 4 137 69.00 39.69 69.00
## Progress 5 137 100.00 0.00 100.00
## Duration (in seconds) 6 137 783.45 511.62 642.00
## Finished 7 137 NaN NA NA
## RecordedDate 8 137 NaN NA NA
## ResponseId* 9 137 69.00 39.69 69.00
## RecipientLastName 10 0 NaN NA NA
## RecipientFirstName 11 0 NaN NA NA
## RecipientEmail 12 0 NaN NA NA
## ExternalReference 13 0 NaN NA NA
## LocationLatitude 14 137 37.97 5.58 39.17
## LocationLongitude 15 137 -95.83 17.15 -90.85
## DistributionChannel* 16 137 1.00 0.00 1.00
## UserLanguage* 17 137 1.00 0.00 1.00
## consent* 18 137 1.00 0.00 1.00
## prolific_id* 19 137 69.00 39.69 69.00
## hiringexperience* 20 137 1.00 0.00 1.00
## companychk* 21 137 2.00 0.21 2.00
## candatchk* 22 137 2.16 0.39 2.00
## Location* 23 137 2.00 0.12 2.00
## interview* 24 137 3.12 1.92 4.00
## offer* 25 137 2.54 1.87 2.00
## opencncrn* 26 136 68.50 39.40 68.50
## openresolv* 27 134 66.75 38.50 66.50
## referral* 28 137 1.72 0.45 2.00
## competent* 29 137 2.17 1.37 1.00
## productive* 30 137 2.17 1.29 1.00
## skilled* 31 137 2.42 1.67 1.00
## finexp* 32 137 3.19 2.08 4.00
## committed* 33 137 2.39 1.40 2.00
## commit_ind* 34 137 3.78 1.86 4.00
## extrhrs* 35 137 2.47 1.37 3.00
## teamwork* 36 137 3.66 1.62 4.00
## stay* 37 137 3.15 1.51 3.00
## flightrsk* 38 137 3.09 1.90 3.00
## oe_qual* 39 137 69.00 39.69 69.00
## oe_comm* 40 137 69.00 39.69 69.00
## oe_interview* 41 136 68.50 39.40 68.50
## oe_offer* 42 136 68.50 39.40 68.50
## oe_referral* 43 135 68.00 39.12 68.00
## ethnicity* 44 136 2.88 0.37 3.00
## race_1* 45 126 1.00 0.00 1.00
## race_2* 46 5 1.00 0.00 1.00
## race_3* 47 8 1.00 0.00 1.00
## race_4* 48 5 1.00 0.00 1.00
## race_5* 49 3 1.00 0.00 1.00
## race_6 50 0 NaN NA NA
## race_7* 51 1 1.00 NA 1.00
## race_8 52 0 NaN NA NA
## race_8_TEXT 53 0 NaN NA NA
## gender* 54 137 1.25 0.45 1.00
## gender_5_TEXT 55 0 NaN NA NA
## Age 56 137 39.18 13.09 37.00
## educ* 57 135 3.53 2.02 2.00
## workexperience 58 137 18.71 11.35 17.00
## country* 59 137 10.06 3.43 12.00
## PROLIFIC_ID 60 0 NaN NA NA
## gc 61 0 NaN NA NA
## term 62 0 NaN NA NA
## Profile_DO_high_overqual_profil 63 68 1.00 0.00 1.00
## Profile_DO_low_overqual_profile 64 69 1.00 0.00 1.00
## CommitmentManipulation_DO_highcommitment_HR 65 73 1.00 0.00 1.00
## CommitmentManipulation_DO_neutralcommitment_HR 66 64 1.00 0.00 1.00
## Competence_DO_skilled 67 137 3.47 1.13 3.00
## Competence_DO_competent 68 137 3.45 1.10 3.00
## Competence_DO_introcomp 69 137 1.00 0.00 1.00
## Competence_DO_finexp 70 137 3.52 1.09 4.00
## Competence_DO_productive 71 137 3.56 1.16 4.00
## commitment_DO_teamwork 72 137 4.53 1.70 4.00
## commitment_DO_committed 73 137 4.66 1.66 5.00
## commitment_DO_intrcommt 74 137 1.00 0.00 1.00
## commitment_DO_extrhrs 75 137 4.30 1.65 4.00
## commitment_DO_stay 76 137 4.50 1.80 5.00
## commitment_DO_flightrsk 77 137 4.49 1.72 5.00
## commitment_DO_commit_ind 78 137 4.53 1.72 5.00
## trimmed mad min max
## StartDate NaN NA Inf -Inf
## EndDate NaN NA Inf -Inf
## Status* 1.00 0.00 1.00 1.00
## IPAddress* 69.00 50.41 1.00 137.00
## Progress 100.00 0.00 100.00 100.00
## Duration (in seconds) 685.05 309.86 202.00 2869.00
## Finished NaN NA Inf -Inf
## RecordedDate NaN NA Inf -Inf
## ResponseId* 69.00 50.41 1.00 137.00
## RecipientLastName NaN NA Inf -Inf
## RecipientFirstName NaN NA Inf -Inf
## RecipientEmail NaN NA Inf -Inf
## ExternalReference NaN NA Inf -Inf
## LocationLatitude 37.98 5.23 25.79 61.18
## LocationLongitude -95.02 19.55 -149.95 -71.12
## DistributionChannel* 1.00 0.00 1.00 1.00
## UserLanguage* 1.00 0.00 1.00 1.00
## consent* 1.00 0.00 1.00 1.00
## prolific_id* 69.00 50.41 1.00 137.00
## hiringexperience* 1.00 0.00 1.00 1.00
## companychk* 2.00 0.00 1.00 3.00
## candatchk* 2.09 0.00 1.00 3.00
## Location* 2.00 0.00 1.00 3.00
## interview* 3.14 1.48 1.00 6.00
## offer* 2.31 1.48 1.00 7.00
## opencncrn* 68.50 50.41 1.00 136.00
## openresolv* 66.69 49.67 1.00 133.00
## referral* 1.77 0.00 1.00 2.00
## competent* 2.09 0.00 1.00 4.00
## productive* 2.09 0.00 1.00 4.00
## skilled* 2.29 0.00 1.00 5.00
## finexp* 3.12 2.97 1.00 6.00
## committed* 2.30 1.48 1.00 6.00
## commit_ind* 3.80 1.48 1.00 7.00
## extrhrs* 2.26 1.48 1.00 6.00
## teamwork* 3.56 1.48 1.00 7.00
## stay* 3.03 1.48 1.00 7.00
## flightrsk* 3.00 2.97 1.00 6.00
## oe_qual* 69.00 50.41 1.00 137.00
## oe_comm* 69.00 50.41 1.00 137.00
## oe_interview* 68.50 50.41 1.00 136.00
## oe_offer* 68.50 50.41 1.00 136.00
## oe_referral* 68.00 50.41 1.00 135.00
## ethnicity* 2.99 0.00 1.00 3.00
## race_1* 1.00 0.00 1.00 1.00
## race_2* 1.00 0.00 1.00 1.00
## race_3* 1.00 0.00 1.00 1.00
## race_4* 1.00 0.00 1.00 1.00
## race_5* 1.00 0.00 1.00 1.00
## race_6 NaN NA Inf -Inf
## race_7* 1.00 0.00 1.00 1.00
## race_8 NaN NA Inf -Inf
## race_8_TEXT NaN NA Inf -Inf
## gender* 1.18 0.00 1.00 3.00
## gender_5_TEXT NaN NA Inf -Inf
## Age 37.98 13.34 19.00 79.00
## educ* 3.41 1.48 1.00 7.00
## workexperience 17.86 11.86 2.00 56.00
## country* 10.34 1.48 1.00 15.00
## PROLIFIC_ID NaN NA Inf -Inf
## gc NaN NA Inf -Inf
## term NaN NA Inf -Inf
## Profile_DO_high_overqual_profil 1.00 0.00 1.00 1.00
## Profile_DO_low_overqual_profile 1.00 0.00 1.00 1.00
## CommitmentManipulation_DO_highcommitment_HR 1.00 0.00 1.00 1.00
## CommitmentManipulation_DO_neutralcommitment_HR 1.00 0.00 1.00 1.00
## Competence_DO_skilled 3.46 1.48 2.00 5.00
## Competence_DO_competent 3.44 1.48 2.00 5.00
## Competence_DO_introcomp 1.00 0.00 1.00 1.00
## Competence_DO_finexp 3.52 1.48 2.00 5.00
## Competence_DO_productive 3.58 1.48 2.00 5.00
## commitment_DO_teamwork 4.53 1.48 2.00 7.00
## commitment_DO_committed 4.70 1.48 2.00 7.00
## commitment_DO_intrcommt 1.00 0.00 1.00 1.00
## commitment_DO_extrhrs 4.25 1.48 2.00 7.00
## commitment_DO_stay 4.50 2.97 2.00 7.00
## commitment_DO_flightrsk 4.49 2.97 2.00 7.00
## commitment_DO_commit_ind 4.53 1.48 2.00 7.00
## range skew kurtosis se
## StartDate -Inf NA NA NA
## EndDate -Inf NA NA NA
## Status* 0.00 NaN NaN 0.00
## IPAddress* 136.00 0.00 -1.23 3.39
## Progress 0.00 NaN NaN 0.00
## Duration (in seconds) 2667.00 1.98 4.05 43.71
## Finished -Inf NA NA NA
## RecordedDate -Inf NA NA NA
## ResponseId* 136.00 0.00 -1.23 3.39
## RecipientLastName -Inf NA NA NA
## RecipientFirstName -Inf NA NA NA
## RecipientEmail -Inf NA NA NA
## ExternalReference -Inf NA NA NA
## LocationLatitude 35.39 0.36 1.47 0.48
## LocationLongitude 78.83 -0.51 -0.74 1.47
## DistributionChannel* 0.00 NaN NaN 0.00
## UserLanguage* 0.00 NaN NaN 0.00
## consent* 0.00 NaN NaN 0.00
## prolific_id* 136.00 0.00 -1.23 3.39
## hiringexperience* 0.00 NaN NaN 0.00
## companychk* 2.00 0.00 19.50 0.02
## candatchk* 2.00 1.45 1.29 0.03
## Location* 2.00 0.00 64.50 0.01
## interview* 5.00 -0.09 -1.92 0.16
## offer* 6.00 0.93 -0.64 0.16
## opencncrn* 135.00 0.00 -1.23 3.38
## openresolv* 132.00 0.01 -1.23 3.33
## referral* 1.00 -0.98 -1.04 0.04
## competent* 3.00 0.43 -1.69 0.12
## productive* 3.00 0.39 -1.61 0.11
## skilled* 4.00 0.60 -1.33 0.14
## finexp* 5.00 0.15 -1.62 0.18
## committed* 5.00 0.37 -1.34 0.12
## commit_ind* 6.00 -0.38 -1.19 0.16
## extrhrs* 5.00 0.99 0.87 0.12
## teamwork* 6.00 0.26 -0.91 0.14
## stay* 6.00 0.62 0.02 0.13
## flightrsk* 5.00 0.49 -1.19 0.16
## oe_qual* 136.00 0.00 -1.23 3.39
## oe_comm* 136.00 0.00 -1.23 3.39
## oe_interview* 135.00 0.00 -1.23 3.38
## oe_offer* 135.00 0.00 -1.23 3.38
## oe_referral* 134.00 0.00 -1.23 3.37
## ethnicity* 2.00 -3.20 10.23 0.03
## race_1* 0.00 NaN NaN 0.00
## race_2* 0.00 NaN NaN 0.00
## race_3* 0.00 NaN NaN 0.00
## race_4* 0.00 NaN NaN 0.00
## race_5* 0.00 NaN NaN 0.00
## race_6 -Inf NA NA NA
## race_7* 0.00 NA NA NA
## race_8 -Inf NA NA NA
## race_8_TEXT -Inf NA NA NA
## gender* 2.00 1.39 0.56 0.04
## gender_5_TEXT -Inf NA NA NA
## Age 60.00 0.74 -0.14 1.12
## educ* 6.00 0.54 -1.15 0.17
## workexperience 54.00 0.65 -0.13 0.97
## country* 14.00 -0.62 -0.99 0.29
## PROLIFIC_ID -Inf NA NA NA
## gc -Inf NA NA NA
## term -Inf NA NA NA
## Profile_DO_high_overqual_profil 0.00 NaN NaN 0.00
## Profile_DO_low_overqual_profile 0.00 NaN NaN 0.00
## CommitmentManipulation_DO_highcommitment_HR 0.00 NaN NaN 0.00
## CommitmentManipulation_DO_neutralcommitment_HR 0.00 NaN NaN 0.00
## Competence_DO_skilled 3.00 0.02 -1.41 0.10
## Competence_DO_competent 3.00 0.09 -1.34 0.09
## Competence_DO_introcomp 0.00 NaN NaN 0.00
## Competence_DO_finexp 3.00 -0.06 -1.30 0.09
## Competence_DO_productive 3.00 -0.05 -1.47 0.10
## commitment_DO_teamwork 5.00 0.01 -1.26 0.15
## commitment_DO_committed 5.00 -0.08 -1.26 0.14
## commitment_DO_intrcommt 0.00 NaN NaN 0.00
## commitment_DO_extrhrs 5.00 0.12 -1.19 0.14
## commitment_DO_stay 5.00 -0.03 -1.45 0.15
## commitment_DO_flightrsk 5.00 0.03 -1.26 0.15
## commitment_DO_commit_ind 5.00 -0.05 -1.29 0.15# add id column
data <- tibble::rowid_to_column(data, "id")Prepare Data for Analysis - Create Columns, etc.
### Clean data (first step)
# select needed columns
data <- data %>%
dplyr::select(c('ResponseId', 'id', 'companychk', 'candatchk', 'Location', 'interview', 'offer', 'opencncrn', 'openresolv', 'referral', 'competent', 'productive', 'skilled', 'finexp', 'committed', 'commit_ind', 'extrhrs', 'teamwork', 'stay', 'flightrsk', 'oe_qual', 'oe_comm', 'oe_interview', 'oe_offer', 'oe_referral', 'ethnicity', 'race_1', 'race_2', 'race_3', 'race_4', 'race_5', 'race_6', 'race_7', 'race_8', 'race_8_TEXT', 'gender', 'Age', 'educ', 'workexperience', 'Profile_DO_high_overqual_profil', 'Profile_DO_low_overqual_profile', 'CommitmentManipulation_DO_highcommitment_HR', 'CommitmentManipulation_DO_neutralcommitment_HR'))
# data manipulation - scores
# likelihood of getting an interview (score)
data <- data %>%
mutate(interview_score = ifelse(interview == "Very Unlikely", 1,
ifelse(interview == "Unlikely", 2,
ifelse(interview == "Somewhat Unlikely", 3,
ifelse(interview == "Undecided", 4,
ifelse(interview == "Somewhat Likely", 5,
ifelse(interview == "Likely", 6,
ifelse(interview == "Very Likely", 7, NA))))))))
# likelihood of getting an offer (score)
data <- data %>%
mutate(offer_score = ifelse(offer == "Very Unlikely", 1,
ifelse(offer == "Unlikely", 2,
ifelse(offer == "Somewhat Unlikely", 3,
ifelse(offer == "Undecided", 4,
ifelse(offer == "Somewhat Likely", 5,
ifelse(offer == "Likely", 6,
ifelse(offer == "Very Likely", 7, NA))))))))
# referral or not
data <- data %>%
mutate(refer = ifelse(referral == "Yes", 1, 0)) %>%
mutate(refer = as.factor(refer))
### Manipulation checks
# competent
data <- data %>%
mutate(competent_score = ifelse(competent == "Strongly disagree", 1,
ifelse(competent == "Disagree", 2,
ifelse(competent == "Somewhat disagree", 3,
ifelse(competent == "Neither agree nor disagree", 4,
ifelse(competent == "Somewhat agree", 5,
ifelse(competent == "Agree", 6,
ifelse(competent == "Strongly agree", 7, NA))))))))
# productive
data <- data %>%
mutate(productive_score = ifelse(productive == "Strongly disagree", 1,
ifelse(productive == "Disagree", 2,
ifelse(productive == "Somewhat disagree", 3,
ifelse(productive == "Neither agree nor disagree", 4,
ifelse(productive == "Somewhat agree", 5,
ifelse(productive == "Agree", 6,
ifelse(productive == "Strongly agree", 7, NA))))))))
# skilled
data <- data %>%
mutate(skilled_score = ifelse(skilled == "Strongly disagree", 1,
ifelse(skilled == "Disagree", 2,
ifelse(skilled == "Somewhat disagree", 3,
ifelse(skilled == "Neither agree nor disagree", 4,
ifelse(skilled == "Somewhat agree", 5,
ifelse(skilled == "Agree", 6,
ifelse(skilled == "Strongly agree", 7, NA))))))))
# finexp
data <- data %>%
mutate(finexp_score = ifelse(finexp == "Strongly disagree", 1,
ifelse(finexp == "Disagree", 2,
ifelse(finexp == "Somewhat disagree", 3,
ifelse(finexp == "Neither agree nor disagree", 4,
ifelse(finexp == "Somewhat agree", 5,
ifelse(finexp == "Agree", 6,
ifelse(finexp == "Strongly agree", 7, NA))))))))
# committed
data <- data %>%
mutate(commit_score = ifelse(committed == "Highly uncommitted", 1,
ifelse(committed == "Uncommitted", 2,
ifelse(committed == "Somewhat uncommitted", 3,
ifelse(committed == "Neither committed nor uncommitted", 4,
ifelse(committed == "Somewhat committed", 5,
ifelse(committed == "Committed", 6,
ifelse(committed == "Highly committed", 7, NA))))))))
# commit_ind
data <- data %>%
mutate(commit_ind_score = ifelse(commit_ind == "Highly uncommitted", 1,
ifelse(commit_ind == "Uncommitted", 2,
ifelse(commit_ind == "Somewhat uncommitted", 3,
ifelse(commit_ind == "Neither committed nor uncommitted", 4,
ifelse(commit_ind == "Somewhat committed", 5,
ifelse(commit_ind == "Committed", 6,
ifelse(commit_ind == "Highly committed", 7, NA))))))))
# extra hours score
data <- data %>%
mutate(extrhrs_score = ifelse(extrhrs == "Very unlikely", 1,
ifelse(extrhrs == "Unlikely", 2,
ifelse(extrhrs == "Somewhat unlikely", 3,
ifelse(extrhrs == "Neither likely nor unlikely", 4,
ifelse(extrhrs == "Somewhat likely", 5,
ifelse(extrhrs == "Likely", 6,
ifelse(extrhrs == "Very likely", 7, NA))))))))
# teamwork score
data <- data %>%
mutate(teamwork_score = ifelse(teamwork == "Very Unlikely", 1,
ifelse(teamwork == "Unlikely", 2,
ifelse(teamwork == "Somewhat unlikely", 3,
ifelse(teamwork == "Neither likely nor unlikely", 4,
ifelse(teamwork == "Somewhat likely", 5,
ifelse(teamwork == "Likely", 6,
ifelse(teamwork == "Very likely", 7, NA))))))))
# likelihood of staying score
data <- data %>%
mutate(stay_score = ifelse(stay == "Less than 1 year", 1,
ifelse(stay == "1-2 years", 2,
ifelse(stay == "2-3 years", 3,
ifelse(stay == "3-4 years", 4,
ifelse(stay == "4-5 years", 5,
ifelse(stay == "5-6 years", 6,
ifelse(stay == "Greater than 6 years", 7, NA))))))))
# flight risk score (flightrsk)
data <- data %>%
mutate(flight_score = ifelse(flightrsk == "Very unlikely", 1,
ifelse(flightrsk == "Unlikely", 2,
ifelse(flightrsk == "Somewhat unlikely", 3,
ifelse(flightrsk == "Neither likely nor unlikely", 4,
ifelse(flightrsk == "Somewhat likely", 5,
ifelse(flightrsk == "Likely", 6,
ifelse(flightrsk == "Very likely", 7, NA))))))))### Data cleaning continuted - demographic information
#summary(as.factor(data$race_1))
#summary(as.factor(data$race_2))
#summary(as.factor(data$race_3))
#summary(as.factor(data$race_4))
#summary(as.factor(data$race_5))
##summary(as.factor(data$race_6))
#summary(as.factor(data$race_7))
##summary(as.factor(data$race_8))
##summary(as.factor(data$race_8_TEXT))
# ethnicity
data <- data %>%
mutate(ethnic_hispanic = ifelse(is.na(ethnicity), 0, ifelse(ethnicity == "Hispanic", 1, 0)))
# race
data <- data %>%
mutate(White = ifelse(!is.na(race_1), 1, 0)) %>%
mutate(Black = ifelse(!is.na(race_2), 1, 0)) %>%
mutate(Hispanic = ifelse(!is.na(race_3), 1, 0)) %>%
mutate(Asian = ifelse(!is.na(race_4), 1, 0)) %>%
mutate(Native = ifelse(!is.na(race_5), 1, 0)) %>%
mutate(Decline = ifelse(!is.na(race_7), 1, 0))
# variables with no obs dropped (race_6, race_8, race_8_TEXT)
# create multiple race variable
data <- data %>%
mutate(Race_sum = White + Black + Hispanic + Asian + Native + Decline)
data <- data %>%
mutate(Race_multi = ifelse(Race_sum > 1, 1, 0))
# create one race variable (factor)
data <- data %>%
mutate(race = ifelse(Race_multi == 1, "Multirace",
ifelse(White == 1, "White",
ifelse(Black == 1, "Black",
ifelse(Hispanic == 1, "Hispanic",
ifelse(Asian == 1, "Asian",
ifelse(Native == 1, "Native",
ifelse(Decline == 1, "Decline", NA)))))))) %>%
mutate(race = as.factor(race))
# gender
data <- data %>%
mutate(gender = as.factor(gender))
# education
data <- data %>%
mutate(educ = as.factor(educ))
#summary(data$educ)
# add education level (ordinal variable)
# educ_ordinal
data <- data %>%
mutate(educ_ordinal = ifelse(educ == "Less than High School", 1,
ifelse(educ == "High School / GED", 2,
ifelse(educ == "Some College", 3,
ifelse(educ == "2-year College Degree", 4,
ifelse(educ == "4-year College Degree", 5,
ifelse(educ == "Masters Degree", 6,
ifelse(educ == "Doctoral Degree", 7,
ifelse(educ == "Professional Degree (JD, MD)", 8, NA)))))))))# Treatment conditions
# Overqual condition (Capability)
data <- data %>%
mutate(Overqual = ifelse(is.na(Profile_DO_high_overqual_profil), "Low", "High")) %>%
mutate(Overqual = as.factor(Overqual))
data <- data %>%
mutate(Commitment = ifelse(is.na(CommitmentManipulation_DO_highcommitment_HR), "Neutral", "High")) %>%
mutate(Commitment = as.factor(Commitment))# Before selecting out variables, let's save this version as a csv file
#write.csv(data, "/Users/jbyun/Dropbox/2021 Fall/PSYCH 251/galperin2020/data/data_fin_wrangled.csv", row.names = F)# clean data (select needed columns only - exclude open-ended questions as well)
data_clean <- data %>%
dplyr::select(c('id', 'Commitment', 'Overqual', 'offer_score', 'interview_score', 'refer', 'competent_score', 'productive_score', 'skilled_score', 'finexp_score', 'commit_score', 'commit_ind_score', 'extrhrs_score', 'teamwork_score', 'stay_score', 'flight_score', 'gender', 'Age', 'ethnic_hispanic', 'race', 'educ_ordinal', 'workexperience', 'White', 'Black', 'Hispanic', 'Asian', 'Native', 'Decline', 'educ'))
data_clean <- data_clean %>%
mutate(Commitment = factor(Commitment, levels = c("Neutral", "High"), labels = c("Neutral", "High"))) %>%
mutate(Overqual = factor(Overqual, levels = c("Low", "High"), labels = c("Low", "High")))
# save the clean data
#write.csv(data_clean, "/Users/jbyun/Dropbox/2021 Fall/PSYCH 251/galperin2020/data/data_fin_clean.csv", row.names = F)# Data with neutral commitment condition (for t-test)
data_neutral_commit <- data_clean %>%
filter(Commitment == "Neutral")
# Data with high commitment condition (fot t-test)
data_high_commit <- data_clean %>%
filter(Commitment == "High")Descriptive Statistics
# Check number of participants in each condition
kable(table(data$Overqual, data$Commitment), caption = "Number of participants in each condition")| High | Neutral | |
|---|---|---|
| High | 37 | 31 |
| Low | 36 | 33 |
kable(describe(data_clean), caption = "Overall Descriptive Statistics")| vars | n | mean | sd | median | trimmed | mad | min | max | range | skew | kurtosis | se | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | 1 | 137 | 69.0000000 | 39.6925686 | 69 | 69.000000 | 50.4084 | 1 | 137 | 136 | 0.0000000 | -1.2263075 | 3.3911650 |
| Commitment* | 2 | 137 | 1.5328467 | 0.5007508 | 2 | 1.540541 | 0.0000 | 1 | 2 | 1 | -0.1302323 | -1.9974601 | 0.0427820 |
| Overqual* | 3 | 137 | 1.4963504 | 0.5018215 | 1 | 1.495496 | 0.0000 | 1 | 2 | 1 | 0.0144394 | -2.0143352 | 0.0428735 |
| offer_score | 4 | 137 | 5.5255474 | 1.1119689 | 6 | 5.612613 | 1.4826 | 1 | 7 | 6 | -1.0034573 | 1.4793318 | 0.0950019 |
| interview_score | 5 | 137 | 6.2554745 | 0.9854023 | 6 | 6.441441 | 1.4826 | 1 | 7 | 6 | -2.1702834 | 6.6260062 | 0.0841886 |
| refer* | 6 | 137 | 1.7226277 | 0.4493447 | 2 | 1.774775 | 0.0000 | 1 | 2 | 1 | -0.9836667 | -1.0398305 | 0.0383901 |
| competent_score | 7 | 137 | 6.1240876 | 0.7712219 | 6 | 6.216216 | 0.0000 | 4 | 7 | 3 | -0.8811495 | 0.8579431 | 0.0658899 |
| productive_score | 8 | 137 | 5.9270073 | 0.8457075 | 6 | 6.000000 | 0.0000 | 4 | 7 | 3 | -0.5870463 | -0.1364169 | 0.0722537 |
| skilled_score | 9 | 137 | 5.9562044 | 0.8213029 | 6 | 6.018018 | 0.0000 | 3 | 7 | 4 | -0.7106299 | 0.6174569 | 0.0701686 |
| finexp_score | 10 | 137 | 5.8613139 | 0.9329151 | 6 | 5.927928 | 1.4826 | 2 | 7 | 5 | -0.9114626 | 1.6547759 | 0.0797043 |
| commit_score | 11 | 137 | 5.4306569 | 1.0346612 | 6 | 5.477477 | 1.4826 | 2 | 7 | 5 | -0.6460966 | 0.1665554 | 0.0883971 |
| commit_ind_score | 12 | 137 | 4.8321168 | 1.3148996 | 5 | 4.882883 | 1.4826 | 1 | 7 | 6 | -0.5006282 | 0.0434987 | 0.1123395 |
| extrhrs_score | 13 | 137 | 5.2408759 | 0.9356151 | 5 | 5.252252 | 1.4826 | 2 | 7 | 5 | -0.4362044 | 0.8888275 | 0.0799350 |
| teamwork_score | 14 | 137 | 3.3357664 | 1.3894422 | 3 | 3.252252 | 1.4826 | 1 | 7 | 6 | 0.4689766 | -0.0638776 | 0.1187081 |
| stay_score | 15 | 137 | 3.8978102 | 1.4311897 | 4 | 3.846847 | 1.4826 | 1 | 7 | 6 | 0.1781473 | -0.3211508 | 0.1222748 |
| flight_score | 16 | 137 | 5.5328467 | 1.1760546 | 6 | 5.612613 | 1.4826 | 2 | 7 | 5 | -0.4685564 | -0.4432663 | 0.1004771 |
| gender* | 17 | 137 | 1.2481752 | 0.4501801 | 1 | 1.180180 | 0.0000 | 1 | 3 | 2 | 1.3909193 | 0.5608722 | 0.0384615 |
| Age | 18 | 137 | 39.1824818 | 13.0863754 | 37 | 37.981982 | 13.3434 | 19 | 79 | 60 | 0.7396705 | -0.1447146 | 1.1180445 |
| ethnic_hispanic | 19 | 137 | 0.0875912 | 0.2837369 | 0 | 0.000000 | 0.0000 | 0 | 1 | 1 | 2.8857608 | 6.3743024 | 0.0242413 |
| race* | 20 | 137 | 5.6788321 | 0.9388221 | 6 | 5.927928 | 0.0000 | 1 | 6 | 5 | -3.5634947 | 13.1931028 | 0.0802090 |
| educ_ordinal | 21 | 135 | 4.5259259 | 1.2976996 | 5 | 4.623853 | 1.4826 | 2 | 8 | 6 | -0.4836182 | -0.4968797 | 0.1116882 |
| workexperience | 22 | 137 | 18.7080292 | 11.3507972 | 17 | 17.855856 | 11.8608 | 2 | 56 | 54 | 0.6482872 | -0.1341063 | 0.9697640 |
| White | 23 | 137 | 0.9197080 | 0.2727419 | 1 | 1.000000 | 0.0000 | 0 | 1 | 1 | -3.0552288 | 7.3885131 | 0.0233019 |
| Black | 24 | 137 | 0.0364964 | 0.1882098 | 0 | 0.000000 | 0.0000 | 0 | 1 | 1 | 4.8894416 | 22.0678782 | 0.0160798 |
| Hispanic | 25 | 137 | 0.0583942 | 0.2353478 | 0 | 0.000000 | 0.0000 | 0 | 1 | 1 | 3.7254011 | 11.9661164 | 0.0201071 |
| Asian | 26 | 137 | 0.0364964 | 0.1882098 | 0 | 0.000000 | 0.0000 | 0 | 1 | 1 | 4.8894416 | 22.0678782 | 0.0160798 |
| Native | 27 | 137 | 0.0218978 | 0.1468870 | 0 | 0.000000 | 0.0000 | 0 | 1 | 1 | 6.4622801 | 40.0535860 | 0.0125494 |
| Decline | 28 | 137 | 0.0072993 | 0.0854358 | 0 | 0.000000 | 0.0000 | 0 | 1 | 1 | 11.4496399 | 130.0436358 | 0.0072993 |
| educ* | 29 | 135 | 3.5333333 | 2.0215259 | 2 | 3.412844 | 1.4826 | 1 | 7 | 6 | 0.5410094 | -1.1542112 | 0.1739853 |
kable(table(data_clean$gender, data_clean$race), caption = "Participant Race and Gender")| Asian | Black | Decline | Hispanic | Multirace | White | |
|---|---|---|---|---|---|---|
| Female | 3 | 1 | 0 | 5 | 7 | 88 |
| Male | 0 | 0 | 1 | 1 | 3 | 27 |
| Non-binary / Third Gender | 0 | 0 | 0 | 0 | 0 | 1 |
kable(table(data_clean$educ), caption = "Participant Education Level")| Var1 | Freq |
|---|---|
| 2-year College Degree | 12 |
| 4-year College Degree | 60 |
| Doctoral Degree | 1 |
| High School / GED | 13 |
| Masters Degree | 26 |
| Professional Degree (JD, MD) | 1 |
| Some College | 22 |
desc_tab <- data_clean %>%
dplyr::group_by(Commitment, Overqual) %>%
dplyr::summarise(Age = mean(Age, na.rm = T),
WorkExperience = mean(workexperience, na.rm = T),
Offer = mean(offer_score),
Interview = mean(interview_score),
Educ_level = mean(educ_ordinal, na.rm = T))## `summarise()` has grouped output by 'Commitment'. You can override using the `.groups` argument.kable(desc_tab, caption = "Key Descriptive Statistics for Each Condition")| Commitment | Overqual | Age | WorkExperience | Offer | Interview | Educ_level |
|---|---|---|---|---|---|---|
| Neutral | Low | 41.09091 | 20.36364 | 5.242424 | 6.030303 | 4.545454 |
| Neutral | High | 36.77419 | 17.06452 | 5.645161 | 6.548387 | 4.387097 |
| High | Low | 39.83333 | 19.58333 | 5.555556 | 6.138889 | 4.485714 |
| High | High | 38.86486 | 17.75676 | 5.648649 | 6.324324 | 4.666667 |
summ_mean_sd <- data_clean %>%
group_by(Commitment, Overqual) %>%
get_summary_stats(offer_score, type = "mean_sd")
kable(summ_mean_sd, caption = "Mean and s.d. of Likelihood of Offer across All Conditions")| Commitment | Overqual | variable | n | mean | sd |
|---|---|---|---|---|---|
| Neutral | Low | offer_score | 33 | 5.242 | 1.173 |
| Neutral | High | offer_score | 31 | 5.645 | 0.985 |
| High | Low | offer_score | 36 | 5.556 | 1.340 |
| High | High | offer_score | 37 | 5.649 | 0.889 |
ggplot(data = data_clean, aes(x = Age)) +
geom_histogram(aes(y = ..density..)) +
geom_density(col = "blue") +
facet_grid(Overqual ~ Commitment) +
theme_classic() +
ylab("Density") +
ggtitle("Histogram of participant age in each condition")## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.
ggplot(data = data_clean, aes(x = workexperience)) +
geom_histogram(aes(y = ..density..)) +
geom_density(col = "blue") +
facet_grid(Overqual ~ Commitment) +
theme_classic() +
ylab("Density") +
xlab("Work Experience (in years)") +
ggtitle("Histogram of participant work experience in each condition")## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.
ggplot(data = data_clean, aes(x = offer_score)) +
geom_histogram(aes(y = ..density..)) +
geom_density(col = "blue") +
theme_classic() +
facet_wrap(~ Commitment) +
ylab("Density") +
xlab("Likelihood of Offer") +
ggtitle("Likelihood of getting an offer in each commitment condition")## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.
plot_box <- ggboxplot(
data = data_clean, x = "Overqual", y = "offer_score",
color = "Commitment", palette = "jco"
)
plot_box
commit_labels <- c("Neutral Commitment", "High Commitment")
names(commit_labels) <- c("Neutral", "High")
ggplot(data = data_clean, aes(x = Overqual, y = offer_score, group = Overqual, fill = Overqual)) +
geom_boxplot(alpha = 0.6) +
geom_jitter(width = 0.1, alpha = 0.6) +
guides(fill = "none") +
scale_y_continuous(breaks = 1:7, limits = c(1, 7)) +
theme(legend.position = c(0.9, 0.15),
legend.direction = "vertical",
legend.background = element_rect(fill = "transparent"),
legend.title = element_blank(),
axis.line = element_line(),
panel.grid = element_blank(),
panel.background = element_blank(),
plot.title = element_text(hjust = 0.5)) +
facet_wrap(~Commitment, labeller = labeller(Commitment = commit_labels)) +
ggtitle("Final Results (N = 137)") +
labs(x = "Capability", y = "Likelihood of an Offer")## Warning: Removed 11 rows containing missing values (geom_point).
ggsave("/Users/jbyun/Dropbox/2021 Fall/PSYCH 251/galperin2020/data/final_data_boxplot.png")## Saving 7 x 5 in image## Warning: Removed 13 rows containing missing values (geom_point).Manipulation Checks
# Capability manipulation check (Overqual)
require(dplyr)
require(psych)
capa_check <- data_clean %>%
dplyr::select(c('competent_score', 'productive_score', 'skilled_score', 'finexp_score'))
alpha(capa_check)##
## Reliability analysis
## Call: alpha(x = capa_check)
##
## raw_alpha std.alpha G6(smc) average_r S/N ase mean sd median_r
## 0.82 0.83 0.79 0.54 4.7 0.026 6 0.68 0.55
##
## lower alpha upper 95% confidence boundaries
## 0.77 0.82 0.87
##
## Reliability if an item is dropped:
## raw_alpha std.alpha G6(smc) average_r S/N alpha se var.r
## competent_score 0.75 0.75 0.68 0.50 3.0 0.038 0.01021
## productive_score 0.77 0.78 0.71 0.54 3.5 0.034 0.00910
## skilled_score 0.74 0.75 0.68 0.50 3.0 0.039 0.01493
## finexp_score 0.83 0.83 0.77 0.63 5.0 0.025 0.00063
## med.r
## competent_score 0.51
## productive_score 0.51
## skilled_score 0.46
## finexp_score 0.63
##
## Item statistics
## n raw.r std.r r.cor r.drop mean sd
## competent_score 137 0.83 0.85 0.79 0.71 6.1 0.77
## productive_score 137 0.81 0.81 0.73 0.64 5.9 0.85
## skilled_score 137 0.85 0.85 0.79 0.71 6.0 0.82
## finexp_score 137 0.75 0.73 0.58 0.53 5.9 0.93
##
## Non missing response frequency for each item
## 2 3 4 5 6 7 miss
## competent_score 0.00 0.00 0.05 0.09 0.55 0.31 0
## productive_score 0.00 0.00 0.07 0.18 0.50 0.25 0
## skilled_score 0.00 0.01 0.04 0.18 0.52 0.25 0
## finexp_score 0.01 0.01 0.03 0.26 0.43 0.26 0# run t-test to examine if perceived capabilities are different across conditions
capa_check_t <- data_clean %>%
dplyr::select(c('Overqual', 'competent_score', 'productive_score', 'skilled_score', 'finexp_score'))
capa_check_t$mean_capa <- rowMeans(capa_check_t[2:5], na.rm = T)
capa_test <- t.test(mean_capa ~ Overqual, data = capa_check_t, paired = F, var.equal = T)
capa_test##
## Two Sample t-test
##
## data: mean_capa by Overqual
## t = -1.9622, df = 135, p-value = 0.0518
## alternative hypothesis: true difference in means between group Low and group High is not equal to 0
## 95 percent confidence interval:
## -0.453402594 0.001782815
## sample estimates:
## mean in group Low mean in group High
## 5.855072 6.080882require(broom)
tidy(t.test(data = capa_check_t, mean_capa ~ Overqual, paired = F, var.equal = T))## # A tibble: 1 × 10
## estimate estimate1 estimate2 statistic p.value parameter conf.low conf.high
## <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 -0.226 5.86 6.08 -1.96 0.0518 135 -0.453 0.00178
## # … with 2 more variables: method <chr>, alternative <chr># Commitment manipulation check (Commitment)
commit_check <- data_clean %>%
dplyr::select(c('commit_score', 'commit_ind_score', 'stay_score', 'extrhrs_score'))
alpha(commit_check)##
## Reliability analysis
## Call: alpha(x = commit_check)
##
## raw_alpha std.alpha G6(smc) average_r S/N ase mean sd median_r
## 0.74 0.74 0.71 0.42 2.9 0.034 4.9 0.9 0.42
##
## lower alpha upper 95% confidence boundaries
## 0.68 0.74 0.81
##
## Reliability if an item is dropped:
## raw_alpha std.alpha G6(smc) average_r S/N alpha se var.r
## commit_score 0.63 0.63 0.55 0.36 1.7 0.051 0.0177
## commit_ind_score 0.60 0.61 0.54 0.34 1.6 0.055 0.0204
## stay_score 0.70 0.69 0.64 0.43 2.2 0.042 0.0389
## extrhrs_score 0.77 0.79 0.72 0.55 3.7 0.033 0.0074
## med.r
## commit_score 0.34
## commit_ind_score 0.29
## stay_score 0.34
## extrhrs_score 0.51
##
## Item statistics
## n raw.r std.r r.cor r.drop mean sd
## commit_score 137 0.80 0.81 0.75 0.65 5.4 1.03
## commit_ind_score 137 0.84 0.83 0.78 0.66 4.8 1.31
## stay_score 137 0.79 0.75 0.61 0.54 3.9 1.43
## extrhrs_score 137 0.56 0.62 0.39 0.34 5.2 0.94
##
## Non missing response frequency for each item
## 1 2 3 4 5 6 7 miss
## commit_score 0.00 0.01 0.04 0.14 0.27 0.43 0.12 0
## commit_ind_score 0.01 0.04 0.10 0.20 0.34 0.23 0.09 0
## stay_score 0.04 0.14 0.21 0.28 0.22 0.05 0.06 0
## extrhrs_score 0.00 0.01 0.01 0.16 0.43 0.31 0.07 0# run t-test to examine if perceived commitments are different across conditions
commit_check_t <- data_clean %>%
dplyr::select(c('Commitment', 'commit_score', 'commit_ind_score', 'stay_score', 'extrhrs_score'))
commit_check_t$mean_commit <- rowMeans(capa_check_t[2:5], na.rm = T)
commit_test <- t.test(mean_commit ~ Commitment, data = commit_check_t, paired = F, var.equal = T)
commit_test##
## Two Sample t-test
##
## data: mean_commit by Commitment
## t = -1.2349, df = 135, p-value = 0.219
## alternative hypothesis: true difference in means between group Neutral and group High is not equal to 0
## 95 percent confidence interval:
## -0.37363503 0.08639188
## sample estimates:
## mean in group Neutral mean in group High
## 5.890625 6.034247require(broom)
tidy(t.test(mean_commit ~ Commitment, data = commit_check_t, paired = F, var.equal = T))## # A tibble: 1 × 10
## estimate estimate1 estimate2 statistic p.value parameter conf.low conf.high
## <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 -0.144 5.89 6.03 -1.23 0.219 135 -0.374 0.0864
## # … with 2 more variables: method <chr>, alternative <chr>I precisely followed the original work’s manipulation check procedure. I combined the four measures used in the original work to create a measure for candidate’s perceived capability. (Cronbach’s alpha = 0.81) While the overall Cronbach’s alpha value showed the consistency among the four measures was good, the participants randomly assigned to the extremely high-capability conditions did not rate the candidate higher in perceived capability than those assigned to the moderately high-capability condition. The t-test results were significant only at alpha = 0.1 level. (t = -1.96, d.f. = 135, p-value = 0.0518) I combined four commitment measures used in the original work to create a perceived organizational commitment score. (Cronbach’s alpha = 0.74) The overall Cronbach’s alpha value fell under acceptable range. However, the commitment manipulation did not seem to be effective for the participants in this study. Participants randomly assigned to the high-commitment conditions did not rate the candidate higher in commitment than those assinged to the neutral-commitment conditions. (t = -1.23, d.f. = 135, p-value = 0.219) The fact that manipulations were not effective (especially commitment conditions) to the participants in this replication study may suggest that the sample used for this study was substantially different than the original study’s sample. The original work’s manipulations were very effective. (All manipulation checks were significant with p < 0.001).
Confirmatory Analysis
I ran two t-tests: one for neutral commitment condition, the other for high commitment condition. The choice of key statistical model is determined by the original work, as the original paper ran two t-tests.
T-test for Neutral-Commitment Conditions
t.test(offer_score ~ Overqual, data = data_neutral_commit, paired = F, var.equal = T)##
## Two Sample t-test
##
## data: offer_score by Overqual
## t = -1.4823, df = 62, p-value = 0.1433
## alternative hypothesis: true difference in means between group Low and group High is not equal to 0
## 95 percent confidence interval:
## -0.9458523 0.1403782
## sample estimates:
## mean in group Low mean in group High
## 5.242424 5.645161require(broom)
tidy(t.test(data = data_neutral_commit, offer_score ~ Overqual, paired = F, var.equal = T))## # A tibble: 1 × 10
## estimate estimate1 estimate2 statistic p.value parameter conf.low conf.high
## <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 -0.403 5.24 5.65 -1.48 0.143 62 -0.946 0.140
## # … with 2 more variables: method <chr>, alternative <chr>I found no statistically significant results from the first t-test (neutral-commitment condition). While the mean value of likelihood of getting an offer for the Extremely-high capability condition (mean = 5.65) was greater than that for the Moderately-high contition (mean = 5.24), the t-test results were not significant. (t = -1.48, df = 62, p-value = 0.14)
T-test for High-Commitment Conditions
t.test(offer_score ~ Overqual, data = data_high_commit, paired = F, var.equal = T)##
## Two Sample t-test
##
## data: offer_score by Overqual
## t = -0.35063, df = 71, p-value = 0.7269
## alternative hypothesis: true difference in means between group Low and group High is not equal to 0
## 95 percent confidence interval:
## -0.6224871 0.4363010
## sample estimates:
## mean in group Low mean in group High
## 5.555556 5.648649require(broom)
tidy(t.test(data = data_high_commit, offer_score ~ Overqual, paired = F, var.equal = T))## # A tibble: 1 × 10
## estimate estimate1 estimate2 statistic p.value parameter conf.low conf.high
## <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 -0.0931 5.56 5.65 -0.351 0.727 71 -0.622 0.436
## # … with 2 more variables: method <chr>, alternative <chr>I found no statistically significant results from the second t-test (high-commitment condition). While the mean value of likelihood of getting an offer for the Extremely-high capability condition (mean = 5.65) was greater than that for the Moderately-high contition (mean = 5.56), the t-test results were not significant. (t = -0.35, df = 71, p-value = 0.73)
Summary: Capability x Commitment and Likelihood of an Offer across All 4 Conditions
summ_mean_sd## # A tibble: 4 × 6
## Commitment Overqual variable n mean sd
## <fct> <fct> <chr> <dbl> <dbl> <dbl>
## 1 Neutral Low offer_score 33 5.24 1.17
## 2 Neutral High offer_score 31 5.64 0.985
## 3 High Low offer_score 36 5.56 1.34
## 4 High High offer_score 37 5.65 0.889png("test.png")
p <- tableGrob(summ_mean_sd)
grid.arrange(p)
dev.off## function (which = dev.cur())
## {
## if (which == 1)
## stop("cannot shut down device 1 (the null device)")
## .External(C_devoff, as.integer(which))
## dev.cur()
## }
## <bytecode: 0x7fe417f7b9b8>
## <environment: namespace:grDevices>ggplot(data = summ_mean_sd, aes(x = Commitment, y = mean,
ymin = mean - sqrt(sd), ymax = mean + sqrt(sd), fill = Overqual)) +
geom_bar(stat = "identity", position = position_dodge()) +
geom_errorbar(position=position_dodge(0.9), width = .2) +
scale_y_continuous(breaks = 1:7) +
theme(legend.direction = "vertical",
legend.position = "right",
axis.line = element_line(),
panel.grid.major.x = element_blank(),
panel.grid.major.y = element_line(color = "grey", size = 0.15),
panel.background = element_blank(),
plot.title = element_text(hjust = 0.5)) +
labs(x = "Commitment", y = "Likelihood of Offer", fill = "Capability") +
ggtitle("Capability x Commitment: Likelihood of Offer across all Four Conditions")
ggsave("/Users/jbyun/Dropbox/2021 Fall/PSYCH 251/galperin2020/data/final_bargraph.png")## Saving 7 x 5 in imageFor reference, the original work’s corresponding figure:
Exploratory Analyses
Two-way ANOVA
# Check normality assumption (QQ plot)
#ggqqplot(data_clean, "offer_score", ggtheme = theme_bw()) +
# facet_grid(Overqual ~ Commitment)result_aov <- data_clean %>% anova_test(offer_score ~ Overqual*Commitment)## Coefficient covariances computed by hccm()result_aov## ANOVA Table (type II tests)
##
## Effect DFn DFd F p p<.05 ges
## 1 Overqual 1 133 1.564 0.213 0.012
## 2 Commitment 1 133 0.705 0.403 0.005
## 3 Overqual:Commitment 1 133 0.661 0.418 0.005The two-way ANOVA results show that there was no statitically significant interaction between Capability and Commitment condition for the likelihood of getting an offer.
Two-way ANCOVA
Since the results of t-tests and two-way ANOVA were not statistically significant, I examined the possibility of other factors, such as work experience and age, interfering with the results.
Check Linearity Assumptions
ggscatter(
data = data_clean, x = "workexperience", y = "offer_score",
facet.by = c("Commitment", "Overqual"),
short.panel.labs = F,
xlab = c("Work Experience (in years)"),
ylab = c("Likelihood of Offer"),
) +
stat_smooth(method = "loess", span = 0.9)## `geom_smooth()` using formula 'y ~ x'
Homogeneity of Regression Slopes
This assumption checks that there is not significant interaction betwen the covariate and the grouping variables.
data_clean %>%
anova_test(
offer_score ~ workexperience + Overqual + Commitment + Overqual*Commitment + workexperience*Overqual + workexperience*Commitment + workexperience*Commitment*Overqual
)## Coefficient covariances computed by hccm()## ANOVA Table (type II tests)
##
## Effect DFn DFd F p p<.05 ges
## 1 workexperience 1 129 5.647 0.019 * 0.042
## 2 Overqual 1 129 1.022 0.314 0.008
## 3 Commitment 1 129 0.679 0.412 0.005
## 4 Overqual:Commitment 1 129 0.720 0.398 0.006
## 5 workexperience:Overqual 1 129 0.814 0.369 0.006
## 6 workexperience:Commitment 1 129 0.806 0.371 0.006
## 7 workexperience:Overqual:Commitment 1 129 2.002 0.160 0.015I already know that there is no statistically significant interaction effect between Capability (Overqual) and Commitment. However, just as a sanity check, I ran another ANOVA to show that there is no significant interaction between Capability (Overqual) and Commitment here.
Interaction Effect after Controlling for Participant’s Work Experience (ANCOVA)
work_aov <- data_clean %>%
anova_test(offer_score ~ workexperience + Overqual*Commitment)## Coefficient covariances computed by hccm()get_anova_table(work_aov)## ANOVA Table (type II tests)
##
## Effect DFn DFd F p p<.05 ges
## 1 workexperience 1 132 5.629 0.019 * 0.041
## 2 Overqual 1 132 0.995 0.320 0.007
## 3 Commitment 1 132 0.720 0.398 0.005
## 4 Overqual:Commitment 1 132 0.561 0.455 0.004There was no statistically significant interaction between Capability and Commitment on the likelihood of getting an offer after controlling for participant’s work experience (in years).
Interaction Effect after Controlling for Participant’s Age (ANCOVA)
age_aov <- data_clean %>%
anova_test(offer_score ~ Age + Overqual*Commitment)## Coefficient covariances computed by hccm()get_anova_table(age_aov)## ANOVA Table (type II tests)
##
## Effect DFn DFd F p p<.05 ges
## 1 Age 1 132 4.492 0.036 * 0.033
## 2 Overqual 1 132 1.103 0.295 0.008
## 3 Commitment 1 132 0.781 0.379 0.006
## 4 Overqual:Commitment 1 132 0.470 0.494 0.004After adjusting for participant’s age, I found no statistically significant interaction between Capability and Commitment on the likelihood of getting an offer.
Ordinal Logistic Regression
data_clean <- data_clean %>%
mutate(offer_score = as.factor(offer_score))
model <- polr(offer_score ~ Commitment + Overqual + Age + workexperience + gender, data = data_clean, Hess = T)
summary(model)## Call:
## polr(formula = offer_score ~ Commitment + Overqual + Age + workexperience +
## gender, data = data_clean, Hess = T)
##
## Coefficients:
## Value Std. Error t value
## CommitmentHigh 0.4273632 0.32440 1.31742
## OverqualHigh 0.0731818 0.32266 0.22681
## Age 0.0003693 0.03287 0.01124
## workexperience -0.0349398 0.03926 -0.88987
## genderMale 0.0855793 0.38905 0.21997
## genderNon-binary / Third Gender -1.6547902 1.54093 -1.07389
##
## Intercepts:
## Value Std. Error t value
## 1|2 -5.3672 1.2271 -4.3739
## 2|3 -4.6619 1.0032 -4.6472
## 3|4 -3.5316 0.8217 -4.2981
## 4|5 -2.0190 0.7407 -2.7259
## 5|6 -0.7734 0.7205 -1.0734
## 6|7 1.3483 0.7314 1.8433
##
## Residual Deviance: 384.0399
## AIC: 408.0399ctable <- coef(summary(model))p <- pnorm(abs(ctable[, "t value"]), lower.tail = F) * 2
# combined table
ctable <- cbind(ctable, "p value" = p)
ctable## Value Std. Error t value
## CommitmentHigh 0.4273631916 0.32439525 1.31741508
## OverqualHigh 0.0731817779 0.32265500 0.22681123
## Age 0.0003693194 0.03286835 0.01123632
## workexperience -0.0349398326 0.03926403 -0.88986878
## genderMale 0.0855792829 0.38905061 0.21996954
## genderNon-binary / Third Gender -1.6547901689 1.54093250 -1.07388881
## 1|2 -5.3672374464 1.22710542 -4.37390086
## 2|3 -4.6618626499 1.00315792 -4.64718720
## 3|4 -3.5316194534 0.82166521 -4.29812460
## 4|5 -2.0190232853 0.74069081 -2.72586516
## 5|6 -0.7733973811 0.72051194 -1.07339982
## 6|7 1.3482594203 0.73141872 1.84334826
## p value
## CommitmentHigh 1.876995e-01
## OverqualHigh 8.205705e-01
## Age 9.910349e-01
## workexperience 3.735363e-01
## genderMale 8.258949e-01
## genderNon-binary / Third Gender 2.828725e-01
## 1|2 1.220458e-05
## 2|3 3.364918e-06
## 3|4 1.722493e-05
## 4|5 6.413319e-03
## 5|6 2.830918e-01
## 6|7 6.527818e-02Results of ordinal logistic regression are also statistically not significant
Discussion
Summary of Replication Attempt
I ran two t-tests (one for high-commitment conditions, another for neutral-commitment conditions) as confirmatory analyses. Confirmatory analyses results are statistically insignificant at \(\alpha = 0.05\) level. To sum up, this study failed to replicate the original results.
Commentary
I conducted multiple exploratory analyses: 2-way ANOVA (Capability x Commitment), 2 ANCOVA’s (one adjusting for age, the other adjusting for work experience), and ordinal logistic regression. I found no statitically significant effect for the main effects (Capability x Commitment). However, participant’s age and work experience seemed to have statistically significant associations with the outcome variable. This indicates that participants’ work experience and age could be important when they are evaluating job candidates’ resumes in the hiring processes. This failure to replicate indicates importance of the sample and timing in hiring studies. I confirmed that the same stimuli failed to manipulate conditions by manipulation checks. This may be attributed to the different natures of the samples. It might require some experience in the part of hiring managers when it comes to evaluating nuances in resumes. While the original study relied on Qualtrics to verify the participants’ experience as hiring managers, I had to rely on self-reports on Prolific. Since different individuals may interpret the term “hiring managers” differently, I could have ended up with a very different sample compared to that of original study. Furthermore, the 5-year time difference should be accounted for. While the original experiment was conducted in December 2016, this study was conducted in November 2021. With hiring, timing is very important because there are a lot of time sentivities that could relate to economy, labor market conditions, etc. How people think about hiring could have changed substantially over the 5-year time period, especially with the COVID-19 pandemic. (It is widely known that the outbreak of COVID-19 pandemic in late 2019 has changed the labor market conditions, the meaning of work, etc.) The future research should account for the importance of the timing and the nature of sample for studies about hiring in the labor market.