Employers’ selection of job applicants is an important labor market proesses for both employers and individuals. For organizations, access to high-quality workers is a source of competitive advantage. For individuals, access to certain jobs at certain timing can have a large impact on one’s career trajectories. Job signaling is the most fundamental and important aspect of selection processes in labor market. Information asymmetries between employers and job applicants provide challenges for both ends.
Most studies have focused on job signaling and applicant capabilities when trying to understand selection processes in labor markets. On the other hand, increasing number of studies suggest that selection decisions are affected not only by perceptions of applicantss’ capability but also by perceptions of applicants’ commitment to the organization.
While previous literature investigated both the effect of perceptions of applicants’ capability and the effect of perceptions of applicants’ commitment on the likelihood of a candidate’s being selected for a job, the relationship between the two dimensions had been understudied. In an attempt to shed light on such a gap in the literature, Galperin, Hahl, Sterling, and Guo (Gelperin et al. henceforth) conducted a series of online field experiments and published an article in ASQ in the year of 2020. (For reference, see Links section in the below.)
Galperin et al. (2020) suggested that signals that influence perceptions of capability in labor markets may have discernable effects of the perceptions of a candidate’s commitment, which also affects the likelihood of the candidate’s being selected for the job. The paper included four studies, three of which were online field experiments. In this study, I plan to replicate Study 2.
In the original paper, the authors conducted a direct test of the effect of applicants’ commitment and high-capability rejection.
Study 2 was conducted to directly manipulated perceived commitment of an applicant to complement one of the other studies where they could not directly manipulate commitment. This is important because the main contribution of the study is suggested (and proved) mediation effect of commitment on likelihood of a job offer.
I anticipate two challenges while attempting to replicate the results.
First, I may run into sample issue. This study is about job signals and hiring managers’ decision making in making job offers. Hence, it is important that I only have hiring managers (defined by someone with experiences as a hiring manager) in the sample. Since I have to rely on Qualtrics to do the sampling, less appropriate sample might acquired at the end. Similar issue applies to random assignment of participants.
Second, survey design might be difficult. The main part of this replication study is manipulating commitment of a job applicant. Since commitment could mean different things both conceptually and practically, it is essential that I manipulate the main concepts elaborately. To overcome this challenge, I plan to reach out to original authors and try to acquire survey items that they used for the original study.
To replicate the results of Study 2, I plan to run the same online experiment on Prolific. Prior to running the main study, I plan to run two pilot studies. The first pilot study will involve non-naive participants, whereas the second pilot study will have naive participants on MTurk.
For the main study, participants will be recruited to fill out a “survey for hiring managers” through Prolific in October-November 2021. I intend to have a sample size of 200 in order to have a similar sample size with the original study, but the sample size may be changed depending on the results of the power analysis. (Power analysis is to be conducted.)
The study will have the same structure as that of the original study: 2 by 2 between-subjects design. Participants will be randomly assigned to one of the four conditions. (2 capability conditions X 2 commitment conditions) Finally given the same stimuli as the original study.
I plan to conduct a manipulation check for both candidate’s perceived capability and perceived commitment by combining the questionnaires included in the survey.
With the data acquired from the survey experiment, I plan to conduct two t-tests to replicate the key results of Study 2.
Explicitly describe known differences in the analysis pipeline between the original paper and yours (e.g., computing environment). The goal, of course, is to minimize those differences, but differences may occur. Also, note whether such differences are anticipated to influence your ability to reproduce the original results.
First of all, the most important difference would be that I will be using a different sample. While the original work recruited participants through Qualtrics panel, I will recruit participants through Prolific. While I do not foresee any systematic differences between two platforms, some substantial differences may exist when filtering for the target participants. This may cause differences between the sample of this study and that of original paper.
Second, I intend to conduct statistical analyses using programming language R, while the original work seems to have used Stata. R and Stata should not have systematic differences for t-tests or ANOVA, but the use of different statistical software may result in trivial divergences.
I acknowledge that any unforeseen issues are the sole responsiblitity of mine.
Please describe the outcome measure for the success or failure of your reproduction and how this outcome will be computed.
The objective of this replication project is to re-run the Study 2 of the original paper and compute the main effect. The main effect will be computed by running two separate t-tests. One for neutral-level commitment group, and the other for high-level commitment group.
Earlier in this report, you described the steps necessary to reproduce the key result(s) of this study. Please describe your progress on each of these steps (e.g., data preprocessing, model fitting, model evaluation).
(October 24, 2021) I have survey experiment ready in the Qualtrics form. I ran the first pilot (Pilot A) with 8 people and confirmed that the survey experiment runs and collects data. I identified some minor problems with Qualtrics survey, so I am trying to fix the issues.
The original survey experiment includes quite a few items and I have not been able to use all the variables. The original work included manipulation checks for each item (commitment and capability), and I plan to include them in the next draft.
I was able to run one of the main models (t-test), but I could not run the other model because I did not have all the conditions in the pilot A sample.
First, when the study is done, download data from Qualtrics using qualtRics package. (Other than qualtRics package, load other essential packages for data analysis and visualization, such as: tidyverse, knitr, ggplot2, stats, and etc.)
Second, clean the data. Check for missing values and exclude observations with missing values. If needed, conduct data manipulation. Clean and manipulate data so that I have a clean tidy data.
Third, conduct descriptive analyses. Before running the main statistical models, run some descriptive analysis and show the descriptives as needed.
Fourth, conduct main statistical analysis. Present the findings in a table.
Fifth, do some data visualization so that the readers can view the results easily.
Lastly, if needed, run some additional analysis. (Check for homoscedasticity, conduct robustness checks, more explaratory analysis, etc.)
library(qualtRics)
library(tidyverse)## ── Attaching packages ─────────────────────────────────────── tidyverse 1.3.1 ──## ✓ ggplot2 3.3.5 ✓ purrr 0.3.4
## ✓ tibble 3.1.5 ✓ dplyr 1.0.7
## ✓ tidyr 1.1.4 ✓ stringr 1.4.0
## ✓ readr 2.0.2 ✓ forcats 0.5.1## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## x dplyr::filter() masks stats::filter()
## x dplyr::lag() masks stats::lag()library(knitr)
library(psych)##
## Attaching package: 'psych'## The following objects are masked from 'package:ggplot2':
##
## %+%, alpha
# load survey
qualtricsData_pilotA <- fetch_survey(surveyID = "SV_3CP4lUkXb73NVSm", force_request = T)##
|
| | 0%
|
|======================================================================| 100%##
## ── Column specification ────────────────────────────────────────────────────────
## cols(
## .default = col_character(),
## StartDate = col_datetime(format = ""),
## EndDate = col_datetime(format = ""),
## Progress = col_double(),
## `Duration (in seconds)` = col_double(),
## Finished = col_logical(),
## RecordedDate = col_datetime(format = ""),
## RecipientLastName = col_logical(),
## RecipientFirstName = col_logical(),
## RecipientEmail = col_logical(),
## ExternalReference = col_logical(),
## LocationLatitude = col_double(),
## LocationLongitude = col_double(),
## race_2 = col_logical(),
## race_3 = col_logical(),
## race_5 = col_logical(),
## race_6 = col_logical(),
## Age = col_double(),
## id = col_logical(),
## gc = col_double(),
## term = col_logical()
## # ... with 16 more columns
## )
## ℹ Use `spec()` for the full column specifications.# save data
write_csv(qualtricsData_pilotA, "/Users/jbyun/Dropbox/2021 Fall/PSYCH 251/galperin2020/data/pilotA.csv")# load data from csv
pilotA_df <- read_csv("/Users/jbyun/Dropbox/2021 Fall/PSYCH 251/galperin2020/data/pilotA.csv")## Rows: 8 Columns: 70## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## chr (34): Status, IPAddress, ResponseId, DistributionChannel, UserLanguage,...
## dbl (22): Progress, Duration (in seconds), LocationLatitude, LocationLongit...
## lgl (11): Finished, RecipientLastName, RecipientFirstName, RecipientEmail, ...
## dttm (3): StartDate, EndDate, RecordedDate##
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
# clean data (first step)
pilotA_cleaning <- pilotA_df |>
select(c('Progress', 'Duration (in seconds)', 'Finished', 'companychk', 'candatchk', 'Location', 'interview', 'offer', 'referral', 'competent', 'productive', 'skilled', 'finexp', 'committed', 'commit_ind', 'extrhrs', 'teamwork', 'stay', 'flightrsk', 'gender', 'race_1', 'race_2', 'race_3', 'race_4', 'race_5', 'race_6', 'race_7', 'educ', 'Age', 'Profile_DO_high_overqual_profil', 'Profile_DO_low_overqual_profile', 'CommitmentManipulation_DO_highcommitment_HR', 'CommitmentManipulation_DO_neutralcommitment_HR', 'Competence_DO_skilled', 'Competence_DO_competent', 'Competence_DO_introcomp', 'Competence_DO_finexp', 'Competence_DO_productive', 'commitment_DO_teamwork', 'commitment_DO_committed', 'commitment_DO_intrcommt', 'commitment_DO_extrhrs', 'commitment_DO_stay', 'commitment_DO_flightrsk', 'commitment_DO_commit_ind'))
# add id column
pilotA_cleaning <- tibble::rowid_to_column(pilotA_cleaning, "id")
# likelyhood of getting an offer (score)
pilotA_cleaning <- pilotA_cleaning |>
mutate(offer_score = ifelse(offer == "Very Unlikely", 1,
ifelse(offer == "Unlikely", 2,
ifelse(offer == "Somewhat Unlikely", 3,
ifelse(offer == "Undecided", 4,
ifelse(offer == "Somewhat Likely", 5,
ifelse(offer == "Likely", 6,
ifelse(offer == "Very Likely", 7, NA))))))))
# overqual - high (condition1)
pilotA_cleaning <- pilotA_cleaning |>
mutate(cond1_high_overqual = ifelse(is.na(Profile_DO_high_overqual_profil), 0, 1))
# commit - high (condition2)
pilotA_cleaning <- pilotA_cleaning |>
mutate(cond2_high_commit = ifelse(is.na(CommitmentManipulation_DO_highcommitment_HR), 0, 1))# clean data and make a clean df
pilotA_offer_condition <- pilotA_cleaning |>
select(c('offer_score', 'cond1_high_overqual', 'cond2_high_commit', 'Profile_DO_high_overqual_profil', 'Profile_DO_low_overqual_profile', 'CommitmentManipulation_DO_highcommitment_HR', 'CommitmentManipulation_DO_neutralcommitment_HR'))
# checked the conditions. now drop original columns
pilotA_offer_condition <- pilotA_offer_condition |>
select(c('offer_score', 'cond1_high_overqual', 'cond2_high_commit'))# df only for neutral commitment condition
pilotA_neutral <- pilotA_offer_condition |>
filter(cond2_high_commit == 0)
# df only for high commitment condition
pilotA_high_commit <- pilotA_offer_condition |>
filter(cond2_high_commit == 1)
pilotA_summary <- pilotA_cleaning |>
select(gender, Age, cond1_high_overqual, cond2_high_commit)describe(pilotA_summary)## vars n mean sd median trimmed mad min max range skew
## gender* 1 8 1.12 0.35 1 1.12 0.00 1 2 1 1.86
## Age 2 7 30.43 2.94 29 30.43 2.97 27 35 8 0.32
## cond1_high_overqual 3 8 0.38 0.52 0 0.38 0.00 0 1 1 0.42
## cond2_high_commit 4 8 0.62 0.52 1 0.62 0.00 0 1 1 -0.42
## kurtosis se
## gender* 1.70 0.12
## Age -1.72 1.11
## cond1_high_overqual -2.03 0.18
## cond2_high_commit -2.03 0.18
The main statistical model is going to be two t-tests, the same as that from original study. One t-test is to be conducted for Capability and the other t-test is to be conducted for Commitment.
Below are the key analysis results from Pilot A.
#neutral_t <- t.test(offer_score ~ cond1_high_overqual, data = pilotA_neutral)
#neutral_t
# could not run the model because I do not have different conditions for neutral commitment
For the first t-test, I could not run the model because I did not have both condition groups (highly overqualified / not overqualified) for neutral commitment groups. This is due to small sample size and randomization. I expect this problem to be resolved itself when I get enough sample size for the study.
high_t <- t.test(offer_score ~ cond1_high_overqual, data = pilotA_high_commit)
high_t##
## Welch Two Sample t-test
##
## data: offer_score by cond1_high_overqual
## t = 0.86603, df = 1.6842, p-value = 0.4921
## alternative hypothesis: true difference in means between group 0 and group 1 is not equal to 0
## 95 percent confidence interval:
## -4.976997 6.976997
## sample estimates:
## mean in group 0 mean in group 1
## 6 5The above shows the results for the second t-test. (For high commitment group.)
(This part comes from a previous draft. To be deleted)
The main statistical model is going to be a two-way ANOVA analysis. Two-way ANOVA is the best statitical model since I am trying to examine the interation effect of candidate's capability and commitment (Capability X Commitment) on the likelihood of receiving an offer. By using the two-way factorial design, I can effectively examine the potential interaction effect between Capability and Commitment (given by the hypothesis).
Since the original paper ran two t-tests, I also plan to conduct the two t-tests as a supplement.###Exploratory analyses
Any follow-up analyses desired (not required).
## Discussion
### Summary of Reproduction Attempt
Open the discussion section with a paragraph summarizing the primary result from the key analysis and assess whether you successfully reproduced it, partially reproduced it, or failed to reproduce it.
### Commentary
Add open-ended commentary (if any) reflecting (a) insights from follow-up exploratory analysis of the dataset, (b) assessment of the meaning of the successful or unsuccessful reproducibility attempt - e.g., for a failure to reproduce the original findings, are the differences between original and present analyses ones that definitely, plausibly, or are unlikely to have been moderators of the result, and (c) discussion of any objections or challenges raised by the current and original authors about the reproducibility attempt (if you contacted them). None of these need to be long.