Sleep Deprivation and Advice Taking

Journal Article

The journal article that this report is based on is: Häusser, J. A., Leder, J., Ketturat, C., Dresler, M., & Faber, N. S. (2016). Sleep deprivation and advice taking. Scientific Reports, 6(1), 24386-24386. https://doi.org/10.1038/srep24386 Before using the journal article for this report, it was verified that the article was peer-reviewed. A copy of the journal article with comments regarding how simulations were undertaken in this report is attached to this submission.

Background

Research shows that total sleep deprivation for even a single night can harm a broad range of decisions and judgments (Harrison & Horne, 2000). This applies to tasks that are simple and highly monotonous as well as those that are highly complex (Baranski et al., 2007). The negative influence of sleep deprivation on judgment and decision-making is potentially associated with the negative effects of sleep deprivation on cognitive functions. Studies regarding the cognitive influence of sleep deprivation found that speed and accuracy in performing several cognitive functions are negatively impacted by sleep deprivation, with working memory and attention having the highest impacts (Lim & Dinges, 2010).

Despite the findings, it is also noticeable that several medical, economic, and political decisions are often made when the decision-makers are sleep-deprived. During these events, the decision-makers are expected to synthesize information given by advisors to come up with the most rational decisions. The overnight EU summits held in 2015 regarding the last-minute packages for the financial rescue of Greece indicated that several significant decisions are often made under severe sleep deprivation (Faber et al., 2017). Decisions such as those made during the EU summit are usually made after evaluations of a variety of information provided by advisors such as consultants, lobbyists, and experts. The accuracy of such key decisions made under sleep deprivation can be questioned as studies have shown that deprivation of sleep alters decision-making and judgment, and can even have a more severe influence on individual functioning.

The article used in this study sought to evaluate the social aspect of sleep deprivation, as several studies only concentrated on its impact on individual functioning. Specifically, the social aspect that is evaluated in the article is advice-taking, as the key objective of the article was to determine whether sleep deprivation has any influence on advice-taking. The study used a laboratory-based experimental Judge Advisor System (JAS) by manipulating the competency of advisers among the subjects. Sleep deprivation was manipulated among the subjects by dividing them into two groups: one group was deprived of sleep for 24 hours and the control group was given a normal night’s sleep. Moreover, it was hypothesised that sleep deprivation from highly competitive advisers was generally relied upon at greater extents than advice from those considered as medium competent advisors. This research shifts focus from individual impairments to the social dynamics of sleep deprivation, examining its interaction with advisor competency and its influence on judgment accuracy. # Research Questions and hypothesis

The study aimed to explore the way sleep deprivation influences social decision-making in terms of the utilisation of advice and the accuracy of judgment. The research questions for the study included:

  1. To what extent does sleep deprivation influence the way individuals take advice?
  2. In what ways do competency levels of advisors affect advise-taking?
  3. How does the interaction between sleep deprivation and advisor competency influence advice-and the accuracy of judgment?

From the research questions, the study sought to test two hypotheses. These included:

Hypothesis 1: Sleep-deprived individuals often show increased advice-taking behavior compared to individuals who are well-rested.

Hypothesis 2: Individuals usually rely more on the advice that is provided by highly competent advisors than those provided by moderately competent advisors.

Research Variables

The research had two independent and two dependent variables. The independent variables included sleep deprivation and advisory competency. Sleep deprivation was manipulated between subjects, with two levels that included sleep-deprived versus well-rested individuals. Advisor competency was equally manipulated in two levels among the subjects. This included medium competency versus high competency.

The dependent variables included advice-taking and judgement accuracy. Advise-taking was measured using a Weight of Advice (WOA) score that indicated the extent to which the advisors’ input influenced the judgments of the participants. The judgment accuracy, on the other hand, was measured using the mean absolute percentage error (MAPE) that represented the deviations of estimates from the correct values. Below is a summary of all the variables used in the study and the findings.

  1. Name: Condition (Sleep deprivation)

    Type: Binary

    Values it can take: 0 = Well-rested, 1 = Sleep-deprived

    Descriptive statistics: The study involved 96 participants. Among them, the condition of 50 participants were “well-rested’ and that of 46 participants was “sleep deprived”. Those within the sleep-deprived group reported significantly higher mental fatigue mean scores of 4.33 (SD = 0.17), and the participants within the well-rested group had an average fatigue score of 1.73 (SD=0.16). The findings showed that sleep deprivation led to a substantial rise in perceived mental fatigue.

  2. Name: Advisory Competency

    Type: Categorical

    Values it can take: 1= Medium Competency, 2= High Competency

    Descriptive Statistics: There were two types of advisory competency. Every participant was advised by two advisors. One was a medium-competency advisor (Rank 52) and the other was a high-competency advisor (Rank 7). The deviations of the medium-competency advisors were by about 55%, and the deviation of the estimates of high-competency advisors was by about 16%. The participants showed a higher reception of the advice provided by the high-competency advisors, indicating that the quality of advisors influenced the levels of advice taking.

  3. Name: Weight of Advice (WOA)

    Type: Continuous

    Values it can take: Range of 0 to 1, where 0= no reliance on advice, 1= complete reliance on advice.

    Descriptive statistics: Among the sleep-deprived group, the average WOA score was higher at 0.43, than the well-rested group which had a mean WOA score of 0.34. The findings indicated that there is a major reliance on advice among the participants who are sleep-deprived. WOA was 0.5 for higher competency advisors, while the medium competency advisers had a WOA of 0.28. This showed that both sleep deprivation and advisor competency levels had a significant influence on the reliance on advice by participants.

  4. Name: Judgement Accuracy (Mean Absolute Percentage Error-MAPE)

    Type: Continuous

    Values it can take: Percentage of accuracy ranges from 0% to any positive digit, in which higher values show less accuracy

    Descriptive Statistics: The initial MAPE scores were higher for participants who were sleep-deprived (M= 51.74, SD = 2.45). This was unlike the well-rested students whose MAPE scores had a mean of 45.51, and a standard deviation of 2.42. This shows that sleep deprivation results in a decline in the accuracy of judgment. The change in MAPE from the initial to final estimates was -23.43 for advisors with higher competency and -6.46 for advisors with medium competency. This reflects a rise in accuracy when advice is received from highly competent sources.

  5. Name: Mental Fatigue

    Type: Continuous

    Values it can take: 0 to 6 on a Likert scale with 0 being not fatigued at all and 6 being extremely fatigued.

    Descriptive statistics: There were higher levels of mental fatigue among the participants who were sleep-deprived with a mean of 4.33 (SD= 0.17). Those who were well-rested had a mean mental fatigue of 1.73 (SD = 0.16). The findings show that a significant rise in fatigue is caused by sleep deprivation.

  6. Name: Chronotype

    Type: Ordinal

    Values it may take: 0 to 6, in which 0 shows extreme morning type, and 6 indicates extreme evening type.

    Descriptive statistics: The participants showed variations in chronotypes, with an average score of 3.5 (mid-range). This suggested a balanced mix of mid-evening and mid-morning types. There were no significant correlations between chronotypes and WOA or Judgment accuracy. As such, its indication did not have a noticeable role in the influence that sleep deprivation has on advice-taking or decision accuracy.

Research Design

The study employed a 2 x 2 mixed factorial experiemntal research design, which involved between-subjects factor (sleep condition) and within subjects’ factor (advisor competency) of 96 participantse. Sleep conditions were of two levels, sleep deprived versus well-rested conditions, and the advisor competency was of two levels, medium competency versus high competency. In the study, every participant was randomly assigned to either a sleep-deprived or well-rested group to create the between-subjects factor. Each advisor also received advice from both high-competency and medium-competency advisors during the experiment to create the within-subjects factor.

Model formula: WOA ~ Condition * AdvisorCompetency + (1 | ParticipantID) + (1 | Trial)

Simulation code for dataset

The simulation involved approximation of parameters based on the findings from the article such as the means and standard deviations for Weight of Advice (WOA) and Judgement Accuracy (ΔMAPE) under different conditions. Data for 96 participants was simulated, with every participant undergoing trials with both a high-competency and medium-competency advisor.

# Load necessary library
library(dplyr)
## 
## Attaching package: 'dplyr'
## The following objects are masked from 'package:stats':
## 
##     filter, lag
## The following objects are masked from 'package:base':
## 
##     intersect, setdiff, setequal, union
# Set seed for reproducibility
set.seed(123)
# Define parameters based on findings
n_participants <- 96
n_trials <- 15  # Assume 15 trials per condition per participant
# Means for Weight of Advice (WOA) and ΔMAPE based on study findings
woa_means <- list(
  sleep_deprived_medium = 0.34,
  sleep_deprived_high = 0.53,
  well_rest_medium = 0.22,
  well_rest_high = 0.47
)
delta_mape_means <- list(
  sleep_deprived_medium = -4.67,
  sleep_deprived_high = -26.94,
  well_rest_medium = -8.25,
  well_rest_high = -19.92
)
# Standard deviations for WOA and ΔMAPE
woa_sd <- 0.1
delta_mape_sd <- 5
# Simulate data
data <- data.frame(
  ParticipantID = rep(1:n_participants, each = n_trials * 2),
  Condition = rep(c("Sleep_Deprived", "Well_Rested"), each = n_trials * n_participants),
  AdvisorCompetency = rep(c("Medium", "High"), times = n_trials * n_participants),
  Trial = rep(1:n_trials, times = 2 * n_participants)
)
# Assign mean values for each condition combination
data <- data %>%
  mutate(
    WOA_mean = case_when(
      Condition == "Sleep_Deprived" & AdvisorCompetency == "Medium" ~ woa_means$sleep_deprived_medium,
      Condition == "Sleep_Deprived" & AdvisorCompetency == "High" ~ woa_means$sleep_deprived_high,
      Condition == "Well_Rested" & AdvisorCompetency == "Medium" ~ woa_means$well_rest_medium,
      Condition == "Well_Rested" & AdvisorCompetency == "High" ~ woa_means$well_rest_high
    ),
    Delta_MAPE_mean = case_when(
      Condition == "Sleep_Deprived" & AdvisorCompetency == "Medium" ~ delta_mape_means$sleep_deprived_medium,
      Condition == "Sleep_Deprived" & AdvisorCompetency == "High" ~ delta_mape_means$sleep_deprived_high,
      Condition == "Well_Rested" & AdvisorCompetency == "Medium" ~ delta_mape_means$well_rest_medium,
      Condition == "Well_Rested" & AdvisorCompetency == "High" ~ delta_mape_means$well_rest_high
    ),
    # Generate random values based on means and standard deviations
    WOA = rnorm(n = nrow(data), mean = WOA_mean, sd = woa_sd),
    Delta_MAPE = rnorm(n = nrow(data), mean = Delta_MAPE_mean, sd = delta_mape_sd)
  )
Sample of Simulated Data
ParticipantID Condition AdvisorCompetency Trial WOA_mean Delta_MAPE_mean WOA Delta_MAPE
1 Sleep_Deprived Medium 1 0.34 -4.67 0.2839524 -10.034123
1 Sleep_Deprived High 2 0.53 -26.94 0.5069823 -28.089144
1 Sleep_Deprived Medium 3 0.34 -4.67 0.4958708 -1.561814
1 Sleep_Deprived High 4 0.53 -26.94 0.5370508 -22.752773
1 Sleep_Deprived Medium 5 0.34 -4.67 0.3529288 -6.184440
1 Sleep_Deprived High 6 0.53 -26.94 0.7015065 -27.697763
Summary Table of Simulated WOA and Delta_MAPE by Condition and Advisor Competency
Condition AdvisorCompetency Mean_WOA SD_WOA Mean_Delta_MAPE SD_Delta_MAPE
Sleep_Deprived High 0.5278759 0.1017543 -26.937842 5.041654
Sleep_Deprived Medium 0.3450618 0.0962820 -4.767597 4.903897
Well_Rested High 0.4718001 0.0985134 -19.999789 4.904848
Well_Rested Medium 0.2192905 0.0996459 -8.152742 5.032684

Code to check simulation resembles original study data

To verify if the data that was simulated is a representation of the original data used in the study, a comparison of the key summary statistics such as means and standard deviations is undertaken alongside distributions of the original and simulate data. This is first undertaken through the calculation of descriptive statistics. The means and standard deviations of WOA and ΔMAPE for every condition in the data that is simulated is compared to the original values in the study article. The second step is the visual comparison, which involves the generation of boxplots and density plots from the simulated data and comparing the patterns to the results reported in the study article.

Descriptive Statistics

# Load necessary libraries for table display
library(knitr)
library(dplyr)

# Simulated data's descriptive statistics for each group
summary_stats <- data %>%
  group_by(Condition, AdvisorCompetency) %>%
  summarize(
    Mean_WOA = mean(WOA),
    SD_WOA = sd(WOA),
    Mean_Delta_MAPE = mean(Delta_MAPE),
    SD_Delta_MAPE = sd(Delta_MAPE),
    .groups = 'drop'
  )

# Compare summary stats to original means from the article
original_means <- list(
  WOA_sleep_deprived_medium = 0.34,
  WOA_sleep_deprived_high = 0.53,
  WOA_well_rest_medium = 0.22,
  WOA_well_rest_high = 0.47,
  Delta_MAPE_sleep_deprived_medium = -4.67,
  Delta_MAPE_sleep_deprived_high = -26.94,
  Delta_MAPE_well_rest_medium = -8.25,
  Delta_MAPE_well_rest_high = -19.92
)

comparison <- summary_stats %>%
  mutate(
    Original_Mean_WOA = case_when(
      Condition == "Sleep_Deprived" & AdvisorCompetency == "Medium" ~ original_means$WOA_sleep_deprived_medium,
      Condition == "Sleep_Deprived" & AdvisorCompetency == "High" ~ original_means$WOA_sleep_deprived_high,
      Condition == "Well_Rested" & AdvisorCompetency == "Medium" ~ original_means$WOA_well_rest_medium,
      Condition == "Well_Rested" & AdvisorCompetency == "High" ~ original_means$WOA_well_rest_high
    ),
    Original_Mean_Delta_MAPE = case_when(
      Condition == "Sleep_Deprived" & AdvisorCompetency == "Medium" ~ original_means$Delta_MAPE_sleep_deprived_medium,
      Condition == "Sleep_Deprived" & AdvisorCompetency == "High" ~ original_means$Delta_MAPE_sleep_deprived_high,
      Condition == "Well_Rested" & AdvisorCompetency == "Medium" ~ original_means$Delta_MAPE_well_rest_medium,
      Condition == "Well_Rested" & AdvisorCompetency == "High" ~ original_means$Delta_MAPE_well_rest_high
    )
  )

# Display the comparison table
kable(comparison, caption = "Comparison of Simulated and Original Means for WOA and Delta_MAPE")
Comparison of Simulated and Original Means for WOA and Delta_MAPE
Condition AdvisorCompetency Mean_WOA SD_WOA Mean_Delta_MAPE SD_Delta_MAPE Original_Mean_WOA Original_Mean_Delta_MAPE
Sleep_Deprived High 0.5278759 0.1017543 -26.937842 5.041654 0.53 -26.94
Sleep_Deprived Medium 0.3450618 0.0962820 -4.767597 4.903897 0.34 -4.67
Well_Rested High 0.4718001 0.0985134 -19.999789 4.904848 0.47 -19.92
Well_Rested Medium 0.2192905 0.0996459 -8.152742 5.032684 0.22 -8.25

From the comparison between the means and standard deviations of the original data and the simulated data, the descriptive statistics show that there is a close resemblance of the data. Among the well-rested participants, the mean of WOA from those who received advice from highly competent advisors was 0.47 in the original data and 0.4718001 in the simulated data, while those who received advice from medium competency advisors had a WOA mean of 0.22 in the original data and 0.2192905 in the simulated data. Among the participants who were sleep deprived, the WOA mean for those who received advice from highly competent advisors was 0.53 in the original data and 0.527879 in the simulated data, while for those who received advice from medium competency advisors was 0.34 in the original data and 0.3450618 in the simulated data. The close resemblance is also seen in means of change in MAPE. For instance, -19.92 is close to -19.9997, -8.25 is comparable to -8.152742, and -26.94 from the original data is close to -26.937842 from the simulated data.

Visual Comparison

# Load necessary libraries
library(dplyr)
library(ggplot2)

# Assuming 'data' is the simulated dataset
# Calculate Group Means and Standard Errors (SEM) for WOA
summary_data_woa <- data %>%
  group_by(Condition, AdvisorCompetency) %>%
  summarize(
    Mean_WOA = mean(WOA),
    SEM_WOA = sd(WOA) / sqrt(n()),
    .groups = 'drop'
  )

# Display summary data
print(summary_data_woa)
## # A tibble: 4 × 4
##   Condition      AdvisorCompetency Mean_WOA SEM_WOA
##   <chr>          <chr>                <dbl>   <dbl>
## 1 Sleep_Deprived High                 0.528 0.00379
## 2 Sleep_Deprived Medium               0.345 0.00359
## 3 Well_Rested    High                 0.472 0.00367
## 4 Well_Rested    Medium               0.219 0.00371
# Generate the figure with error bars and custom x-axis labels
ggplot(summary_data_woa, aes(x = AdvisorCompetency, y = Mean_WOA, fill = Condition)) +
  geom_bar(stat = "identity", position = position_dodge(width = 0.9), width = 0.7) +
  geom_errorbar(aes(ymin = Mean_WOA - SEM_WOA, ymax = Mean_WOA + SEM_WOA),
                position = position_dodge(width = 0.9), width = 0.25) +
  labs(
    title = "WOA by Advisor Competency and Experimental Condition",
    x = "",  # Leave x-axis label blank for custom labeling
    y = "Weight of Advice (WOA)"
  ) +
  scale_x_discrete(labels = c("Medium" = "Medium", "High" = "High")) +
  scale_fill_manual(values = c("Sleep_Deprived" = "lightblue", "Well_Rested" = "lightgreen"),
                    name = "Condition") +
  theme_minimal() +
  theme(
    plot.title = element_text(hjust = 0.5),
    axis.title.y = element_text(vjust = 1.5),
    axis.title.x = element_text(vjust = -0.5)
  ) +
  # Optional: Add significance markers
  geom_text(aes(label = ifelse(abs(Mean_WOA) < 0.05, "*", "")),
            vjust = -1, color = "red", position = position_dodge(0.9))

The first chart in the original article displayed the influence of sleep deprivation on the accuracy of estimates. This was done through an examination of the influence of sleep deprivation on MAPE scores of initial estimates. The findings from the simulated data closely resemble the original findings displayed in the chart. For those who received advice from highly competent advisors, the means of WOA for sleep-deprived participants was slightly above 50, and for those who were well-rested was about 45 for both the original and simulated data. The resemblance is similar to those who received advice from medium competency advisors.

# Load necessary libraries
library(dplyr)
library(ggplot2)

# simulated data is already available in `data`
# Calculate Group Means and Standard Errors (SEM) for ΔMAPE
summary_data <- data %>%
  group_by(Condition, AdvisorCompetency) %>%
  summarize(
    Mean_Delta_MAPE = mean(Delta_MAPE),
    SEM_Delta_MAPE = sd(Delta_MAPE) / sqrt(n()),
    .groups = 'drop'
  )

# Display summary data for verification
print(summary_data)
## # A tibble: 4 × 4
##   Condition      AdvisorCompetency Mean_Delta_MAPE SEM_Delta_MAPE
##   <chr>          <chr>                       <dbl>          <dbl>
## 1 Sleep_Deprived High                       -26.9           0.188
## 2 Sleep_Deprived Medium                      -4.77          0.183
## 3 Well_Rested    High                       -20.0           0.183
## 4 Well_Rested    Medium                      -8.15          0.188
# Generate the bar chart with error bars
ggplot(summary_data, aes(x = AdvisorCompetency, y = Mean_Delta_MAPE, fill = Condition)) +
  geom_bar(stat = "identity", position = position_dodge(width = 0.9), width = 0.7) +
  geom_errorbar(aes(ymin = Mean_Delta_MAPE - SEM_Delta_MAPE, ymax = Mean_Delta_MAPE + SEM_Delta_MAPE),
                position = position_dodge(width = 0.9), width = 0.25) +
  labs(
    title = expression("Improvements in MAPE Scores by Condition and Advisor Competency"),
    x = "Advisor Competency",
    y = expression("Mean Improvement in MAPE (" * Delta * "MAPE)")
  ) +
  scale_fill_manual(values = c("Sleep_Deprived" = "lightblue", "Well_Rested" = "lightgreen"),
                    name = "Condition") +
  theme_minimal() +
  theme(
    plot.title = element_text(hjust = 0.5),
    axis.title.y = element_text(vjust = 1.5),
    axis.title.x = element_text(vjust = -0.5)
  ) +
  # Optional: Add significance markers if Mean_Delta_MAPE < 0.05
  geom_text(aes(label = ifelse(abs(Mean_Delta_MAPE) < 0.05, "*", "")),
            vjust = -1, color = "red", position = position_dodge(0.9))

The second chart displayed the improvement in MAPE scores from initial estimates to final estimates to measure the competency of the advisors. There is a close resemblance between the chart from the original article and the chart generated from the simulated data. For instance, for those highly competent advisors, the advisory competency increased by 27 for the sleep-deprived participants and 20 for the well-rested participants. This resemblance is similar to the findings of the advisory competency of the medium competency advisors.

Model Output

The analysis involved the application of a mediation model to explore the increase in advice taking those results from sleep deprivation cause changes in judgment accuracy. Through a bootstrapping procedure, the indirect influence of sleep deprivation (independent variable) on change in MAPE (dependent variable is tested via advice-taking (mediator). A similar mediation model analysis is applied to the simulated data and the results compared.

library(dplyr)
library(mediation)
library(openxlsx)
# Recode Condition as numeric for mediation analysis
data <- data %>%
  mutate(Condition_numeric = ifelse(Condition == "Sleep_Deprived", 1, 0))
# Fit the mediator model (a path) predicting WOA from Condition
mediator_model <- lm(WOA ~ Condition_numeric, data = data)
# Fit the outcome model (c and c′ paths) predicting Delta_MAPE from Condition and WOA
outcome_model <- lm(Delta_MAPE ~ Condition_numeric + WOA, data = data)
# Perform the mediation analysis using the numeric Condition
mediation_results <- mediate(mediator_model, outcome_model, treat = "Condition_numeric", mediator = "WOA", boot = TRUE, sims = 1000)
# Extract coefficients for paths (a, b, c, c′)
# Extract the results for the mediation paths to create the table
summary_mediation <- summary(mediation_results)
# Fit the c-path model separately for the condition effect
c_model <- lm(Delta_MAPE ~ Condition_numeric, data = data)
# Extract coefficients with correct naming
a_path <- coef(summary(mediator_model))["Condition_numeric", ]  # a-path (effect of Condition on WOA)
b_path <- coef(summary(outcome_model))["WOA", ]  # b-path (effect of WOA on Delta_MAPE)
c_path <- coef(summary(c_model))["Condition_numeric", ]  # c-path (direct effect of Condition on Delta_MAPE)
c_prime_path <- coef(summary(outcome_model))["Condition_numeric", ]  # c′-path (direct effect of Condition on Delta_MAPE with mediator)
# Create the table with mediation results for a, b, and indirect paths
# Manually check if `summary_mediation$d.avg.se` is available, otherwise set to NA
d_avg_se <- ifelse(length(summary_mediation$d.avg.se) > 0, summary_mediation$d.avg.se, NA)
d_avg_p <- ifelse(length(summary_mediation$d.avg.p) > 0, summary_mediation$d.avg.p, NA)
# Now create the table with mediation results, ensuring all columns have 3 elements
mediation_table <- data.frame(
  Predictor = c("Experimental Condition (IV)", "WOA-values (M)", "Indirect"),
  Outcome = c("WOA-values (M)", "MAPE difference (DV)", "MAPE difference (DV)"),
  B = c(a_path["Estimate"], b_path["Estimate"], summary_mediation$d.avg),
  SE = c(a_path["Std. Error"], b_path["Std. Error"], d_avg_se),
  T_or_Z = c(a_path["t value"], b_path["t value"], summary_mediation$d.avg / d_avg_se),
  p_value = c(a_path["Pr(>|t|)"], b_path["Pr(>|t|)"], d_avg_p)
)

# Create the table with direct effects (c and c′ paths)
direct_effects_table <- data.frame(
  Predictor = c("Experimental Condition (IV)", "Experimental Condition (IV)"),
  Outcome = c("MAPE difference (DV)", "MAPE difference (DV)"),
  B = c(c_path["Estimate"], c_prime_path["Estimate"]),
  SE = c(c_path["Std. Error"], c_prime_path["Std. Error"]),
  T_or_Z = c(c_path["t value"], c_prime_path["t value"]),
  p_value = c(c_path["Pr(>|t|)"], c_prime_path["Pr(>|t|)"])
)
# Combine the mediation and direct effects tables
full_table <- rbind(mediation_table, direct_effects_table)

# Display the mediation table
kable(full_table, caption = "Mediation Analysis Results: Coefficients and Significance Levels for a, b, c, and Indirect Paths")
Mediation Analysis Results: Coefficients and Significance Levels for a, b, c, and Indirect Paths
Predictor Outcome B SE T_or_Z p_value
Experimental Condition (IV) WOA-values (M) 0.0909236 0.0055232 16.462174 0e+00
WOA-values (M) MAPE difference (DV) -39.8760738 1.0435174 -38.213138 0e+00
Indirect MAPE difference (DV) -3.6256745 NA NA 0e+00
Experimental Condition (IV) MAPE difference (DV) -1.7764538 0.3795735 -4.680132 3e-06
Experimental Condition (IV) MAPE difference (DV) 1.8492207 0.3234264 5.717594 0e+00

The model output from the original article also resembled the model output from the simulated data. The mediation model examined whether WOA values mediate the influence of Experimental Conditions on MAPE difference. On the impact of Experimental Condition (IV) on WOA values, the positive coefficient (0.091), which is statistically significant (p < 0.01) shows that participants who are sleep-deprived have significantly higher WOA values compared to the well-rested group. This is similar to the original table and table with simulated data. The row representing the influence of WOA values on MAPE differences shows a statistically significant (p < 0.01) negative coefficient (-21.96) in the original table and -39.87 in the simulated table. These indicate that higher WOA values are linked with greater improvements in judgment accuracy. The coefficients and statistical significances in the simulated table have similar resemblance and interpretation for the indirect effect of Experimental Condition on MAPE difference (p < 0.01, original coefficient= -2.00, simulated coefficient = -3.626); the Total Effect of Experimental Condition on MAPE difference (p < 0.01, original coefficient= -1.72, simulated coefficient = -1.776); and the Direct Effect of Experimental Condition on MAPE difference (p < 0.01, original coefficient= 0.28, simulated coefficient = 1.849). These show a close resemblance of the simulated data and the data from the original article.

Power analysis

The study applied a 2 x 2 mixed design: a between-subjects factor for sleep deprivation and a within-subjects factor for advisory competency. The original article stated an expected effect size of ¦= 0.2 (Cohen’s small-to-medium) and a power level of 0.90. The power analysis was performed to detect a significance of α=0.05 using simulation by running a Monte Carlo power analysis. This involved a simulation of data multiple times and calculating the percentage of times we detected a significant influence at the α- level. The analysis assumes a mixed-design ANOVA structure similar to the original study, with a between-subjects factor (sleep deprivation) and within-subjects factor (advisor competency).

# Load necessary libraries
library(MASS)    # For multivariate data simulation
library(lme4)    # For linear mixed effects models
## Warning: package 'lme4' was built under R version 4.4.2
# Simulation parameters
n_simulations <- 1000    # Number of simulations to estimate power
n_between <- 50          # Number of participants per group (sleep-deprived vs well-rested)
n_within <- 2            # Number of levels in within-subject factor (advisor competency)
alpha <- 0.05            # Significance level
effect_size <- 0.2       # Cohen's f effect size for small-to-medium effect
# Store results
results <- numeric(n_simulations)
simulate_experiment <- function() {
  # Generate data 
  condition <- rep(c("Sleep_Deprived", "Well_Rested"), each = n_between)
  advisor_competency <- factor(rep(c("Medium", "High"), times = n_between * n_within))
  participant <- rep(1:(n_between * 2), each = n_within)
    # Generate outcome variable with effect of condition and competency
  baseline <- 0.5
  outcome <- baseline +
    ifelse(condition == "Sleep_Deprived", effect_size, 0) +
    ifelse(advisor_competency == "High", effect_size / 2, 0) +
    rnorm(length(condition), mean = 0, sd = 1)
    # Combine data
  data <- data.frame(participant, condition, advisor_competency, outcome)
    # Fit model without random effects (using ANOVA)
  model <- aov(outcome ~ condition * advisor_competency, data = data)
    # Extract p-value for the interaction effect
  p_value <- summary(model)[[1]]["condition:advisor_competency", "Pr(>F)"]
    # Return 1 if the effect is significant, otherwise 0
  return(ifelse(p_value < alpha, 1, 0))
}
# Run simulations
set.seed(123)  # For reproducibility
for (i in 1:n_simulations) {
  results[i] <- simulate_experiment()
}
# Calculate power as the proportion of significant results
power <- mean(results)
cat("Estimated Power:", power, "\n")
## Estimated Power: 0.178

The estimated power of 0.178 from the power analysis indicates that given the simulation parameters, there is only a 17.8% chance of detecting a significant effect. While this is below the conventional threshold for adequate power, which is often 80%, the original article stated an expected effect size of ¦= 0.2 (Cohen’s small-to-medium).

Sensitivity analysis

In undertaking sensitivity analysis for the simulated data, power analysis is applied to determine the smallest detectable effect size with the sample size and the power levels. This helps in the detection of the effect size that the study would be able to detect with adequate statistical power, often set at 0.80.

# Load the required package
if (!require("pwr")) install.packages("pwr", dependencies=TRUE)
## Loading required package: pwr
## Warning: package 'pwr' was built under R version 4.4.2
library(pwr)
# Sensitivity analysis parameters
n_between <- 50           # Sample size per between-subjects group (e.g., 50 participants in each condition)
n_total <- n_between * 2   # Total sample size for the two groups
alpha <- 0.05              # Significance level
power <- 0.80              # Desired power level
# Sensitivity analysis for between-subjects effect (Sleep Condition)
sensitivity_between <- pwr.anova.test(k = 2,     # Two levels: sleep-deprived and well-rested
                                      n = n_total / 2,  # Sample size per group
                                      sig.level = alpha,
                                      power = power)
# Sensitivity analysis for within-subjects effect (Advisor Competency)
# Since this is within-subjects, we'll estimate the effect size needed for a paired t-test
sensitivity_within <- pwr.t.test(n = n_total,  # Total number of observations (both groups)
                                 sig.level = alpha,
                                 power = power,
                                 type = "paired")
# Display results of the sensitivity analysis
cat("Sensitivity Analysis Results:\n")
## Sensitivity Analysis Results:
cat("1. Between-Subjects Effect (Sleep Condition):\n")
## 1. Between-Subjects Effect (Sleep Condition):
cat("Minimum Detectable Effect Size (Cohen's f):", sensitivity_between$f, "\n\n")
## Minimum Detectable Effect Size (Cohen's f): 0.2829415
cat("2. Within-Subjects Effect (Advisor Competency):\n")
## 2. Within-Subjects Effect (Advisor Competency):
cat("Minimum Detectable Effect Size (Cohen's d):", sensitivity_within$d, "\n\n")
## Minimum Detectable Effect Size (Cohen's d): 0.2829005
# Summary table for sensitivity analysis
sensitivity_results <- data.frame(
  Analysis = c("Between-Subjects (Sleep Condition)", "Within-Subjects (Advisor Competency)"),
  Minimum_Detectable_Effect_Size = c(sensitivity_between$f, sensitivity_within$d)
)
# Print the summary table
print(sensitivity_results)
##                               Analysis Minimum_Detectable_Effect_Size
## 1   Between-Subjects (Sleep Condition)                      0.2829415
## 2 Within-Subjects (Advisor Competency)                      0.2829005

The sensitivity analysis results indicate that for the between-subjects factor (sleep condition) the minimum detectable size (Cohen’s ¦) is 0.283. This is similar to the minimum detectable effect size for the within-subjects factor (Advisor Competency) (Cohen’s d = 0.283). These fall within the small-to-medium range, as mentioned in the original study.

Reflections

This coursework has provided me with deep insights into how to undertake simulation analysis, power analysis, and sensitivity analysis. The replication of the dataset of the real study provides the opportunity to control specific parameters such as variances and effect sizes. It also provides the opportunity to comprehend expected patterns under different experimental conditions, including those that were not performed in the original study. Moreover, power analysis provides information regarding the decisions on the sample size while sensitivity analysis helps in the determination of the smallest effect size that can be reliably detected. While my simulated data had findings that closely resembled descriptive statistics, visual output and model output of the original article, the power analysis showed an estimated power of only 17.8%, which is way below the recommended 80% level. This is likely influenced by the nature of the study, as the original article outlined that the independent variables would likely have small to medium influence on the dependent variables.

I learned that it is significant to understand the study design and technical details of the simulation functions in order to correctly replicate the dataset of a study. Setting up factors, groups, and interaction terms was one of the most challenging issues I faced in the simulation of the random effects and ensuring that they matched the expected distribution. The generation of codes for data simulation, power analysis, and sensitivity analysis can sometimes be complex, as issues such as singular fits in mixed models, mismatched vectors, and understanding error messages are consistent. For instance, I encountered boundary singularity errors in generating the model output for the simulated data and a perfumed a lot of troubleshooting before finding and rectifying the error.

A key lesson that I learned is the significance of parameter checking, as it is always important to ensure realistic effect sizes, interaction terms, and sample sizes. Small errors in the specification of parameters can result in incorrect results, making it important to double-check parameters all the time. An iterative approach to coding is another issue that is learned in the coursework. In the creation of complex analysis workflows in R, it is best to have a step-by-step approach in which each part is tested individually. This helps in the isolation of errors and makes troubleshooting easy.

For improvements in the future, I will concentrate on optimisation of simulation and analysis code. These will include comprehension of ways in which code efficiency is improved through the use of vectorised operations, minimisation for -loops, and applying parallel computing for simulations. Code optimisation can improve reproducibility by saving time. I will also work on the documentation and code readability, as the documentation of codes makes it easier to understand complex analysis scripts. I intend to improve my understanding of advanced mixed modeling techniques, visualisation packages such as ggplot2 for more customised plots, and bootstrapping for power analysis.

References

Lim, J. & Dinges, D. F. (2010). A meta-analysis of the impact of short-term sleep deprivation on cognitive variables. Psychol Bull 136(3), 375–389.

Harrison, Y. & Horne, J. A. (2000). The impact of sleep deprivation on decision making: A review. J Exp Psychol: Appl. 6(3), 236–249.

Baranski, J. V. at al. (2007). Effects of Sleep Loss on Team Decision Making: Motivational Loss or Motivational Gain? Hum Factors 49(4), 646–660.

Faber, N. S., Häusser, J. A., & Kerr, N. L. (2017). Sleep deprivation impairs and caffeine enhances my performance, but not always our performance: How acting in a group can change the effects of impairments and enhancements. Personality and Social Psychology Review, 21(1), 3-28.