suppressWarnings({
library(githubinstall)
library(evaluate)
library(tidyverse)
gh_install_packages("konfound", ref = "newitcv_2by2_update", force = TRUE)
library(konfound) })2x2, logistic output_091224
Overall summary
With the revised output language (and corrected calculation of primary and supplemental fragility), the overall output where
needtworows == Tscenarios align well with the key output values, including replacement and transfer interpretations.One question remains regarding the addition of output language for the
replaceoption in the 2x2 and logistic model. Forneedtworows == Tscenarios, if the focus of the output language is fragility, the current printed output would be sufficient and would not require further modification.
Importing Package
2x2 Model Output (when needtworows == T)
- For the 2x2 model, the printed output under the specific condition
needtworows == Tis primarily related to the input options likeswitch_trm(True/False),replace(entire/control), andtest(Fisher’s exact test/chi-square).
The following eight cases represent a subset of the scenarios when
needtworows == T(1610 cases in total), based on various combinations of these options.These cases were generated using the following big-loop code snippet, which covers all combinations of the parameters (Total cases = 6^4 * 2 * 2 * 3 * 2 = 31,104, considering variations in
switch_trm,replace,test, and alpha values of .05, .01, and .001).
# Generate the sequence
num_pool <- seq(1, 3000, by = 1)
# Define the intervals
intervals <- list(
`1-5` = num_pool[num_pool >= 1 & num_pool <= 4],
`6-9` = num_pool[num_pool >= 5 & num_pool <= 9],
`10-99` = num_pool[num_pool >= 10 & num_pool <= 99],
`100-999` = num_pool[num_pool >= 100 & num_pool <= 999],
`1000-1999` = num_pool[num_pool >= 1000 & num_pool <= 1999],
`2000-3000` = num_pool[num_pool >= 2000 & num_pool <= 3000]
)
# Function to safely sample from an interval
safe_sample <- function(x) {
if (length(x) > 0) {
return(sample(x, 1))
} else {
return(NA)
}
}
# Function to generate a list of random numbers for each interval
generate_random_list <- function(seed) {
set.seed(seed)
random_list <- sapply(intervals, safe_sample)
return(as.numeric(random_list))
}
# Generate sets of random numbers
a_list <- generate_random_list(123)
b_list <- generate_random_list(456)
c_list <- generate_random_list(789)
d_list <- generate_random_list(101112)
a_list[1] 3 7 23 294 1937 2817b_list[1] 1 9 44 649 1468 2251c_list[1] 1 8 51 765 1623 2994d_list[1] 2 7 36 397 1115 2467switch_trm_list = c("TRUE", "FALSE")
replace_list = c("entire", "control")
alpha_list = c(0.05, 0.01, 0.001)
test_list = c("fisher", "chisq")Printed Output
Overall, the printed output looks fine for me. However, there is no indication of whether
replace = "entire"orreplace = "control"in cases whereneedtworows == T, as compared toneedtworows == Fcases.Should we consider adding text to clarify whether the replacement is based on the entire group or the control group for
needtworows == Tcases?
# switch_trm = TRUE / replace = "entire"
# chisq
pkonfound(a = 7, b = 9, c = 8, d = 7,
switch_trm = TRUE, replace = "entire", alpha = 0.001, test = "chisq", to_return = "print")Warning in chisq.test(table, correct = FALSE): Chi-squared approximation may be
incorrect
Warning in chisq.test(table, correct = FALSE): Chi-squared approximation may be
incorrectRobustness of Inference to Replacement (RIR):
RIR = 13 + 4 = 17
Total RIR = Primary RIR in treatment row + Supplemental RIR in control row
Fragility = 6 + 2 = 8
Total Fragility = Primary Fragility in treatment row + Supplemental Fragility in control row
This function calculates the number of data points that would have to be replaced with
zero effect data points (RIR) to invalidate an inference made about the association
between the rows and columns in a 2x2 table.
One can also interpret this as switches (Fragility) from one cell to another, such as from the
treatment success cell to the treatment failure cell.
To sustain an inference, only transferring 6 data points from treatment success to treatment failure
is not enough to change the inference.
We also need to transfer 2 data points from control failure to control success as shown, from the
User-entered Table to the Transfer Table.
These switches would require one to replace 13 of treatment success with null hypothesis data points;
and replace 4 control failure with null hypothesis data points to change the inference.
For the User-entered Table, the Pearson's chi square is 0.285, with p-value of 0.594:
User-entered Table:
Fail Success Success_Rate
Control 7 9 56.25%
Treatment 8 7 46.67%
Total 15 16 51.61%
For the Transfer Table, the Pearson's chi square is 12.577, with p-value of 0.000:
Transfer Table:
Fail Success Success_Rate
Control 5 11 68.75%
Treatment 14 1 6.67%
Total 19 12 38.71%For other forms of output, run
?pkonfound and inspect the to_return argumentFor models fit in R, consider use of konfound().# fisher
pkonfound(a = 7, b = 1, c = 55, d = 4,
switch_trm = TRUE, replace = "entire", alpha = 0.05, test = "fisher")Robustness of Inference to Replacement (RIR):
RIR = 5 + 14 = 19
Total RIR = Primary RIR in treatment row + Supplemental RIR in control row
Fragility = 4 + 1 = 5
Total Fragility = Primary Fragility in treatment row + Supplemental Fragility in control row
This function calculates the number of data points that would have to be replaced with
zero effect data points (RIR) to invalidate an inference made about the association
between the rows and columns in a 2x2 table.
One can also interpret this as switches (Fragility) from one cell to another, such as from the
treatment success cell to the treatment failure cell.
To sustain an inference, only transferring 4 data points from treatment success to treatment failure
is not enough to change the inference.
We also need to transfer 1 data points from control failure to control success as shown, from the
User-entered Table to the Transfer Table.
These switches would require one to replace 5 of treatment success with null hypothesis data points;
and replace 14 control failure with null hypothesis data points to change the inference.
For the User-entered Table, the estimated odds ratio is 0.515, with p-value of 0.482:
User-entered Table:
Fail Success Success_Rate
Control 7 1 12.50%
Treatment 55 4 6.78%
Total 62 5 7.46%
For the Transfer Table, the estimated odds ratio is 0.000, with p-value of 0.013:
Transfer Table:
Fail Success Success_Rate
Control 6 2 25.00%
Treatment 59 0 0.00%
Total 65 2 2.99%For other forms of output, run
?pkonfound and inspect the to_return argument
For models fit in R, consider use of konfound().# switch_trm = TRUE / replace = "control"
# chisq
pkonfound(a = 294, b = 44, c = 51, d = 7,
switch_trm = TRUE, replace = "control", alpha = 0.01, test = "chisq")Robustness of Inference to Replacement (RIR):
RIR = 7 + 16 = 23
Total RIR = Primary RIR in treatment row + Supplemental RIR in control row
Fragility = 6 + 2 = 8
Total Fragility = Primary Fragility in treatment row + Supplemental Fragility in control row
This function calculates the number of data points that would have to be replaced with
zero effect data points (RIR) to invalidate an inference made about the association
between the rows and columns in a 2x2 table.
One can also interpret this as switches (Fragility) from one cell to another, such as from the
treatment success cell to the treatment failure cell.
To sustain an inference, only transferring 6 data points from treatment success to treatment failure
is not enough to change the inference.
We also need to transfer 2 data points from control failure to control success as shown, from the
User-entered Table to the Transfer Table.
These switches would require one to replace 7 of treatment success with null hypothesis data points;
and replace 16 control failure with null hypothesis data points to change the inference.
For the User-entered Table, the Pearson's chi square is 0.040, with p-value of 0.842:
User-entered Table:
Fail Success Success_Rate
Control 294 44 13.02%
Treatment 51 7 12.07%
Total 345 51 12.88%
For the Transfer Table, the Pearson's chi square is 6.686, with p-value of 0.010:
Transfer Table:
Fail Success Success_Rate
Control 292 46 13.61%
Treatment 57 1 1.72%
Total 349 47 11.87%For other forms of output, run
?pkonfound and inspect the to_return argumentFor models fit in R, consider use of konfound().# fisher
pkonfound(a = 2584, b = 22, c = 901, d = 4,
switch_trm = TRUE, replace = "control", alpha = 0.001, test = "fisher")Robustness of Inference to Replacement (RIR):
RIR = 5 + 356 = 361
Total RIR = Primary RIR in treatment row + Supplemental RIR in control row
Fragility = 4 + 3 = 7
Total Fragility = Primary Fragility in treatment row + Supplemental Fragility in control row
This function calculates the number of data points that would have to be replaced with
zero effect data points (RIR) to invalidate an inference made about the association
between the rows and columns in a 2x2 table.
One can also interpret this as switches (Fragility) from one cell to another, such as from the
treatment success cell to the treatment failure cell.
To sustain an inference, only transferring 4 data points from treatment success to treatment failure
is not enough to change the inference.
We also need to transfer 3 data points from control failure to control success as shown, from the
User-entered Table to the Transfer Table.
These switches would require one to replace 5 of treatment success with null hypothesis data points;
and replace 356 control failure with null hypothesis data points to change the inference.
For the User-entered Table, the estimated odds ratio is 0.521, with p-value of 0.267:
User-entered Table:
Fail Success Success_Rate
Control 2584 22 0.84%
Treatment 901 4 0.44%
Total 3485 26 0.74%
For the Transfer Table, the estimated odds ratio is 0.000, with p-value of 0.001:
Transfer Table:
Fail Success Success_Rate
Control 2581 25 0.96%
Treatment 905 0 0.00%
Total 3486 25 0.71%For other forms of output, run
?pkonfound and inspect the to_return argument
For models fit in R, consider use of konfound().# switch_trm = FALSE / replace = "entire"
# chisq
pkonfound(a = 7, b = 22, c = 901, d = 2347,
switch_trm = FALSE, replace = "entire", alpha = 0.001, test = "chisq")Robustness of Inference to Replacement (RIR):
RIR = 9 + 506 = 515
Total RIR = Primary RIR in control row + Supplemental RIR in treatment row
Fragility = 6 + 140 = 146
Total Fragility = Primary Fragility in control row + Supplemental Fragility in treatment row
This function calculates the number of data points that would have to be replaced with
zero effect data points (RIR) to invalidate an inference made about the association
between the rows and columns in a 2x2 table.
One can also interpret this as switches (Fragility) from one cell to another, such as from the
treatment success cell to the treatment failure cell.
To sustain an inference, only transferring 6 data points from control failure to control success
is not enough to change the inference.
We also need to transfer 140 data points from treatment success to treatment failure as shown, from the
User-entered Table to the Transfer Table.
These switches would require one to replace 9 of control failure with null hypothesis data points;
and replace 506 treatment success with null hypothesis data points to change the inference.
For the User-entered Table, the Pearson's chi square is 0.186, with p-value of 0.666:
User-entered Table:
Fail Success Success_Rate
Control 7 22 75.86%
Treatment 901 2347 72.26%
Total 908 2369 72.29%
For the Transfer Table, the Pearson's chi square is 10.843, with p-value of 0.001:
Transfer Table:
Fail Success Success_Rate
Control 1 28 96.55%
Treatment 1041 2207 67.95%
Total 1042 2235 68.20%For other forms of output, run
?pkonfound and inspect the to_return argumentFor models fit in R, consider use of konfound().# fisher
pkonfound(a = 7, b = 211, c = 1, d = 16,
switch_trm = FALSE, replace = "entire", alpha = 0.05, test = "fisher")Robustness of Inference to Replacement (RIR):
RIR = 7 + 30 = 37
Total RIR = Primary RIR in control row + Supplemental RIR in treatment row
Fragility = 6 + 1 = 7
Total Fragility = Primary Fragility in control row + Supplemental Fragility in treatment row
This function calculates the number of data points that would have to be replaced with
zero effect data points (RIR) to invalidate an inference made about the association
between the rows and columns in a 2x2 table.
One can also interpret this as switches (Fragility) from one cell to another, such as from the
treatment success cell to the treatment failure cell.
To sustain an inference, only transferring 6 data points from control failure to control success
is not enough to change the inference.
We also need to transfer 1 data points from treatment success to treatment failure as shown, from the
User-entered Table to the Transfer Table.
These switches would require one to replace 7 of control failure with null hypothesis data points;
and replace 30 treatment success with null hypothesis data points to change the inference.
For the User-entered Table, the estimated odds ratio is 0.533, with p-value of 0.457:
User-entered Table:
Fail Success Success_Rate
Control 7 211 96.79%
Treatment 1 16 94.12%
Total 8 227 96.60%
For the Transfer Table, the estimated odds ratio is 0.000, with p-value of 0.005:
Transfer Table:
Fail Success Success_Rate
Control 0 218 100.00%
Treatment 2 15 88.24%
Total 2 233 99.15%For other forms of output, run
?pkonfound and inspect the to_return argument
For models fit in R, consider use of konfound().# switch_trm = FALSE / replace = "control"
# chisq
pkonfound(a = 2817, b = 9, c = 1623, d = 7,
switch_trm = FALSE, replace = "control", alpha = 0.001, test = "chisq")Warning in chisq.test(table, correct = FALSE): Chi-squared approximation may be
incorrect
Warning in chisq.test(table, correct = FALSE): Chi-squared approximation may be
incorrect
Warning in chisq.test(table, correct = FALSE): Chi-squared approximation may be
incorrect
Warning in chisq.test(table, correct = FALSE): Chi-squared approximation may be
incorrect
Warning in chisq.test(table, correct = FALSE): Chi-squared approximation may be
incorrect
Warning in chisq.test(table, correct = FALSE): Chi-squared approximation may be
incorrect
Warning in chisq.test(table, correct = FALSE): Chi-squared approximation may be
incorrect
Warning in chisq.test(table, correct = FALSE): Chi-squared approximation may be
incorrect
Warning in chisq.test(table, correct = FALSE): Chi-squared approximation may be
incorrectRobustness of Inference to Replacement (RIR):
RIR = 9 + 628 = 637
Total RIR = Primary RIR in control row + Supplemental RIR in treatment row
Fragility = 8 + 2 = 10
Total Fragility = Primary Fragility in control row + Supplemental Fragility in treatment row
This function calculates the number of data points that would have to be replaced with
zero effect data points (RIR) to invalidate an inference made about the association
between the rows and columns in a 2x2 table.
One can also interpret this as switches (Fragility) from one cell to another, such as from the
treatment success cell to the treatment failure cell.
To sustain an inference, only transferring 8 data points from control success to control failure
is not enough to change the inference.
We also need to transfer 2 data points from treatment failure to treatment success as shown, from the
User-entered Table to the Transfer Table.
These switches would require one to replace 9 of control success with null hypothesis data points;
and replace 628 treatment failure with null hypothesis data points to change the inference.
For the User-entered Table, the Pearson's chi square is 0.356, with p-value of 0.551:
User-entered Table:
Fail Success Success_Rate
Control 2817 9 0.32%
Treatment 1623 7 0.43%
Total 4440 16 0.36%
For the Transfer Table, the Pearson's chi square is 12.329, with p-value of 0.000:
Transfer Table:
Fail Success Success_Rate
Control 2825 1 0.04%
Treatment 1621 9 0.55%
Total 4446 10 0.22%For other forms of output, run
?pkonfound and inspect the to_return argumentFor models fit in R, consider use of konfound().# fisher
pkonfound(a = 7, b = 1, c = 1, d = 4,
switch_trm = FALSE, replace = "control", alpha = 0.001, test = "fisher")Robustness of Inference to Replacement (RIR):
RIR = 2 + 8 = 10
Total RIR = Primary RIR in control row + Supplemental RIR in treatment row
Fragility = 1 + 1 = 2
Total Fragility = Primary Fragility in control row + Supplemental Fragility in treatment row
This function calculates the number of data points that would have to be replaced with
zero effect data points (RIR) to invalidate an inference made about the association
between the rows and columns in a 2x2 table.
One can also interpret this as switches (Fragility) from one cell to another, such as from the
treatment success cell to the treatment failure cell.
To sustain an inference, only transferring 1 data points from control success to control failure
is not enough to change the inference.
We also need to transfer 1 data points from treatment failure to treatment success as shown, from the
User-entered Table to the Transfer Table.
These switches would require one to replace 2 of control success with null hypothesis data points;
and replace 8 treatment failure with null hypothesis data points to change the inference.
For the User-entered Table, the estimated odds ratio is 18.534, with p-value of 0.032:
User-entered Table:
Fail Success Success_Rate
Control 7 1 12.50%
Treatment 1 4 80.00%
Total 8 5 38.46%
For the Transfer Table, the estimated odds ratio is Inf, with p-value of 0.001:
Transfer Table:
Fail Success Success_Rate
Control 8 0 0.00%
Treatment 0 5 100.00%
Total 8 5 38.46%For other forms of output, run
?pkonfound and inspect the to_return argument
For models fit in R, consider use of konfound().needtworows == F Scenario
- In the
needtworows == Fscenario, the printed output explicitly reflects thereplaceoption.
pkonfound(a = 2817, b = 9, c = 1623, d = 7,
switch_trm = TRUE, replace = "entire", alpha = 0.05, test = "chisq")Robustness of Inference to Replacement (RIR):
RIR = 1393
Fragility = 5
This function calculates the number of data points that would have to be replaced with
zero effect data points (RIR) to invalidate an inference made about the association
between the rows and columns in a 2x2 table.
One can also interpret this as switches (Fragility) from one cell to another, such as from the
treatment success cell to the treatment failure cell.
To sustain an inference, one would need to replace 1393 treatment failure data points with data points
for which the probability of failure in the entire group applies (RIR = 1393).
This is equivalent to transferring 5 data points from treatment failure to treatment success (Fragility = 5).
For the User-entered Table, the Pearson's chi square is 0.356, with p-value of 0.551:
User-entered Table:
Fail Success Success_Rate
Control 2817 9 0.32%
Treatment 1623 7 0.43%
Total 4440 16 0.36%
For the Transfer Table, the Pearson's chi square is 3.846, with p-value of 0.050:
Transfer Table:
Fail Success Success_Rate
Control 2817 9 0.32%
Treatment 1618 12 0.74%
Total 4435 21 0.47%For other forms of output, run
?pkonfound and inspect the to_return argumentFor models fit in R, consider use of konfound().Logistic Model
- Similar to the 2x2 model,
needtworows == Tcases are selected from a set of 160 scenarios, derived from the broader data set of 21,600 cases (5 * 3 * 3 * 4 * 5 * 2 * 2 * 3 * 2 * 1), generated by the big-loop input process.
est_eff_list = c(-3, -0.3, 0.1, 0.3, 3)
std_err = c(0.01, 0.5, 1)
n_obs_list = c(5000, 500, 20)
n_covariates_list = c(0, 5, 20, 50)
n_treat_list = c(0, 20, 50, 500, 2500)
switch_trm_list = c("TRUE", "FALSE")
replace_list = c("entire", "control")
alpha_list = c(0.05, 0.01, 0.001)
tail_list = c(2, 1)
nu_list = c(0)Printed Output
The logistic model produces fewer
needtworows == Tcases compared to the 2x2 model.Additionally, only the sustaining scenario is observed, with variations occurring solely based on the
changeSEoption (True or False).
# switch_trm = TRUE / replace = "entire"
pkonfound(-0.3, 0.01, 500, n_covariates = 20,
alpha = .001, tails = 2, n_treat = 20,
switch_trm = TRUE, replace = "entire",
model_type = 'logistic', to_return = "print")Robustness of Inference to Replacement (RIR):
RIR = 21 + 39 = 60
Total RIR = Primary RIR in treatment row + Supplemental RIR in control row
Fragility = 9 + 22 = 31
Total Fragility = Primary Fragility in treatment row + Supplemental Fragility in control row
The table implied by the parameter estimates and sample sizes you entered:
Fail Success Success_Rate
Control 206 274 57.08%
Treatment 10 10 50.00%
Total 216 284 56.80%
The reported log odds = -0.300, SE = 0.010, and p-value = 0.532.
The SE has been adjusted to 0.457 to generate real numbers in the
implied table for which the p-value would be 0.532. Numbers in
the table cells have been rounded to integers, which may slightly
alter the estimated effect from the value originally entered.
The inference cannot be sustained merely by switching 9 data points in
the treatment row. Therefore, 22 additional data points have been
switched from control failure to control success.
The final Fragility(= 31) and RIR(= 60) reflect both sets of changes.
Please compare the after transfer table with the implied table.
Fail Success Success_Rate
Control 184 296 61.67%
Treatment 19 1 5.00%
Total 203 297 59.40%
The log odds = -3.420, SE = 1.030, p-value = 0.001.
This is based on t = estimated effect/standard error
See Frank et al. (2013) for a description of the method.
Citation: Frank, K.A., Maroulis, S., Duong, M., and Kelcey, B. (2013).
What would it take to change an inference?
Using Rubin's causal model to interpret the robustness of causal inferences.
Education, Evaluation and Policy Analysis, 35 , 437-460.
Accuracy of results increases with the number of decimals entered.For other forms of output, run
?pkonfound and inspect the to_return argumentFor models fit in R, consider use of konfound().# switch_trm = TRUE / replace = "control"
pkonfound(0.1, 0.5, 5000, n_covariates = 50,
alpha = .01, tails = 2, n_treat = 2500,
switch_trm = TRUE, replace = "control",
model_type = 'logistic', to_return = "print")Robustness of Inference to Replacement (RIR):
RIR = 8 + 2188 = 2196
Total RIR = Primary RIR in treatment row + Supplemental RIR in control row
Fragility = 7 + 7 = 14
Total Fragility = Primary Fragility in treatment row + Supplemental Fragility in control row
The table implied by the parameter estimates and sample sizes you entered:
Fail Success Success_Rate
Control 8 2492 99.68%
Treatment 8 2492 99.68%
Total 16 4984 99.68%
The reported log odds = 0.100, SE = 0.500, and p-value = 1.000.
Values have been rounded to the nearest integer. This may cause
a small change to the estimated effect for the table.
The inference cannot be sustained merely by switching 7 data points in
the treatment row. Therefore, 7 additional data points have been
switched from control success to control failure.
The final Fragility(= 14) and RIR(= 2196) reflect both sets of changes.
Please compare the after transfer table with the implied table.
Fail Success Success_Rate
Control 15 2485 99.40%
Treatment 1 2499 99.96%
Total 16 4984 99.68%
The log odds = 2.714, SE = 1.033, p-value = 0.009.
This is based on t = estimated effect/standard error
See Frank et al. (2013) for a description of the method.
Citation: Frank, K.A., Maroulis, S., Duong, M., and Kelcey, B. (2013).
What would it take to change an inference?
Using Rubin's causal model to interpret the robustness of causal inferences.
Education, Evaluation and Policy Analysis, 35 , 437-460.
Accuracy of results increases with the number of decimals entered.For other forms of output, run
?pkonfound and inspect the to_return argumentFor models fit in R, consider use of konfound().# switch_trm = FALSE / replace = "entire"
pkonfound(0.3, 0.5, 5000, n_covariates = 50,
alpha = .01, tails = 2, n_treat = 2500,
switch_trm = FALSE, replace = "entire",
model_type = 'logistic', to_return = "print")Robustness of Inference to Replacement (RIR):
RIR = 7 + 1875 = 1882
Total RIR = Primary RIR in control row + Supplemental RIR in treatment row
Fragility = 6 + 6 = 12
Total Fragility = Primary Fragility in control row + Supplemental Fragility in treatment row
The table implied by the parameter estimates and sample sizes you entered:
Fail Success Success_Rate
Control 2493 7 0.28%
Treatment 2491 9 0.36%
Total 4984 16 0.32%
The reported log odds = 0.300, SE = 0.500, and p-value = 0.617.
Values have been rounded to the nearest integer. This may cause
a small change to the estimated effect for the table.
The inference cannot be sustained merely by switching 6 data points in
the control row. Therefore, 6 additional data points have been
switched from treatment failure to treatment success.
The final Fragility(= 12) and RIR(= 1882) reflect both sets of changes.
Please compare the after transfer table with the implied table.
Fail Success Success_Rate
Control 2499 1 0.04%
Treatment 2485 15 0.60%
Total 4984 16 0.32%
The log odds = 2.714, SE = 1.033, p-value = 0.009.
This is based on t = estimated effect/standard error
See Frank et al. (2013) for a description of the method.
Citation: Frank, K.A., Maroulis, S., Duong, M., and Kelcey, B. (2013).
What would it take to change an inference?
Using Rubin's causal model to interpret the robustness of causal inferences.
Education, Evaluation and Policy Analysis, 35 , 437-460.
Accuracy of results increases with the number of decimals entered.For other forms of output, run
?pkonfound and inspect the to_return argumentFor models fit in R, consider use of konfound().# switch_trm = FALSE / replace = "control"
pkonfound(-0.3, 0.5, 5000, n_covariates = 5,
alpha = .01, tails = 2, n_treat = 2500,
switch_trm = FALSE, replace = "control",
model_type = 'logistic', to_return = "print")Robustness of Inference to Replacement (RIR):
RIR = 7 + 2143 = 2150
Total RIR = Primary RIR in control row + Supplemental RIR in treatment row
Fragility = 6 + 6 = 12
Total Fragility = Primary Fragility in control row + Supplemental Fragility in treatment row
The table implied by the parameter estimates and sample sizes you entered:
Fail Success Success_Rate
Control 7 2493 99.72%
Treatment 9 2491 99.64%
Total 16 4984 99.68%
The reported log odds = -0.300, SE = 0.500, and p-value = 0.617.
Values have been rounded to the nearest integer. This may cause
a small change to the estimated effect for the table.
The inference cannot be sustained merely by switching 6 data points in
the control row. Therefore, 6 additional data points have been
switched from treatment success to treatment failure.
The final Fragility(= 12) and RIR(= 2150) reflect both sets of changes.
Please compare the after transfer table with the implied table.
Fail Success Success_Rate
Control 1 2499 99.96%
Treatment 15 2485 99.40%
Total 16 4984 99.68%
The log odds = -2.714, SE = 1.033, p-value = 0.009.
This is based on t = estimated effect/standard error
See Frank et al. (2013) for a description of the method.
Citation: Frank, K.A., Maroulis, S., Duong, M., and Kelcey, B. (2013).
What would it take to change an inference?
Using Rubin's causal model to interpret the robustness of causal inferences.
Education, Evaluation and Policy Analysis, 35 , 437-460.
Accuracy of results increases with the number of decimals entered.For other forms of output, run
?pkonfound and inspect the to_return argumentFor models fit in R, consider use of konfound().needtworows == F Scenario
- Similar to 2x2 model, in the
needtworows == Fscenario, the printed output explicitly reflects thereplaceoption.
# switch_trm = TRUE / replace = "entire"
pkonfound(-0.3, 0.01, 500, n_covariates = 20,
alpha = .05, tails = 2, n_treat = 20,
switch_trm = TRUE, replace = "entire",
model_type = 'logistic', to_return = "print")Robustness of Inference to Replacement (RIR):
RIR = 10
Fragility = 4
The table implied by the parameter estimates and sample sizes you entered:
Fail Success Success_Rate
Control 206 274 57.08%
Treatment 10 10 50.00%
Total 216 284 56.80%
The reported log odds = -0.300, SE = 0.010, and p-value = 0.532.
The SE has been adjusted to 0.457 to generate real numbers in the
implied table for which the p-value would be 0.532. Numbers in
the table cells have been rounded to integers, which may slightly
alter the estimated effect from the value originally entered.
To reach the threshold that would sustain an inference that the
effect is different from 0 (alpha = 0.050) one would need to replace 10
(100.000%) treatment success data points with data points for which the probability of
failure in the entire sample (41.200%) applies (RIR = 10). This is equivalent
to transferring 4 data points from treatment success to treatment failure
(Fragility = 4).
Note that RIR = Fragility/[1-P(success in the entire sample)]
The transfer of 4 data points yields the following table:
Fail Success Success_Rate
Control 206 274 57.08%
Treatment 14 6 30.00%
Total 220 280 56.00%
The log odds = -1.133, SE = 0.497, p-value = 0.023.
This is based on t = estimated effect/standard error
See Frank et al. (2013) for a description of the method.
Citation: Frank, K.A., Maroulis, S., Duong, M., and Kelcey, B. (2013).
What would it take to change an inference?
Using Rubin's causal model to interpret the robustness of causal inferences.
Education, Evaluation and Policy Analysis, 35 , 437-460.
Accuracy of results increases with the number of decimals entered.For other forms of output, run
?pkonfound and inspect the to_return argumentFor models fit in R, consider use of konfound().