0) Summary

1) STATA Output Language for RIR in Linear Model (model 0 in STATA)

1.1) Invalidate

  • It appears that the STATA version output simultaneously displays both RIR and ITCV.
  • User entered effect threshold (eff_thr != NA) is going to be updated this week.
// eff_thr = NA
pkonfound_v2 -0.3 0.01 5000 10, nu(0.00) sig(0.05) onetail(0) model_type(0) switch_trm(1) 
pkonfound is working
------------------
Impact Threshold for Omitted Variable

An omitted variable would have to be correlated at 0.611 with the outcome and at -.611 with the predictor
of interest (conditioning on observed covariates. Signs are interchangeable) to invalidate an inference.
Correspondingly the impact of an omitted variable (as defined in Frank 2000) must be
0.611 x -.611=-0.3736 to invalidate an inference.
------------------
The Threshold for % Bias to Invalidate/Sustain the Inference

You entered an estimated effect of -0.3. To invalidate the inference of an effect using the threshold of -.
> 02
for statistical significance with alpha = .05, 93.33% of the (-0.3) estimate would have to be due to bias.
This implies that to invalidate the inference one would expect to have to replace 4667 (93.33%) observation
> s
with cases for which the treatment effect is 0 (RIR = 4667)

Note:
For non-linear models, the impact threshold should not be used.
The % bias calculation is based on the original coefficient,
compare with the use of average partial effects as in the [konfound] command.

See Frank et al. (2013) for a description of the method.

Citation: Frank, K.A., Maroulis, S., Duong, M., and Kelcey, B. (2013).
What would it take to change an inference?
Using Rubin's causal model to interpret the robustness of causal inferences.
Education, Evaluation and Policy Analysis, 35 437-460.


1.2) Sustain

// eff_thr = NA
pkonfound_v2 0.003 0.1 5000 10, nu(0.00) sig(0.05) onetail(0) model_type(0) switch_trm(1) 
pkonfound is working
------------------
Your estimate is not statistically significant.
------------------
------------------
Impact Threshold for Omitted Variable

An omitted variable would have to be correlated at 0.163 with the outcome and at -.163 with the predictor
of interest (conditioning on observed covariates. Signs are interchangeable) to sustain an inference.
Correspondingly the impact of an omitted variable (as defined in Frank 2000) must be
0.163 x -.163=-0.0266 to sustain an inference.
------------------
The Threshold for % Bias to Invalidate/Sustain the Inference

You entered an estimated effect of 0.003. The threshold value for statistical significance is .196 (alpha =
>  .05).
To reach that threshold, 98.47% of the (0.003) estimate would have to be due to bias. This implies that
to sustain the inference one would expect to have to replace 4924 (98.47%) observations with effect of 0
with cases with effect of .196 (RIR = 4924)

Note:
For non-linear models, the impact threshold should not be used.
The % bias calculation is based on the original coefficient,
compare with the use of average partial effects as in the [konfound] command.

See Frank et al. (2013) for a description of the method.

Citation: Frank, K.A., Maroulis, S., Duong, M., and Kelcey, B. (2013).
What would it take to change an inference?
Using Rubin's causal model to interpret the robustness of causal inferences.
Education, Evaluation and Policy Analysis, 35 437-460.


2) STATA Output Language for RIR in Non-linear Model (model 1 in STATA)

2.1) Invalidate, changeSE = TRUE

// Case 2-1: changeSE = TRUE, Invalidate
pkonfound_v2 -0.3 0.01 5000 0 2500, nu(0.00) sig(0.5) onetail(0) model_type(1) switch_trm(1) replace(0)
test_sensitivity_ln is working
RIR = 330

The table implied by the parameter estimates and sample sizes you entered:

             |      Fail    Success 
-------------+----------------------
     Control |      1156            
   Treatment |      1344       1156 

The reported effect size =-0.300, SE = 0.010, p-value = 0.000.
The SE has been adjusted to 0.057 to generate real numbers in the implied table.
Numbers in the table cells have been rounded to integers, which may slightly alter
the estimated effect from the value originally entered.

To invalidate the inference that the effect is different from 0 (alpha = .5)
one would need to replace 330 (24.55%) treatment failure cases
with cases for which the probability of failure in the entire sample (23.12%) applies (RIR = 330).
This is equivalent to transferring 165 cases from treatment failure to treatment success
(Fragility = 165).

Note that RIR = Fragility/[1-P(failure in the entire sample)]

The transfer of 165 cases yields the following table:

             |      Fail    Success 
-------------+----------------------
     Control |      1156       1344 
   Treatment |      1179       1321 
Effect size =-0.037, SE = 0.057, p-value = 0.514.
This is based on t = estimated effect/standard error


2.2) Invalidate, changeSE = FALSE

// Case 2-2: changeSE = FALSE, Invalidate
pkonfound_v2 -0.3 0.1 5000 10 2500, nu(0.00) sig(0.05) onetail(0) model_type(1) switch_trm(1) replace(0)
test_sensitivity_ln is working
RIR = 235

The table implied by the parameter estimates and sample sizes you entered:

             |      Fail    Success 
-------------+----------------------
     Control |      2246        254 
   Treatment |      2307        193 

The reported effect size =-0.300, SE = 0.100, p-value = 0.003.
Values have been rounded to the nearest integer. This may cause little change
to the estimated effect for the table.

To invalidate the inference that the effect is different from 0 (alpha = .05)
one would need to replace 235 (10.19%) treatment failure cases
with cases for which the probability of failure in the entire sample (44.92%) applies (RIR = 235).
This is equivalent to transferring 21 cases from treatment failure to treatment success
(Fragility = 21).

Note that RIR = Fragility/[1-P(failure in the entire sample)]

The transfer of 21 cases yields the following table:

             |      Fail    Success 
-------------+----------------------
     Control |      2246        254 
   Treatment |      2286        214 
Effect size =-0.189, SE = 0.097, p-value = 0.052.
This is based on t = estimated effect/standard error


2.3) Sustain, changeSE = TRUE, RIR exceeds 100%

// Case 2-3: changeSE = TRUE, Sustain
pkonfound_v2 0.05 0.3 50 5 25, nu(0.00) sig(0.01) onetail(0) model_type(1) switch_trm(1) replace(0)
test_sensitivity_ln is working
RIR = 19

The table implied by the parameter estimates and sample sizes you entered:

             |      Fail    Success 
-------------+----------------------
     Control |        13            
   Treatment |        12         13 

The reported effect size = 0.050, SE = 0.300, p-value = 0.930.
The SE has been adjusted to 0.566 to generate real numbers in the implied table.
Numbers in the table cells have been rounded to integers, which may slightly alter
the estimated effect from the value originally entered.

To reach the threshold that would sustain an inference that the effect is different
from 0 (alpha = .01) one would need to replace 19 (158.33%) treatment failure cases
with cases for which the probability of failure in the entire sample (26%) applies (RIR = 19).
This is equivalent to transferring 9 cases from treatment failure to treatment success
(Fragility = 9).

Note that RIR = Fragility/[1-P(failure in the entire sample)]

Note the RIR exceeds 100%. Generating the transfer of 9 cases would
require replacing more cases than are in the treatment failure condition.

The transfer of 9 cases yields the following table:

             |      Fail    Success 
-------------+----------------------
     Control |        13         12 
   Treatment |         3         22 
Effect size = 2.072, SE = 0.734, p-value = 0.007.
This is based on t = estimated effect/standard error


2.4) Sustain, changeSE = FALSE

// Case 2-4: changeSE = FALSE, Sustain
pkonfound_v2 -0.03 0.2 500 10 250, nu(0.00) sig(0.05) onetail(0) model_type(1) switch_trm(1) replace(0)
test_sensitivity_ln is working
RIR = 24

The table implied by the parameter estimates and sample sizes you entered:

             |      Fail    Success 
-------------+----------------------
     Control |       180         70 
   Treatment |       182         68 

The reported effect size =-0.030, SE = 0.200, p-value = 0.881.
Values have been rounded to the nearest integer. This may cause little change
to the estimated effect for the table.

To reach the threshold that would sustain an inference that the effect is different
from 0 (alpha = .05) one would need to replace 24 (35.29%) treatment success cases
with cases for which the probability of failure in the entire sample (36%) applies (RIR = 24).
This is equivalent to transferring 17 cases from treatment success to treatment failure
(Fragility = 17).

Note that RIR = Fragility/[1-P(success in the entire sample)]

The transfer of 17 cases yields the following table:

             |      Fail    Success 
-------------+----------------------
     Control |       180         70 
   Treatment |       199         51 
Effect size =-0.417, SE = 0.211, p-value = 0.049.
This is based on t = estimated effect/standard error


2.5) needtworow = TRUE, Invalidate, changeSE = FALSE

// Case 2-5: changeSE = FALSE, needtworow = T, Invalidate, right
pkonfound_v2 0.25 0.5 70 20 20, nu(0.00) sig(0.001) onetail(0) model_type(1) switch_trm(1) replace(0)
test_sensitivity_ln is working
RIR = 35

The table implied by the parameter estimates and sample sizes you entered:

             |      Fail    Success 
-------------+----------------------
     Control |        27         23 
   Treatment |        10         10 

The reported effect size = 0.250, SE = 0.500, p-value = 0.639.
The SE has been adjusted to 0.530 to generate real numbers in the implied table.
Numbers in the table cells have been rounded to integers, which may slightly alter
the estimated effect from the value originally entered.

The inference cannot be invalidated merely by switching cases in only one treatment
condition. Therefore, cases have been switched from treatment failure to treatment success
and from control success to control failure. The final Fragility (= 16) and RIR (= 35)
reflect both sets of changes.

Please compare the after transfer table with the implied table.

             |      Fail    Success 
-------------+----------------------
     Control |        35         15 
   Treatment |         1         19 
Effect size = 3.813, SE = 1.072, p-value = 0.001.
This is based on t = estimated effect/standard error


3) tkonfound (model 2 in STATA)

3.1) chisq p

  • Due to an error in R markdown, please refer to the figure that directly obtain the output from STATA for the chi-squared p-value option.
// chisq p
pkonfound_v2 35 17 17 38, sig(0.05) model_type(2) switch_trm(1) replace(0) test1(1)
Figure
Figure


3.2) fisher p

// fisher p
pkonfound_v2 35 17 17 38, sig(0.05) model_type(2) switch_trm(1) replace(0) test1(0)
tkonfound is working

Background Information:
This function calculates the number of cases that would have to be replaced 
with zero effect cases (RIR) to invalidate an inference made about the association 
between the rows and columns in a 2x2 table. 
One can also interpret this as switches from one cell to another, such as from 
the treatment success cell to the treatment failure cell.

Conclusion:
To invalidate the inference, one would need to replace 14 treatment success
cases for which the probability of failure in the control group applies (RIR = 14).
This is equivalent to transferring 9 cases from treatment success to treatment failure.

For the User-entered Table, the estimated odds ratio is 4.602, with p-value of 0

             |      Fail    Success 
-------------+----------------------
     Control |        35            
   Treatment |        17         38 

For the Transfer Table, the estimated odds ratio is 2.296 with p-value of .051

             |      Fail    Success 
-------------+----------------------
     Control |        35         17 
   Treatment |        26         29 

RIR = 14