This document confirms that there is no numerical difference between newitcv_output, which was recently merged with newitcv to update its output language.
Since newitcv_output is a merged version with newitcv, and there is no difference between the previous newitcv_output(can check in Output_final_240202) that I created, it can be concluded that there are no differences in numerical values or output languages.
pkonfound(), Linear
For the linear model, indexes “RIR” and “IT” have updated language outputs.
In the “IT” section, the overall language remains the same (except for some edits in the non-covariate condition), with slight editing applied for line alignment.
You can check the “Click here for the article” hyperlink attached to each citation in the Viewer panel (note: this functionality may not be supported in this HTML document).
Robustness of Inference to Replacement (RIR):
TO INVALIDATE:
RIR = 4673
You entered an estimated effect of -0.3. To invalidate
the inference of an effect using the threshold of -0.02 for
statistical significance with alpha = 0.05, 93.465% of the
(-0.3) estimate would have to be due to bias. This implies
that to invalidate the inference one would expect to have to
replace 4673 (93.465%) observations with cases for which the
treatment effect is 0 (RIR = 4673).
See Frank et al. (2013) for a description of the method.
Citation: Frank, K.A., Maroulis, S., Duong, M., and Kelcey, B. (2013).
What would it take to change an inference?
Using Rubin's causal model to interpret the robustness of causal inferences.
Education, Evaluation and Policy Analysis, 35 437-460.
Accuracy of results increases with the number of decimals reported.
For other forms of output, run ?pkonfound and inspect the to_return argument
Robustness of Inference to Replacement (RIR):
TO INVALIDATE:
RIR = 4673
You entered an effect of -0.3, and specified a threshold
for inference of 0.2. To invalidate the inference based on your
estimate, 93.465% of the (-0.3) estimate would have to be due to
bias. This implies that to invalidate the inference one would
expect to have to replace 4673 (93.465%) observations with
cases for which the treatment effect is 0 (RIR = 4673).
See Frank et al. (2013) for a description of the method.
Citation: Frank, K.A., Maroulis, S., Duong, M., and Kelcey, B. (2013).
What would it take to change an inference?
Using Rubin's causal model to interpret the robustness of causal inferences.
Education, Evaluation and Policy Analysis, 35 437-460.
Accuracy of results increases with the number of decimals reported.
For other forms of output, run ?pkonfound and inspect the to_return argument
Robustness of Inference to Replacement (RIR):
TO SUSTAIN:
RIR = 4923
You entered an estimated effect of 0.003. The threshold value for
statistical significance is 0.196 (alpha = 0.05). To reach that
threshold, 98.47% of the (0.003) estimate would have to be due to
bias. This implies that to sustain an inference one would expect to
have to replace 4923 (98.47%) observations with effect of 0 with
cases with effect of 0.196 (RIR = 4923).
See Frank et al. (2013) for a description of the method.
Citation: Frank, K.A., Maroulis, S., Duong, M., and Kelcey, B. (2013).
What would it take to change an inference?
Using Rubin's causal model to interpret the robustness of causal inferences.
Education, Evaluation and Policy Analysis, 35 437-460.
Accuracy of results increases with the number of decimals reported.
For other forms of output, run ?pkonfound and inspect the to_return argument
Robustness of Inference to Replacement (RIR):
TO SUSTAIN:
RIR = 4923
You entered an effect size of 0.003, and specified a threshold
for inference of 0.001. To reach that threshold, 98.47% of the
(0.003) estimate would have to be due to bias. This implies
that to sustain an inference one would expect to have to replace
4923 (98.47%) observations with effect of 0 with cases
with effect of 0.196 (RIR = 4923).
See Frank et al. (2013) for a description of the method.
Citation: Frank, K.A., Maroulis, S., Duong, M., and Kelcey, B. (2013).
What would it take to change an inference?
Using Rubin's causal model to interpret the robustness of causal inferences.
Education, Evaluation and Policy Analysis, 35 437-460.
Accuracy of results increases with the number of decimals reported.
For other forms of output, run ?pkonfound and inspect the to_return argument
Impact Threshold for a Confounding Variable:
The maximum impact (in absolute value) of an omitted variable to
sustain an inference for a null hypothesis of 0 effect is based on
a correlation of -0.163 with the outcome an at 0.163 with the predictor
of interest (conditioning on all observed covariates in the model;
signs are interchangeable) based on a threshold of 0.196 for
statistical significance (alpha = 0.05).
Correspondingly the impact of an omitted variable (as defined in Frank 2000) must be
-0.163 X 0.163 = -0.027 to sustain an inference for a null hypothesis of 0 effect.
See Frank (2000) for a description of the method.
Citation:
Frank, K. (2000). Impact of a confounding variable on the inference of a
regression coefficient. Sociological Methods and Research, 29 (2), 147-194
Accuracy of results increases with the number of decimals reported.
For other forms of output, run ?pkonfound and inspect the to_return argument
Impact Threshold for a Confounding Variable:
The maximum impact of an omitted variable to sustain an inference
for a null hypothesis of 0 effect is based on a correlation of 0.163
with the outcome and at 0.163 with the predictor of interest (conditioning
on all observed covariates in the model) based on a threshold of -0.196
for statistical significance (alpha = 0.05).
Correspondingly the impact of an omitted variable (as defined in Frank 2000) must be
0.163 X 0.163 = 0.027 to sustain an inference for a null hypothesis of 0 effect.
See Frank (2000) for a description of the method.
Citation:
Frank, K. (2000). Impact of a confounding variable on the inference of a
regression coefficient. Sociological Methods and Research, 29 (2), 147-194
Accuracy of results increases with the number of decimals reported.
For other forms of output, run ?pkonfound and inspect the to_return argument
Impact Threshold for a Confounding Variable:
The minimum impact of an omitted variable to invalidate an inference
for a null hypothesis of 0 effect is based on a correlation of 0.123
with the outcome and at 0.123 with the predictor of interest (conditioning
on all observed covariates in the model) based on a threshold of 0.028
for statistical significance (alpha = 0.05).
Correspondingly the impact of an omitted variable (as defined in Frank 2000) must be
0.123 X 0.123 = 0.015 to invalidate an inference for a null hypothesis of 0 effect.
See Frank (2000) for a description of the method.
Citation:
Frank, K. (2000). Impact of a confounding variable on the inference of a
regression coefficient. Sociological Methods and Research, 29 (2), 147-194
Accuracy of results increases with the number of decimals reported.
For other forms of output, run ?pkonfound and inspect the to_return argument
Impact Threshold for a Confounding Variable:
The minimum (in absolute value) impact of an omitted variable to
invalidate an inference for a null hypothesis of 0 effect is based on
a correlation of -0.123 with the outcome and at 0.123 with the predictor
of interest (conditioning on all observed covariates in the model;
signs are interchangeable) based on a threshold of -0.196 for
statistical significance (alpha = 0.05).
Correspondingly the impact of an omitted variable (as defined in Frank 2000) must be
-0.123 X 0.123 = -0.015 to invalidate an inference for a null hypothesis of 0 effect.
See Frank (2000) for a description of the method.
Citation:
Frank, K. (2000). Impact of a confounding variable on the inference of a
regression coefficient. Sociological Methods and Research, 29 (2), 147-194
Accuracy of results increases with the number of decimals reported.
For other forms of output, run ?pkonfound and inspect the to_return argument
For models fit in R, consider use of konfound().
pkonfound(), non-linear
There are no further updates on the non-linear model, just for your reference.
RIR = 220
The table implied by the parameter estimates and sample sizes you entered:
Fail Success
Control 1157 1343
Treatment 1344 1156
The reported effect size = -0.300, SE = 0.010, p-value = 0.000.
The SE has been adjusted to 0.057 to generate real numbers in the
implied table for which the p-value would be 0.000. Numbers in
the table cells have been rounded to integers, which may slightly
alter the estimated effect from the value originally entered.
To invalidate the inference that the effect is different from 0
(alpha = 0.050) one would need to replace 220 (16.37%) treatment failure
cases with cases for which the probability of failure in the control
group (46.28%) applies (RIR = 220). This is equivalent to transferring
118 cases from treatment failure to treatment success (Fragility = 118).
Note that RIR = Fragility/[1-P(failure in the control group)]
The transfer of 118 cases yields the following table:
Fail Success
Control 1157 1343
Treatment 1226 1274
Effect size = -0.111, SE = 0.057, p-value = 0.051.
This is based on t = estimated effect/standard error
For other forms of output, run ?pkonfound and inspect the to_return argument
RIR = 207
The table implied by the parameter estimates and sample sizes you entered:
Fail Success
Control 2246 254
Treatment 2307 193
The reported effect size = -0.300, and SE = 0.100, p-value = 0.003.
Values have been rounded to the nearest integer. This may cause
a little change to the estimated effect for the table.
To invalidate the inference that the effect is different from 0
(alpha = 0.050) one would need to replace 207 (8.97%) treatment failure
cases with cases for which the probability of failure in the control
group (89.84%) applies (RIR = 207). This is equivalent to transferring
21 cases from treatment failure to treatment success (Fragility = 21).
Note that RIR = Fragility/[1-P(failure in the control group)]
The transfer of 21 cases yields the following table:
Fail Success
Control 2246 254
Treatment 2286 214
Effect size = -0.189, SE = 0.097, p-value = 0.052.
This is based on t = estimated effect/standard error
For other forms of output, run ?pkonfound and inspect the to_return argument
RIR = 19
The table implied by the parameter estimates and sample sizes you entered:
Fail Success
Control 13 12
Treatment 12 13
The reported effect size = 0.050, SE = 0.300, p-value = 0.779.
The SE has been adjusted to 0.566 to generate real numbers in the
implied table for which the p-value would be 0.779. Numbers in
the table cells have been rounded to integers, which may slightly
alter the estimated effect from the value originally entered.
To reach the threshold that would sustain an inference that the
effect is different from 0 (alpha = 0.010) one would need to replace 19
(158.33%) treatment failure cases with cases for which the probability of
failure in the control group (52.00%) applies (RIR = 19). This is equivalent
to transferring 9 cases from treatment failure to treatment success
(Fragility = 9).
Note that RIR = Fragility/[1-P(failure in the control group)]
Note the RIR exceeds 100%. Generating the transfer of 9 cases would
require replacing more cases than are in the treatment failure condition.
The transfer of 9 cases yields the following table:
Fail Success
Control 13 12
Treatment 3 22
Effect size = 2.072, SE = 0.734, p-value = 0.007.
This is based on t = estimated effect/standard error
For other forms of output, run ?pkonfound and inspect the to_return argument
RIR = 24
The table implied by the parameter estimates and sample sizes you entered:
Fail Success
Control 180 70
Treatment 182 68
The reported effect size = -0.030, and SE = 0.200, p-value = 0.841.
Values have been rounded to the nearest integer. This may cause
a little change to the estimated effect for the table.
To reach the threshold that would sustain an inference that the
effect is different from 0 (alpha = 0.050) one would need to replace 24
(35.29%) treatment success cases with cases for which the probability of
failure in the control group (72.00%) applies (RIR = 24). This is equivalent
to transferring 17 cases from treatment success to treatment failure
(Fragility = 17).
Note that RIR = Fragility/[1-P(success in the control group)]
The transfer of 17 cases yields the following table:
Fail Success
Control 180 70
Treatment 199 51
Effect size = -0.417, SE = 0.211, p-value = 0.049.
This is based on t = estimated effect/standard error
For other forms of output, run ?pkonfound and inspect the to_return argument
RIR = 52
The table implied by the parameter estimates and sample sizes you entered:
Fail Success
Control 27 23
Treatment 10 10
The reported effect size = 0.250, SE = 0.500, p-value = 0.763.
The SE has been adjusted to 0.530 to generate real numbers in the
implied table for which the p-value would be 0.763. Numbers in
the table cells have been rounded to integers, which may slightly
alter the estimated effect from the value originally entered.
The inference cannot be sustained merely by switching cases in
only one treatment condition. Therefore, cases have been switched from
treatment failure to treatment success and from control success to control failure.
The final Fragility(= 25) and RIR(= 52) reflect both sets of changes.
Please compare the after transfer table with the implied table.
Fail Success
Control 35 15
Treatment 1 19
Effect size = 3.792, SE = 1.071, p-value = 0.001.
This is based on t = estimated effect/standard error
For other forms of output, run ?pkonfound and inspect the to_return argument