Start Using the neg.twocors
Function
From the negligible
R package
Test for Evaluating Negligible Effects of Two Independent or Dependent Correlation Coefficients: Based on Counsell & Cribbie (2015)
Introduction
Goal
This function evaluates whether the difference between two correlation coefficients can be considered statistically and practically negligible according to a predefined interval. i.e., minimally meaningful effect size (MMES)/smallest effect size of interest (SESOI). The effect size tested is the difference between two correlation coefficients (i.e., r1 - r2).
Background
Unlike the most common null hypothesis significance tests looking to detect a difference or the existence of an effect statistically different than zero, in negligible effect testing, the hypotheses are flipped: In essence, \(H_0\) states that the effect is non-negligible, whereas \(H_1\) states that the effect is in fact statistically and practically negligible.
The statistical tests are based on Anderson-Hauck (1983) and
Schuirmann’s (1987) Two One-Sided Test (TOST) equivalence testing
procedures; namely addressing the question of whether the estimated
effect size (and its associated uncertainty) of a difference between two
correlation coefficients (i.e., \(r_1\)
and \(r_2\)) is smaller than the what
the user defines as negligible effect size. Defining what is considered
negligible effect is done by specifying the negligible (equivalence)
interval: its upper (eiu) and lower (eil)
bounds.
The negligible (equivalence) interval should be set based on the
context of the research. Because the two correlations (and, therefore
their difference) are in standardized units, setting eil
and eiu is a matter of determining what is the smallest
difference between the two correlations that can be considered of
practical significance. For example, if the user determines that the
smallest effect of interest is 0.1 – that is, any difference between the
two correlation coefficient larger than 0.1 is meaningful in this
context - then eil will be set to eil = -0.1
and eiu = 0.1. Therefore, any observable difference that is
larger than -0.1 and smaller than 0.1, will be considered
practically negligible.
Instructions
There are two main approaches to using neg.twocors:
- The first (and more recommended) is by inserting a dataset (with the
data =argument) into the function. If the user/s have access to the dataset, they should use the following set of arguments:data =a data.frame or matrix which includes the variables in \(r_1\) and \(r_2\)r1v1 =the name of the 1st variable included in the 1st correlation coefficient (r1, variable 1)r1v2 =the name of the 2nd variable included in the 1st correlation coefficient (r1, variable 2)r2v1 =the name of the 1st variable included in the 2nd correlation coefficient (r2, variable 1)r2v2 =the name of the 2nd variable included in the 2st correlation coefficient (r2, variable 2)dep =(if applicable) are the correlation coefficients dependent (overlapping)?bootstrap =(optional) logical, default isTRUE, incorporating bootstrapping when calculating regression coefficients, SE, and CIsnboot =(optional) 1000 is the default. indicate if other number of bootstrapping iterations is desiredseed =(optional) to reproduce previous analyses using bootstrapping, the user can set their seed of choice
- This function also accommodates cases where no dataset is available.
In this case, users should use the following set of arguments instead:
r1 =entered 1st correlation coefficient manually, without a datasetn1 =entered sample size associated with r1 manually, without a datasetr2 =entered 2nd correlation coefficient manually, without a datasetn2 =entered sample size associated with r2 manually, without a datasetr3 =(if applicable). if the correlation coefficients are dependent and no datasets were entered, specify the correlation between the two, non-intersecting variables (e.g. if \(r_1\) = \(r_{12}\) and \(r_2\) = \(r_{13}\), then \(r_3\) = \(r_{23}\))
In either situation, users must specify the negligible interval
bounds (eiu = and eil =).
Other optional arguments and features include:
alpha =desired alpha level, defualt is .05test =AHis the default based on recommendation in Counsell & Cribbie (2015),TOSTis an additional (albeit, more conservative) option.plots =logical, plotting the results.TRUEis set as defaultsaveplots =FALSEfor no,pngandjpegfor different formats
Independent vs. Dependent Correlation Coefficients
This function accommodates both independent and dependent
correlations. A user might want to compare two independent correlations.
For example, the correlation between \(X\) and \(Y\) in one group (e.g., Control group;
\(r_{XYC}\)) with the correlation
between X and Y in a different, independent group (e.g., Treatment
group; \(r_{XYT}\)). The ‘independent
correlations’ setting (i.e., dep = FALSE) is the default in
this function. However, in other cases, a user might want to compare two
dependent correlation coefficients. That is, the two correlations share
a common variable (i.e., same variable values). For example, the
correlation between \(X\) and \(Y\) in one group (e.g., Treatment group;
\(r_{XYT}\)) with the correlation
between \(X\) and \(B\) in the same group (e.g., Treatment
group; \(r_{XBT}\)). Because values in
variable \(X\) are shared among the two
correlations, the two correlations (e.g., \(r_{XYT}\) and \(r_{XBT}\)) are not independent from one
another, but, in fact, dependent.
To compare two dependent correlation coefficients, users need only to
specify dep = TRUE. **If no dataset is entered into the
function, users should also use the argument r3 =, which
will hold the correlation between the two non-shared variables. In
the example above (i.e., \(r_{XYT}\)
and \(r_{XBT}\)), the two non-shared
variables are \(Y\) and \(B\). In this case, \(r_3 = r_{YBT}\). If dep = TRUE
is entered into the function, test statistics and p*-values will be
calculated differently to account for the shared variable. The
negligible testing methods for comparing dependent correlations in this
function are based on Williams’s (1959) modification to Hotelling’s
(1931) test for comparing overlapping dependent correlations. For more
details see Counsell and Cribbie (2015).
Applied Examples
Example 1A: Data available, Independent Coefficients
Say you want to assess whether the difference between two independent
correlation coefficients is negligible. The first coefficient depicts
the correlation between Automatic Thoughts (labeled
atqpost.total, the first variable in the first correlation;
r1v1) and Oriented Perfectionism (labeled
mpshfpost.oop, the second variable in the first
correlation; r1v2). Their correlation is \(r_1 = 0.224\). The second coefficient
depicts the correlation between Concern Over Mistakes (labeled
mpsfpost.cm, the first variable in the second correlation;
r1v2) and Depression (labeled cesdpost.total,
the second variable in the second correlation; r2v2). Their
correlation is \(r_2 = 0.517\).
Let’s consider the equivalence interval to be [-.15, .15]. Thus, we
will set eil = -.15 and eiu = .15.
Finally, because the two coefficients are independent, we will set
dep = FALSE.
Putting It All Together (when data is available; independent coefficients)
library(negligible) # load the package
neg.twocors(data=perfectionism,
r1v1=atqpost.total, # Automatic Thoughts Questionnaire
r1v2=mpshfpost.oop, # Oriented Perfectionism
r2v1=mpsfpost.cm, # Concern Over Mistakes
r2v2=cesdpost.total, # CESD Depression Scale
eiu=.15, # upper bound of SESOI (standardized)
eil=-.15, # lower bound of SESOI (standardized)
seed = 123,
dep=FALSE) # correlation coefficients are NOT dependent
Test for Evaluating Negligible Difference Between Two Correlation Coefficients
*** Comparison of Independent Correlation Coefficients ***
Correlation coefficients:
Variables: atqpost.total & mpshfpost.oop, r1 = 0.224
Variables: mpsfpost.cm & cesdpost.total, r2 = 0.517
**********************
Correlation coefficients' difference and confidence interval using 1000 bootstrap iterations (seed=123):
r1-r2 = -0.293, 95% CI: [-0.54, -0.055]
std. error = 0.133
**********************
AH-ρ: Counsell-Cribbie Test for Comparing Two Independent Correlation Coefficients
Equivalence Interval: Lower = -0.15, Upper = 0.15
p value = 0.858
NHST Decision: The null hypothesis that the difference between the two correlation coefficients is non-negligible (i.e., beyond the equivalence interval), was NOT rejected: There is insufficient evidence that the difference between the two correlation coefficients is negligible in the population. Be sure to interpret the magnitude (and precision) of the effect size.
*Note that NHST decisions using the AH-ρ and AH-ρ procedures may not match KTOST-ρ and TOST-ρ-D results or the Symmetric CI Approach at 100*(1-2α)% illustrated in the plot.
**********************
Proportional Distance
Proportional distance: -1.955
95% confidence interval for the proportional distance: (-3.603, -0.369)
*Note that the confidence interval for the proportional distance may not be precise with small sample sizes
******************* In Example 1A above, we used the neg.twocors function
with access to the raw data.
The results were NOT statistically significant, \(r_1 - r_2 = -0.293, \ 95\% \ CI \ [-0.528, -0.054], \ p = 0.858\) indicating that the null hypothesis that the difference between the two correlation coefficients is non-negligible (i.e., beyond the equivalence interval), was NOT rejected. Thus, there is insufficient evidence that the difference between the two correlation coefficients is negligible in the population.
Example 1B: Data Available, Dependent Coefficients
Now, say you want to evaluate the difference between two correlation
coefficients, similar to Example 1A. However, unlike in the previous
example, here, the two correlations are dependent because the data
yielding both coefficients use the same varible: the first coefficient
depicts the correlation between Anxiety (labeled
baipre.total) at pretest and Depression at pretest (labeled
cesdpre.total). The second coefficient depicts the
association of the same variable of Anxiety at pretest with another
variable, Automatic Thoughts at pretest (atqpre.total),
Let’s consider the same equivalence interval: eil = -1.5
and eiu = 1.5.
Importantly, because the two coefficients are dependent, we will set
dep = TRUE.
Putting It All Together (when data is available; dependent variables)
library(negligible) # load the package
neg.twocors(data = perfectionism,
r1v1 = baipre.total, # Anxiety
r1v2 = cesdpre.total, # Depression
r2v1 = baipre.total, # Same anxiety
r2v2 = atqpre.total, # Automatic thoughts
eiu = .15, # upper bound of SESOI (standardized)
eil= -.15, # lower bound of SESOI (standardized)
seed = 123,
dep=T) # correlation coefficients are NOT dependent
Test for Evaluating Negligible Difference Between Two Correlation Coefficients
*** Comparison of Dependent Correlation Coefficients ***
Correlation coefficients:
Variables: baipre.total & cesdpre.total, r1 = 0.702
Variables: baipre.total & atqpre.total, r2 = 0.661
r3 = 0.8183588
**********************
Correlation coefficients' difference and confidence interval using 1000 bootstrap iterations (seed=123):
r1-r2 = 0.042, 95% CI: [-0.12, 0.206]
std. error = 0.084
**********************
AH-ρ-D: Counsell-Cribbie Test for Comparing Two Dependent Correlation Coefficients
Equivalence Interval: Lower = -0.15, Upper = 0.15
p value < 0.001
NHST Decision: The null hypothesis that the difference between the two correlation coefficients is non-negligible (i.e., beyond the specified equivalence interval), can be rejected. A negligible difference between the two correlation coefficients in the population can be concluded. Be sure to interpret the magnitude (and precision) of the effect size.
*Note that NHST decisions using the AH-ρ and AH-ρ procedures may not match KTOST-ρ and TOST-ρ-D results or the Symmetric CI Approach at 100*(1-2α)% illustrated in the plot.
**********************
Proportional Distance
Proportional distance: 0.277
95% confidence interval for the proportional distance: (-0.799, 1.375)
*Note that the confidence interval for the proportional distance may not be precise with small sample sizes
******************* Here, the difference between the two correlation coefficients is statistically and practically negligible, \(r_1 - r_2 = 0.042, \ 95 \% \ CI \ [-0.12, 0.206], \ p < 0.001\) suggesting that the null hypothesis that the difference between the two correlation coefficients is non-negligible (i.e., beyond the specified equivalence interval), can be rejected. A negligible difference between the two correlation coefficients in the population can be concluded!
Example 2: No Dataset Available
Suppose you do not have access to the full data. Still, you might look at the reported results in a research article and gather the required details: Say you observe that \(r_1 = 0.22\) with a sample size of \(n=825\), \(r_2 = 0.18\) with a sample size of \(n=789\).
Let’s consider the same equivalence interval: eil = -.15
and eiu = .15.
Putting It All Together (when data is NOT available)
neg.twocors(r1=0.22, # first coefficient
n1=825, # sample size for 1st coefficient
r2=0.18, # second coefficient
n2=789, # sample size for 2nd coefficient
eiu=.15,
eil=-0.15,
dep=FALSE)
Test for Evaluating Negligible Difference Between Two Correlation Coefficients
*** Comparison of Independent Correlation Coefficients ***
Correlation coefficients:
r1 = 0.22
r2 = 0.18
**********************
Correlation coefficients' raw difference:
r1-r2 =0.04, 95% CI [-0.039, 0.119]
std. error = 0.048
**********************
AH-ρ: Counsell-Cribbie Test for Comparing Two Independent Correlation Coefficients
Equivalence Interval: Lower = -0.15, Upper = 0.15
p value = 0.011
NHST Decision: The null hypothesis that the difference between the two correlation coefficients is non-negligible (i.e., beyond the specified equivalence interval), can be rejected. A negligible difference between the two correlation coefficients in the population can be concluded. Be sure to interpret the magnitude (and precision) of the effect size.
*Note that NHST decisions using the AH-ρ and AH-ρ procedures may not match KTOST-ρ and TOST-ρ-D results or the Symmetric CI Approach at 100*(1-2α)% illustrated in the plot.
**********************
Proportional Distance
Proportional distance: 0.267
95% confidence interval for the proportional distance: (-0.258, 0.791)
*Note that the confidence interval for the proportional distance may not be precise with small sample sizes
******************* Results from the negligible effect testing indicate that the difference between the two correlation coefficients is indeed negligible, \(r_1 - r_2 = 0.04, \ 95 \% \ CI \ [-0.039, 0.119], \ p = 0.011\) suggesting that the null hypothesis that the difference between the two correlation coefficients is non-negligible (i.e., beyond the specified equivalence interval), can be rejected. A negligible difference between the two correlation coefficients in the population can be concluded!