Cohen's Kappa

Cohen's kappa measures the agreement between two raters who each classify N items into C mutually exclusive categories. The definition of \({\textstyle \kappa }\) is

\[{\displaystyle \kappa \equiv {\frac {p_{o}-p_{e}}{1-p_{e}}}=1-{\frac {1-p_{o}}{1-p_{e}}},}\] where po is the relative observed agreement among raters, and pe is the hypothetical probability of chance agreement, using the observed data to calculate the probabilities of each observer randomly seeing each category. If the raters are in complete agreement then \({\textstyle \kappa =1}\).

vcd::kappa()

Kappa Cohen's Kappa and Weighted Kappa

Desription

Computes two agreement rates: Cohen's kappa and weighted kappa, and confidence bands.

Usage



Kappa(x, weights = c("Equal-Spacing", "Fleiss-Cohen"))
## S3 method for class 'Kappa'
print(x, digits=max(getOption("digits") - 3, 3), CI=FALSE, level=0.95, ...)
## S3 method for class 'Kappa'
confint(object, parm, level = 0.95, ...)
## S3 method for class 'Kappa'
summary(object, ...)
## S3 method for class 'summary.Kappa'
print(x, ...)

Arguments

  • x For Kappa: a confusion matrix. For the print methods: object of class "Kappa" or "summary.Kappa"
  • weights either one of the character strings given in the default value, or a user-specified matrix with same dimensions as x.
  • digits minimal number of significant digits.
  • CI logical; shall confidence limits be added to the output?
  • level : confidence level between 0 and 1 used for the confidence interval.
  • object: object of class "Kappa".
  • parm: Currently, ignored.
  • ...: Further arguments passed to the default print method.

Examples

data("SexualFun")
K <- Kappa(SexualFun)
K
##             value     ASE     z Pr(>|z|)
## Unweighted 0.1293 0.06860 1.885 0.059387
## Weighted   0.2374 0.07832 3.031 0.002437
confint(K)
##             
## Kappa                 lwr       upr
##   Unweighted -0.005120399 0.2637809
##   Weighted    0.083883432 0.3908778
summary(K)
##             value     ASE     z Pr(>|z|)
## Unweighted 0.1293 0.06860 1.885 0.059387
## Weighted   0.2374 0.07832 3.031 0.002437
## 
## Weights:
##           [,1]      [,2]      [,3]      [,4]
## [1,] 1.0000000 0.6666667 0.3333333 0.0000000
## [2,] 0.6666667 1.0000000 0.6666667 0.3333333
## [3,] 0.3333333 0.6666667 1.0000000 0.6666667
## [4,] 0.0000000 0.3333333 0.6666667 1.0000000
print(K, CI = TRUE)
##             value     ASE     z Pr(>|z|)    lower  upper
## Unweighted 0.1293 0.06860 1.885 0.059387 -0.00512 0.2638
## Weighted   0.2374 0.07832 3.031 0.002437  0.08388 0.3909

Details

  • Cohen's kappa is the diagonal sum of the (possibly weighted) relative frequencies, corrected for expected Value and standardized by its maximum value.
  • The equal-spacing weights are defined by \(1-|i-j|/ (r-1)\), r number of columns/rows, and the Fleiss-Cohen weights by \(1-|i-j|^2/ (r-1)^2\). The latter one attaches greater importance to near disagreements.

Value

An object of class "Kappa" with three components:

  • Unweighted numeric vector of length 2 with the kappa statistic (value component), along with Approximate Standard Error (ASE component)
  • Weighted idem for the weighted kappa.
  • Weights numeric matrix with weights used.

Note

The summary method also prints the weights. There is a confint method for computing approximate confidence intervals.

Author(s)

David Meyer David.Meyer@R-project.org

References

  • Cohen, J. (1960), A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46.
  • Everitt, B.S. (1968), Moments of statistics kappa and weighted kappa. The British Journal of Mathematical and Statistical Psychology, 21, 97-103.
  • Fleiss, J.L., Cohen, J., and Everitt, B.S. (1969), Large sample standard errors of kappa and weighted kappa. Psychological Bulletin, 72, 332-327.