1. Traditionally, we may use Cohen's kappa or Fleiss's kappa to measure interobserver agreemnet.

2. But I use a generalized estimating equations approach instead.

3. A generalized estimating equation (GEE) is used to estimate the parameters of a generalized linear model with a possible unknown correlation between outcomes.

1. Case records of the eye-testing

2. 7477 female employees in Royal Ordnance factories

3. 2 variables : r.eye, l.eye

4. 4 levels : 1st grade, 2nd grade, 3rd grade, 4th Grade

library(pacman)
p_load(tidyverse,irr,slidify,aspect, MASS, gee, 
       repolr, geepack)

data(vision)
dta <- vision
# add ID variable
dta$ID <- factor(paste0("S", 1001:8477))
head(dta)

##       r.eye     l.eye    ID
## 1 1st grade 1st grade S1001
## 2 1st grade 1st grade S1002
## 3 1st grade 1st grade S1003
## 4 1st grade 1st grade S1004
## 5 1st grade 1st grade S1005
## 6 1st grade 1st grade S1006

with(dta,table(r.eye,l.eye))

##            l.eye
## r.eye       1st grade 2nd grade 3rd grade 4th Grade
##   1st grade      1520       266       124        66
##   2nd grade       234      1512       432        78
##   3rd grade       117       362      1772       205
##   4th Grade        36        82       179       492

# data table
show(prop.table(with(dtaL, ftable(Type, Rank)), m=1))

##       Rank 1st grade 2nd grade 3rd grade 4th Grade
## Type                                              
## l.eye      0.2550488 0.2971780 0.3352949 0.1124783
## r.eye      0.2642771 0.3017253 0.3284740 0.1055236

1. \[logit(P\{Y_t <= j\}) = \alpha_k + \beta_l*Eye\]

2. \[j = 1,2,3,4; k = j-1; t = 1,2; l = t-1\]

## 
## repolr: 2016-02-26 version 3.4 
## 
## Call:
## repolr(formula = Rank ~ Type, subjects = "ID", data = dtaL, times = c(1, 
##     2), categories = 4, corr.mod = "uniform")
## 
## Coefficients: 
##            coeff     se.robust  z.robust  p.value 
## cuts1|2     -1.0757    0.0257   -41.8560    0.0000
## cuts2|3      0.2098    0.0226     9.2832    0.0000
## cuts3|4      2.0734    0.0336    61.7083    0.0000
## Typer.eye    0.0553    0.0160     3.4563    0.0005
## 
## Correlation Structure:  uniform 
## Estimated Correlation:  0.5298

## 
## Re-fitting to get Hessian

## Call:
## polr(formula = ordered(Rank) ~ Type, data = dtaL)
## 
## Coefficients:
##              Value Std. Error t value
## Typer.eye -0.05532    0.02957  -1.871
## 
## Intercepts:
##     Value    Std. Error t value 
## 1|2  -1.0756   0.0239   -45.0150
## 2|3   0.2099   0.0221     9.4858
## 3|4   2.0736   0.0300    69.0808
## 
## Residual Deviance: 39440.64 
## AIC: 39448.64

kappa2(dta[,1:2],weight = "unweighted")

##  Cohen's Kappa for 2 Raters (Weights: unweighted)
## 
##  Subjects = 7477 
##    Raters = 2 
##     Kappa = 0.595 
## 
##         z = 84.6 
##   p-value = 0

1. GEE approach let us know the degree of criterion changing between two or more raters, but kappa doesn's

2. We don't assumes the distribution of the observation