Carregamento de biblioteca
library(RTextTools)
Dados seguindo a mesma distribuição
data(USCongress)
Alguns pontos de dados
head(USCongress)
## ID cong billnum h_or_sen major
## 1 1 107 4499 HR 18
## 2 2 107 4500 HR 18
## 3 3 107 4501 HR 18
## 4 4 107 4502 HR 18
## 5 5 107 4503 HR 5
## 6 6 107 4504 HR 21
## text
## 1 To suspend temporarily the duty on Fast Magenta 2 Stage.
## 2 To suspend temporarily the duty on Fast Black 286 Stage.
## 3 To suspend temporarily the duty on mixtures of Fluazinam.
## 4 To reduce temporarily the duty on Prodiamine Technical.
## 5 To amend the Immigration and Nationality Act in regard to Caribbean-born immigrants.
## 6 To amend title 38, United States Code, to extend the eligibility for housing loans guaranteed by the Secretary of Veterans Affairs under the Native American Housing Loan Pilot Program to veterans who are married to Native Americans.
Notar que a remoção de termos esparsos está em .998, e portanto a simplificação da matrix é alta na remoção de termos esparsos, mas isso prejudica o desempenho do classificador (na pratica usaremos um valor menor). Além disso, o idioma está definido para ingles, mas o idioma em portugues está disponível. Também é neste momento que o pré-processamento dos dados é feito (remoção de números, stem, entre outros estão também disponíveis neste pacote).
doc_matrix <- create_matrix(USCongress$text, language = "english", removeNumbers = TRUE,
stemWords = TRUE, removeSparseTerms = 0.998)
O container será o input final para os algoritmos de aprendizagem. Ele utiliza a matrix de term-document e separa os dados em treino e teste (i.e. modelo HOLDOUT de generalização), bem como os labels associados a cada documento. Observar que o conjunto de treino foi definido como os 4k primeiros documentos e os remanescentes como dados de teste. Ao se definir virgin=FALSE estamos avaliando a generalizacão do modelo, enquanto em virgin=TRUE estamos querendo de fato a classificação de novos labels que não possuimos os valores ground truth.
container <- create_container(doc_matrix, USCongress$major, trainSize = 1:4000,
testSize = 4001:4449, virgin = FALSE)
Existem trade-off na performance dos modelos de acordo com a distribuição dos dados e limite de processamento desejado, isso será dependente dos dados que queremos avaliar o treinamento. Aqui estao listado apenas os de baixo custo de memoria (os outros demoraram demais em meu mac).
SVM <- train_model(container, "SVM")
MAXENT <- train_model(container, "MAXENT")
SVM_CLASSIFY <- classify_model(container, SVM)
MAXENT_CLASSIFY <- classify_model(container, MAXENT)
A biblioteca utiliza varias funções de performace. Eu particularmente prefiro apesar de mais trabalhosas as matrizes de confusão, que a biblioteca não implementa. De toda forma, fica a gosto do freguês!
analytics <- create_analytics(container, cbind(SVM_CLASSIFY, MAXENT_CLASSIFY))
summary(analytics)
## ENSEMBLE SUMMARY
##
## n-ENSEMBLE COVERAGE n-ENSEMBLE RECALL
## n >= 1 1.00 0.75
## n >= 2 0.77 0.87
##
##
## ALGORITHM PERFORMANCE
##
## SVM_PRECISION SVM_RECALL SVM_FSCORE
## 0.6685 0.6530 0.6505
## MAXENTROPY_PRECISION MAXENTROPY_RECALL MAXENTROPY_FSCORE
## 0.6145 0.6325 0.6190
# CREATE THE data.frame SUMMARIES
topic_summary <- analytics@label_summary
alg_summary <- analytics@algorithm_summary
ens_summary <- analytics@ensemble_summary
doc_summary <- analytics@document_summary
Sumario de topicos
topic_summary
## NUM_MANUALLY_CODED NUM_CONSENSUS_CODED NUM_PROBABILITY_CODED
## 1 9 8 8
## 2 5 4 4
## 3 66 62 62
## 4 24 26 26
## 5 33 36 36
## 6 27 22 22
## 7 24 26 26
## 8 4 6 6
## 10 23 22 22
## 12 28 35 35
## 13 5 6 6
## 14 8 9 9
## 15 24 23 23
## 16 16 19 19
## 17 14 14 14
## 18 19 16 16
## 19 7 10 10
## 20 52 50 50
## 21 60 55 55
## 99 1 0 0
## PCT_CONSENSUS_CODED PCT_PROBABILITY_CODED PCT_CORRECTLY_CODED_CONSENSUS
## 1 88.89 88.89 55.56
## 2 80.00 80.00 0.00
## 3 93.94 93.94 83.33
## 4 108.33 108.33 87.50
## 5 109.09 109.09 81.82
## 6 81.48 81.48 70.37
## 7 108.33 108.33 66.67
## 8 150.00 150.00 50.00
## 10 95.65 95.65 73.91
## 12 125.00 125.00 64.29
## 13 120.00 120.00 60.00
## 14 112.50 112.50 75.00
## 15 95.83 95.83 62.50
## 16 118.75 118.75 68.75
## 17 100.00 100.00 64.29
## 18 84.21 84.21 78.95
## 19 142.86 142.86 57.14
## 20 96.15 96.15 82.69
## 21 91.67 91.67 81.67
## 99 0.00 0.00 0.00
## PCT_CORRECTLY_CODED_PROBABILITY
## 1 55.56
## 2 0.00
## 3 83.33
## 4 87.50
## 5 81.82
## 6 70.37
## 7 66.67
## 8 50.00
## 10 73.91
## 12 64.29
## 13 60.00
## 14 75.00
## 15 62.50
## 16 68.75
## 17 64.29
## 18 78.95
## 19 57.14
## 20 82.69
## 21 81.67
## 99 0.00
Sumario de algoritmos
alg_summary
## SVM_PRECISION SVM_RECALL SVM_FSCORE MAXENTROPY_PRECISION
## 1 0.38 0.67 0.48 0.62
## 2 0.67 0.40 0.50 0.00
## 3 0.89 0.86 0.87 0.89
## 4 0.84 0.88 0.86 0.81
## 5 0.73 0.73 0.73 0.75
## 6 0.88 0.85 0.86 0.86
## 7 0.72 0.75 0.73 0.62
## 8 0.67 0.50 0.57 0.33
## 10 0.77 0.74 0.75 0.77
## 12 0.56 0.71 0.63 0.51
## 13 0.60 0.60 0.60 0.50
## 14 0.67 0.50 0.57 0.67
## 15 0.74 0.71 0.72 0.65
## 16 0.53 0.56 0.54 0.58
## 17 0.73 0.57 0.64 0.64
## 18 0.88 0.74 0.80 0.94
## 19 0.45 0.71 0.55 0.40
## 20 0.84 0.73 0.78 0.86
## 21 0.82 0.85 0.83 0.89
## 99 NaN 0.00 NaN NaN
## MAXENTROPY_RECALL MAXENTROPY_FSCORE
## 1 0.56 0.59
## 2 0.00 NaN
## 3 0.83 0.86
## 4 0.88 0.84
## 5 0.82 0.78
## 6 0.70 0.77
## 7 0.67 0.64
## 8 0.50 0.40
## 10 0.74 0.75
## 12 0.64 0.57
## 13 0.60 0.55
## 14 0.75 0.71
## 15 0.62 0.63
## 16 0.69 0.63
## 17 0.64 0.64
## 18 0.79 0.86
## 19 0.57 0.47
## 20 0.83 0.84
## 21 0.82 0.85
## 99 0.00 NaN
Sumario de ensembles
ens_summary
## n-ENSEMBLE COVERAGE n-ENSEMBLE RECALL
## n >= 1 1.00 0.75
## n >= 2 0.77 0.87
Sumario de documentos
doc_summary
## SVM_LABEL SVM_PROB MAXENTROPY_LABEL MAXENTROPY_PROB MANUAL_CODE
## 1 12 0.7164 12 1.0000 12
## 2 5 0.7672 5 1.0000 5
## 3 15 0.7716 15 1.0000 15
## 4 10 0.9037 10 1.0000 10
## 5 3 0.9991 3 1.0000 3
## 6 18 0.6284 18 1.0000 18
## 7 21 0.9992 21 1.0000 21
## 8 16 0.3400 12 1.0000 12
## 9 20 0.9899 20 1.0000 20
## 10 3 0.6280 3 1.0000 3
## 11 5 0.4966 5 1.0000 5
## 12 21 0.9930 21 1.0000 21
## 13 21 0.7480 10 0.9991 21
## 14 21 0.7346 21 1.0000 21
## 15 12 0.8758 12 1.0000 12
## 16 3 0.9562 3 1.0000 3
## 17 16 0.6113 16 0.6964 3
## 18 21 0.4063 7 0.6985 7
## 19 17 0.2797 6 0.7673 3
## 20 16 0.5937 16 1.0000 16
## 21 3 0.9999 3 1.0000 3
## 22 15 0.9659 15 1.0000 15
## 23 20 0.7574 20 1.0000 20
## 24 20 0.8300 20 1.0000 20
## 25 21 0.6625 21 1.0000 21
## 26 3 0.9978 3 1.0000 3
## 27 5 0.4292 5 0.8623 5
## 28 3 0.8961 3 1.0000 4
## 29 15 0.6374 15 1.0000 15
## 30 5 0.1630 3 1.0000 1
## 31 4 0.9209 4 1.0000 4
## 32 12 0.3184 15 1.0000 12
## 33 4 0.9985 4 1.0000 4
## 34 12 0.5217 12 1.0000 20
## 35 20 0.7767 20 1.0000 20
## 36 4 0.3241 15 0.5041 4
## 37 4 0.9893 4 1.0000 4
## 38 1 0.6743 1 1.0000 1
## 39 20 0.8333 20 0.9482 20
## 40 20 0.6637 20 1.0000 20
## 41 20 0.7005 20 1.0000 20
## 42 21 0.8940 21 1.0000 21
## 43 4 0.8837 4 1.0000 12
## 44 13 0.8924 13 1.0000 16
## 45 21 0.6559 21 0.9999 21
## 46 6 0.9300 6 1.0000 6
## 47 20 0.6623 20 1.0000 20
## 48 19 0.4707 19 1.0000 18
## 49 7 0.9232 7 1.0000 20
## 50 7 0.2611 7 0.9995 8
## 51 3 0.9674 3 1.0000 3
## 52 1 0.3145 15 1.0000 16
## 53 21 0.4050 20 1.0000 20
## 54 1 0.3963 16 1.0000 16
## 55 21 0.3362 21 1.0000 21
## 56 10 0.8465 10 1.0000 10
## 57 21 0.8491 21 1.0000 21
## 58 16 0.2575 20 0.6219 16
## 59 21 0.7509 21 1.0000 21
## 60 1 0.1941 1 0.8838 1
## 61 7 0.9982 7 1.0000 7
## 62 3 1.0000 3 1.0000 3
## 63 12 0.6572 12 1.0000 12
## 64 3 0.9909 3 1.0000 3
## 65 18 0.9551 18 1.0000 18
## 66 20 0.8493 20 1.0000 20
## 67 15 0.4411 5 1.0000 5
## 68 21 0.8752 21 1.0000 21
## 69 4 0.9794 4 1.0000 4
## 70 4 0.9383 4 1.0000 4
## 71 1 0.5730 16 0.9999 16
## 72 4 0.9496 4 1.0000 4
## 73 21 0.9189 21 1.0000 21
## 74 10 0.5489 12 1.0000 10
## 75 16 0.8068 16 1.0000 16
## 76 16 0.7246 16 1.0000 16
## 77 10 0.6057 10 0.9998 10
## 78 20 0.9188 20 1.0000 20
## 79 3 0.1238 5 0.9996 17
## 80 21 0.9844 21 1.0000 21
## 81 5 0.2424 5 1.0000 20
## 82 7 0.8447 7 1.0000 7
## 83 4 0.9616 4 1.0000 4
## 84 7 0.8147 7 1.0000 7
## 85 1 0.4612 16 1.0000 16
## 86 17 0.5383 17 1.0000 17
## 87 3 0.9090 3 1.0000 3
## 88 3 0.9991 3 1.0000 3
## 89 15 0.1765 5 0.9974 15
## 90 3 0.8658 3 1.0000 3
## 91 15 0.4582 15 1.0000 15
## 92 12 0.1660 12 0.9830 21
## 93 15 0.5855 15 0.6691 15
## 94 1 0.6305 20 1.0000 20
## 95 12 0.7024 12 1.0000 12
## 96 15 0.3136 1 1.0000 1
## 97 20 0.8116 20 1.0000 20
## 98 20 0.3698 20 1.0000 20
## 99 4 0.9794 4 1.0000 4
## 100 5 0.8192 5 1.0000 5
## 101 18 0.1367 7 0.9986 20
## 102 5 0.5340 20 0.9993 20
## 103 4 0.9901 4 1.0000 4
## 104 3 0.9441 3 1.0000 3
## 105 18 0.9288 18 1.0000 18
## 106 14 0.8074 14 0.9954 14
## 107 21 0.7407 21 1.0000 21
## 108 2 0.5054 17 1.0000 2
## 109 20 0.9409 20 1.0000 20
## 110 15 0.2922 14 1.0000 14
## 111 14 0.4515 14 1.0000 13
## 112 15 0.8643 12 1.0000 15
## 113 6 0.2296 5 1.0000 6
## 114 19 0.3906 19 0.9995 5
## 115 12 0.4737 12 1.0000 2
## 116 19 0.7423 19 1.0000 19
## 117 3 0.9941 3 1.0000 3
## 118 3 0.6064 3 1.0000 3
## 119 15 0.6173 15 1.0000 15
## 120 6 0.9098 6 1.0000 6
## 121 5 0.6280 5 0.9990 5
## 122 21 0.8677 21 1.0000 21
## 123 3 0.9567 3 1.0000 3
## 124 3 0.9969 3 1.0000 3
## 125 5 0.6437 5 1.0000 5
## 126 3 0.9904 3 1.0000 3
## 127 21 0.6710 21 0.9999 10
## 128 21 0.5154 21 1.0000 21
## 129 3 0.9237 3 1.0000 3
## 130 10 0.2433 10 1.0000 10
## 131 15 0.5570 12 1.0000 12
## 132 17 0.1423 17 1.0000 17
## 133 12 0.3474 12 0.9019 20
## 134 21 0.8758 21 1.0000 21
## 135 6 0.4960 20 1.0000 20
## 136 4 0.9811 4 1.0000 4
## 137 7 0.2661 16 1.0000 21
## 138 17 0.5498 17 1.0000 17
## 139 20 0.2360 16 1.0000 20
## 140 21 0.9068 19 1.0000 7
## 141 8 0.8365 8 1.0000 8
## 142 3 0.7946 3 1.0000 3
## 143 20 0.6142 20 1.0000 20
## 144 5 0.7844 5 1.0000 5
## 145 13 0.9877 13 1.0000 13
## 146 20 0.3308 5 1.0000 5
## 147 13 0.7187 12 1.0000 13
## 148 3 0.7593 3 1.0000 3
## 149 21 0.9416 21 1.0000 21
## 150 5 0.6226 5 0.9944 5
## 151 1 0.3787 10 0.9385 6
## 152 6 0.9374 6 1.0000 6
## 153 3 0.2009 3 1.0000 10
## 154 15 0.2080 16 1.0000 15
## 155 5 0.1728 21 0.9982 2
## 156 12 0.9179 12 1.0000 12
## 157 20 0.7673 20 1.0000 20
## 158 3 0.7879 3 1.0000 3
## 159 3 0.9625 3 1.0000 3
## 160 5 0.4542 5 1.0000 15
## 161 5 0.6716 5 1.0000 5
## 162 21 0.9898 21 1.0000 21
## 163 12 0.2513 12 1.0000 2
## 164 12 0.3127 15 0.9998 10
## 165 12 0.6516 12 1.0000 12
## 166 19 0.7896 7 1.0000 19
## 167 5 0.6062 5 0.7369 5
## 168 21 0.7262 21 1.0000 21
## 169 20 0.9527 20 1.0000 20
## 170 20 0.2537 15 1.0000 15
## 171 6 0.9926 6 1.0000 6
## 172 3 0.8710 3 1.0000 3
## 173 5 0.6251 5 0.6184 3
## 174 6 0.8554 6 1.0000 6
## 175 10 0.9065 10 1.0000 10
## 176 20 0.8498 20 1.0000 20
## 177 16 0.9311 16 1.0000 16
## 178 1 0.9777 1 1.0000 6
## 179 21 0.5553 21 1.0000 21
## 180 6 0.4044 6 1.0000 6
## 181 6 0.9971 6 1.0000 6
## 182 3 0.9298 3 1.0000 3
## 183 3 0.9862 3 1.0000 3
## 184 6 0.8222 6 1.0000 6
## 185 6 0.7343 6 1.0000 6
## 186 12 0.9803 12 1.0000 12
## 187 20 0.9314 20 1.0000 20
## 188 5 0.6062 5 0.7369 5
## 189 12 0.2215 13 1.0000 21
## 190 21 0.7443 21 1.0000 21
## 191 15 0.2944 15 1.0000 17
## 192 3 0.5909 3 1.0000 3
## 193 20 0.6847 20 1.0000 20
## 194 10 0.4540 1 1.0000 18
## 195 20 0.8674 20 1.0000 20
## 196 8 0.8678 8 1.0000 8
## 197 21 0.6358 21 1.0000 21
## 198 20 0.9003 20 1.0000 20
## 199 21 0.5530 12 0.9998 21
## 200 20 0.9234 20 1.0000 20
## 201 6 0.7294 6 1.0000 6
## 202 12 0.2373 14 1.0000 12
## 203 20 0.2692 16 1.0000 16
## 204 3 0.9455 3 1.0000 3
## 205 20 0.6393 20 1.0000 20
## 206 21 0.8114 21 1.0000 21
## 207 20 0.1981 5 1.0000 5
## 208 20 0.8205 20 1.0000 20
## 209 10 0.5283 10 1.0000 10
## 210 3 0.4199 3 0.9587 3
## 211 3 0.4439 13 1.0000 13
## 212 20 0.2615 12 1.0000 4
## 213 1 0.5817 8 1.0000 7
## 214 17 0.6109 17 1.0000 17
## 215 15 0.3088 18 1.0000 18
## 216 15 0.5768 2 1.0000 15
## 217 19 0.7243 19 0.9866 18
## 218 10 0.5590 10 1.0000 10
## 219 10 0.7418 10 1.0000 10
## 220 10 0.4271 10 1.0000 10
## 221 21 0.8420 21 1.0000 21
## 222 20 0.4034 20 1.0000 20
## 223 6 0.9988 6 1.0000 6
## 224 3 0.5669 3 1.0000 3
## 225 12 0.4672 20 1.0000 12
## 226 18 0.9288 18 1.0000 18
## 227 18 0.9288 18 1.0000 18
## 228 18 0.9288 18 1.0000 18
## 229 18 0.9288 18 1.0000 18
## 230 18 0.9288 18 1.0000 18
## 231 18 0.9288 18 1.0000 18
## 232 21 0.4160 21 1.0000 21
## 233 7 0.6411 7 1.0000 7
## 234 3 0.9906 3 1.0000 3
## 235 10 0.5265 10 1.0000 17
## 236 14 0.4270 14 1.0000 14
## 237 21 0.9117 21 1.0000 21
## 238 6 0.8881 6 1.0000 6
## 239 3 0.9839 3 1.0000 3
## 240 21 0.5040 21 1.0000 21
## 241 7 0.8199 7 1.0000 7
## 242 20 0.8639 20 1.0000 20
## 243 21 0.9542 21 1.0000 21
## 244 21 0.3262 21 1.0000 21
## 245 4 0.8933 4 1.0000 4
## 246 21 0.3901 16 0.9959 19
## 247 21 0.5126 7 1.0000 21
## 248 10 0.5907 10 1.0000 10
## 249 12 0.1725 18 0.9999 3
## 250 18 0.9957 18 1.0000 18
## 251 21 0.2489 15 1.0000 5
## 252 21 0.5739 21 1.0000 21
## 253 7 0.6934 7 1.0000 7
## 254 7 0.2690 2 1.0000 7
## 255 21 0.9836 21 1.0000 21
## 256 16 0.8050 16 1.0000 16
## 257 16 0.1251 14 0.5783 7
## 258 21 0.9995 21 1.0000 21
## 259 3 0.7565 3 1.0000 3
## 260 5 0.6062 5 0.7369 5
## 261 21 0.9956 21 1.0000 21
## 262 6 0.9976 6 1.0000 6
## 263 1 0.8594 1 1.0000 1
## 264 4 0.4901 4 1.0000 4
## 265 12 0.4668 15 1.0000 15
## 266 20 0.8358 20 1.0000 20
## 267 12 0.9763 12 1.0000 12
## 268 21 0.5348 7 1.0000 7
## 269 3 0.6389 3 1.0000 3
## 270 4 0.9357 4 1.0000 4
## 271 21 0.7934 21 1.0000 21
## 272 16 0.4079 21 1.0000 21
## 273 4 0.9834 4 1.0000 4
## 274 21 0.9320 21 1.0000 21
## 275 15 0.1628 17 0.9997 15
## 276 16 0.7704 16 1.0000 16
## 277 6 0.9998 6 1.0000 6
## 278 5 0.6716 5 1.0000 5
## 279 5 0.7265 5 1.0000 5
## 280 12 0.4610 12 0.7684 10
## 281 12 0.3048 12 1.0000 1
## 282 3 0.3201 3 1.0000 3
## 283 5 0.5119 20 1.0000 5
## 284 20 0.7279 20 1.0000 20
## 285 4 0.6567 4 1.0000 14
## 286 12 0.2718 15 0.8582 12
## 287 3 0.9999 3 1.0000 3
## 288 21 0.8797 21 1.0000 21
## 289 16 0.8789 16 1.0000 16
## 290 7 0.9555 7 1.0000 7
## 291 7 0.7948 7 1.0000 15
## 292 7 0.4068 21 0.9505 7
## 293 21 0.9474 21 1.0000 21
## 294 7 0.8463 7 1.0000 21
## 295 12 0.2847 12 1.0000 12
## 296 20 0.7079 12 0.9998 12
## 297 5 0.8862 5 1.0000 5
## 298 3 0.9859 3 1.0000 3
## 299 20 0.9877 20 1.0000 20
## 300 10 0.5726 10 1.0000 10
## 301 14 0.6245 19 1.0000 19
## 302 3 0.9253 3 1.0000 3
## 303 15 0.6293 15 0.9856 15
## 304 2 0.1869 2 0.5221 3
## 305 16 0.6248 16 1.0000 21
## 306 7 0.9036 7 1.0000 7
## 307 7 0.9529 7 1.0000 7
## 308 15 0.6972 15 1.0000 15
## 309 21 0.9980 21 1.0000 21
## 310 3 0.8405 20 0.9941 3
## 311 3 0.7071 3 1.0000 3
## 312 3 0.8503 3 1.0000 3
## 313 20 0.6497 20 1.0000 20
## 314 7 0.9088 7 1.0000 7
## 315 21 0.7014 21 1.0000 21
## 316 21 0.9848 21 1.0000 21
## 317 3 0.9531 3 1.0000 3
## 318 7 0.8009 7 0.9973 7
## 319 12 0.4618 12 1.0000 12
## 320 5 0.7027 5 1.0000 5
## 321 6 0.1430 13 1.0000 12
## 322 5 0.6092 5 1.0000 5
## 323 20 0.6944 20 1.0000 20
## 324 15 0.7120 15 1.0000 15
## 325 10 0.8404 10 1.0000 10
## 326 12 0.6229 12 1.0000 12
## 327 8 0.5508 5 1.0000 5
## 328 21 0.3324 21 1.0000 21
## 329 3 0.9797 3 1.0000 3
## 330 19 0.3213 20 1.0000 8
## 331 4 0.8540 4 1.0000 4
## 332 3 0.9908 3 1.0000 3
## 333 2 0.2805 3 0.9997 2
## 334 5 0.1803 14 1.0000 14
## 335 21 0.8009 21 1.0000 21
## 336 12 0.4436 17 0.9717 17
## 337 4 0.9925 4 1.0000 4
## 338 5 0.6062 5 0.7369 5
## 339 17 0.2219 17 0.9998 20
## 340 10 0.3585 10 1.0000 10
## 341 20 0.5225 20 1.0000 20
## 342 3 0.1788 16 0.8573 15
## 343 5 0.1543 6 0.9741 12
## 344 19 0.7513 19 1.0000 19
## 345 17 0.2752 17 1.0000 20
## 346 20 0.8729 20 1.0000 20
## 347 21 0.9583 21 1.0000 21
## 348 16 0.3750 5 1.0000 5
## 349 6 0.9729 6 1.0000 6
## 350 19 0.2607 19 1.0000 12
## 351 19 0.5367 19 1.0000 19
## 352 3 0.9681 3 1.0000 3
## 353 3 0.8027 3 1.0000 3
## 354 21 0.2995 21 1.0000 21
## 355 3 0.9584 3 1.0000 3
## 356 6 0.8225 6 0.9993 6
## 357 21 0.3849 12 1.0000 16
## 358 18 0.9853 18 1.0000 18
## 359 7 0.5999 4 0.9979 4
## 360 3 0.9989 3 1.0000 3
## 361 13 0.3048 3 1.0000 3
## 362 17 0.5488 17 1.0000 17
## 363 4 0.9564 4 1.0000 4
## 364 4 0.8297 4 1.0000 4
## 365 12 0.8880 12 1.0000 12
## 366 10 0.3314 10 1.0000 10
## 367 3 0.9892 3 1.0000 3
## 368 4 0.9794 4 1.0000 4
## 369 12 0.8155 12 1.0000 12
## 370 12 0.3186 3 1.0000 12
## 371 17 0.5298 17 1.0000 17
## 372 17 0.7174 17 1.0000 17
## 373 7 0.9423 7 1.0000 7
## 374 3 0.9083 7 1.0000 3
## 375 10 0.7866 10 1.0000 10
## 376 10 0.5316 10 1.0000 10
## 377 16 0.7547 21 1.0000 21
## 378 16 0.8355 21 1.0000 16
## 379 7 0.1591 2 0.8225 7
## 380 4 0.8756 4 1.0000 4
## 381 1 0.8203 1 0.9707 17
## 382 5 0.7410 5 1.0000 3
## 383 20 0.9016 20 1.0000 20
## 384 21 0.8200 21 1.0000 21
## 385 12 0.7551 12 1.0000 12
## 386 5 0.3147 5 0.9971 5
## 387 7 0.7238 7 1.0000 7
## 388 3 0.9991 3 1.0000 3
## 389 20 0.6194 20 1.0000 20
## 390 19 0.5543 10 0.9935 19
## 391 5 0.5649 5 1.0000 5
## 392 18 0.9742 18 1.0000 18
## 393 20 0.2953 20 0.9993 17
## 394 21 0.3275 20 0.8639 20
## 395 14 0.7883 14 1.0000 14
## 396 7 0.4054 3 1.0000 7
## 397 16 0.5186 20 1.0000 20
## 398 21 0.5925 8 0.9989 21
## 399 4 0.5587 4 1.0000 3
## 400 1 0.3705 1 1.0000 1
## 401 3 0.1217 10 0.8326 10
## 402 1 0.3180 4 0.9999 10
## 403 10 0.6947 10 0.6293 14
## 404 7 0.6736 7 1.0000 7
## 405 3 0.3234 3 1.0000 5
## 406 20 0.1892 20 0.8347 20
## 407 21 0.9328 21 1.0000 21
## 408 5 0.9802 5 1.0000 5
## 409 12 0.2847 12 1.0000 12
## 410 5 0.2935 5 1.0000 5
## 411 3 0.9988 3 1.0000 3
## 412 15 0.4570 15 1.0000 15
## 413 13 0.9336 13 1.0000 13
## 414 6 0.9634 6 1.0000 6
## 415 21 0.2209 20 1.0000 12
## 416 12 0.7982 12 1.0000 3
## 417 3 0.5785 3 1.0000 3
## 418 18 0.1427 17 0.9758 21
## 419 18 0.9293 18 1.0000 18
## 420 18 0.9293 18 1.0000 18
## 421 7 0.6871 7 1.0000 21
## 422 1 0.3009 15 0.9175 1
## 423 10 0.2880 8 0.8164 18
## 424 19 0.7704 19 1.0000 6
## 425 21 0.2207 21 1.0000 20
## 426 21 0.9634 21 1.0000 21
## 427 3 0.6810 3 1.0000 3
## 428 5 0.3990 12 0.9998 5
## 429 4 0.9794 4 1.0000 15
## 430 20 0.3381 20 0.9967 20
## 431 10 0.3807 8 1.0000 7
## 432 6 0.9997 6 1.0000 6
## 433 6 0.9973 6 1.0000 5
## 434 12 0.1809 5 0.9151 6
## 435 6 0.2632 12 1.0000 6
## 436 6 0.4293 21 1.0000 6
## 437 15 0.5924 15 0.9983 15
## 438 6 0.7854 16 0.9999 6
## 439 12 0.5360 12 1.0000 99
## 440 6 0.9984 6 1.0000 6
## 441 15 0.7715 15 0.9589 15
## 442 3 0.6830 5 0.8119 3
## 443 12 0.4711 15 1.0000 15
## 444 1 0.6473 7 1.0000 1
## 445 14 0.9892 14 1.0000 14
## 446 17 0.9388 17 1.0000 17
## 447 3 0.8999 3 1.0000 3
## 448 3 0.9965 3 1.0000 3
## 449 3 0.9535 3 1.0000 3
## CONSENSUS_CODE CONSENSUS_AGREE CONSENSUS_INCORRECT PROBABILITY_CODE
## 1 12 2 0 12
## 2 5 2 0 5
## 3 15 2 0 15
## 4 10 2 0 10
## 5 3 2 0 3
## 6 18 2 0 18
## 7 21 2 0 21
## 8 12 1 0 12
## 9 20 2 0 20
## 10 3 2 0 3
## 11 5 2 0 5
## 12 21 2 0 21
## 13 10 1 1 10
## 14 21 2 0 21
## 15 12 2 0 12
## 16 3 2 0 3
## 17 16 2 1 16
## 18 7 1 0 7
## 19 6 1 1 6
## 20 16 2 0 16
## 21 3 2 0 3
## 22 15 2 0 15
## 23 20 2 0 20
## 24 20 2 0 20
## 25 21 2 0 21
## 26 3 2 0 3
## 27 5 2 0 5
## 28 3 2 1 3
## 29 15 2 0 15
## 30 3 1 1 3
## 31 4 2 0 4
## 32 15 1 1 15
## 33 4 2 0 4
## 34 12 2 1 12
## 35 20 2 0 20
## 36 15 1 1 15
## 37 4 2 0 4
## 38 1 2 0 1
## 39 20 2 0 20
## 40 20 2 0 20
## 41 20 2 0 20
## 42 21 2 0 21
## 43 4 2 1 4
## 44 13 2 1 13
## 45 21 2 0 21
## 46 6 2 0 6
## 47 20 2 0 20
## 48 19 2 1 19
## 49 7 2 1 7
## 50 7 2 1 7
## 51 3 2 0 3
## 52 15 1 1 15
## 53 20 1 0 20
## 54 16 1 0 16
## 55 21 2 0 21
## 56 10 2 0 10
## 57 21 2 0 21
## 58 20 1 1 20
## 59 21 2 0 21
## 60 1 2 0 1
## 61 7 2 0 7
## 62 3 2 0 3
## 63 12 2 0 12
## 64 3 2 0 3
## 65 18 2 0 18
## 66 20 2 0 20
## 67 5 1 0 5
## 68 21 2 0 21
## 69 4 2 0 4
## 70 4 2 0 4
## 71 16 1 0 16
## 72 4 2 0 4
## 73 21 2 0 21
## 74 12 1 1 12
## 75 16 2 0 16
## 76 16 2 0 16
## 77 10 2 0 10
## 78 20 2 0 20
## 79 5 1 1 5
## 80 21 2 0 21
## 81 5 2 1 5
## 82 7 2 0 7
## 83 4 2 0 4
## 84 7 2 0 7
## 85 16 1 0 16
## 86 17 2 0 17
## 87 3 2 0 3
## 88 3 2 0 3
## 89 5 1 1 5
## 90 3 2 0 3
## 91 15 2 0 15
## 92 12 2 1 12
## 93 15 2 0 15
## 94 20 1 0 20
## 95 12 2 0 12
## 96 1 1 0 1
## 97 20 2 0 20
## 98 20 2 0 20
## 99 4 2 0 4
## 100 5 2 0 5
## 101 7 1 1 7
## 102 20 1 0 20
## 103 4 2 0 4
## 104 3 2 0 3
## 105 18 2 0 18
## 106 14 2 0 14
## 107 21 2 0 21
## 108 17 1 1 17
## 109 20 2 0 20
## 110 14 1 0 14
## 111 14 2 1 14
## 112 12 1 1 12
## 113 5 1 1 5
## 114 19 2 1 19
## 115 12 2 1 12
## 116 19 2 0 19
## 117 3 2 0 3
## 118 3 2 0 3
## 119 15 2 0 15
## 120 6 2 0 6
## 121 5 2 0 5
## 122 21 2 0 21
## 123 3 2 0 3
## 124 3 2 0 3
## 125 5 2 0 5
## 126 3 2 0 3
## 127 21 2 1 21
## 128 21 2 0 21
## 129 3 2 0 3
## 130 10 2 0 10
## 131 12 1 0 12
## 132 17 2 0 17
## 133 12 2 1 12
## 134 21 2 0 21
## 135 20 1 0 20
## 136 4 2 0 4
## 137 16 1 1 16
## 138 17 2 0 17
## 139 16 1 1 16
## 140 19 1 1 19
## 141 8 2 0 8
## 142 3 2 0 3
## 143 20 2 0 20
## 144 5 2 0 5
## 145 13 2 0 13
## 146 5 1 0 5
## 147 12 1 1 12
## 148 3 2 0 3
## 149 21 2 0 21
## 150 5 2 0 5
## 151 10 1 1 10
## 152 6 2 0 6
## 153 3 2 1 3
## 154 16 1 1 16
## 155 21 1 1 21
## 156 12 2 0 12
## 157 20 2 0 20
## 158 3 2 0 3
## 159 3 2 0 3
## 160 5 2 1 5
## 161 5 2 0 5
## 162 21 2 0 21
## 163 12 2 1 12
## 164 15 1 1 15
## 165 12 2 0 12
## 166 7 1 1 7
## 167 5 2 0 5
## 168 21 2 0 21
## 169 20 2 0 20
## 170 15 1 0 15
## 171 6 2 0 6
## 172 3 2 0 3
## 173 5 2 1 5
## 174 6 2 0 6
## 175 10 2 0 10
## 176 20 2 0 20
## 177 16 2 0 16
## 178 1 2 1 1
## 179 21 2 0 21
## 180 6 2 0 6
## 181 6 2 0 6
## 182 3 2 0 3
## 183 3 2 0 3
## 184 6 2 0 6
## 185 6 2 0 6
## 186 12 2 0 12
## 187 20 2 0 20
## 188 5 2 0 5
## 189 13 1 1 13
## 190 21 2 0 21
## 191 15 2 1 15
## 192 3 2 0 3
## 193 20 2 0 20
## 194 1 1 1 1
## 195 20 2 0 20
## 196 8 2 0 8
## 197 21 2 0 21
## 198 20 2 0 20
## 199 12 1 1 12
## 200 20 2 0 20
## 201 6 2 0 6
## 202 14 1 1 14
## 203 16 1 0 16
## 204 3 2 0 3
## 205 20 2 0 20
## 206 21 2 0 21
## 207 5 1 0 5
## 208 20 2 0 20
## 209 10 2 0 10
## 210 3 2 0 3
## 211 13 1 0 13
## 212 12 1 1 12
## 213 8 1 1 8
## 214 17 2 0 17
## 215 18 1 0 18
## 216 2 1 1 2
## 217 19 2 1 19
## 218 10 2 0 10
## 219 10 2 0 10
## 220 10 2 0 10
## 221 21 2 0 21
## 222 20 2 0 20
## 223 6 2 0 6
## 224 3 2 0 3
## 225 20 1 1 20
## 226 18 2 0 18
## 227 18 2 0 18
## 228 18 2 0 18
## 229 18 2 0 18
## 230 18 2 0 18
## 231 18 2 0 18
## 232 21 2 0 21
## 233 7 2 0 7
## 234 3 2 0 3
## 235 10 2 1 10
## 236 14 2 0 14
## 237 21 2 0 21
## 238 6 2 0 6
## 239 3 2 0 3
## 240 21 2 0 21
## 241 7 2 0 7
## 242 20 2 0 20
## 243 21 2 0 21
## 244 21 2 0 21
## 245 4 2 0 4
## 246 16 1 1 16
## 247 7 1 1 7
## 248 10 2 0 10
## 249 18 1 1 18
## 250 18 2 0 18
## 251 15 1 1 15
## 252 21 2 0 21
## 253 7 2 0 7
## 254 2 1 1 2
## 255 21 2 0 21
## 256 16 2 0 16
## 257 14 1 1 14
## 258 21 2 0 21
## 259 3 2 0 3
## 260 5 2 0 5
## 261 21 2 0 21
## 262 6 2 0 6
## 263 1 2 0 1
## 264 4 2 0 4
## 265 15 1 0 15
## 266 20 2 0 20
## 267 12 2 0 12
## 268 7 1 0 7
## 269 3 2 0 3
## 270 4 2 0 4
## 271 21 2 0 21
## 272 21 1 0 21
## 273 4 2 0 4
## 274 21 2 0 21
## 275 17 1 1 17
## 276 16 2 0 16
## 277 6 2 0 6
## 278 5 2 0 5
## 279 5 2 0 5
## 280 12 2 1 12
## 281 12 2 1 12
## 282 3 2 0 3
## 283 20 1 1 20
## 284 20 2 0 20
## 285 4 2 1 4
## 286 15 1 1 15
## 287 3 2 0 3
## 288 21 2 0 21
## 289 16 2 0 16
## 290 7 2 0 7
## 291 7 2 1 7
## 292 21 1 1 21
## 293 21 2 0 21
## 294 7 2 1 7
## 295 12 2 0 12
## 296 12 1 0 12
## 297 5 2 0 5
## 298 3 2 0 3
## 299 20 2 0 20
## 300 10 2 0 10
## 301 19 1 0 19
## 302 3 2 0 3
## 303 15 2 0 15
## 304 2 2 1 2
## 305 16 2 1 16
## 306 7 2 0 7
## 307 7 2 0 7
## 308 15 2 0 15
## 309 21 2 0 21
## 310 20 1 1 20
## 311 3 2 0 3
## 312 3 2 0 3
## 313 20 2 0 20
## 314 7 2 0 7
## 315 21 2 0 21
## 316 21 2 0 21
## 317 3 2 0 3
## 318 7 2 0 7
## 319 12 2 0 12
## 320 5 2 0 5
## 321 13 1 1 13
## 322 5 2 0 5
## 323 20 2 0 20
## 324 15 2 0 15
## 325 10 2 0 10
## 326 12 2 0 12
## 327 5 1 0 5
## 328 21 2 0 21
## 329 3 2 0 3
## 330 20 1 1 20
## 331 4 2 0 4
## 332 3 2 0 3
## 333 3 1 1 3
## 334 14 1 0 14
## 335 21 2 0 21
## 336 17 1 0 17
## 337 4 2 0 4
## 338 5 2 0 5
## 339 17 2 1 17
## 340 10 2 0 10
## 341 20 2 0 20
## 342 16 1 1 16
## 343 6 1 1 6
## 344 19 2 0 19
## 345 17 2 1 17
## 346 20 2 0 20
## 347 21 2 0 21
## 348 5 1 0 5
## 349 6 2 0 6
## 350 19 2 1 19
## 351 19 2 0 19
## 352 3 2 0 3
## 353 3 2 0 3
## 354 21 2 0 21
## 355 3 2 0 3
## 356 6 2 0 6
## 357 12 1 1 12
## 358 18 2 0 18
## 359 4 1 0 4
## 360 3 2 0 3
## 361 3 1 0 3
## 362 17 2 0 17
## 363 4 2 0 4
## 364 4 2 0 4
## 365 12 2 0 12
## 366 10 2 0 10
## 367 3 2 0 3
## 368 4 2 0 4
## 369 12 2 0 12
## 370 3 1 1 3
## 371 17 2 0 17
## 372 17 2 0 17
## 373 7 2 0 7
## 374 7 1 1 7
## 375 10 2 0 10
## 376 10 2 0 10
## 377 21 1 0 21
## 378 21 1 1 21
## 379 2 1 1 2
## 380 4 2 0 4
## 381 1 2 1 1
## 382 5 2 1 5
## 383 20 2 0 20
## 384 21 2 0 21
## 385 12 2 0 12
## 386 5 2 0 5
## 387 7 2 0 7
## 388 3 2 0 3
## 389 20 2 0 20
## 390 10 1 1 10
## 391 5 2 0 5
## 392 18 2 0 18
## 393 20 2 1 20
## 394 20 1 0 20
## 395 14 2 0 14
## 396 3 1 1 3
## 397 20 1 0 20
## 398 8 1 1 8
## 399 4 2 1 4
## 400 1 2 0 1
## 401 10 1 0 10
## 402 4 1 1 4
## 403 10 2 1 10
## 404 7 2 0 7
## 405 3 2 1 3
## 406 20 2 0 20
## 407 21 2 0 21
## 408 5 2 0 5
## 409 12 2 0 12
## 410 5 2 0 5
## 411 3 2 0 3
## 412 15 2 0 15
## 413 13 2 0 13
## 414 6 2 0 6
## 415 20 1 1 20
## 416 12 2 1 12
## 417 3 2 0 3
## 418 17 1 1 17
## 419 18 2 0 18
## 420 18 2 0 18
## 421 7 2 1 7
## 422 15 1 1 15
## 423 8 1 1 8
## 424 19 2 1 19
## 425 21 2 1 21
## 426 21 2 0 21
## 427 3 2 0 3
## 428 12 1 1 12
## 429 4 2 1 4
## 430 20 2 0 20
## 431 8 1 1 8
## 432 6 2 0 6
## 433 6 2 1 6
## 434 5 1 1 5
## 435 12 1 1 12
## 436 21 1 1 21
## 437 15 2 0 15
## 438 16 1 1 16
## 439 12 2 1 12
## 440 6 2 0 6
## 441 15 2 0 15
## 442 5 1 1 5
## 443 15 1 0 15
## 444 7 1 1 7
## 445 14 2 0 14
## 446 17 2 0 17
## 447 3 2 0 3
## 448 3 2 0 3
## 449 3 2 0 3
## PROBABILITY_INCORRECT
## 1 0
## 2 0
## 3 0
## 4 0
## 5 0
## 6 0
## 7 0
## 8 0
## 9 0
## 10 0
## 11 0
## 12 0
## 13 1
## 14 0
## 15 0
## 16 0
## 17 1
## 18 0
## 19 1
## 20 0
## 21 0
## 22 0
## 23 0
## 24 0
## 25 0
## 26 0
## 27 0
## 28 1
## 29 0
## 30 1
## 31 0
## 32 1
## 33 0
## 34 1
## 35 0
## 36 1
## 37 0
## 38 0
## 39 0
## 40 0
## 41 0
## 42 0
## 43 1
## 44 1
## 45 0
## 46 0
## 47 0
## 48 1
## 49 1
## 50 1
## 51 0
## 52 1
## 53 0
## 54 0
## 55 0
## 56 0
## 57 0
## 58 1
## 59 0
## 60 0
## 61 0
## 62 0
## 63 0
## 64 0
## 65 0
## 66 0
## 67 0
## 68 0
## 69 0
## 70 0
## 71 0
## 72 0
## 73 0
## 74 1
## 75 0
## 76 0
## 77 0
## 78 0
## 79 1
## 80 0
## 81 1
## 82 0
## 83 0
## 84 0
## 85 0
## 86 0
## 87 0
## 88 0
## 89 1
## 90 0
## 91 0
## 92 1
## 93 0
## 94 0
## 95 0
## 96 0
## 97 0
## 98 0
## 99 0
## 100 0
## 101 1
## 102 0
## 103 0
## 104 0
## 105 0
## 106 0
## 107 0
## 108 1
## 109 0
## 110 0
## 111 1
## 112 1
## 113 1
## 114 1
## 115 1
## 116 0
## 117 0
## 118 0
## 119 0
## 120 0
## 121 0
## 122 0
## 123 0
## 124 0
## 125 0
## 126 0
## 127 1
## 128 0
## 129 0
## 130 0
## 131 0
## 132 0
## 133 1
## 134 0
## 135 0
## 136 0
## 137 1
## 138 0
## 139 1
## 140 1
## 141 0
## 142 0
## 143 0
## 144 0
## 145 0
## 146 0
## 147 1
## 148 0
## 149 0
## 150 0
## 151 1
## 152 0
## 153 1
## 154 1
## 155 1
## 156 0
## 157 0
## 158 0
## 159 0
## 160 1
## 161 0
## 162 0
## 163 1
## 164 1
## 165 0
## 166 1
## 167 0
## 168 0
## 169 0
## 170 0
## 171 0
## 172 0
## 173 1
## 174 0
## 175 0
## 176 0
## 177 0
## 178 1
## 179 0
## 180 0
## 181 0
## 182 0
## 183 0
## 184 0
## 185 0
## 186 0
## 187 0
## 188 0
## 189 1
## 190 0
## 191 1
## 192 0
## 193 0
## 194 1
## 195 0
## 196 0
## 197 0
## 198 0
## 199 1
## 200 0
## 201 0
## 202 1
## 203 0
## 204 0
## 205 0
## 206 0
## 207 0
## 208 0
## 209 0
## 210 0
## 211 0
## 212 1
## 213 1
## 214 0
## 215 0
## 216 1
## 217 1
## 218 0
## 219 0
## 220 0
## 221 0
## 222 0
## 223 0
## 224 0
## 225 1
## 226 0
## 227 0
## 228 0
## 229 0
## 230 0
## 231 0
## 232 0
## 233 0
## 234 0
## 235 1
## 236 0
## 237 0
## 238 0
## 239 0
## 240 0
## 241 0
## 242 0
## 243 0
## 244 0
## 245 0
## 246 1
## 247 1
## 248 0
## 249 1
## 250 0
## 251 1
## 252 0
## 253 0
## 254 1
## 255 0
## 256 0
## 257 1
## 258 0
## 259 0
## 260 0
## 261 0
## 262 0
## 263 0
## 264 0
## 265 0
## 266 0
## 267 0
## 268 0
## 269 0
## 270 0
## 271 0
## 272 0
## 273 0
## 274 0
## 275 1
## 276 0
## 277 0
## 278 0
## 279 0
## 280 1
## 281 1
## 282 0
## 283 1
## 284 0
## 285 1
## 286 1
## 287 0
## 288 0
## 289 0
## 290 0
## 291 1
## 292 1
## 293 0
## 294 1
## 295 0
## 296 0
## 297 0
## 298 0
## 299 0
## 300 0
## 301 0
## 302 0
## 303 0
## 304 1
## 305 1
## 306 0
## 307 0
## 308 0
## 309 0
## 310 1
## 311 0
## 312 0
## 313 0
## 314 0
## 315 0
## 316 0
## 317 0
## 318 0
## 319 0
## 320 0
## 321 1
## 322 0
## 323 0
## 324 0
## 325 0
## 326 0
## 327 0
## 328 0
## 329 0
## 330 1
## 331 0
## 332 0
## 333 1
## 334 0
## 335 0
## 336 0
## 337 0
## 338 0
## 339 1
## 340 0
## 341 0
## 342 1
## 343 1
## 344 0
## 345 1
## 346 0
## 347 0
## 348 0
## 349 0
## 350 1
## 351 0
## 352 0
## 353 0
## 354 0
## 355 0
## 356 0
## 357 1
## 358 0
## 359 0
## 360 0
## 361 0
## 362 0
## 363 0
## 364 0
## 365 0
## 366 0
## 367 0
## 368 0
## 369 0
## 370 1
## 371 0
## 372 0
## 373 0
## 374 1
## 375 0
## 376 0
## 377 0
## 378 1
## 379 1
## 380 0
## 381 1
## 382 1
## 383 0
## 384 0
## 385 0
## 386 0
## 387 0
## 388 0
## 389 0
## 390 1
## 391 0
## 392 0
## 393 1
## 394 0
## 395 0
## 396 1
## 397 0
## 398 1
## 399 1
## 400 0
## 401 0
## 402 1
## 403 1
## 404 0
## 405 1
## 406 0
## 407 0
## 408 0
## 409 0
## 410 0
## 411 0
## 412 0
## 413 0
## 414 0
## 415 1
## 416 1
## 417 0
## 418 1
## 419 0
## 420 0
## 421 1
## 422 1
## 423 1
## 424 1
## 425 1
## 426 0
## 427 0
## 428 1
## 429 1
## 430 0
## 431 1
## 432 0
## 433 1
## 434 1
## 435 1
## 436 1
## 437 0
## 438 1
## 439 1
## 440 0
## 441 0
## 442 1
## 443 0
## 444 1
## 445 0
## 446 0
## 447 0
## 448 0
## 449 0