Inter-rater agreement for Dichotomous data (Relevant and
Observable)
We want to determine the agreement for single Facets for single
Tasks, irrespective of other facets or tasks. It is not possible to
calculate any forms of kappa or alpha for single items, and comparisons
across facets and tasks would not be relevant for answering the research
question. Therefore, the best approach is to report the proportion of
raters in agreement.
Presentation
Presentation relevant facets
This table presents the 15 most relevant facets on the Presentation
task as well as the number of raters who responded that the facet was
relevant (in lieu of % agreement):
Relevance of NEO Facets on Presentation Task
| Pres_Assertiveness_Relevant |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
9 |
| Pres_SelfEfficacy_Relevant |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
9 |
| Pres_Cooperation_Relevant |
1 |
0 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
8 |
| Pres_Friendliness_Relevant |
0 |
1 |
1 |
1 |
0 |
1 |
1 |
1 |
1 |
7 |
| Pres_Sympathy_Relevant |
1 |
1 |
1 |
1 |
0 |
1 |
1 |
1 |
0 |
7 |
| Pres_Anxiety_Relevant |
0 |
1 |
1 |
1 |
0 |
1 |
1 |
1 |
1 |
7 |
| Pres_Liberalism_Relevant |
1 |
1 |
1 |
1 |
0 |
0 |
1 |
1 |
1 |
7 |
| Pres_Gregariousness_Relevant |
0 |
0 |
0 |
1 |
1 |
1 |
1 |
1 |
1 |
6 |
| Pres_Cheerfulnessful_Relevant |
1 |
1 |
0 |
1 |
0 |
1 |
1 |
1 |
0 |
6 |
| Pres_Morality_Relevant |
0 |
1 |
1 |
1 |
0 |
0 |
1 |
1 |
1 |
6 |
| Pres_Vulnerability_Relevant |
1 |
1 |
0 |
1 |
0 |
1 |
1 |
1 |
0 |
6 |
| Pres_Emotionality_Relevant |
0 |
0 |
0 |
1 |
1 |
1 |
1 |
1 |
1 |
6 |
| Pres_Intellect_Relevant |
1 |
1 |
0 |
1 |
0 |
0 |
1 |
1 |
1 |
6 |
| Pres_Altruism_Relevant |
1 |
1 |
0 |
0 |
0 |
1 |
1 |
1 |
0 |
5 |
| Pres_Modesty_Relevant |
0 |
0 |
0 |
1 |
0 |
1 |
1 |
1 |
1 |
5 |
Presentation observable facets
This table presents the 15 most observable facets on the Presentation
task:
Observability of NEO Facets on Presentation Task
| Pres_Assertiveness_Observable |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
9 |
| Pres_Gregariousness_Observable |
0 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
8 |
| Pres_Cheerfulnessful_Observable |
0 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
8 |
| Pres_SelfEfficacy_Observable |
1 |
1 |
1 |
1 |
0 |
1 |
1 |
1 |
1 |
8 |
| Pres_Friendliness_Observable |
0 |
1 |
1 |
1 |
0 |
1 |
1 |
1 |
1 |
7 |
| Pres_Cooperation_Observable |
1 |
0 |
1 |
1 |
1 |
1 |
0 |
1 |
1 |
7 |
| Pres_Sympathy_Observable |
1 |
1 |
0 |
1 |
1 |
1 |
0 |
1 |
1 |
7 |
| Pres_Anxiety_Observable |
0 |
1 |
1 |
1 |
0 |
1 |
1 |
1 |
1 |
7 |
| Pres_Altruism_Observable |
1 |
0 |
1 |
0 |
1 |
1 |
0 |
1 |
1 |
6 |
| Pres_Modesty_Observable |
0 |
1 |
0 |
1 |
0 |
1 |
1 |
1 |
1 |
6 |
| Pres_Depression_Observable |
0 |
1 |
1 |
1 |
0 |
1 |
0 |
1 |
1 |
6 |
| Pres_SelfConsciousness_Observable |
0 |
0 |
1 |
1 |
0 |
1 |
1 |
1 |
1 |
6 |
| Pres_Vulnerability_Observable |
1 |
0 |
0 |
1 |
0 |
1 |
1 |
1 |
1 |
6 |
| Pres_Emotionality_Observable |
0 |
0 |
0 |
1 |
1 |
1 |
1 |
1 |
1 |
6 |
| Pres_Orderliness_Observable |
0 |
0 |
0 |
1 |
1 |
1 |
0 |
1 |
1 |
5 |
Presentation optimal levels
This table presents the average of optimal level ratings for the
presentation task.
Optimal Level of Relevant NEO Facets on Presentation
Task
| Pres_Friendliness_OptLvl |
NA |
4 |
4 |
5 |
NA |
4 |
5 |
4 |
4 |
4.285714 |
| Pres_Assertiveness_OptLvl |
6 |
6 |
5 |
6 |
4 |
6 |
5 |
6 |
6 |
5.555556 |
| Pres_Cooperation_OptLvl |
4 |
NA |
4 |
5 |
6 |
5 |
6 |
6 |
2 |
4.750000 |
| Pres_Sympathy_OptLvl |
6 |
6 |
4 |
4 |
NA |
5 |
5 |
5 |
1 |
4.500000 |
| Pres_SelfEfficacy_OptLvl |
6 |
6 |
4 |
6 |
6 |
6 |
6 |
6 |
4 |
5.555556 |
| Pres_Anxiety_OptLvl |
NA |
5 |
3 |
1 |
NA |
1 |
2 |
1 |
1 |
2.000000 |
The optimal level data is Likert scale ordinal/interval. Therefore,
it can be analysed using Kendall’s W, which is a correlation among
raters. It is also possible to use an ICC.
ICC used a two-way random effects model based on the average of
ratings. The ICC here reflects good inter-rater reliability!
## Average Score Intraclass Correlation
##
## Model: twoway
## Type : consistency
##
## Subjects = 6
## Raters = 9
## ICC(C,9) = 0.875
##
## F-Test, H0: r0 = 0 ; H1: r0 > 0
## F(5,40) = 8.01 , p = 2.65e-05
##
## 95%-Confidence Interval for ICC Population Values:
## 0.637 < ICC < 0.98
Group Discussion
Group Discussion: Relevance
Relevance of NEO Facets for Group Discussion Task
| Grp_Friendliness_Relevant |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
9 |
| Grp_Assertiveness_Relevant |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
9 |
| Grp_Cooperation_Relevant |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
9 |
| Grp_Gregariousness_Relevant |
0 |
1 |
0 |
1 |
1 |
1 |
1 |
1 |
1 |
7 |
| Grp_Modesty_Relevant |
1 |
1 |
1 |
1 |
0 |
1 |
1 |
1 |
0 |
7 |
| Grp_AchievementStriving_Relevant |
1 |
0 |
0 |
1 |
1 |
1 |
1 |
1 |
1 |
7 |
| Grp_Anxiety_Relevant |
0 |
1 |
0 |
1 |
1 |
1 |
1 |
1 |
1 |
7 |
| Grp_Intellect_Relevant |
1 |
1 |
0 |
1 |
1 |
1 |
0 |
1 |
1 |
7 |
| Grp_SelfEfficacy_Relevant |
1 |
1 |
0 |
1 |
1 |
1 |
1 |
0 |
0 |
6 |
| Grp_SelfConsciousness_Relevant |
0 |
0 |
1 |
1 |
0 |
1 |
1 |
1 |
1 |
6 |
| Grp_Anger_Relevant |
0 |
0 |
0 |
1 |
0 |
1 |
1 |
1 |
1 |
5 |
| Grp_Vulnerability_Relevant |
0 |
1 |
0 |
1 |
0 |
1 |
0 |
1 |
1 |
5 |
| Grp_Imagination_Relevant |
0 |
0 |
0 |
1 |
1 |
1 |
0 |
1 |
1 |
5 |
| Grp_Cheerfulness_Relevant |
0 |
0 |
0 |
1 |
0 |
1 |
1 |
1 |
0 |
4 |
| Grp_Trust_Relevant |
1 |
1 |
0 |
1 |
0 |
0 |
1 |
0 |
0 |
4 |
Group Discussion: Observability
Observability of NEO Facets on Group Discussion Task
| Grp_Friendliness_Observable |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
9 |
| Grp_Gregariousness_Observable |
0 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
8 |
| Grp_Assertiveness_Observable |
0 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
8 |
| Grp_Cooperation_Observable |
0 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
8 |
| Grp_Anxiety_Observable |
0 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
8 |
| Grp_Cheerfulness_Observable |
0 |
0 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
7 |
| Grp_Modesty_Observable |
0 |
1 |
1 |
1 |
0 |
1 |
1 |
1 |
1 |
7 |
| Grp_SelfEfficacy_Observable |
0 |
1 |
1 |
1 |
1 |
1 |
0 |
0 |
1 |
6 |
| Grp_Anger_Observable |
0 |
0 |
1 |
1 |
0 |
1 |
1 |
1 |
1 |
6 |
| Grp_Intellect_Observable |
0 |
1 |
0 |
1 |
1 |
1 |
0 |
1 |
1 |
6 |
| Grp_Morality_Observable |
0 |
1 |
0 |
1 |
1 |
1 |
0 |
0 |
1 |
5 |
| Grp_AchievementStriving_Observable |
0 |
0 |
0 |
1 |
0 |
1 |
1 |
1 |
1 |
5 |
| Grp_SelfConsciousness_Observable |
0 |
0 |
1 |
1 |
0 |
1 |
0 |
1 |
1 |
5 |
| Grp_Vulnerability_Observable |
0 |
1 |
0 |
1 |
0 |
1 |
0 |
1 |
1 |
5 |
| Grp_Trust_Observable |
0 |
1 |
0 |
1 |
0 |
0 |
1 |
0 |
1 |
4 |
Group Discussion: Optimal level
Optimal Level of Relevant NEO Facets on Group Discussion
Task
| Grp_Friendliness_OptLvl |
5 |
6 |
4 |
4 |
6 |
5 |
5 |
4 |
5 |
4.888889 |
| Grp_Gregariousness_OptLvl |
NA |
6 |
NA |
4 |
6 |
6 |
6 |
5 |
5 |
5.428571 |
| Grp_Assertiveness_OptLvl |
6 |
6 |
4 |
6 |
5 |
5 |
6 |
5 |
6 |
5.444444 |
| Grp_Cooperation_OptLvl |
6 |
2 |
3 |
3 |
3 |
4 |
5 |
3 |
6 |
3.888889 |
| Grp_Modesty_OptLvl |
6 |
3 |
3 |
2 |
NA |
2 |
4 |
2 |
1 |
2.875000 |
| Grp_AchievementStriving_OptLvl |
6 |
NA |
NA |
6 |
5 |
5 |
5 |
5 |
6 |
5.428571 |
| Grp_Anxiety_OptLvl |
NA |
4 |
NA |
1 |
2 |
1 |
2 |
1 |
1 |
1.714286 |
| Grp_Intellect_OptLvl |
6 |
6 |
NA |
4 |
6 |
5 |
NA |
6 |
4 |
5.285714 |
## Average Score Intraclass Correlation
##
## Model: twoway
## Type : consistency
##
## Subjects = 8
## Raters = 9
## ICC(C,9) = 0.806
##
## F-Test, H0: r0 = 0 ; H1: r0 > 0
## F(7,56) = 5.16 , p = 0.000141
##
## 95%-Confidence Interval for ICC Population Values:
## 0.511 < ICC < 0.955
Critique
Critique: Relevance
Relevance of NEO Facets for Critique Task
| Crit_Assertiveness_Relevant |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
9 |
| Crit_Anger_Relevant |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
9 |
| Crit_Friendliness_Relevant |
0 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
8 |
| Crit_Altruism_Relevant |
0 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
8 |
| Crit_Cooperation_Relevant |
1 |
1 |
1 |
1 |
0 |
1 |
1 |
1 |
1 |
8 |
| Crit_Dutifulness_Relevant |
1 |
1 |
0 |
1 |
1 |
1 |
1 |
1 |
1 |
8 |
| Crit_Morality_Relevant |
0 |
1 |
1 |
1 |
1 |
1 |
0 |
1 |
1 |
7 |
| Crit_Cautiousness_Relevant |
1 |
0 |
1 |
1 |
1 |
1 |
0 |
1 |
1 |
7 |
| Crit_Anxiety_Relevant |
1 |
1 |
1 |
1 |
1 |
1 |
0 |
1 |
0 |
7 |
| Crit_Sympathy_Relevant |
0 |
0 |
1 |
1 |
0 |
1 |
1 |
1 |
1 |
6 |
| Crit_Cheerfulness_Relevant |
1 |
1 |
1 |
0 |
0 |
1 |
1 |
0 |
0 |
5 |
| Crit_SelfEfficacy_Relevant |
1 |
0 |
1 |
1 |
1 |
0 |
1 |
0 |
0 |
5 |
| Crit_Orderliness_Relevant |
1 |
0 |
0 |
1 |
0 |
1 |
0 |
1 |
1 |
5 |
| Crit_AchievementStriving_Relevant |
1 |
0 |
0 |
1 |
0 |
1 |
1 |
1 |
0 |
5 |
| Crit_SelfConsciousness_Relevant |
1 |
1 |
0 |
1 |
0 |
1 |
0 |
1 |
0 |
5 |
Critique: Observability
Observability of NEO Facets on Critique Task
| Crit_Anger_Observable |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
9 |
| Crit_Assertiveness_Observable |
0 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
8 |
| Crit_Altruism_Observable |
0 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
8 |
| Crit_Cooperation_Observable |
1 |
1 |
1 |
1 |
0 |
1 |
1 |
1 |
1 |
8 |
| Crit_Anxiety_Observable |
1 |
1 |
1 |
1 |
1 |
1 |
0 |
1 |
1 |
8 |
| Crit_Friendliness_Observable |
0 |
1 |
1 |
1 |
0 |
1 |
1 |
1 |
1 |
7 |
| Crit_Sympathy_Observable |
0 |
0 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
7 |
| Crit_Dutifulness_Observable |
0 |
1 |
0 |
1 |
1 |
1 |
1 |
1 |
1 |
7 |
| Crit_Cheerfulness_Observable |
0 |
1 |
1 |
0 |
1 |
1 |
1 |
0 |
1 |
6 |
| Crit_Orderliness_Observable |
0 |
0 |
1 |
1 |
1 |
1 |
0 |
1 |
1 |
6 |
| Crit_SelfConsciousness_Observable |
0 |
1 |
0 |
1 |
1 |
1 |
0 |
1 |
1 |
6 |
| Crit_Vulnerability_Observable |
0 |
1 |
1 |
1 |
0 |
1 |
0 |
1 |
1 |
6 |
| Crit_SelfEfficacy_Observable |
0 |
0 |
1 |
1 |
1 |
0 |
1 |
0 |
1 |
5 |
| Crit_AchievementStriving_Observable |
0 |
0 |
0 |
1 |
0 |
1 |
1 |
1 |
1 |
5 |
| Crit_SelfDiscipline_Observable |
0 |
1 |
1 |
0 |
0 |
1 |
1 |
0 |
1 |
5 |
Critique: Optimal level
Optimal Level of Relevant NEO Facets on Critique Task
| Crit_Friendliness_OptLvl |
NA |
3 |
5 |
4 |
5 |
4 |
5 |
4 |
3 |
4.125 |
| Crit_Assertiveness_OptLvl |
4 |
6 |
4 |
5 |
4 |
5 |
5 |
6 |
6 |
5.000 |
| Crit_Altruism_OptLvl |
NA |
5 |
4 |
5 |
4 |
5 |
5 |
5 |
4 |
4.625 |
| Crit_Cooperation_OptLvl |
4 |
1 |
3 |
6 |
NA |
4 |
6 |
3 |
2 |
3.625 |
| Crit_Dutifulness_OptLvl |
6 |
5 |
NA |
5 |
6 |
6 |
4 |
5 |
3 |
5.000 |
| Crit_Anger_OptLvl |
2 |
4 |
2 |
1 |
2 |
1 |
1 |
1 |
4 |
2.000 |
The ICC is lower here likely because NA values are recoded as 0.
## Average Score Intraclass Correlation
##
## Model: twoway
## Type : consistency
##
## Subjects = 6
## Raters = 9
## ICC(C,9) = 0.734
##
## F-Test, H0: r0 = 0 ; H1: r0 > 0
## F(5,40) = 3.75 , p = 0.00703
##
## 95%-Confidence Interval for ICC Population Values:
## 0.226 < ICC < 0.957
Teaching task
Teaching: Relevance
Relevance of NEO Facets for Teaching Task
| Teach_Friendliness_Relevant |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
9 |
| Teach_Assertiveness_Relevant |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
9 |
| Teach_Cheerfulness_Relevant |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
9 |
| Teach_Altruism_Relevant |
0 |
1 |
1 |
1 |
0 |
1 |
1 |
1 |
1 |
7 |
| Teach_Modesty_Relevant |
1 |
0 |
1 |
1 |
0 |
1 |
1 |
1 |
1 |
7 |
| Teach_Intellect_Relevant |
1 |
1 |
0 |
1 |
1 |
0 |
1 |
1 |
1 |
7 |
| Teach_Cooperation_Relevant |
0 |
0 |
0 |
1 |
1 |
1 |
1 |
1 |
1 |
6 |
| Teach_Anger_Relevant |
0 |
1 |
0 |
1 |
1 |
1 |
1 |
1 |
0 |
6 |
| Teach_Gregariousness_Relevant |
0 |
0 |
0 |
1 |
0 |
1 |
1 |
1 |
1 |
5 |
| Teach_Sympathy_Relevant |
0 |
0 |
0 |
1 |
0 |
1 |
1 |
1 |
1 |
5 |
| Teach_SelfEfficacy_Relevant |
1 |
0 |
0 |
1 |
1 |
0 |
1 |
1 |
0 |
5 |
| Teach_Orderliness_Relevant |
0 |
1 |
0 |
1 |
1 |
0 |
0 |
1 |
1 |
5 |
| Teach_SelfDiscipline_Relevant |
0 |
1 |
0 |
1 |
0 |
0 |
1 |
1 |
1 |
5 |
| Teach_Imagination_Relevant |
1 |
1 |
0 |
1 |
0 |
0 |
0 |
1 |
1 |
5 |
| Teach_ActivityLevel_Relevant |
1 |
0 |
0 |
1 |
0 |
1 |
1 |
0 |
0 |
4 |
Teaching: Observability
Observability of NEO Facets on Teaching Task
| Teach_Friendliness_Observable |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
9 |
| Teach_Cheerfulness_Observable |
0 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
8 |
| Teach_Anger_Observable |
0 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
8 |
| Teach_Altruism_Observable |
0 |
1 |
1 |
1 |
0 |
1 |
1 |
1 |
1 |
7 |
| Teach_Cooperation_Observable |
0 |
0 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
7 |
| Teach_Gregariousness_Observable |
0 |
0 |
1 |
1 |
0 |
1 |
1 |
1 |
1 |
6 |
| Teach_Assertiveness_Observable |
0 |
0 |
1 |
1 |
0 |
1 |
1 |
1 |
1 |
6 |
| Teach_Modesty_Observable |
0 |
0 |
1 |
1 |
0 |
1 |
0 |
1 |
1 |
5 |
| Teach_Sympathy_Observable |
0 |
0 |
0 |
1 |
0 |
1 |
1 |
1 |
1 |
5 |
| Teach_SelfConsciousness_Observable |
0 |
0 |
1 |
1 |
0 |
0 |
1 |
1 |
1 |
5 |
| Teach_Intellect_Observable |
0 |
1 |
0 |
1 |
0 |
0 |
1 |
1 |
1 |
5 |
| Teach_Orderliness_Observable |
0 |
0 |
0 |
1 |
1 |
0 |
0 |
1 |
1 |
4 |
| Teach_Anxiety_Observable |
0 |
0 |
1 |
1 |
0 |
0 |
0 |
1 |
1 |
4 |
| Teach_Depression_Observable |
0 |
1 |
1 |
0 |
0 |
0 |
0 |
1 |
1 |
4 |
| Teach_Vulnerability_Observable |
0 |
1 |
0 |
1 |
0 |
0 |
0 |
1 |
1 |
4 |
Teaching: Optimal level
Optimal Levels of most relevant NEO Facets for Teaching
Task
| Teach_Friendliness_OptLvl |
6 |
6 |
4 |
4 |
4 |
5 |
6 |
6 |
6 |
5.222222 |
| Teach_Assertiveness_OptLvl |
4 |
5 |
2 |
5 |
4 |
4 |
5 |
4 |
5 |
4.222222 |
| Teach_Cheerfulness_OptLvl |
5 |
5 |
4 |
5 |
4 |
6 |
5 |
5 |
5 |
4.888889 |
| Teach_Altruism_OptLvl |
NA |
5 |
4 |
6 |
NA |
5 |
5 |
6 |
5 |
5.142857 |
| Teach_Modesty_OptLvl |
4 |
NA |
4 |
3 |
NA |
4 |
4 |
5 |
3 |
3.857143 |
| Teach_Intellect_OptLvl |
6 |
5 |
NA |
6 |
4 |
NA |
4 |
5 |
6 |
5.142857 |
The NA values in the last 3 facets lead to a low ICC here. The last 3
facets each had two NA values, which I transformed to 0 in order to run
the ICC on all the data (it will casewise delete otherwise).
## Average Score Intraclass Correlation
##
## Model: twoway
## Type : consistency
##
## Subjects = 6
## Raters = 9
## ICC(C,9) = 0.558
##
## F-Test, H0: r0 = 0 ; H1: r0 > 0
## F(5,40) = 2.26 , p = 0.0666
##
## 95%-Confidence Interval for ICC Population Values:
## -0.284 < ICC < 0.928
Only analysing the top 3 (which had unanimous agreement on the
Relevant question) gives a much better ICC.
## Average Score Intraclass Correlation
##
## Model: twoway
## Type : consistency
##
## Subjects = 3
## Raters = 9
## ICC(C,9) = 0.821
##
## F-Test, H0: r0 = 0 ; H1: r0 > 0
## F(2,16) = 5.6 , p = 0.0143
##
## 95%-Confidence Interval for ICC Population Values:
## 0.163 < ICC < 0.995