Testing intergroup concordance in ranking experiments with two groups of judges
Dekle, Dawn J., Leung, Denis H. Y., & Zhu, Min (2008) Testing intergroup concordance in ranking experiments with two groups of judges. Psychological Methods, 13(1), pp. 58-71.
Across many areas of psychology, concordance is commonly used to measure the (intragroup) agreement in ranking a number of items by a group of judges. Sometimes, however, the judges come from multiple groups, and in those situations, the interest is to measure the concordance between groups, under the assumption that there is some within-group concordance. In this investigation, existing methods are compared under a variety of scenarios. Permutation theory is used to calculate the error rates and the power of the methods. Missing data situations are also studied. The results indicate that the performance of the methods depend on (a) the number of items to be ranked, (b) the level of within-group agreement, and (c) the level of between-group agreement. Overall, using the actual ranks of the items gives better results than using the pairwise comparison of rankings. Missing data lead to loss in statistical power, and in some cases, the loss is substantial. The degree of power loss depends on the missing mechanism and the method of imputing the missing data, among other factors. © 2008 American Psychological Association.
Impact and interest:
Citation counts are sourced monthly from and citation databases.
These databases contain citations from different subsets of available publications and different time periods and thus the citation count from each is usually different. Some works are not in either database and no count is displayed. Scopus includes citations from articles published in 1996 onwards, and Web of Science® generally from 1980 onwards.
Citations counts from theindexing service can be viewed at the linked Google Scholar™ search.
|Item Type:||Journal Article|
|Keywords:||concordance, intergroup, Kendall's W, missing data, ranking experiment|
|Subjects:||Australian and New Zealand Standard Research Classification > PSYCHOLOGY AND COGNITIVE SCIENCES (170000) > PSYCHOLOGY (170100)
Australian and New Zealand Standard Research Classification
|Divisions:||Current > QUT Faculties and Divisions > QUT Business School
Current > Schools > School of Economics & Finance
|Copyright Owner:||Copyright 2008 American Psychological Association|
|Deposited On:||27 Sep 2016 23:10|
|Last Modified:||27 Sep 2016 23:10|
Repository Staff Only: item control page