Testing intergroup concordance in ranking experiments with two groups of judges

Dekle, Dawn J., Leung, Denis H. Y., & Zhu, Min (2008) Testing intergroup concordance in ranking experiments with two groups of judges. Psychological Methods, 13(1), pp. 58-71.

View at publisher


Across many areas of psychology, concordance is commonly used to measure the (intragroup) agreement in ranking a number of items by a group of judges. Sometimes, however, the judges come from multiple groups, and in those situations, the interest is to measure the concordance between groups, under the assumption that there is some within-group concordance. In this investigation, existing methods are compared under a variety of scenarios. Permutation theory is used to calculate the error rates and the power of the methods. Missing data situations are also studied. The results indicate that the performance of the methods depend on (a) the number of items to be ranked, (b) the level of within-group agreement, and (c) the level of between-group agreement. Overall, using the actual ranks of the items gives better results than using the pairwise comparison of rankings. Missing data lead to loss in statistical power, and in some cases, the loss is substantial. The degree of power loss depends on the missing mechanism and the method of imputing the missing data, among other factors. © 2008 American Psychological Association.

Impact and interest:

2 citations in Web of Science®
Search Google Scholar™

Citation counts are sourced monthly from Scopus and Web of Science® citation databases.

These databases contain citations from different subsets of available publications and different time periods and thus the citation count from each is usually different. Some works are not in either database and no count is displayed. Scopus includes citations from articles published in 1996 onwards, and Web of Science® generally from 1980 onwards.

Citations counts from the Google Scholar™ indexing service can be viewed at the linked Google Scholar™ search.

ID Code: 99589
Item Type: Journal Article
Refereed: Yes
Keywords: concordance, intergroup, Kendall's W, missing data, ranking experiment
DOI: 10.1037/1082-989X.13.1.58
ISSN: 1082-989X
Subjects: Australian and New Zealand Standard Research Classification > PSYCHOLOGY AND COGNITIVE SCIENCES (170000) > PSYCHOLOGY (170100)
Australian and New Zealand Standard Research Classification
Divisions: Current > QUT Faculties and Divisions > QUT Business School
Current > Schools > School of Economics & Finance
Copyright Owner: Copyright 2008 American Psychological Association
Deposited On: 27 Sep 2016 23:10
Last Modified: 27 Sep 2016 23:10

Export: EndNote | Dublin Core | BibTeX

Repository Staff Only: item control page