Comparison of scoring systems for invasive pests using ROC analysis and Monte Carlo simulation

Makowski, David & Mittinty, Murthy (2010) Comparison of scoring systems for invasive pests using ROC analysis and Monte Carlo simulation. Risk Analysis, 30(6), pp. 906-915.

View at publisher


Different international plant protection organisations advocate different schemes for conducting pest risk assessments. Most of these schemes use structured questionnaire in which experts are asked to score several items using an ordinal scale. The scores are then combined using a range of procedures, such as simple arithmetic mean, weighted averages, multiplication of scores, and cumulative sums. The most useful schemes will correctly identify harmful pests and identify ones that are not. As the quality of a pest risk assessment can depend on the characteristics of the scoring system used by the risk assessors (i.e., on the number of points of the scale and on the method used for combining the component scores), it is important to assess and compare the performance of different scoring systems. In this article, we proposed a new method for assessing scoring systems. Its principle is to simulate virtual data using a stochastic model and, then, to estimate sensitivity and specificity values from these data for different scoring systems. The interest of our approach was illustrated in a case study where several scoring systems were compared. Data for this analysis were generated using a probabilistic model describing the pest introduction process. The generated data were then used to simulate the outcome of scoring systems and to assess the accuracy of the decisions about positive and negative introduction. The results showed that ordinal scales with at most 5 or 6 points were sufficient and that the multiplication-based scoring systems performed better than their sum-based counterparts. The proposed method could be used in the future to assess a great diversity of scoring systems.

Impact and interest:

7 citations in Scopus
Search Google Scholar™
5 citations in Web of Science®

Citation counts are sourced monthly from Scopus and Web of Science® citation databases.

These databases contain citations from different subsets of available publications and different time periods and thus the citation count from each is usually different. Some works are not in either database and no count is displayed. Scopus includes citations from articles published in 1996 onwards, and Web of Science® generally from 1980 onwards.

Citations counts from the Google Scholar™ indexing service can be viewed at the linked Google Scholar™ search.

ID Code: 41734
Item Type: Journal Article
Refereed: Yes
Additional URLs:
Keywords: invasive species, pest risk assessment, ROC, scoring systems, sensitivity
DOI: 10.1111/j.1539-6924.2010.01393.x
ISSN: 0272-4332
Subjects: Australian and New Zealand Standard Research Classification > MATHEMATICAL SCIENCES (010000) > STATISTICS (010400) > Applied Statistics (010401)
Australian and New Zealand Standard Research Classification > ENVIRONMENTAL SCIENCES (050000) > ECOLOGICAL APPLICATIONS (050100) > Invasive Species Ecology (050103)
Australian and New Zealand Standard Research Classification > AGRICULTURAL AND VETERINARY SCIENCES (070000) > CROP AND PASTURE PRODUCTION (070300) > Crop and Pasture Protection (Pests Diseases and Weeds) (070308)
Divisions: Past > QUT Faculties & Divisions > Faculty of Science and Technology
Past > Institutes > Institute for Sustainable Resources
Past > Schools > Mathematical Sciences
Deposited On: 19 May 2011 06:52
Last Modified: 10 Aug 2011 15:52

Export: EndNote | Dublin Core | BibTeX

Repository Staff Only: item control page