Model selection and error estimation

Bartlett, P. L., Boucheron, S., & Lugosi, G. (2002) Model selection and error estimation. Machine Learning, 48(1-3), pp. 85-113.

View at publisher


We study model selection strategies based on penalized empirical loss minimization. We point out a tight relationship between error estimation and data-based complexity penalization: any good error estimate may be converted into a data-based penalty function and the performance of the estimate is governed by the quality of the error estimate. We consider several penalty functions, involving error estimates on independent test data, empirical VC dimension, empirical VC entropy, and margin-based quantities. We also consider the maximal difference between the error on the first half of the training data and the second half, and the expected maximal discrepancy, a closely related capacity estimate that can be calculated by Monte Carlo integration. Maximal discrepancy penalty functions are appealing for pattern classification problems, since their computation is equivalent to empirical risk minimization over the training data with some labels flipped.

Impact and interest:

169 citations in Scopus
Search Google Scholar™
132 citations in Web of Science®

Citation counts are sourced monthly from Scopus and Web of Science® citation databases.

These databases contain citations from different subsets of available publications and different time periods and thus the citation count from each is usually different. Some works are not in either database and no count is displayed. Scopus includes citations from articles published in 1996 onwards, and Web of Science® generally from 1980 onwards.

Citations counts from the Google Scholar™ indexing service can be viewed at the linked Google Scholar™ search.

ID Code: 43932
Item Type: Journal Article
Refereed: Yes
Keywords: Concentration inequalities, Empirical penalties, Model selection, Penalization, Calculations, Computational complexity, Error analysis, Estimation, Integration, Mathematical models, Monte Carlo methods, Pattern recognition, Error estimation, Learning systems
DOI: 10.1023/A:1013999503812
ISSN: 0885-6125
Subjects: Australian and New Zealand Standard Research Classification > MATHEMATICAL SCIENCES (010000) > APPLIED MATHEMATICS (010200)
Australian and New Zealand Standard Research Classification > INFORMATION AND COMPUTING SCIENCES (080000) > ARTIFICIAL INTELLIGENCE AND IMAGE PROCESSING (080100)
Australian and New Zealand Standard Research Classification > PSYCHOLOGY AND COGNITIVE SCIENCES (170000) > COGNITIVE SCIENCE (170200)
Divisions: Past > QUT Faculties & Divisions > Faculty of Science and Technology
Past > Schools > Mathematical Sciences
Copyright Owner: Springer
Deposited On: 12 Aug 2011 02:46
Last Modified: 12 Aug 2011 02:46

Export: EndNote | Dublin Core | BibTeX

Repository Staff Only: item control page