Fast rates for estimation error and oracle inequalities for model section

Bartlett, Peter L. (2008) Fast rates for estimation error and oracle inequalities for model section. Econometric Theory, 24(2), pp. 545-552.

View at publisher


We consider complexity penalization methods for model selection. These methods aim to choose a model to optimally trade off estimation and approximation errors by minimizing the sum of an empirical risk term and a complexity penalty. It is well known that if we use a bound on the maximal deviation between empirical and true risks as a complexity penalty, then the risk of our choice is no more than the approximation error plus twice the complexity penalty. There are many cases, however, where complexity penalties like this give loose upper bounds on the estimation error. In particular, if we choose a function from a suitably simple convex function class with a strictly convex loss function, then the estimation error (the difference between the risk of the empirical risk minimizer and the minimal risk in the class) approaches zero at a faster rate than the maximal deviation between empirical and true risks. In this paper, we address the question of whether it is possible to design a complexity penalized model selection method for these situations. We show that, provided the sequence of models is ordered by inclusion, in these cases we can use tight upper bounds on estimation error as a complexity penalty. Surprisingly, this is the case even in situations when the difference between the empirical risk and true risk (and indeed the error of any estimate of the approximation error) decreases much more slowly than the complexity penalty. We give an oracle inequality showing that the resulting model selection method chooses a function with risk no more than the approximation error plus a constant times the complexity penalty.

Impact and interest:

5 citations in Scopus
Search Google Scholar™
7 citations in Web of Science®

Citation counts are sourced monthly from Scopus and Web of Science® citation databases.

These databases contain citations from different subsets of available publications and different time periods and thus the citation count from each is usually different. Some works are not in either database and no count is displayed. Scopus includes citations from articles published in 1996 onwards, and Web of Science® generally from 1980 onwards.

Citations counts from the Google Scholar™ indexing service can be viewed at the linked Google Scholar™ search.

ID Code: 43979
Item Type: Journal Article
Refereed: Yes
Keywords: complexity penalization methods, estimation error
DOI: 10.1017/S0266466608080225
ISSN: 0266-4666
Subjects: Australian and New Zealand Standard Research Classification > ECONOMICS (140000) > ECONOMETRICS (140300)
Divisions: Past > QUT Faculties & Divisions > Faculty of Science and Technology
Past > Schools > Mathematical Sciences
Copyright Owner: Copyright 2008 Cambridge University Press
Deposited On: 18 Aug 2011 00:50
Last Modified: 29 Feb 2012 14:34

Export: EndNote | Dublin Core | BibTeX

Repository Staff Only: item control page