Principles of experimental design for Big Data analysis

Drovandi, Christopher C., Holmes, Christopher, McGree, James, Mengersen, Kerrie, Richardson, Sylvia, & Ryan, Elizabeth (2017) Principles of experimental design for Big Data analysis. Statistical Science. (In Press)

View at publisher (open access)

Abstract

Big Datasets are endemic, but are often notoriously difficult to analyse because of their size, heterogeneity and quality. The purpose of this paper is to open a discourse on the potential for modern decision theoretic optimal experimental design methods, which by their very nature have traditionally been applied prospectively, to improve the analysis of Big Data through retrospective designed sampling in order to answer particular questions of interest. By appealing to a range of examples, it is suggested that this perspective on Big Data modelling and analysis has the potential for wide generality and advantageous inferential and computational properties. We highlight current hurdles and open research questions surrounding efficient computational optimisation in using retrospective designs, and in part this paper is a call to the optimisation and experimental design communities to work together in the field of Big Data analysis.

Impact and interest:

Search Google Scholar™

Citation counts are sourced monthly from Scopus and Web of Science® citation databases.

These databases contain citations from different subsets of available publications and different time periods and thus the citation count from each is usually different. Some works are not in either database and no count is displayed. Scopus includes citations from articles published in 1996 onwards, and Web of Science® generally from 1980 onwards.

Citations counts from the Google Scholar™ indexing service can be viewed at the linked Google Scholar™ search.

Full-text downloads:

385 since deposited on 01 Oct 2015
248 in the past twelve months

Full-text downloads displays the total number of times this work’s files (e.g., a PDF) have been downloaded from QUT ePrints as well as the number of downloads in the previous 365 days. The count includes downloads for all files if a work has more than one.

ID Code: 87946
Item Type: Journal Article
Refereed: Yes
Keywords: Big Data, Sub-sampling, Experimental design, Active learning, Dimension reduction, Subset
ISSN: 2168-8745
Subjects: Australian and New Zealand Standard Research Classification > MATHEMATICAL SCIENCES (010000) > STATISTICS (010400)
Divisions: Current > QUT Faculties and Divisions > Science & Engineering Faculty
Copyright Owner: Copyright 2017 The Author(s)
Deposited On: 01 Oct 2015 01:55
Last Modified: 21 Jun 2017 14:51

Export: EndNote | Dublin Core | BibTeX

Repository Staff Only: item control page