The use of various data mining and feature selection methods in the analysis of a population survey dataset

Pitt, Ellen & Nayak, Richi (2007) The use of various data mining and feature selection methods in the analysis of a population survey dataset. In Ong, K.-L., Li, W., & Gao, J. (Eds.) 2nd International Workshop on Integrating Artificial Intelligence and Data Mining (AIDM 2007), 2 December 2007, Gold Coast.


This paper reports the results of feature reduction in the analysis of a population based dataset for which there were no specific target variables. All attributes were assessed as potential targets in models derived from the full dataset and from subsets of it. The feature selection methods used were of the filter and wrapper types as well as clustering techniques. The predictive accuracy and the complexity of models based on the reduced datasets for each method were compared both amongst the methods and with those of the complete dataset. Analysis showed a marked similarity in the correlated features chosen by the supervised (filter) methods and moderate consistency in those chosen by the clustering methods (unsupervised). The breadth of distribution of the correlated features amongst the attribute groups was related in large part to the number of attributes selected by the given algorithm or elected by the user. Characteristics related to Health and Home, Paid and Volunteer Work and Demographics were the targets for which predictive accuracy was highest in both the reduced and full datasets. These attributes and a limited number of characteristics from the Learning, Social and Emotional attribute groups were important in clustering the population with Health and Home characteristics being most consistently important. Misclassification rates for models associated with most targets decreased with the use of subsets derived via filter methods but were increased for subsets derived using clustering methods.

Impact and interest:

Search Google Scholar™

Citation counts are sourced monthly from Scopus and Web of Science® citation databases.

These databases contain citations from different subsets of available publications and different time periods and thus the citation count from each is usually different. Some works are not in either database and no count is displayed. Scopus includes citations from articles published in 1996 onwards, and Web of Science® generally from 1980 onwards.

Citations counts from the Google Scholar™ indexing service can be viewed at the linked Google Scholar™ search.

Full-text downloads:

276 since deposited on 04 Aug 2008
10 in the past twelve months

Full-text downloads displays the total number of times this work’s files (e.g., a PDF) have been downloaded from QUT ePrints as well as the number of downloads in the previous 365 days. The count includes downloads for all files if a work has more than one.

ID Code: 14268
Item Type: Conference Paper
Refereed: Yes
Additional URLs:
ISBN: 9781920682651
Subjects: Australian and New Zealand Standard Research Classification > INFORMATION AND COMPUTING SCIENCES (080000) > LIBRARY AND INFORMATION STUDIES (080700) > Information Retrieval and Web Search (080704)
Divisions: Past > QUT Faculties & Divisions > Faculty of Science and Technology
Copyright Owner: Copyright 2007 Australian Computer Society
Copyright Statement: Copyright © 2007, Australian Computer Society, Inc. This
paper appeared at the Second Workshop on Integrating AI and
Data Mining (AIDM 2007), Gold Coast, Australia. Conferences
in Research and Practice in Information Technology (CRPIT),
Vol. 84, Kok-Leong Ong, Junbin Gao and Wenyuan Li, Ed.
Reproduction for academic, not-for profit purposes permitted provided this text is included.
Deposited On: 04 Aug 2008 00:00
Last Modified: 29 Feb 2012 13:35

Export: EndNote | Dublin Core | BibTeX

Repository Staff Only: item control page