FlexAnalytics: A flexible data analytics framework for big data applications with I/O performance improvement
Zou, Hongbo, Yu, Yongen, Tang, Wei, & Chen, Hsuan-Wei Michelle (2014) FlexAnalytics: A flexible data analytics framework for big data applications with I/O performance improvement. Big Data Research, 1, pp. 4-13.
Increasingly larger scale applications are generating an unprecedented amount of data. However, the increasing gap between computation and I/O capacity on High End Computing machines makes a severe bottleneck for data analysis. Instead of moving data from its source to the output storage, in-situ analytics processes output data while simulations are running. However, in-situ data analysis incurs much more computing resource contentions with simulations. Such contentions severely damage the performance of simulation on HPE. Since different data processing strategies have different impact on performance and cost, there is a consequent need for flexibility in the location of data analytics. In this paper, we explore and analyze several potential data-analytics placement strategies along the I/O path. To find out the best strategy to reduce data movement in given situation, we propose a flexible data analytics (FlexAnalytics) framework in this paper. Based on this framework, a FlexAnalytics prototype system is developed for analytics placement. FlexAnalytics system enhances the scalability and flexibility of current I/O stack on HEC platforms and is useful for data pre-processing, runtime data analysis and visualization, as well as for large-scale data transfer. Two use cases – scientific data compression and remote visualization – have been applied in the study to verify the performance of FlexAnalytics. Experimental results demonstrate that FlexAnalytics framework increases data transition bandwidth and improves the application end-to-end transfer performance.
Impact and interest:
Citation counts are sourced monthly from and citation databases.
These databases contain citations from different subsets of available publications and different time periods and thus the citation count from each is usually different. Some works are not in either database and no count is displayed. Scopus includes citations from articles published in 1996 onwards, and Web of Science® generally from 1980 onwards.
Citations counts from theindexing service can be viewed at the linked Google Scholar™ search.
|Item Type:||Journal Article|
|Additional Information:||Special Issue on Scalable Computing for Big Data|
|Keywords:||I/O bottlenecks; In-situ analytics; Data preparation; Big data; High-end computing|
|Divisions:||Current > Schools > School of Information Systems
Current > QUT Faculties and Divisions > Science & Engineering Faculty
|Copyright Owner:||Copyright 2014 Elsevier Inc.|
|Deposited On:||04 Nov 2015 00:00|
|Last Modified:||04 Nov 2015 00:00|
Repository Staff Only: item control page