Network based buffer overflow detection by exploit code analysis

Andersson, Stig, Clark, Andrew, & Mohay, George (2004) Network based buffer overflow detection by exploit code analysis. In Mohay, George M., Clark, Andrew J., & Kerr, Kathryn (Eds.) AusCERT Asia Pacific Information Technology Security Conference: R&D Stream, 23-27 May 2004, Gold Coast, Australia.

View at publisher


Buffer overflow attacks continue to be a major security problem and detecting attacks of this nature is therefore crucial to network security. Signature based network based intrusion detection systems (NIDS) compare network traffic to signatures modelling suspicious or attack traffic to detect network attacks. Since detection is based on pattern matching, a signature modelling the attack must exist for the NIDS to detect it, and it is therefore only capable of detecting known attacks. This paper proposes a method to detect buffer overflow attacks by parsing the payload of network packets in search of shellcode which is the remotely executable component of a buffer overflow attack. By analysing the shellcode it is possible to determine which system calls the exploit uses, and hence the operation of the exploit. Current NIDS-based buffer overflow detection techniques mainly rely upon specific signatures for each new attack. Our approach is able to detect previously unseen buffer overflow attacks, in addition to existing ones, without the need for specific signatures for each new attack. The method has been implemented and tested for buffer overflow attacks on Linux on the Intel x86 architecture using the Snort NIDS.

Impact and interest:

Search Google Scholar™

Citation counts are sourced monthly from Scopus and Web of Science® citation databases.

These databases contain citations from different subsets of available publications and different time periods and thus the citation count from each is usually different. Some works are not in either database and no count is displayed. Scopus includes citations from articles published in 1996 onwards, and Web of Science® generally from 1980 onwards.

Citations counts from the Google Scholar™ indexing service can be viewed at the linked Google Scholar™ search.

Full-text downloads:

736 since deposited on 14 Jun 2009
104 in the past twelve months

Full-text downloads displays the total number of times this work’s files (e.g., a PDF) have been downloaded from QUT ePrints as well as the number of downloads in the previous 365 days. The count includes downloads for all files if a work has more than one.

ID Code: 21172
Item Type: Conference Paper
Refereed: Yes
Additional URLs:
Keywords: intrusion detection, buffer overflow detection, network monitoring, network security
ISBN: 1864997745
Subjects: Australian and New Zealand Standard Research Classification > INFORMATION AND COMPUTING SCIENCES (080000) > COMPUTER SOFTWARE (080300) > Computer System Security (080303)
Divisions: Past > QUT Faculties & Divisions > Faculty of Science and Technology
Past > Institutes > Information Security Institute
Deposited On: 14 Jun 2009 22:46
Last Modified: 29 Feb 2012 13:07

Export: EndNote | Dublin Core | BibTeX

Repository Staff Only: item control page