Detection of Buffer Overflow Attacks with Memoization-based Rule Set

Authors

  • Oğuz Özger

    Electrical-Electronics Engineering, Sakarya University of Applied Sciences, Sakarya, 54050, Turkey
  • Halit Öztekİn

    Computer Engineering, Sakarya University of Applied Sciences, Sakarya, 54050, Turkey

DOI:

https://doi.org/10.30564/jcsr.v5i4.6044
Received: 28 October 2023 | Revised: 15 November 2023 | Accepted: 22 November 2023 | Published Online: 30 November 2023

Abstract

Different abnormalities are commonly encountered in computer network systems. These types of abnormalities can lead to critical data losses or unauthorized access in the systems. Buffer overflow anomaly is a prominent issue among these abnormalities, posing a serious threat to network security. The primary objective of this study is to identify the potential risks of buffer overflow that can be caused by functions frequently used in the PHP programming language and to provide solutions to minimize these risks. Static code analyzers are used to detect security vulnerabilities, among which SonarQube stands out with its extensive library, flexible customization options, and reliability in the industry. In this context, a customized rule set aimed at automatically detecting buffer overflows has been developed on the SonarQube platform. The memoization optimization technique used while creating the customized rule set enhances the speed and efficiency of the code analysis process. As a result, the code analysis process is not repeatedly run for code snippets that have been analyzed before, significantly reducing processing time and resource utilization. In this study, a memoization-based rule set was utilized to detect critical security vulnerabilities that could lead to buffer overflow in source codes written in the PHP programming language. Thus, the analysis process is not repeatedly run for code snippets that have been analyzed before, leading to a significant reduction in processing time and resource utilization. In a case study conducted to assess the effectiveness of this method, a significant decrease in the source code analysis time was observed.

Keywords:

Buffer overflow, Cybersecurity, Anomaly, SonarQube, Memoization

References

[1] Spafford, E.H., 1989. The Internet worm program: An analysis. ACM SIGCOMM Computer Communication Review. 19(1), 17-57. DOI: https://doi.org/10.1145/66093.66095

[2] Moore, D., Paxson, V., Savage, S., et al., 2003. Inside the slammer worm. IEEE Security & Privacy. 1(4), 33-39. DOI: https://doi.org/10.1109/MSECP.2003.1219056

[3] Springall, D., Durumeric, Z., Halderman, J.A. (editors), 2016. Measuring the security harm of TLS crypto shortcuts. IMC' 16: Proceedings of the 2016 Internet Measurement Conference; 2016 Nov 14-16; Santa Monica California USA. New York: Association for Computing Machinery. p. 33-47. DOI: https://doi.org/10.1145/2987443.2987480

[4] Oversight and Government Reform [Internet]. Available from: https://oversight.house.gov/wp-content/uploads/2018/12/Equifax-Report.pdf

[5] Kocher, P., Horn, J., Fogh, A., et al. (editors), 2019. Spectre attacks: Exploiting speculative execution. 2019 IEEE Symposium on Security and Privacy (SP); 2019 May 19-23; San Francisco, CA, USA. New York: IEEE. DOI: https://doi.org/10.1109/SP.2019.00002

[6] Remote Desktop Services Remote Code Execution Vulnerability [Internet]. Available from: https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2019-0708

[7] Bhutan Built a Bitcoin Mine on the Site of Its Failed 'Education City'; 2019.

[8] On-Premises Exchange Server Vulnerabilities Resource Center—updated March 25, 2021 [Internet]. Available from: https://msrc.microsoft.com/blog/2021/03/multiple-security-updates-released-for-exchange-server/

[9] Dowd, M., McDonald, J., Schuh, J., 2006. The art of software security assessment: Identifying and preventing software vulnerabilities. Pearson Education: London.

[10] SEI CERT Oracle Coding Standard for Java [Internet]. Available from: https://wiki.sei.cmu.edu/confluence/display/java/SEI+CERT+Oracle+Coding+Standard+for+Java

[11] Conklin, W.A., White, G., Cothren, C., et al., 2022. Principles of Computer Security: CompTIA Security+TM and Beyond [Internet]. Available from: https://sisis.rz.htw-berlin.de/inh2010/12389366.pdf

[12] Xu, J., Patel, S., Iyer, R., et al., 2002. Architecture Support for Defending Against Buffer Overflow Attacks [Internet]. Available from: https://www.ideals.illinois.edu/items/100089/bitstreams/319746/data.pdf

[13] Anley, C., Heasman, J., Lindner, F., et al., 2007. The shellcoder's handbook: Discovering and exploiting security holes. Wiley: Hoboken.

[14] Intel® 64 and IA-32 Architectures Software Developer Manuals [Internet]. Available from: https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html

[15] One, A., 1996. Smashing the stack for fun and profit. Phrack Magazine. 7(49), 14-16.

[16] Wagner, D., Foster, J.S., Brewer, E.A., et al., 2000. A First Step Towards Automated Detection of Buffer Overrun Vulnerabilities [Internet]. Available from: https://www.cs.umd.edu/class/spring2021/cmsc614/papers/automated-buffer.pdf

[17] Baratloo, A., Singh, N., Tsai, T. (editors), 2000. Transparent run-time defense against stack smashing attacks. Proceedings of the 2000 USENIX Annual Technical Conference; 2000 Jun 18-23; San Diego, California, USA.

[18] Chiueh, T.C., Hsu, F.H. (editors), 2001. RAD: A compile-time solution to buffer overflow attacks. Proceedings of the 21st International Conference on Distributed Computing Systems; 2001 Apr 16-19; Mesa, AZ, USA. New York: IEEE. p. 409-417. DOI: https://doi.org/10.1109/ICDSC.2001.918971

[19] Kuperman, B., Brodley, C., Ozdoganoglu, H., et al., 2005. Detection and prevention of stack buffer overflow attacks. Communications of the ACM. 48(11), 50-56. DOI: https://doi.org/10.1145/1096000.1096004

[20] Le, W., Soffa, M.L. (editors), 2008. Marple: A demand-driven path-sensitive buffer overflow detector. Proceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering; 2008 Nov 9-14; Atlanta Georgia. New York: Association for Computing Machinery. p. 272-282. DOI: https://doi.org/10.1145/1453101.1453137

[21] Brooks, T.N., 2017. Survey of automated vulnerability detection and exploit generation techniques in cyber reasoning systems. Advances in intelligent systems and computing. Springer: Cham. pp. 1083-1102. DOI: https://doi.org/10.1007/978-3-030-01177-2_79

[22] Chess, W., 1998. Secure programming with static analysis. Pearson Education: London.

[23] Cowan, C., Beattie, S., Johansen, J., et al. (editors), 2003. Pointguard TM: Protecting pointers from buffer overflow vulnerabilities. Proceedings of the 12th USENIX Security Symposium; 2003 Aug 4-8; Washington D.C., USA.

[24] Yong, S.H., Horwitz, S. (editors), 2003. Protecting C programs from attacks via invalid pointer dereferences. Proceedings of the 9th European Software Engineering Conference Held Jointly with 11th ACM SIGSOFT International Symposium on Foundations of Software Engineering; 2003 Sep 1-5; Helsinki Finland. New York: Association for Computing Machinery. p. 307-316. DOI: https://doi.org/10.1145/940071.940113

[25] Nethercote, N., Seward, J., 2003. Valgrind: A program supervision framework. Electronic Notes in Theoretical Computer Science. 89(2), 44-66. DOI: https://doi.org/10.1016/S1571-0661(04)81042-9

[26] Rinard, M., Cadar, C., Dumitran, D., et al. (editors), 2004. A dynamic technique for eliminating buffer overflow vulnerabilities (and other memory errors). 20th Annual Computer Security Applications Conference; 2004 Dec 6-10; Tucson, AZ, USA. New York: IEEE. p. 82-90. DOI: https://doi.org/10.1109/CSAC.2004.2

[27] Cowan, C., Barringer, M., Beattie, S., et al. (editors), 2001. FormatGuard: Automatic protection from printf format string vulnerabilities. Proceedings of the 10th USENIX Security Symposium; 2001 Aug 13-17; Washington D.C., USA. p. 191-200.

[28] Fen, Y., Fuchao, Y., Xiaobing, S., et al., 2012. A new data randomization method to defend buffer overflow attacks. Physics Procedia. 24, 1757-1764. DOI: https://doi.org/10.1016/j.phpro.2012.02.259

[29] Ruwase, O., Lam, M.S., 2003. A Practical Dynamic Buffer Overflow Detector [Internet]. Available from: http://www.cs.cmu.edu/afs/cs.cmu.edu/Web/People/oor/papers/cred.pdf

[30] Jha, S. (editor), 2010. Retrofitting legacy code for security. Computer Aided Verification, 22nd International Conference, CAV 2010; 2010 Jul 15-19; Edinburgh, UK. Berlin: Springer. DOI: https://doi.org/10.1007/978-3-642-14295-6_2

[31] Emami, M., Ghiya, R., Hendren, L. (editors), 1994. Context-sensitive interprocedural points-to analysis in the presence of function pointers. Proceedings of the ACM SIGPLAN 1994 Conference on Programming Language Design and Implementation; 1994 Jun 20-24; Orlando Florida USA. New York: Association for Computing Machinery. DOI: https://doi.org/10.1145/178243.178264

[32] Liang, Z., Sekar, R. (editors), 2005. Automatic generation of buffer overflow attack signatures: An approach based on program behavior models. 21st Annual Computer Security Applications Conference (ACSAC' 05); 2005 Dec 5-9; Tucson, AZ, USA. New York: IEEE. p. 10-224. DOI: https://doi.org/10.1109/CSAC.2005.12

[33] Newsome, J., Song, D., 2005. Dynamic Taint Analysis for Automatic Detection, Analysis, and Signature Generation of Exploits on Commodity Software [Internet]. Available from: https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=5803709632cf010d3923e8f85416bb95db0dd8ea

[34] Qin, F., Lu, S., Zhou, Y. (editors), 2005. SafeMem: Exploiting ECC-memory for detecting memory leaks and memory corruption during production runs. 11th International Symposium on High-Performance Computer Architecture; 2005 Feb 12-16; San Francisco, CA, USA. New York: IEEE. p. 291-302. DOI: https://doi.org/10.1109/HPCA.2005.29

[35] Seward, J., Nethercote, N. (editors), 2005. Using Valgrind to detect undefined value errors with bit-precision. Proceedings of the USENIX' 05 Annual Technical Conference; 2005 Apr 10-15; Anaheim, California, USA.

[36] Executable-space Protection [Internet]. Available from: https://en.wikipedia.org/wiki/Executable-space_protection

[37] Nethercote, N., Seward, J., 2007. Valgrind: A framework for heavyweight dynamic binary instrumentation. ACM Sigplan Notices. 42(6), 89-100. DOI: https://doi.org/10.1145/1273442.1250746

[38] Costa, M. (editor), 2008. Bouncer: Securing software by blocking bad input. Proceedings of the 2nd Workshop on Recent Advances on Intrusiton-tolerant Systems; 2008 Apr 1; Glasgow United Kingdom. New York: Association for Computing Machinery. DOI: https://doi.org/10.1145/1413901.1413902

[39] Song, D., Brumley, D., Yin, H., et al. (editors), 2008. BitBlaze: A new approach to computer security via binary analysis. Information Systems Security, 4th International Conference, ICISS 2008; 2008 Dec 16-20; Hyderabad, India. Berlin: Springer. p. 1-25. DOI: https://doi.org/10.1007/978-3-540-89862-7_1

[40] Liu, G.H., Wu, G., Tao, Z., et al. (editors), 2008. Vulnerability analysis for x86 executables using genetic algorithm and fuzzing. 2008 Third International Conference on Convergence and Hybrid Information Technology; 2008 Nov 11-13; Busan, Korea (South). New York: IEEE. p. 491-497. DOI: https://doi.org/10.1109/ICCIT.2008.9

[41] Kroes, T., Koning, K., Kouwe, E., et al. (editors), 2018. Delta pointers: Buffer overflow checks without the checks. EuroSys' 18: Proceedings of the Thirteenth EuroSys Conference; 2018 Apr 23-26; Porto Portugal. New York: Association for Computing Machinery. p. 1-14. DOI: https://doi.org/10.1145/3190508.3190553

[42] Frantzen, M., Shuey, M. (editors), 2001. StackGhost: Hardware facilitated stack protection. 10th USENIX Security Symposium; 2001 Aug 13-17; Washington, D.C., USA.

[43] Novark, G., Berger, E. (editors), 2010. DieHarder: Securing the heap. Proceedings of the 17th ACM Conference on Computer and Communications Security; 2010 Oct 4-8; Chicago, Illinois, USA. New York: Association for Computing Machinery. p. 573-584. DOI: https://doi.org/10.1145/1866307.1866371

[44] Sayeed, S., Marco-Gisbert, H., Ripoll, I., et al., 2019. Control-flow integrity: Attacks and protections. Applied Sciences. 9(20), 4229. DOI: https://doi.org/10.3390/app9204229

[45] Andriesse, D., Bos, H., Slowinska, A. (editors), 2015. Parallax: Implicit code integrity verification using return-oriented programming. 2015 45th Annual IEEE/IFIP International Conference on Dependable Systems and Networks; 2015 Jun 22-25; Rio de Janeiro, Brazil. New York: IEEE. p. 125-135. DOI: https://doi.org/10.1109/DSN.2015.12

[46] Mukkamala, S., Janoski, G., Sung, A. (editors), 2002. Intrusion detection using neural networks and support vector machines. Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN' 02 (Cat. No. 02CH37290); 2002 May 12-17; Honolulu, HI, USA. New York: IEEE. p. 1702-1707. DOI: https://doi.org/10.1109/IJCNN.2002.1007774

[47] Thottan, M., Ji, C., 2003. Anomaly detection in IP networks. IEEE Transactions on Signal Processing. 51(8), 2191-2204. DOI: https://doi.org/10.1109/TSP.2003.814797

[48] Cova, M., Felmetsger, V., Banks, G., et al. (editors), 2006. Static detection of vulnerabilities in x86 executables. 2006 22nd Annual Computer Security Applications Conference (ACSAC' 06); 2006 Dec 11-15; Miami Beach, FL, USA. New York: IEEE. p. 269-278. DOI: https://doi.org/10.1109/ACSAC.2006.50

[49] Lanzi, A., Martignoni, L., Monga, M., et al. (editors), 2007. A smart fuzzer for x86 executables. Third International Workshop on Software Engineering for Secure Systems (SESS' 07: ICSE Workshops 2007); 2007 May 20-26; Minneapolis, MN, USA. New York: IEEE. DOI: https://doi.org/10.1109/SESS.2007.1

[50] Miller, B.P., Cooksey, G., Moore, F., 2007. An empirical study of the robustness of MacOS applications using random testing. Operating Systems Review. 41(1), 78-86. DOI: https://doi.org/10.1145/1228291.1228308

[51] SonarQube Homepage [Internet]. Available from: https://www.sonarqube.org/

Downloads

How to Cite

Özger, O., & Öztekİn, H. (2023). Detection of Buffer Overflow Attacks with Memoization-based Rule Set. Journal of Computer Science Research, 5(4), 13–26. https://doi.org/10.30564/jcsr.v5i4.6044

Issue

Vol. 5 , Iss. 4 (October 2023)

Article Type

Article

License

Copyright © 2023 Author(s)

Creative Commons License

This is an open access article under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) License.

Crossref
Scopus
Google Scholar
Europe PMC