Data Analytics of an Information System Based on a Markov Decision Process and a Partially Observable Markov Decision Process

Authors

  • Lidong Wang Institute for Systems Engineering Research, Mississippi State University, Vicksburg, MS 39180, USA
  • Reed L. Mosher Institute for Systems Engineering Research, Mississippi State University, Vicksburg, MS 39180, USA
  • Terril C. Falls Institute for Systems Engineering Research, Mississippi State University, Vicksburg, MS 39180, USA
  • Patti Duett Institute for Systems Engineering Research, Mississippi State University, Vicksburg, MS 39180, USA

DOI:

https://doi.org/10.30564/jcsr.v5i1.5434

Abstract

Data analytics of an information system is conducted based on a Markov decision process (MDP) and a partially observable Markov decision process (POMDP) in this paper. Data analytics over a finite planning horizon and an infinite planning horizon for a discounted MDP is performed, respectively. Value iteration (VI), policy iteration (PI), and Q-learning are utilized in the data analytics for a discounted MDP over an infinite planning horizon to evaluate the validity of the MDP model. The optimal policy to minimize the total expected cost of states of the information system is obtained based on the MDP. In the analytics for a discounted POMDP over an infinite planning horizon of the information system, the effects of various parameters on the total expected cost of the information system are studied.

Keywords:

Predictive modelling; Information system; MDP; POMDP; Cybersecurity; Q-learning

References

[1] AlSadhan, T., Park, J.S. (editors), 2021. Leveraging information security continuous monitoring to enhance cybersecurity. 2021 International Conference on Computational Science and Computational Intelligence (CSCI); 2021 Dec 15-17; Las Vegas, NV, USA. USA: IEEE. p. 753-759.

[2] United States Cyber Command, 2018. Achieve and Maintain Cyberspace Superiority, Command Vision for U.S. Cyber Command [Internet]. Available from: https://www.cybercom.mil/Portals/56/Documents/USCYBERCOM%20Vision%20April%202018.pdf?ver=2018-06-14-152556-010

[3] Wendt, D., 2019. Addressing both sides of the cybersecurity equation. Journal of the Cyber Security & Information Systems Information Analysis Center. 7(2).

[4] Anuar, N.B., Sallehudin, H., Gani, A., et al., 2008. Identifying false alarm for network intrusion detection system using hybrid data mining and decision tree. Malaysian Journal of Computer Science. 21(2), 101-115.

[5] Kukielka, P., Kotulski, Z. (editors), 2008. Analysis of different architectures of neural networks for application in intrusion detection systems. 2008 International Multiconference on Computer Science and Information Technology; 2008 Oct 20-22; Wisla, Poland. USA: IEEE. p. 807-811.

[6] Faisal, M.A., Aung, Z., Williams, J.R., et al., 2012. Securing advanced metering infrastructure using intrusion detection system with data stream mining. In: Chau, M., Wang, G.A., Yue, W.T., et al. (editors), intelligence and security informatics. PAISI 2012. Lecture Notes in Computer Science. Springer, Berlin: Heidelberg. pp. 96-111. DOI: https://doi.org/10.1007/978-3-642-30428-6_8

[7] Raiyn, J., 2014. A survey of cyber attack detection strategies. International Journal of Security and Its Applications. 8(1), 247-256.

[8] Singh, J., Nene, M.J., 2013. A survey on machine learning techniques for intrusion detection systems. International Journal of Advanced Research in Computer and Communication Engineering. 2(11), 4349-4355.

[9] Cardenas, A.A., Manadhata, P.K., Rajan, S.P., 2013. Big data analytics for security. IEEE Security & Privacy. 11(6), 74-76.

[10] Wiafe, I., Koranteng, F.N., Obeng, E.N., et al., 2020. Artificial intelligence for cybersecurity: A systematic mapping of literature. IEEE Access. 8, 146598-146612.

[11] Bao, N., Musacchio, J., 2009. Optimizing the decision to expel attackers from an information system. 2009 47th Annual Allerton Conference on Communication, Control, and Computing (Allerton); 2009 Sep 30-Oct 2; Monticello, IL, USA. USA: IEEE. p. 644-651.

[12] Mohri, M., Rostamizadeh, A., Talwalkar, A., 2012. Foundations of machine learning. Adaptive computation and machine learning. MIT Press: USA.

[13] Alsheikh, M.A., Hoang, D.T., Niyato, D., et al., 2015. Markov decision processes with applications in wireless sensor networks: A survey. IEEE Communications Surveys & Tutorials. 17(3), 1239-1267.

[14] Chen, Y., Hong, J., Liu, C.C., 2018. Modeling of intrusion and defense for assessment of cyber security at power substations. IEEE Transactions on Smart Grid. 9(4), 2541-2552.

[15] van Otterlo, M., Wiering, M., 2012. Reinforcement learning and Markov decision processes. Reinforcement Learning. Springer, Berlin: Heidelberg. pp. 3-42.

[16] Sutton R.S., Barto, A.G., 2018. Reinforcement learning: An introduction. MIT press: USA.

[17] Zanini, E., 2014. Markov Decision Processes [Internet]. Available from: https://web.archive.org/web/20170812131743id_/http://www.lancs.ac.uk/~zaninie/MDP.pdf

[18] Liu, D., Khoukhi, L., Hafid, A. (editors), 2017. Data offloading in mobile cloud computing: A Markov decision process approach. 2017 IEEE International Conference on Communications (ICC); 2017 May 21-25; Paris, France. USA: IEEE. p. 1-6.

[19] Xiang, X., Foo, S., 2021. Recent advances in deep reinforcement learning applications for solving partially observable Markov decision processes (POMDP) problems: Part 1—fundamentals and applications in games, robotics and natural language processing. Machine Learning and Knowledge Extraction. 3(3), 554-581.

[20] Russell, S.J., Norvig, P., 2021. Artificial intelligence a modern approach, 4th edition. Pearson Education, Inc: UK.

[21] Kurniawati, H., Hsu, D., Lee, W.S., 2008. Sarsop: Efficient point-based pomdp planning by approximating optimally reachable belief spaces. Robotics: Science and systems. MIT Press: USA. pp. 65-72.

[22] Cassandra, A.R., Kaelbling, L.P., Littman, M.L., 1994. Acting optimally in partially observable stochastic domains. Aaai. 94, 1023-1028.

[23] Kamalzadeh, H., Hahsler, M., 2019. POMDP: Introduction to Partially Observable Markov Decision Processes.

Downloads

How to Cite

Wang, L., Mosher, R. L., Falls, T. C., & Duett, P. (2023). Data Analytics of an Information System Based on a Markov Decision Process and a Partially Observable Markov Decision Process. Journal of Computer Science Research, 5(1), 21–30. https://doi.org/10.30564/jcsr.v5i1.5434

Issue

Vol. 5 , Iss. 1 (January 2023)

Article Type

Article

License

Copyright © 2023 Lidong Wang, Reed L. Mosher, Terril C. Falls, Patti Duett

Creative Commons License

This is an open access article under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) License.

Crossref
Scopus
Google Scholar
Europe PMC