Celebrating 11 Editorial Board Members of Journal of Electronic & Information Systems Recognized in the World's Top 2% Scientists 2025!
2025-09-23
Sentiment Analysis of Short and Incomplete Texts Using Transformer-Based Attention Models
Authors
-
Reza Nouralizadeh Ganji
Artificial Intelligence Department, Faculty of Computer Engineering, K. N. Toosi University of Technology, Tehran 16317-14191, Iran
-
Nasim Tohidi
Department of Computer Science, Wayne State University, Detroit, MI 48202, USA
DOI:
https://doi.org/10.30564/jeis.v7i2.12431Abstract
The sentiment is seen as valuable information that can represent people's opinions, and its analysis is regarded as an essential component of decision-making processes. Social media has led to an exponential increase in the volume of shared textual content, and natural language processing as a potential area provides a variety of cutting-edge, deep learning-based models for analyzing and understanding this content. However, short, incomplete, and noisy text containing misspellings and grammatical errors is prevalent in these postings, making sentiment analysis challenging. This paper proposes a sentiment polarity detection approach using a three-phased methodology for short and incomplete text. First, the model utilizes transformer-based mechanisms to automatically correct and complete texts, thereby eliminating the need for manual human annotation. In the second phase, denoising neural networks are learned to reconstruct representations of short and incomplete texts. In the third phase, outputs and trained weights from previous steps are used to predict the sentiment polarity of input text by applying the attention mechanism, convolution nets, and pooling layers. Experimental evaluations demonstrate that the proposed approach outperformed state-of-the-art models in sentiment classification. It achieved an F1 score of 89.96% on the Sentiment 140 dataset and 76.91% on the ACL 14 dataset, demonstrating that it excels in precise correction, effective learning, and accurate prediction.
Keywords:
Sentiment Analysis; Natural Language Processing; Incomplete Texts; Transformers; Text; Deep Learning; Social Media; Attention MechanismReferences
[1] Zheng, C., 2025. A Comprehensive Review of Probabilistic and Statistical Methods in Social Network Sentiment Analysis. Advances in Engineering Innovation. 16(3), 38–43. DOI: https://doi.org/10.54254/2977-3903/2025.21918
[2] Tohidi, N., Beheshti, M., 2024. Enhanced Explanations in Recommendation Systems. In Proceedings of the 2024 IEEE International Symposium on Systems Engineering (ISSE), Perugia, Italy, 16–18 October 2024; pp. 1–5. DOI: https://doi.org/10.1109/ISSE63315.2024.10741116
[3] Ghezelji, M., Dadkhah, C., Tohidi, N., et al., 2022. Personality-Based Matrix Factorization for Personalization in Recommender Systems. International Journal of Information & Commununication Technology Research. 14(1), 48–55. DOI: https://doi.org/10.52547/itrc.14.1.48
[4] Liu, B., 2020. Sentiment Analysis: Mining Opinions, Sentiments, and Emotions. Cambridge University Press: Cambridge, UK.
[5] Shaik, T., Tao, X., Dann, C., et al., 2023. Sentiment Analysis and Opinion Mining on Educational Data: A Survey. Natural Language Processing Journal. 2, 100003. DOI: https://doi.org/10.1016/J.NLP.2022.100003
[6] Ganji, R.N., Dadkhah, C., Tohidi, N., 2023. Improving Sentiment Classification for Hotel Recommender System through Deep Learning and Data Balancing. Computación y Sistemas. 27(3), 811–825. DOI: https://doi.org/10.13053/cys-27-3-4655
[7] Tang, J., Chang, Y., Liu, H., 2014. Mining Social Media with Social Theories: A Survey. ACM SIGKDD Explorations Newsletter. 15(2), 20–29. DOI: https://doi.org/10.1145/2641190.2641195
[8] Dong, L., Wei, F., Tan, C., et al., 2014. Adaptive Recursive Neural Network for Target-Dependent Twitter Sentiment Classification. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, Baltimore, MD, USA, 22–27 June 2014; pp. 49–54. DOI: https://doi.org/http://dx.doi.org/10.3115/v1/P14-2009
[9] Cunha Sergio, G., Lee, M., 2021. Stacked DeBERT: All Attention in Incomplete Data for Text Classification. Neural Networks. 136, 87–96. DOI: https://doi.org/10.1016/j.neunet.2020.12.018
[10] Vaswani, A., Shazeer, N., Parmar, N., et al., 2017. Attention Is All You Need. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, 4–7 December 2017; pp. 6000–6010. Available from: https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
[11] Devlin, J., Chang, M.-W., Lee, K., et al., 2019. BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA, 2–7 June 2019; pp. 4171–4186. DOI: https://doi.org/10.18653/v1/N19-1423
[12] Liu, Y., Ott, M., Goyal, N., et al., 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint. arXiv:1907.11692. DOI: https://doi.org/10.48550/arxiv.1907.11692
[13] Shetty, S.M., Pushpa, D., 2024. An Overview of the Literature on Sentiment Analysis Methods for Online and Social Media Platforms. In Proceedings of the 12th International Conference on Intelligent Systems and Embedded Design (ISED 2024), Rourkela, India, 20–22 December 2024; pp. 1–5. DOI: https://doi.org/10.1109/ISED63599.2024.10956498
[14] Alnasrawi, A.M., Alzubaidi, A.M.N., Al-Moadhen, A.A., 2024. Improving Sentiment Analysis Using Text Network Features within Different Machine Learning Algorithms. Bulletin of Electrical Engineering and Informatics. 13(1), 405–412. DOI: https://doi.org/10.11591/EEI.V13I1.5576
[15] Liu, Y., He, H., Han, T., et al., 2025. Understanding LLMs: A Comprehensive Overview from Training to Inference. Neurocomputing. 620, 129190. DOI: https://doi.org/10.1016/J.NEUCOM.2024.129190
[16] Zhang, H., Shafiq, M.O., 2024. Survey of Transformers and Towards Ensemble Learning Using Transformers for Natural Language Processing. Journal of Big Data. 11(1), 25. DOI: https://doi.org/10.1186/s40537-023-00842-0
[17] Shuang, K., Gu, M., Li, R., et al., 2021. Interactive POS-Aware Network for Aspect-Level Sentiment Classification. Neurocomputing. 420, 181–96. DOI: https://doi.org/10.1016/j.neucom.2020.08.013
[18] Liu, N., Shen, B., 2020. ReMemNN: A Novel Memory Neural Network for Powerful Interaction in Aspect-Based Sentiment Analysis. Neurocomputing. 395, 66–77. DOI: https://doi.org/10.1016/j.neucom.2020.02.018
[19] Naseem, U., Razzak, I., Musial, K., et al., 2020. Transformer Based Deep Intelligent Contextual Embedding for Twitter Sentiment Analysis. Future Generation Computer Systems. 113, 58–69. DOI: https://doi.org/10.1016/j.future.2020.06.050
[20] Zhou, J., Huang, J.X., Hu, Q.V., et al., 2020. SK-GCN: Modeling Syntax and Knowledge via Graph Convolutional Network for Aspect-Level Sentiment Classification. Knowledge-Based Systems. 205, 106292. DOI: https://doi.org/10.1016/j.knosys.2020.106292
[21] Liao, W., Zeng, B., Yin, X., et al., 2020. An Improved Aspect-Category Sentiment Analysis Model for Text Sentiment Analysis Based on RoBERTa. Applied Intelligence. 51, 3522–3533. DOI: https://doi.org/10.1007/S10489-020-01964-1
[22] Hochreiter, S., Schmidhuber, J., 1997. Long Short-Term Memory. Neural Computation. 9(8), 1735–1780. DOI: https://doi.org/10.1162/NECO.1997.9.8.1735
[23] Chung, J., Gulcehre, C., Cho, K., et al., 2014. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. arXiv preprint. arXiv:1412.3555. DOI: https://doi.org/10.48550/arXiv.1412.3555
[24] Basiri, M.E., Nemati, S., Abdar, M., et al., 2021. ABCDM: An Attention-Based Bidirectional CNN-RNN Deep Model for Sentiment Analysis. Future Generation Computer Systems. 115, 279–94. DOI: https://doi.org/10.1016/j.future.2020.08.005
[25] Song, Y., Wang, J., Liang, Z., et al., 2020. Utilizing BERT Intermediate Layers for Aspect Based Sentiment Analysis and Natural Language Inference. arXiv preprint. arXiv:2002.04815. DOI: https://doi.org/10.48550/arxiv.2002.04815
[26] Mutinda, J., Mwangi, W., Okeyo, G., 2023. Sentiment Analysis of Text Reviews Using Lexicon-Enhanced Bert Embedding (LeBERT) Model with Convolutional Neural Network. Applied Sciences. 13(3), 1445. DOI: https://doi.org/10.3390/APP13031445
[27] Mirugwe, A., Ashaba, C., Namale, A., et al., 2024. Sentiment Analysis of Social Media Data on Ebola Outbreak Using Deep Learning Classifiers. Life. 14(6), 708. DOI: https://doi.org/10.3390/LIFE14060708
[28] Jayanthi, S.M., Pruthi, D., Neubig, G., 2020. NeuSpell: A Neural Spelling Correction Toolkit. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, online, 16–20 November 2020; pp. 158–164. DOI: https://doi.org/10.18653/V1/2020.EMNLP-DEMOS.21
[29] Xue, L., Barua, A., Constant, N., et al., 2022. ByT5: Towards a Token-Free Future with Pre-Trained Byte-to-Byte Models. Transactions of the Association for Computational Linguistics. 10, 291–306. DOI: https://doi.org/10.1162/tacl_a_00461
[30] Raffel, C., Shazeer, N., Roberts, A., et al., 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research. 21(140), 1–67. Available from: https://jmlr.org/papers/v21/20-074.html
[31] Sokolova, M., Lapalme, G., 2009. A Systematic Analysis of Performance Measures for Classification Tasks. Information Processing & Management. 45(4), 427–437. DOI: https://doi.org/10.1016/J.IPM.2009.03.002
[32] Shridhar, K., Dash, A., Sahu, A., et al., 2019. Subword Semantic Hashing for Intent Classification on Small Datasets. In Proceedings of the 2019 International Joint Conference on Neural Networks (IJCNN), Budapest, Hungary, 14–19 July 2019; pp. 1–6. DOI: https://doi.org/10.1109/IJCNN.2019.8852420
[33] Tang, D., Qin, B., Feng, X., et al., 2016. Effective LSTMs for Target-Dependent Sentiment Classification. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, Osaka, Japan, 11–16 December 2016; pp. 3298–3307. DOI: https://aclanthology.org/C16-1311/
[34] Ma, D., Li, S., Zhang, X., et al., 2017. Interactive Attention Networks for Aspect-Level Sentiment Classification. In Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI’17), Melbourne, Australia; pp. 4068–4074. Available from: https://dl.acm.org/doi/10.5555/3171837.3171854
[35] Sun, J., Han, P., Cheng, Z., et al., 2020. Transformer Based Multi-Grained Attention Network for Aspect-Based Sentiment Analysis. IEEE Access. 8, 211152–211163. DOI: https://doi.org/10.1109/ACCESS.2020.3039470
[36] Zhao, G., Luo, Y., Chen, Q., et al., 2023. Aspect-Based Sentiment Analysis via Multitask Learning for Online Reviews. Knowledge-Based Systems. 264, 110326. DOI: https://doi.org/10.1016/J.KNOSYS.2023.110326
[37] Grattafiori, A., Dubey, A., Jauhri, A., et al., 2024. The Llama 3 Herd of Models. arXiv preprint. arXiv:2407.21783. DOI: https://doi.org/10.48550/arXiv.2407.21783
Downloads
How to Cite
Nouralizadeh Ganji, R., & Tohidi, N. (2025). Sentiment Analysis of Short and Incomplete Texts Using Transformer-Based Attention Models. Journal of Electronic & Information Systems, 7(2), 132–153. https://doi.org/10.30564/jeis.v7i2.12431
Issue
Article Type
ARTICLE
License
Copyright © 2025 Reza Nouralizadeh Ganji, Nasim Tohidi
This is an open access article under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) License.
