Efficacy of AI-Generated Feedback by SmallTalk2Me for Improving Speaking Skill of Saudi EFL Learners

Authors

  • Shahzad ul Hassan Farooqi

    Department of English Language, College of Education, Majmaah University, Majmaah 11952, Saudi Arabia

DOI:

https://doi.org/10.30564/fls.v7i3.8294
Received: 13 December 2024 | Revised: 20 January 2025 | Accepted: 23 January 2025 | Published Online: 14 March 2025

Abstract

Feedback is essential in language teaching and learning as it helps learners identify errors, reinforce correct usage, and guide their progress. Traditionally it was the job of a teacher, however, with the advent of Artificial Intelligence (AI) this task has become automated. This study evaluated the efficacy of AI-generated feedback provided by the SmallTalk2Me platform for improving speaking skill among Saudi EFL learners. Forty-four Saudi EFL learners participated in this quasi-experimental study. The data were derived from the IELTS speaking pre-test and post-test, while in between, the participants completed six practice activities over a period of three weeks (two activities each week). Two data sets were compiled: IELTS speaking band scores and proficiency (measured by words spoken per minute). The data were analyzed using paired t-tests to evaluate statistical significance. The study hypothesized that learners would achieve a higher mean score in the IELTS speaking post-test after completing language practices on SmallTalk2Me. This hypothesis was statistically supported, as the difference between the pre-test and post-test mean scores was significant (p < 0.05). The learners demonstrated a 12.12% improvement in their IELTS speaking test band scores, while an 11.18% increase was observed in their speaking proficiency between the first and the sixth practice activities. Following the intervention, a 25-item questionnaire was administered to evaluate the participants’ perception of AI-generated feedback, the efficacy of SmallTalk2Me, and the overall learning experience. Students found AI-generated feedback effective, motivating, and engaging. The study highlights the efficacy of AI-generated feedback provided by SmallTalk2Me and recommends the use of such AI platforms in the Saudi EFL scenario.

Keywords:

Artificial Intelligence; Feedback; IELTS; Saudi EFL Learners; Speaking Skill; SmallTalk2Me

References

[1] Brown, H.D., 2014. Principles of language learning and teaching: A course in second language acquisition (6th ed.). Routledge: London, UK.

[2] Li, S., 2013. The interactions between the effects of implicit and explicit feedback and individual differences in language analytic ability and working memory. The Modern Language Journal. 97(3), 634–654. DOI: https://doi.org/10.1111/j.1540-4781.2013.12030.x

[3] Shi, H., Aryadoust, V., 2024. A systematic review of AI-based automated written feedback research. ReCALL. 36(2), 187–209. DOI: https://doi.org/10.1017/S0958344023000265

[4] Russell, S., Norvig, P., 2016. Artificial intelligence: A modern approach (3rd ed.). Pearson Education: London, UK.

[5] Zou, B., Du, Y., Wang, Z., et al., 2023. An investigation into artificial intelligence speech evaluation programs with automatic feedback for developing EFL learners' speaking skills. SAGE Open. 13(3). DOI: https://doi.org/10.1177/21582440231193818

[6] Lehman, B., Gu, L., Zhao, J., et al., 2020. Use of adaptive feedback in an app for English language spontaneous speech. In: Bittencourt, I.I., Cukurova, M., Muldner, K., et al., (Eds.). Artificial intelligence in education. Springer International Publishing. Volume 12163, pp. 309–320. DOI: https://doi.org/10.1007/978-3-030-52237-7_25

[7] Kannan, J., Munday, P., 2018. New trends in second language learning and teaching through the lens of ICT, networked learning, and artificial intelligence. Círculo de Linguística Aplicada a La Comunicación. 76, 13–30. DOI: https://doi.org/10.5209/CLAC.62495

[8] Natale, S., Cooke, H., 2021. Browsing with Alexa: Interrogating the impact of voice assistants as web interfaces. Media, Culture & Society. 43(6), 1000–1016. DOI: https://doi.org/10.1177/0163443720983295

[9] Dai, Y., Wu, Z., 2023. Mobile-assisted pronunciation learning with feedback from peers and/or automatic speech recognition: A mixed-methods study. Computer Assisted Language Learning. 36(5–6), 861–884. DOI: https://doi.org/10.1080/09588221.2021.1952272

[10] Dizon, G., 2020. Evaluating intelligent personal assistants for L2 listening and speaking development. Language Learning & Technology. 24(1), 16–26. DOI: https://doi.org/10125/44705

[11] Wang, Y.-H., Young, S.S.-C., 2015. Effectiveness of feedback for enhancing English pronunciation in an ASR-based CALL system: The effectiveness of corrective feedback. Journal of Computer Assisted Learning. 31(6), 493–504.

[12] Xiao, W., Park, M., 2021. Using automatic speech recognition to facilitate English pronunciation assessment and learning in an EFL context: Pronunciation error diagnosis and pedagogical implications. International Journal of Computer-Assisted Language Learning and Teaching. 11(3), 74–91.

[13] Ding, S., Liberatore, C., Sonsaat, S., et al., 2019. Golden speaker builder – An interactive tool for pronunciation training. Speech Communication. 115, 51–66. DOI: https://doi.org/10.1016/j.specom.2019.10.005

[14] Kan, M.S., Ito, A., 2020. Language cognition and pronunciation training using applications. Future Internet. 12(3), 42. DOI: https://doi.org/10.3390/fi12030042

[15] Tai, T.-Y., Chen, H.H.-J., 2023. The impact of Google Assistant on adolescent EFL learners' willingness to communicate. Interactive Learning Environments. 31(3), 1485–1502.

[16] Bashori, M., van Hout, R., Strik, H., et al., 2022. Web-based language learning and speaking anxiety. Computer Assisted Language Learning. 35(5–6), 1058–1089. DOI: https://doi.org/10.1080/09588221.2020.1770293

[17] Al-Harbi, A., 2014. Language learning anxiety among Saudi EFL students: A case study. Journal of Language and Literature. 5(1), 20–29. DOI: https://doi.org/10.1007/s10669-014-9485-0

[18] Al-Hazmi, S., Scholfield, P., 2007. The difficulties of learning English as a foreign language in Saudi Arabia. The Journal of Language and Linguistic Studies. 3(1), 3–16.

[19] Al-Sibai, D., 2004. Difficulties in speaking English as a foreign language: A case study of Saudi EFL learners. Journal of Language Teaching and Research. 5(3), 123–135. DOI: https://doi.org/10.4304/jltr.5.3.123-135

[20] SmallTalk2Me, 2024. SmallTalk2Me: AI-powered simulator to improve spoken English. Available from: https://smalltalk2.me/ (accessed 1 October 2024).

[21] Manggiasih, L.A., Loreana, Y.R., Azizah, A., et al., 2023. Strengths and limitations of SmallTalk2Me app in English language proficiency evaluation. TELL: Teaching of English Language and Literature Journal. 11(2), 134. DOI: https://doi.org/10.30651/tell.v11i2.19560

[22] Macías, A.M.D., Armijos Solano, D.O., Palma Perero, L.M., et al., 2024. The potential of artificial intelligence to improve speaking skills in a second language (English) fluently. Ciencia Latina Revista Científica Multidisciplinar. 8(3), 3826–3836. DOI: https://doi.org/10.37811/cl_rcm.v8i3.11592

[23] García, A., López, M., Martínez, J., 2023. Improving speaking skills through conversational practice with chatbots. Journal of Language Education and Technology. 10(2), 45–58.

[24] Li, M., Chen, J., 2017. The impact of AI-based pronunciation tutoring systems on oral proficiency development. Language Learning & Technology. 21(3), 124–142.

[25] Smith, R., Johnson, L., 2022. Enhancing pronunciation with voice recognition systems. International Journal of Language Learning. 8(1), 112–125.

[26] Zhang, L., Wang, X., 2018. Exploring the use of AI-powered virtual language assistants for speaking practice. Computer-Assisted Language Learning. 31(4), 357–376.

[27] Loewen, S., Crowther, D., Isbell, D.R., et al., 2019. Mobile-assisted language learning: A Duolingo case study. ReCALL. 31(3), 293–311. DOI: https://doi.org/10.1017/S0958344019000065

[28] Loewen, S., Isbell, D.R., Sporn, Z., 2020. The effectiveness of app-based language instruction for developing receptive linguistic knowledge and oral communicative ability. Foreign Language Annals. 53(2), 209–233. DOI: https://doi.org/10.1111/flan.12454

[29] Wu, Y., Liu, Q., 2016. Integrating AI-driven speech recognition technology into English language teaching: A case study. Educational Technology Research and Development. 64(4), 757–776.

[30] Warman, L.a.D., Erlinda, S., Tashid, T., et al., 2023. Empowering introvert students: How artificial intelligence applications enhance speaking ability. Al-Ishlah: Jurnal Pendidikan. 15(4), 4801–4813. DOI: https://doi.org/10.35445/alishlah.v15i4.4435

[31] Yin, X., Wei, Y., 2023. Research on the construction and application of an intelligent learning system to enhance college English listening and speaking. Journal of Education and Teaching. 1(4), 39–45. DOI: https://doi.org/10.59825/jet.2023.1.4.39

[32] Hoang, N.T., Han, D.N., Le, D.H., 2023. Exploring chatbot AI in improving vocational students' English pronunciation. AsiaCALL Online Journal. 14(2), 140–155. DOI: https://doi.org/10.54855/acoj.231429

[33] Zou, B., Liviero, S., Hao, M., et al., 2020. Artificial intelligence technology for EAP speaking skills: Student perceptions of opportunities and challenges. In: Springer eBooks. pp. 433–463. DOI: https://doi.org/10.1007/978-3-030-34212-8_17

[34] Makhlouf, M.K.I., 2021. Effect of artificial intelligence-based application on Saudi preparatory-year students' EFL speaking skills at Albaha University. International Journal of English Language Education. 9(2), 36. DOI: https://doi.org/10.5296/ijele.v9i2.18782

[35] Alsadoon, R., 2021. ‘Chatting with AI Bot: Vocabulary Learning Assistant for Saudi EFL Learners'. Canadian Center of Science and Education. 6(14), 1–23.

[36] Alhalangy, A.G.I., AbdAlgane, M., 2023. Exploring the impact of AI on the EFL context: A case study of Saudi universities. Journal of Intercultural Communication. 23(2), 41–49. DOI: https://doi.org/10.36923/jicc.v23i2.125

[37] Alsaif, O., 2024. The role of artificial intelligence (AI) in developing English language skills in the Saudi EFL context: An analytical study. International Journal of Educational Sciences and Arts. 3(5), 1–10. DOI: https://doi.org/10.59992/IJESA.2024.v3n5p1

[38] Burston, J., 2015. Twenty years of MALL project implementation: A meta-analysis of learning outcomes. ReCALL. 27(1), 4–20. DOI: https://doi.org/10.1017/S0958344014000159

[39] Zhang, R., Zou, D., 2022. Types, purposes, and effectiveness of state-of-the-art technologies for second and foreign language learning. Computer Assisted Language Learning. 35(4), 696–742.

[40] Gao, R., Merzdorf, H.E., Anwar, S., et al., 023. Automatic assessment of text-based responses in post-secondary education: A systematic review. arXiv. DOI: https://arxiv.org/abs/2308.16151

[41] Martens, B.K., Witt, J.C., Elliott, S.N., et al., 1985. Teacher judgments concerning the acceptability of school-based interventions. Professional Psychology: Research and Practice. 16, 191–198. DOI: https://doi.org/10.1037/0735-7028.16.2.191

[42] Petrić, B., Czárl, B., 2003. Validating a writing strategy questionnaire. System. 31(2), 187–215. DOI: https://doi.org/10.1016/S0346-251X(03)00020-4

[43] Raoofi, S., Miri, A., Gharibi, J., et al., 2017. Assessing and validating a writing strategy scale for undergraduate students. Journal of Language Teaching and Research. 8. DOI: https://doi.org/10.17507/jltr.0803.23

[44] Zohrabi, M., 2013. Mixed method research: Instruments, validity, reliability and reporting findings. Theory and Practice in Language Studies. 3(2), 254–262. DOI: https://doi.org/10.4304/tpls.3.2.254-262

Downloads

How to Cite

Farooqi, S. ul H. (2025). Efficacy of AI-Generated Feedback by SmallTalk2Me for Improving Speaking Skill of Saudi EFL Learners. Forum for Linguistic Studies, 7(3), 714–728. https://doi.org/10.30564/fls.v7i3.8294

Issue

Vol. 7 , Iss. 3 (March 2025)

Article Type

Article

License

Copyright © 2025 Shahzad ul Hassan Farooqi

Creative Commons License

This is an open access article under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) License.