Can ESL Instructors Spot Machine Translation? Evidence from Arabic-English Classroom

Authors

  • Anas Alkhofi

    Department of English, College of Arts, King Faisal University, Hofuf 36362, Saudi Arabia

DOI:

https://doi.org/10.30564/fls.v7i3.8652
Received: 4 February 2025 | Revised: 20 February 2025 | Accepted: 24 February 2025 | Published Online: 2 March 2025

Abstract

This study investigates the ability of ESL instructors to differentiate between AI machine-generated and student-generated translations and assesses their confidence in doing so. Twenty instructors evaluated 44 translations (22 student-generated, 22 machine-generated), classifying each as either machine- or student-produced. In total, 434 valid responses were analyzed using a Generalized Linear Mixed Model (GLMM) in R. The responses were coded as 0 (incorrect/unsure) and 1 (correct) to determine whether instructors’ correct identification of machine-generated translations was statistically significant. The results revealed a low identification rate, with instructors having only a 28% probability of correctly distinguishing machine translations, which was significantly below the 50% expected by random chance. Despite this low success rate, 90% of instructors expressed confidence in their ability to detect machine translation. The findings suggest a gap between instructors’ perceived and actual abilities, indicating that reliance on instructors’ judgments may no longer be sufficient for detecting machine translation use. Moreover, instructors may need to prioritize in-class assessments, especially when the focus is on tapping into students’ raw abilities, while take-home assignments should incorporate active post-editing to foster critical engagement with machine translation. These adjustments are crucial for maintaining academic integrity, equipping students with the skills needed to navigate both human and machine-generated translations, and preparing them for the evolving translation marketplace.

Keywords:

AI in TESOL; Machine Translation in Teaching English; Google Translate in the Classroom; English-Arabic Translation

References

[1] Wang, H.F., Wu, H., He, Z.J., et al., 2022. Progress in Machine Translation. Engineering. 18(2), 143–153. DOI: https://doi.org/10.1016/j.eng.2021.03.023

[2] Hassan, H., Aue, A., Chen, C., et al., 2018. Achieving Human Parity on Automatic Chinese to English News Translation. DOI: https://doi.org/10.48550/arXiv.1803.05567

[3] Zhang, J.J., Zong, C.Q., 2020. Neural machine translation: Challenges, progress and future. Sci China Technol Sci. 63, 2028–2050. DOI: https://doi.org/10.1007/s11431-020-1632-x

[4] Almusharraf, A., Bailey, D., 2023. Machine translation in language acquisition: A study on EFL students’ perceptions and practices in Saudi Arabia and South Korea. J Comput Assist Learn. 39(6), 1988–2003. DOI: https://doi.org/10.1111/jcal.12857

[5] Bin Dahmash, N., 2020. ‘I Can’t Live Without Google Translate’: A Close Look at the Use of Google Translate App by Second Language Learners in Saudi Arabia. Arab World English Journal. 11(3), 226–240. DOI: https://dx.doi.org/10.24093/awej/vol11no3.14

[6] Ducar, C., Schocket, D.H., 2018. Machine translation and the L2 classroom: Pedagogical solutions for making peace with Google translate. Foreign Lang Ann. 51(4), 779–795. DOI: https://doi.org/10.1111/flan.12366

[7] Kazemzadeh, A., Kashani, A.F., 2014. The effect of computer-assisted translation on L2 learners’ mastery of writing. Int J Res Stud Lang Learn. 3(3), 29–44. DOI: https://doi.org/10.5861/ijrsll.2013.396

[8] Avila-Chauvet, L., Mejía D., 2023. Can professors and students detect ChatGPT essays? Available from: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4373643 (cited 29 September 2024).

[9] Fleckenstein, J., Meyer, J., Jansen, T., et al., 2024. Do teachers spot AI? Evaluating the detectability of AI-generated texts among student essays. Comput Educ Artif Intell. 6(5), 100209. DOI: https://doi.org/10.1016/j.caeai.2024.100209

[10] Hostetter, A.B., Call, N., Frazier, G., et al., 2023. Student and Faculty Perceptions of Artificial Intelligence in Student Writing. DOI: https://doi.org/10.31234/osf.io/7dnk9

[11] Yamada, M., 2019. The impact of Google neural machine translation on post-editing by student translators. The Journal of Specialised Translation. 31(1), 87–106. Available from: https://www.researchgate.net/profile/Masaru-Yamada/publication/364994185_art_yamada_newpdf/data/63620fc037878b3e87755cf7/art-yamada-new.pdf (cited 19 August 2024).

[12] Elkhatat, A.M., Elsaid, K., Almeer, S., 2023. Evaluating the efficacy of AI content detection tools in differentiating between human and AI-generated text. Int J Educ Integr. 19(1), 1–16. DOI: https://doi.org/10.1007/s40979-023-00140-5

[13] Pan, W.H., Chok, M.J., Wong, J.L.S., et al., 2024. Assessing AI Detectors in Identifying AI-Generated Code: Implications for Education. Proceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET '24); April 14–April 20, 2024; Lisbon, Portugal. pp. 1–11.

[14] Weber-Wulff, D., Anohina-Naumeca, A., Bjelobaba, S., et al., 2023. Testing of detection tools for AI-generated text. Int J Educ Integr. 19(1), 26. DOI: https://doi.org/10.1007/s40979-023-00146-z

[15] Lee, S.-M., Briggs, N., 2021. Effects of using machine translation to mediate the revision process of Korean university students’ academic writing. ReCALL. 33(1), 18–33. DOI: https://doi.org/10.1017/S0958344020000191

[16] Ata, M., Debreli, E., 2021. Machine Translation in the Language Classroom: Turkish EFL Learners’ and Instructors’ Perceptions and Use. IAFOR J Educ. 9(4), 103–122. DOI: https://doi.org/10.22492/ije.9.4.06

[17] Jolley, J.R., Maimone, L., 2015. Free online machine translation: Use and perceptions by Spanish students and instructors. In: Moeller, A.J. (eds.). Learn languages, explore cultures, transform lives. Publisher: Johnson Litho Graphics of Eau Claire, Ltd., Eau Claire, WI, USA. pp. 181–200.

[18] Liu, K.L., Kwok, H.L., Liu, J.W., et al., 2022. Sustainability and influence of machine translation: perceptions and attitudes of translation instructors and learners in Hong Kong. Sustainability. 14(11), 6399. DOI: https://doi.org/10.3390/su14116399

[19] Aslan, E., 2023. Machine Translation: Perception of Translation and Interpreting Students in Turkey. Curr TRENDS Transl Teach Learn E. 10(1), 185–216. DOI: https://doi.org/10.51287/cttl20237

[20] Nygård, A., 2024. Testing teachers’ abilities to distinguish between translations produced by students, Google Translate and ChatGPT [Master Thesis]. Trondheim: Norwegian University of Science and Technology. pp. 1–53.

[21] Innes, A., 2019. Differentiating between machine translation and student translation: red flags and salient lexicogrammatical features. Lub Stud Mod Lang Lit. 43(4), 1–13. DOI: https://doi.org/10.17951/lsmll.2019.43.4.1-13

[22] Maimone, L., Jolley, J., 2023. Looks like google to me: Instructor ability to detect machine translation in L2 Spanish writing. Foreign Lang Ann. 56(3), 627–644. DOI: https://doi.org/10.1111/flan.12690

[23] Stapleton, P., Kin, B.L.K., 2019. Assessing the accuracy and teachers’ impressions of Google Translate: A study of primary L2 writers in Hong Kong. Engl Specif Purp. 56, 18–34. DOI: https://doi.org/10.1016/j.esp.2019.07.001

[24] Waltzer, T., Cox, R.L., Heyman, G.D., 2023. Testing the Ability of Teachers and Students to Differentiate between Essays Generated by ChatGPT and High School Students. Hum Behav Emerg Technol. 2023(1), 1–9. DOI: https://doi.org/10.1155/2023/1923981

[25] Läubli, S., Sennrich, R., Volk, M., 2018. Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing; October 31–November 4, 2018; Brussels, Belgium. pp. 4791–4796.

[26] Graham, Y., Federmann, C., Eskevich, M., et al., 2020. Assessing Human-Parity in Machine Translation on the Segment Level. In: Cohn, T., He, Y.L., Liu, Y. (eds.). Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics: Online. pp. 4199–4207.

[27] Anderson, D.D., 1995. Machine translation as a tool in second language learning. CALICO J. 13(1), 68–97. DOI: https://doi.org/10.1558/cj.v13i1.68-97

[28] Luton, L., 2003. If the Computer Did My Homework, How Come I Didn’t Get an “A"? Fr Rev. 76(4), 766–770.

[29] Steding, S., 2009. Machine Translation in the German Classroom: Detection, Reaction, Prevention. Die Unterrichtspraxis/Teaching Ger. 42(2), 178–189.

[30] Correa, M., 2014. Leaving the “peer” out of peer-editing: Online translators as a pedagogical tool in the Spanish as a second language classroom. Lat Am J Content Lang Integr Learn. 7(1), 1–20. DOI: https://doi.org/10.5294/laclil.2014.7.1.1

Downloads

How to Cite

Alkhofi, A. (2025). Can ESL Instructors Spot Machine Translation? Evidence from Arabic-English Classroom. Forum for Linguistic Studies, 7(3), 340–350. https://doi.org/10.30564/fls.v7i3.8652

Issue

Vol. 7 , Iss. 3 (March 2025)

Article Type

Article

License

Copyright © 2025 Anas Alkhofi

Creative Commons License

This is an open access article under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) License.