Does Student Engagement with Chatbots Enhance English Proficiency?

Tommy Hastomo; Andini Septama  Sari; Utami  Widiati; Francisca Maria  Ivone; Evynurul Laily  Zen; Muhammad Fikri Nugraha Kholid

doi:10.4312/elope.22.1.93-109

Authors

Tommy Hastomo State University of Malang (Universitas Negeri Malang)
Andini Septama Sari State University of Malang (Universitas Negeri Malang)
Utami Widiati State University of Malang (Universitas Negeri Malang)
Francisca Maria Ivone State University of Malang (Universitas Negeri Malang)
Evynurul Laily Zen State University of Malang (Universitas Negeri Malang)
Muhammad Fikri Nugraha Kholid Raden Intan State Islamic University Lampung (UIN Raden Intan Lampung)

DOI:

https://doi.org/10.4312/elope.22.1.93-109

Keywords:

AI, chatbots, cognitive engagement, emotional engagement, English proficieny, Indonesian university students

Abstract

This study examines how Indonesian university students’ engagement with chatbots influences their English proficiency. While AI tools are increasingly used in language education, little research focuses on chatbot interaction dynamics. The research assesses behavioural (active use), cognitive (perceived value), and emotional (attitudinal) engagement across 150 non-English majors at four proficiency levels (A1–B2). Data from engagement surveys and proficiency tests were analysed using ANOV A, correlation, and regression. Results indicated that higherproficiency students (B1/B2) engaged more intensely with chatbots than their lower-level peers. Behavioural and cognitive engagement strongly correlated with improved language skills, while emotional engagement showed no significant link. Regression analysis identified behavioural and cognitive engagement as key predictors of proficiency gains, suggesting that active interaction and perceived utility of chatbots drive language development. The findings underscore chatbots’ potential as effective language-learning aids.

Metrics

Metrics Loading ...

Downloads

Download data is not yet available.

References

Alahmari, Arwa, and Fakieh Alrabai. 2024. “The predictive role of L2 learners’ resilience in language classroom engagement.” Frontiers in Education 9. https://doi.org/10.3389/feduc.2024.1502420.

Al-Obaydi, Liqaa Habeb, Farzaneh Shakki, Ragad M. Tawafak, Marcel Pikhart, and Raed Latif Ugla. 2023. “What I know, what I want to know, what I learned: Activating EFL college students’ cognitive, behavioral, and emotional engagement through structured feedback in an online environment.” Frontiers in Psychology 13:1083673. https://doi.org/10.3389/fpsyg.2022.1083673.

Alsawaier, Raed S. 2018. “The effect of gamification on motivation and engagement.” The International Journal of Information and Learning Technology 35 (1): 56–79. https://doi.org/10.1108/IJILT-02-2017-0009.

Baskara, FX Risang. 2023. “Integrating ChatGPT into EFL writing instruction: Benefits and challenges.” International Journal of Education and Learning 5 (1): 44–55. https://doi.org/10.31763/ijele.v5i1.858.

Casal, J. Elliott, and Matt Kessler. 2023. “Can linguists distinguish between ChatGPT/AI and human writing?: A study of research ethics and academic publishing.” Research Methods in Applied Linguistics 2 (3): 100068. https://doi.org/10.1016/j.rmal.2023.100068.

Chang, Daniel H., Michael Pin Chuan Lin, Shiva Hajian, and Quincy Q. Wang. 2023. “Educational design principles of using AI chatbot that supports self-regulated learning in education: Goal setting, feedback, and personalization.” Sustainability 15 (17): 12921. https://doi.org/10.3390/su151712921.

Gayed, John Maurice, May Kristine Jonson Carlon, Angelu Mari Oriola, and Jeffrey S. Cross. 2022. “Exploring an AI-based writing assistant’s impact on English language learners.” Computers and Education: Artificial Intelligence 3:100055. https://doi.org/10.1016/J.CAEAI.2022.100055.

Guo, Kai, Yuchun Zhong, Danling Li, and Samuel Kai Wah Chu. 2023. “Investigating students’ engagement in chatbot-supported classroom debates.” Interactive Learning Environments 31 (6): 1–17. https://doi.org/10.1080/10494820.2023.2207181.

Hastomo, Tommy, Muhammad Fikri Nugraha Kholid, Pipit Muliyah, Linda Septiyana, and Widi Andewi. 2024. “Exploring how video conferencing impacts students’ cognitive, emotional, and behavioral engagement.” Journal of Educational Management and Instruction 4 (2): 213–25. https://doi.org/10.22515/jemin.v4i2.9335.

Hastomo, Tommy, Berlinda Mandasari, and Utami Widiati. 2024. “Scrutinizing Indonesian pre-service teachers’ technological knowledge in utilizing AI-powered tools.” Journal of Education and Learning (EduLearn) 18 (4): 1572–81. https://doi.org/10.11591/edulearn.v18i4.21644.

Hastomo, Tommy, and Linda Septiyana. 2022. “The investigation of students’ engagement in online class during pandemic COV ID-19.” Jurnal Penelitian Ilmu Pendidikan 15 (2). https://doi.org/10.21831/JPIPFIP.V15I2.49512.

Huang, Weijiao, Khe Foon Hew, and Luke K. Fryer. 2022. “Chatbots for language learning – Are they really useful? A systematic review of chatbot‐supported language learning.” Journal of Computer Assisted Learning 38 (1): 237–57. https://doi.org/10.1111/jcal.12610.

Karabiyik, Ceyhun. 2019. “The relationship between student engagement and tertiary level English language learners’ achievement.” International Online Journal of Education and Teaching 6 (2): 281–93. https://eric.ed.gov/?id=EJ1248494.

Kim, Hea Suk, Yoonjung Cha, and Na Young Kim. 2021. “Effects of AI chatbots on EFL students’ communication skills.” Korean Journal of English Language and Linguistics 21: 712–34. https://doi.org/10.15738/kjell.21.202108.712.

Kim, Susie. 2021. “Generalizability of CEFR criterial grammatical features in a Korean EFL corpus across A1, A2, B1, and B2 levels.” Language Assessment Quarterly 18 (3): 273–95. https://doi.org/10.1080/15434303.2020.1855647.

Logli, Chiara. 2016. “Higher education in Indonesia: Contemporary challenges in governance, access, and quality.” In The Palgrave Handbook of Asia Pacific Higher Education, edited by Christopher S. Collins, Molly N.N. Lee, John N. Hawkins and Deane E. Neubauer, 561–81. Palgrave Macmillan US. https://doi.org/10.1057/978-1-137-48739-1_37.

Mageira, Kleopatra, Dimitra Pittou, Andreas Papasalouros, Konstantinos Kotis, Paraskevi Zangogianni, and Athanasios Daradoumis. 2022. “Educational AI chatbots for content and language integrated learning.” Applied Sciences 12 (7): 3239. https://doi.org/10.3390/app12073239.

Moreira, Paulo A.S., Adelaide Dias, Carla Matias, Jorge Castro, Tânia Gaspar, and Joana Oliveira. 2018. “School effects on students’ engagement with school: Academic performance moderates the effect of school support for learning on students’ engagement.” Learning and Individual Differences 67 (October): 67–77. https://doi.org/10.1016/J.LINDIF.2018.07.007.

Nurchurifiani, Eva, Aksendro Maximilian, Galuh Dwi Ajeng, Purna Wiratno, Tommy Hastomo, and Andri Wicaksono. 2025. “Leveraging AI-powered tools in academic writing and research: Insights from English faculty members in Indonesia.” International Journal of Information and Education Technology 15 (2): 312–22. https://doi.org/10.18178/ijiet.2025.15.2.2244.

Oktarin, Irene Brainnita, Maria Edistianda Eka Saputri, Betty Magdalena, Tommy Hastomo, and Aksendro Maximilian. 2024. “Leveraging ChatGPT to enhance students’ writing skills, engagement, and feedback literacy.” Edelweiss Applied Science and Technology 8 (4): 2306–19. https://doi.org/10.55214/25768484.v8i4.1600.

Rudolph, Jürgen, Samson Tan, and Shannon Tan. 2023. “ChatGPT: Bullshit spewer or the end of traditional assessments in higher education?” Journal of Applied Learning & Teaching 6 (1): 342–63. https://doi.org/10.37074/jalt.2023.6.1.9.

Sari, Lusi Purnama, Tommy Hastomo, and Eva Nurchurifiani. 2023. “Assessing the efficacy of Duolingo for acquiring English vocabulary skills: Experimental research.” Journal of English Teaching Applied Linguistics and Literatures 6 (2): 193–200.

Schuetzler, Ryan M., G. Mark Grimes, and Justin Scott Giboney. 2020. “The impact of chatbot conversational skill on engagement and perceived humanness.” Journal of Management Information Systems 37 (3): 875–900. https://doi.org/10.1080/07421222.2020.1790204.

Shen, Chen, Penghai Shi, Jirong Guo, Suyun Xu, and Jiwei Tian. 2023. “From process to product: Writing engagement and performance of EFL learners under computer-generated feedback instruction.” Frontiers in Psychology 14 (October):1–13. https://doi.org/10.3389/fpsyg.2023.1258286.

Shikun, Shan, Gevorg Grigoryan, Ning Huichun, and Hasmik Harutyunyan. 2024. “AI chatbots: Developing English language proficiency in EFL classroom.” Arab World English Journal 1 (1): 292–305. https://doi.org/10.24093/awej/ChatGPT.20.

Silitonga, Lusia Maryani, Santhy Hawanti, Feisal Aziez, Miftahul Furqon, Dodi Siraj Muamar Zain, Shelia Anjarani, and Ting Ting Wu. 2023. “The impact of AI chatbot-based learning on students’ motivation in English writing classroom.” In Innovative Technologies and Learning, 6th International Conference, ICITL 2023, Porto, Portugal, August 28–30, 2023, Proceedings, ICITL, edited by Yueh-Min Huang and Tánia Rocha, 542–49. Springer. https://doi.org/10.1007/978-3-031-40113-8_53.

Slamet, Joko. 2024. “Potential of ChatGPT as a digital language learning assistant: EFL teachers’ and students’ perceptions.” Discover Artificial Intelligence 4 (1): 46. https://doi.org/10.1007/s44163-024-00143-2.

Thorp, H. Holden. 2023. “ChatGPT is fun, but not an author.” Science 379 (6630): 313. https://doi.org/10.1126/science.adg7879.

Tian, Lili, and Yu Zhou. 2020. “Learner engagement with automated feedback, peer feedback and teacher feedback in an online EFL writing context.” System 91 (July):102247. https://doi.org/10.1016/j.system.2020.102247.

Ward, Wesley S., and Lisa M. Given. 2019. “Assessing intercultural communication: Testing technology tools for information sharing in multinational research teams.” Journal of the Association for Information Science and Technology 70 (4): 338–50. https://doi.org/10.1002/asi.24159.

Waziana, Winia, Widi Andewi, Tommy Hastomo, and Muhamad Hasbi. 2024. “Students’ perceptions about the impact of AI chatbots on their vocabulary and grammar in EFL writing.” Register Journal 17 (2): 328–62. https://doi.org/https://doi.org/10.18326/register.v17i2.352-382.

Xu, Jinfen, and Juan Li. 2024. “Effects of AI affordances on student engagement in EFL classrooms: A structural equation modelling and latent profile analysis.” European Journal of Education 59 (4): e12808. https://doi.org/10.1111/ejed.12808.

Yu, Shulin, Lianjiang Jiang, and Nan Zhou. 2020. “Investigating what feedback practices contribute to students’ writing motivation and engagement in Chinese EFL context: A large scale study.” Assessing Writing 44:100451. https://doi.org/10.1016/j.asw.2020.100451.

Yuan, Lingjie, and Xiaojuan Liu. 2025. “The effect of artificial intelligence tools on EFL learners’ engagement, enjoyment, and motivation.” Computers in Human Behavior 162:108474. https://doi.org/10.1016/j.chb.2024.108474.

Zou, Bin, Xin Guan, Yinghua Shao, and Peng Chen. 2023. “Supporting speaking practice by social network-based interaction in artificial intelligence (AI)-assisted language learning.” Sustainability 15 (4): 2872. https://doi.org/10.3390/su15042872.

Zulianti, Hajjah, Hastuti Hastuti, Eva Nurchurifiani, Tommy Hastomo, Aksendro Maximilian, and Galuh Dwi Ajeng. 2024. “Enhancing novice EFL teachers’ competency in AI-powered tools through a TPACK-based professional development program.” World Journal of English Language 15 (3): 117. https://doi.org/10.5430/wjel.v15n3p117.