Factors determining the efficacy of AI-generated word problems for content-specific math language courses in higher education

Karen Fleischhauer; Kate Friedrich

doi:10.4312/SM.19.1.4-24

Authors

Karen Fleischhauer Technical University of Darmstadt
Kate Friedrich Technical University of Darmstadt

DOI:

https://doi.org/10.4312/SM.19.1.4-24

Keywords:

instructional materials, word problems, co-teaching, screencasting, mathematics

Abstract

The use of artificial intelligence (AI) tools to generate content-specific instructional materials has attracted the interest of Language for Specific Purposes (LSP) educators in higher education, as language courses in this setting typically do not utilize a textbook, requiring the instructor to create independent materials. However, instructors are often not content experts. Collaboration between LSP instructors and content experts in the form of co-teaching is one way in which materials can be generated and benefit instructors and students alike. At the same time, creating instructional materials can be a time-consuming task and can detract from other areas of collaboration. The use of AI tools to generate mathematical content could help instructors save time, enable real-world connections and offer a variety of materials to students. This study examines the generation of word problems with the help of AI tools for a content-based mathematics language course for first-semester bachelor students pursuing a Science, Technology, Engineering or Mathematics (STEM) degree. As they form part of the final exam, a new set of word problems needs to be generated each year. While recent studies (cf. Lu et al., 2022) found that AI methods were effective in generating math word problems that were diverse, relevant, and useable, there have been no studies examining the applicability of AI-generated word problems in terms of their efficacy in an LSP setting. The action research study used screencasting to capture the math content tutors’ content analysis of AI output on math word problems and was followed by a semi-structured group interview. The results showed that word problems generated by AI were generally useful based on factors such as prompting techniques. However, limitations were observed in the areas of accuracy and consistency. Based on initial results, the report suggests first implications for use in LSP instruction and describes measures that need to be taken into account in further studies.

Downloads

Download data is not yet available.

References

Arnold, T. (2023, February 10). ChatGPT für Nicht-Informatiker*innen [Video]. YouTube. https://www.youtube.com/watch?v=-c8ogAwX6KI&t

Arnó-Macià, E. (2013). Information technology and languages for specific purposes in the EHEA: Options and challenges for the knowledge society. In E. Bárcena, T. Read & J. Arús (Eds.), Languages for Specific Purposes in the Digital Era (pp. 3–25). Springer International Publishing. https://doi.org/10.1007/978-3-319-02222-2

Banegas, D.-L., & Consoli, S. (2019). Action research in language education. In J. McKinley & H. Rose H. (Eds.). The Routledge Handbook of Research Methods in Applied Linguistics (pp. 176–187). Routledge. https://doi.org/10.4324/9780367824471

Cohen, L. (2017). Research methods in education (8th ed.). Taylor and Francis. https://doi.org/10.4324/9781315456539

Cook, L., & Friend, M. (1995). Co-teaching: Guidelines for creating effective practices. Focus on Exceptional Children, 28(3), 1–16. https://doi.org/10.17161/foec.v28i3.6852

Davis, E. (2024). Mathematics, word problems, common sense, and artificial intelligence. Bulletin of the American Mathematical Society, 61(2), 287–303. https://doi.org/10.1090/bull/1828

Dörnyei, Z. (2012). Research methods in applied linguistics: Quantitative, qualitative, and mixed methodologies [reprint]. Oxford Univ. Press.

Dresing, T., & Pehl, T. (2018). Praxisbuch Interview, Transkription & Analyse. Anleitungen und Regelsysteme für qualitativ Forschende. 8. Auflage. Eigenverlag: Marburg.

Duden. (n. d.). Chatbot. https://www.duden.de/node/237994/revision/1382866

Eisenmann, M. & Steinbock, J. (2023). Vorwort. ZFF – Zeitschrift für Fremdsprachenforschung, 34(1), 3–7.

Escobar Urmeneta, C. (2020). Coteaching in CLIL in Catalonia. CLIL Journal of Innovation and Research in Plurilingual and Plurilingual Education, 3(2), 37–55. https://doi.org/10.5565/rev/clil.54

Frieder, S., Pinchetti, L., Chevalier, A., Griffiths, R. R., Salvatori, T., Lukasiewicz, T., Petersen, P. C., & Berner, J. (2023). Mathematical capabilities of ChatGPT. https://arxiv.org/abs/2301.13867

Ghassemi, M., Birhane, A., Bilal, M., Kankaria, S., Malone, C., Mollick, E., & Tustumi, F. (2023). ChatGPT one year on: Who is using it, how and why? Nature, 624(7990), 39–41. 10.1038/d41586-023-03798-6

Gunkel, D. J. (2003). Second thoughts: Toward a critique of the digital divide. New media & society, 5(4), 499–522. https://doi.org/10.1177/146144480354003

Hamel, M., & Séror, J. (2016). Video screen capture to document and scaffold the L2 writing process. In C. Caws & M.-J. Hamel (Eds.), Language studies, science and engineering: Vol. 2. Language-learner computer interactions: Theory, methodology and CALL applications (pp. 137–162). John Benjamins Publishing Company. https://doi.org/10.1075/lsse.2.07ham

Helfferich, C. (2019). Leitfaden- und Experteninterviews. In N. Baur & J. Blasius (Eds.), Springer eBook Collection. Handbuch Methoden der empirischen Sozialforschung (pp. 669–686). Springer VS. https://doi.org/10.1007/978-3-658-21308-4_44

Helfrich-Schkarbanenko, A. (2023). Mathematik und ChatGPT: Ein Rendezvous am Fuße der technologischen Singularität. Springer Spektrum. https://doi.org/10.1007/978

Heydarpur, S., & Umhauer, A. (2018). Co-Teaching. Eine Strategie für Nachwuchswissenschaftler*innen zur Erleichterung des Einstiegs in die universitäre Lehre. (Hochschuldidaktischer Berater: Franz Kroath). Universität Innsbruck: Institut für Erziehungswissenschaft. https://www.uibk.ac.at/media/filer_public/92/61/9261372e-f445-4789-b4aa-384b468c3056/2018-12-04_umhauser_andrea_heydarpur_sepideh.pdf

Huang, X., Zou, D., Cheng, G., Chen, X., & Xie, H. (2023). Trends, research issues and applications of artificial intelligence in language education. Educational Technology & Society, 26(1), 112–131. https://doi.org/10.30191/ETS.202301_26(1).0009

Iberdrola. (n. d.). Digital divide. Digital divide throughout the world and why it causes inequality. Iberdrola. https://www.iberdrola.com/social-commitment/what-is-digital-divide

Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G.,

Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., & Stadler, M. (2023). ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education. https://doi.org/10.35542/osf.io/5er8f

Kohnke, N., Moorhouse, B. L., & Zou, D. (2023). Using ChatGPT for language teaching and learning. RELC Journal, 54(2), 537–550. https://doi.org/10.1177/00336882231162868

Kováčiková, E. (2020). English for specific purposes in higher education through content and language integrated learning. Cambridge Scholars Publishing.

Kricke, M., & Reich, K. (2016). Teamteaching. Beltz.

Kuckartz, U., & Rädiker, S. (2023). Qualitative content analysis: Methods, practice and software. SAGE.

Lee, S.-M. (2022). L2 Learners’ strategies for using machine translation as a personalized writing tool. In J. Colpaert & G. Stockwell (Eds.), Smart CALL: Personalization, Contextualization, & Socialization (pp. 184–206). Castledown. 10.29140/9781914291012-pbk

Lew, S., Yang, A. H., & Harklau, L. (2018). Qualitative Methodology. In A. Phakiti, P. I. de Costa, L. Plonsky, & S. Starfield (Eds.), The Palgrave Handbook of Applied Linguistics Research Methodology (pp. 79–101). Palgrave Macmillan. https://doi.org/10.1057/978-1-137-59900-1

Lu, P., Qiu, L., Yu, W., Welleck, S., & Chang, K. W. (2022). A survey of deep learning for mathematical reasoning. arXiv:2212.10535. https://arxiv.org/pdf/2212.10535

Mishan, F. (2022). Language learning materials in the digital era. In J. Norton, & H. Buchanan (Eds.), The Routledge Handbook of Materials Development for Language Teaching (pp. 17–29). Routledge.

Mroz, A. P. (2012). Process research screen capture. The Encyclopedia of Applied Linguistics, 1–7. https://doi.org/10.1002/9781405198431.wbeal1450

Nerdynav. (2022, December 13). 73 Important ChatGPT Statistics & Facts For Mid Feb 2023 + An Infographic – Nerdy Nav. Nerdynav. https://nerdynav.com/chatgpt-statistics/

Pierce, J. (2018). Digital Divide. In R. Hobbs, & P. Mihailidis (Eds.) The International Encyclopedia of Media Literacy. John Wiley & Sons. https://doi.org/10.1002/9781118978238.ieml0052

Pineda, J. (2018). Problemlagen und Herausforderungen internationaler Studierender in Deutschland. Ergebnisse einer qualitativen Vorstudie im Rahmen des SeSaba-Projekts. https://www2.daad.de/medien/der-daad/analysen-studien/sesaba/vorstudie_pineda_2018.pdf

Pineda, J., Kercher, J., Falk, S., Thies, T., Yildirim, H. H., & Zimmermann, J. (2022). Accompanying international students in Germany to academic success: Results and recommendations from the SeSaBa project (DAAD Studien). Bonn. https://static.daad.de/media/daad_de/pdfs_nicht_barrierefrei/derdaad/analysen-studien/daad_sesaba_final_report.pdf

Plevris, V., Papazafeiropoulos, G., & Rios, A. J. (2023). Chatbots put to the test in math and logic problems: A preliminary comparison and assessment of ChatGPT-3.5, ChatGPT-4, and Google Bard. arXiv preprint arXiv:2305.18618

Richards, J. C., Platt, J. T., & Platt, H. (1999). Longman dictionary of language teaching and applied linguistics (New ed., 9. impr). Longman.

Rocha, H. (2019). Mathematical proof: From mathematics to school mathematics. Phil. Trans. R. Soc. A 377: 20180045. http://dx.doi.org/10.1098/rsta.2018.0045

Rooks, R. N., Scandlyn, J., Pelowich, K., & Lor, S. (2022). Co-teaching two interdisciplinary courses in higher education. International Journal for the Scholarship of Teaching and Learning, 16(2), 1–8. https://doi.org/10.20429/ijsotl.2022.160208

Somekh, B. (2008). Action research. In L. M. Given (Ed.), The SAGE Encyclopaedia of Qualitative Research Methods. Volumes 1 & 2 (pp. 4–7). SAGE.

Spannagel, C. (2023). Hat ChatGPT eine Zukunft in der Mathematik? Mitteilungen der Deutschen Mathematiker-Vereinigung, 31(3), 168–172. https://doi.org/10.1515/dmvm-2023-0055

UNESCO. (2023). Education 2030 agenda. https://www.unesco.org/en/digital-education/artificial-intelligence. Wessels, D. (2023, January 27). Was ist ChatGPT und wie funktioniert es? – und welche ähnlichen Tools gibt es? [Video]. YouTube. https://youtu.be/cMuBo_rH15c?si=QuQ_oEgREPXVgTVw