Generative AI in Pragmatics: Assessing the Accuracy of Automated Speech Act Classification in Pinter’s The Birthday Party

Tadej Todorović; Andrej Flogie; Daniel Hari

doi:10.4312/elope.22.1.19-34

Authors

Tadej Todorović University of Maribor
Andrej Flogie
Daniel Hari University of Maribor

DOI:

https://doi.org/10.4312/elope.22.1.19-34

Keywords:

pragmatics, speech act analysis, ChatGPT, DeepSeek, Gemini, Pinter

Abstract

This study explores the feasibility of using generative AI (ChatGPT, Gemini, and DeepSeek) to automate speech act annotation in Harold Pinter’s play The Birthday Party. Three chatbots – ChatGPT, Gemini, and DeepSeek – were tested under three scenarios varying in the amount of theoretical material provided. Each chatbot’s output was compared to a manually annotated reference via a Python script measuring classification accuracy. Scenario 2 produced the highest accuracy overall (75–82%), while Scenario 1 underperformed, owing to incorrect reliance on external typologies, and Scenario 3 showed signs of overfitting. ChatGPT o1 emerged as the most accurate model, achieving 82% accuracy in Scenario 2. The findings suggest that GenAI chatbots can serve as valuable preliminary annotators when good prompt-engineering and well-curated theoretical material are provided. Future research could extend this methodology to more context-dependent texts, further refining promptengineering strategies and exploring larger linguistic corpora.

Metrics

Metrics Loading ...

Downloads

Download data is not yet available.

References

Altmäe, Signe, Alberto Sola-Leyva, and Andres Salumets. 2023. “Artificial intelligence in scientific writing: A friend or a foe?” Reproductive BioMedicine Online 47 (1): 3–9. https://doi.org/10.1016/j.rbmo.2023.04.009.

Chen, Xi, Jun Li, and Yuting Ye. 2024. “A feasibility study for the application of AI-generated conversations in pragmatic analysis.” Journal of Pragmatics 223:14–30. https://doi.org/10.1016/j.pragma.2024.01.003.

Chowdhery, Aakanksha, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, et al. 2022. “PaLM: Scaling language modeling with pathways.” arXiv. https://doi.org/10.48550/arXiv.2204.02311.

DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, et al. 2025. “DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning.” arXiv. https://doi.org/10.48550/ARXIV.2501.12948.

Edmondson, Willis, and Juliane House. 1981. Let’s Talk, and Talk about It: A Pedagogic Interactional Grammar of English. Urban & Schwarzenberg.

Edmondson, Willis J., Juliane House, and Daniel Z. Kádár. 2023. Expressions, Speech Acts and Discourse: A Pedagogic Interactional Grammar of English. Cambridge University Press.

Else, Holly. 2023. “Abstracts written by ChatGPT fool scientists.” Nature 613 (7944): 423. https://doi.org/10.1038/d41586-023-00056-7.

Gavez, Urša. 2016. “The reception of Harold Pinter’s plays in Slovenia between 1999 and 2014.” ELOPE: English Language Overseas Perspectives and Enquiries 13 (2): 51–61. https://doi.org/10.4312/elope.13.2.51-61.

Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, et al. 2024. “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.” arXiv. https://doi.org/10.48550/ARXIV.2403.05530.

Hazemali, David, Janez Osojnik, Tomaž Onič, Tadej Todorović, and Mladen Borovič. 2024. “Evaluating chatbot assistance in historical document analysis.” Moderna arhivistika 7 (2): 53–83. https://doi.org/10.54356/ma/2024/biub3010.

House, Juliane. 1996. “Developing pragmatic fluency in English as a foreign language: Routines and metapragmatic awareness.” Studies in Second Language Acquisition 18 (2): 225–52. https://doi.org/10.1017/S0272263100014893.

House, Juliane, and Dániel Z. Kádár. 2023. “Studying small talk from a pragmatic angle: An introduction.” Acta Linguistica Academica 70 (4): 411–18. https://doi.org/10.1556/2062.2023.00704.

House, Juliane, Dániel Z. Kádár, Tadej Todorović, Matjaž Klemenčič, David Hazemali, Tomaž Onič, and Katja Plemenitaš. 2024. “Capturing power in diplomatic language use: The case of a closed-door mediatory negotiation and its aftermath during the breakup of the former Yugoslavia.” Journal of Language and Politics. https://doi.org/10.1075/jlp.24036.hou.

Hribar, Darja. 2004. “Harold Pinter in Slovene translation.” ELOPE: English Language Overseas Perspectives and Enquiries 1 (1–2): 195–208. https://doi.org/10.4312/elope.1.1-2.195-208.

Kádár, Dániel Z., Juliane House, Tadej Todorović, Tomaž Onič, David Hazemali, Katja Plemenitaš, and Donathan Brown. 2024. “The language of diplomatic mediation – A case study of an emergency meeting in the wake of the Yugoslav wars.” Language & Communication 96:54–66. https://doi.org/10.1016/j.langcom.2024.02.004.

Kaplan, Jared, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. “Scaling laws for neural language models.” arXiv. https://doi.org/10.48550/arXiv.2001.08361.

Kuhn, Harold William. 1955. “The Hungarian method for the assignment problem.” Naval Research Logistics Quarterly 2 (1–2): 83–97.

Liu, Shiyu, Juliane House, and Dániel Z. Kádár. 2024. “Bargaining in Chinese livestream sales events.” Discourse, Context & Media 60:100787. https://doi.org/10.1016/j.dcm.2024.100787.

Mercer, Sarah, Samuel Spillard, and Daniel P. Martin. 2025. “Brief analysis of DeepSeek R1 and its implications for generative AI.” arXiv. https://doi.org/10.48550/ARXIV.2502.02523.

Mohar, Tjaša, Sara Orthaber, and Tomaž Onič. 2020. “Machine translated Atwood: Utopia or dystopia?” ELOPE: English Language Overseas Perspectives and Enquiries 17 (1): 125–41. https://doi.org/10.4312/elope.17.1.125-141.

Monteiro, Mateus De Souza, Vinícius Carvalho Pereira, and Luciana Cardoso De Castro Salgado. 2023. “Investigating Politeness strategies in chatbots through the lens of conversation analysis.” In Proceedings of the XXII Brazilian Symposium on Human Factors in Computing Systems, Maceió, Brazil, 1–12. Association for Computing Machinery. https://doi.org/10.1145/3638067.3638068.

Moors, Sarah. 2025. “DeepSeek changes everything we thought we knew about building smart machines.” Digital Health Insights, January 29. https://www.dhinsights.org/news/deepseek-changes-everything-we-thought-we-knew-about-building-smart-machines.

Onič, Tomaž. 2016. “Slogovne značilnosti … [premolk] … Pinterjevega dialoga.” Primerjalna književnost 39 (2). https://ojs-gr.zrc-sazu.si/primerjalna_knjizevnost/article/view/6367.

Onič, Tomaž, and Nastja Prajnč Kacijan. 2020. “Repetition as a means of verbal and psychological violence in interrogation scenes from contemporary drama.” Ars & Humanitas 14 (1): 13–26. https://doi.org/10.4312/ars.14.1.13-26.

Pinter, Harold. 1991. Plays One. Faber & Faber.

Podbevšek, Katarina, and Nina Žavbi. 2021. “Jezikovna norma v luči odrske govorne estetike.” Jezik in slovstvo 66 (2–3): 145–56. https://doi.org/10.4312/jis.66.2-3.145-156.

Raffel, Colin, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2023. “Exploring the limits of transfer learning with a unified text-totexttransformer.” arXiv. https://doi.org/10.48550/arXiv.1910.10683.

Rane, Nitin, Saurabh Choudhary, and Jayesh Rane. 2024. “Gemini versus ChatGPT: Applications, performance, architecture, capabilities, and implementation.” SSRN Scholarly Paper. Social Science Research Network. https://doi.org/10.2139/ssrn.4723687.

Schechner, Richard. 1966. “Puzzling Pinter.” The Tulane Drama Review 11 (2): 176–84. https://doi.org/10.2307/1125196.

Setlur, Vidya, and Melanie Tory. 2022. “How do you converse with an analytical chatbot? Revisiting Gricean maxims for designing analytical conversational behavior.” arXiv. https://doi.org/10.48550/ARXIV.2203.08420.

Stokel-Walker, Chris. 2023. “ChatGPT listed as author on research papers: Many scientists disapprove.” Nature 613 (7945): 620–21. https://doi.org/10.1038/d41586-023-00107-z.

Taguchi, Naoko, and Dániel Z. Kádár. 2025. “Pragmatics: An overview.” In The Encyclopedia of Applied Linguistics, edited by Carol A. Chapelle, 1st ed., 1–8. Wiley. https://doi.org/10.1002/9781405198431.wbeal1338.pub2.

Wang, Kevin, Junbo Li, Neel P. Bhatt, Yihan Xi, Qiang Liu, Ufuk Topcu, and Zhangyang Wang. 2024. “On the planning abilities of OpenAI’s O1 models: Feasibility, optimality, and generalizability.” arXiv. https://doi.org/10.48550/ARXIV.2409.19924.

Williams, Iwan, and Tim Bayne. 2024. “Chatting with bots: AI, speech acts, and the edge of assertion.” Inquiry: 1–24. https://doi.org/10.1080/0020174X.2024.2434874.