Priprava podatkov pri množičenju v pedagoške namene
Primer igre CrowLL
DOI:
https://doi.org/10.4312/slo2.0.2022.2.62-100Ključne besede:
množičenje, igra z namenom, vzorčni stavki, pedagoški korpusPovzetek
Eden od načinov za spodbujanje uporabe korpusov pri jezikovnem izobraževanju je izdelava pedagoško primernih korpusov, označenih z različnimi vrstami problematik (občutljiva vsebina, žaljiv jezik, strukturne težave). Ker je ročno označevanje korpusov zelo časovno potratno, je potrebno poiskati boljši pristop. Predlagamo kombinacijo dveh pristopov k oblikovanju problemsko označenih pedagoških korpusov nizozemščine, estonščine, slovenščine in brazilske portugalščine: uporabo iger z namenom množičenja. Z udeleženci smo izvedli začetne poskuse, da bi ugotovili, če je naloga množičenja ustrezna, pridobljene izkušnje pa smo uporabili za oblikovanje igre Crowdsourcing for Language Learning (CrowLL), v kateri igralci prepoznavajo problematične povedi in segmente ter jih razvrščajo. V prispevku se osredotočamo na pripravo podatkov, saj ima ta korak ključni pomen pri vsakem projektu množičenja, ki obravnava ustvarjanje jezikovnih učnih virov. Predlagamo metodologijo za pripravo podatkov, podrobno predstavljamo izbiro izvornih korpusov, pedagoško usmerjene konfiguracije GDEX in oblikovanje seznamov lem, s posebnim poudarkom na pogostih in od jezika odvisnih odločitvah. Za konec ponujamo razpravo o izzivih, ki smo jih zasledili, in o rešitvah, ki smo jih do sedaj že uvedli.
Prenosi
Literatura
Aitamurto, T., Leiponen, A., & Tee, R. (2011). The promise of idea crowdsourcing–benefits, contexts, limitations [White paper]. Nokia Ideas project.
Arhar Holdt, Š., Kosem, I., & Gantar, P. (2017). Corpus-based resources for L1 teaching: The case of Slovene. In Handbook on digital learning for K-12 schools (pp. 91–113). Springer, Cham. doi: 10.1007/978-3-319-33808-8_7 DOI: https://doi.org/10.1007/978-3-319-33808-8_7
Arhar Holdt, Š., Kosem, I., Krapš Vodopivec, I., Ledinek, N., Može, S., Stritar Kučuk, M., Svenšek, T., & Zwitter Vitez, A. (2011). Pedagoška slovnica pri projektu Sporazumevanje v slovenskem jeziku: K16 – Standard za korpusno analizo slovničnih pojavov. Ljubljana: Ministrstvo za šolstvo in šport: Amebis. Retrieved from http://projekt.slovenscina.eu/Media/Kazalniki/Kazalnik16/Kazalnik_16_Pedagoska_slovnica_SSJ.pdf
Arhar Holdt, Š., Logar, N., Pori, E., & Kosem, I. (2021). “Game of Words”: Play the game, clean the database. In Z. Gavriilidou, M. Mitsiaki & A. Fliatouras (Eds.), Proceedings of the EURALEX XIX congress: Lexicography for inclusion, 7–11 September, Aleksandroupolis, Greece (Vol I., pp. 41–49). Retrieved from https://www.euralex.org/elx_proceedings/Euralex2020-2021/EURALEX2020-2021_Vol1-p041-049.pdf
Baisa, V., & Suchomel, V. (2014). SkELL: Web interface for English language learning. Proceedings of the eighth workshop on recent advances in Slavonic natural language processing, RASLAN 2014 (pp. 63–70). Retrieved from https://nlp.fi.muni.cz/raslan/2014/12.pdf
Bassignana, E., Basile, V., & Patti, V. (2018). Hurtlex: A multilingual lexicon of words to hurt. CEUR Workshop proceedings, 1–6. Retrieved from http://ceur-ws.org/Vol-2253/paper49.pdf DOI: https://doi.org/10.4000/books.aaccademia.3085
Bédi, B., Chua, C., Habibi, H., Martinez-Lopez, R., & Rayner, M. (2019). Using LARA for language learning: a pilot study for Icelandic. In F. Meunier, J. van de Vyver, L. Bradley & S. Thouësny (Eds.), CALL and complexity: short papers from EUROCALL 2019 (pp. 33–38). Research-publishing.net. doi: 10.14705/rpnet.2019.38.982 DOI: https://doi.org/10.14705/rpnet.2019.38.982
Bonetti, F., & Tonelli. S. (2020). A 3D role-playing game for abusive language annotation. Workshop on games and natural language processing (pp. 39–43). Retrieved from https://aclanthology.org/2020.gamnlp-1.6
Bonetti, F., & Tonelli. S. (2021). Challenges in designing games with a purpose for abusive language annotation. Proceedings of the first workshop on bridging human–computer interaction and natural language processing (pp. 60–65). https://aclanthology.org/2021.hcinlp-1.10
Boulton, A. (2017). Corpora in language teaching and learning: Research timeline. Language Teaching, 50(4), 483–506. doi: 10.1017/S0261444817000167 DOI: https://doi.org/10.1017/S0261444817000167
Braun, S. (2005). From pedagogically relevant corpora to authentic language learning contents. ReCALL, 17(1), 47–64. doi: 10.1017/S0958344005000510 DOI: https://doi.org/10.1017/S0958344005000510
Buecheler, T., Sieg, J. H., Füchslin, R. M., & Pfeifer, R. (2010). Crowdsourcing, open innovation and collective intelligence in the scientific method: a research agenda and operational framework. In H. Fellermann, M. Dörr, M. Hanczyc, L. L. Laursen, S. Maurer, D. Merkle, P-A. Monnard, K. Stoy, S. Rasmussen (Eds.), Artificial live XII: proceedings of the twelfth international conference on the synthesis and simulation of living systems (pp. 679–686). MIT Press. doi: 10.21256/zhaw-4094
Callies, M. (2019). Integrating corpus literacy into language teacher education. In S. Götz, J. Mukherjee (Eds.), Learner corpora and language teaching (pp. 245–263). John Benjamins Publishing Company. doi: 10.1075/scl.92.12cal DOI: https://doi.org/10.1075/slcs.201.12cal
Chamberlain, J., Poesio, M., & Kruschwitz, U. (2008). Phrase detectives: A web-based collaborative annotation game. Proceedings of the international conference on semantic systems (I-Semantics’08) (pp. 42–49). Retrieved from https://www.jonchamberlain.com/media/doc/Chamberlain2008Phrase.pdf
Chambers, A. (2016). Written language corpora and pedagogic applications. In F. Farr, L. Murray (Eds.), The Routledge handbook of language learning and technology (pp. 362–375). Routledge. doi: 10.4324/9781315657899.ch26
Chesbrough, H. W. (2006). Open innovation: The new imperative for creating and profiting from technology. Harvard Business School Press.
Colman, L., & Tiberius C. (2018). A good match: A Dutch collocation, idiom and pattern dictionary combined. Proceedings of the XVIII EURALEX international congress: Lexicography in global contexts (pp. 233–246). Retrieved from https://euralex.org/wp-content/themes/euralex/proceedings/Euralex%202018/118-4-2952-1-10-20180820.pdf
Erelt, M., & Metslang, H. (2017). Eesti keele süntaks. Eesti keele varamu III. Tartu Ülikooli Kirjastus. Retrieved from https://dspace.ut.ee/handle/10062/70510
Eryiğit, G., Şentaş, A., & Monti, J. (2022). Gamified crowdsourcing for idiom corpora construction. Natural Language Engineering (pp. 1–33). doi: 10.1017/S1351324921000401 DOI: https://doi.org/10.1017/S1351324921000401
Gantar, P., Kosem, I., & Krek, S. (2016). Discovering automated lexicography: The case of the Slovene lexical database. International Journal of Lexicography, 29(2), 200–225. doi: 10.1093/ijl/ecw014 DOI: https://doi.org/10.1093/ijl/ecw014
Gorovaia, N. (2018). Behavior of users on the crowdsourcing platforms. [Poster session]. EnetCollect WG3/WG5 meeting, October 24–25, Leiden, Netherlands.
Gries, S. (2009). What is corpus linguistics? Language and Linguistics Compass, 3, 1–17. doi: 10.1111/j.1749-818X.2009.00149.x DOI: https://doi.org/10.1111/j.1749-818X.2009.00149.x
Guillaume, B., Fort, K., & Lefebvre, N. (2016). Crowdsourcing complex language resources: Playing to annotate dependency syntax. Proceedings of COLING 2016, the 26th international conference on computational linguistics: Technical papers (pp. 3041–3052). Retrieved from https://aclanthology.org/C16-1286
Hacker, S., & von Ahn, L. (2009). Matchin: eliciting user preferences with an online game. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1207–1216). doi: 10.1145/1518701.1518882 DOI: https://doi.org/10.1145/1518701.1518882
Harris, C.G. (2014). The beauty contest revisited: measuring consensus rankings of relevance using a game. Proceedings of the first international workshop on gamification for information retrieval – GamifIR@ECIR ‘14 (pp. 17–21). doi: 10.1145/2594776.2594780 DOI: https://doi.org/10.1145/2594776.2594780
Kallas, J., Kilgarriff, A., Koppel, K., Kudritski, E., Langemets, M., Michelfeit, J., Tuulik, M., & Viks, Ü. (2015). Automatic generation of the Estonian Collocations Dictionary database. Proceedings of the eLex 2015 conference (pp. 1−20). Retrieved from https://elex.link/elex2015/proceedings/eLex_2015_01_Kallas+etal.pdf
Kilgarriff, A., Baisa, V., Bušta, J., Jakubíček, M., Kovář, V., Michelfeit, J., Rychlý, P., & Suchomel, V. (2014). The Sketch Engine: ten years on. Lexicography, 1(1), 7–36. doi: 10.1007/s40607-014-0009-9 DOI: https://doi.org/10.1007/s40607-014-0009-9
Kilgarriff, A., Husák, M., McAdam, K., Rundell, M., & Rychlý, P. (2008). GDEX: Automatically finding good dictionary examples in a corpus. Proceedings of the XIII EURALEX international congress (Vol. 1, pp. 425–432). https://tinyurl.com/yckr9w8s
Kilgarriff, A., Rychlý, P., Smrz, P., & D. Tugwell (2004). The Sketch Engine. Proceedings of the eleventh EURALEX international congress, EURALEX 2004 (pp. 105–116). Retrieved from https://tinyurl.com/mvrp4ymy
Koppel, K. (2019). Leksikograafide ja keeleõppijate hinnangud automaatselt tuvastatud korpuslausete sobivusele õppesõnastiku näitelauseks. Lähivõrdlusi. Lähivertailuja, 29, 84−112. doi: 10.5128/LV29.03 DOI: https://doi.org/10.5128/LV29.03
Koppel, K. (2020). Näitelausete korpuspõhine automaattuvastus eesti keele õppesõnastikele. Doktoritöö, Tartu Ülikool. Retrieved from https://dspace.ut.ee/handle/10062/67138
Koppel, K., & Kallas, J. (2022). Eesti keele ühendkorpus 2021. doi: 10.15155/3-00-0000-0000-0000-08D17L
Koppel, K., Kallas, J., Khokhlova, M., Suchomel, V., Baisa, V., & Michelfeit, J. (2019). SkELL corpora as a part of the language portal Sõnaveeb: problems and perspectives. Proceedings of the eLex 2019 conference (pp. 763−782). Retrieved from https://zenodo.org/record/3612933#.Yywd1XZBy70
Koppel, K., Tavast, A., Langemets, M., & Kallas, J. (2019). Aggregating dictionaries into the language portal Sõnaveeb: issues with and without a solution. Proceedings of the eLex 2019 conference (pp. 434−452). Retrieved from https://elex.link/elex2019/wp-content/uploads/2019/09/eLex_2019_24.pdf
Kosem, I. (2012,). Using GDEX in (semi)-automatic creation of database entries [Conference presentation]. SKEW-3, 3rd international Sketch Engine workshop, 21−22 March, 2012.
Kosem, I., Gantar, P., & Krek, S. (2013). Automation of lexicographic work: an opportunity for both lexicographers and crowd-sourcing. Proceedings of the eLex 2013 conference (pp. 32−48). Retrieved from http://eki.ee/elex2013/proceedings/eLex2013_03_Kosem+Gantar+Krek.pdf
Kosem, I., Husák, M., & McCarthy, D. (2011). GDEX for Slovene. Proceedings of eLex 2011 (pp. 151–159). Retrieved from http://www.dianamccarthy.co.uk/files/Kosemetal-paper.pdf
Kosem, I., Koppel, K., Kuhn, T. Z., Michelfeit, J., & Tiberius, C. (2019). Identification and automatic extraction of good dictionary examples: the case(s) of GDEX. International Journal of Lexicography, 32(2), 119−137. doi: 10.1093/ijl/ecy014 DOI: https://doi.org/10.1093/ijl/ecy014
Krek, S., Arhar Holdt, Š., Erjavec, T., Čibej, J., Repar, A., Gantar, P., Ljubešić, N., Kosem, I., & Dobrovoljc, K. (2020). Gigafida 2.0: The reference corpus of written standard Slovene. Proceedings of the twelfth language resources and evaluation conference (pp. 3340–3345). Retrieved from https://aclanthology.org/2020.lrec-1.409
Kuhn, T. Z. (2017). A design proposal of an online corpus-driven dictionary of Portuguese for university students [Doctoral dissertation, Universidade de Lisboa]. Retrieved from http://hdl.handle.net/10451/32013
Kuhn, T. Z., Šandrih Todorović, B., Holdt, Š. A., Zviel-Girshin, R., Koppel, K., Luís, A.R., & Kosem, I. (2021). Crowdsourcing pedagogical corpora for lexicographical purposes. Proceedings of the XIX EURALEX congress: Lexicography for inclusion (Vol. II., pp. 771–779). Retrieved from https://www.euralex.org/elx_proceedings/Euralex2020-2021/EURALEX2020-2021_Vol2-p771-779.pdf
Lafourcade, M. (2007). Making people play for Lexical Acquisition with the JeuxDeMots prototype. Proceedings of SNLP’07: 7th international symposium on natural language processing. Retrieved from https://hal-lirmm.ccsd.cnrs.fr/lirmm-00200883
Langemets, M., Hein, I., Jürviste, M., Kallas, J., Kiisla, O., Koppel, K., Leemets, T., …, & Tubin, V. (2022). EKI ühendsõnastik 2022. doi: 10.15155/3-00-0000-0000-0000-08C0AL
Lévy, P. (1997). Collective intelligence: Mankind’s emerging world in cyberspace. Plenum Trade. New York.
Lew, R. (2014). User-generated content (UGC) in online English dictionaries. OPAL, 4, 8–26. Retrieved from https://pub.ids-mannheim.de//laufend/opal/opal14-4.html
Lyding, V., Nicolas, L., Bédi, B., & Fort, K. (2018). Introducing the European network for combining language learning and crowdsourcing techniques (enetcollect). In P. Taalas, J. Jalkanen, L. Bradley & S. Thouësny (Eds.), Future-proof CALL: language learning as exploration and encounters–short papers from EUROCALL (pp. 176–181). Research-publishing.net. doi: 10.14705/rpnet.2018.26.833 DOI: https://doi.org/10.14705/rpnet.2018.26.833
Morschheuser, B., Hamari, J., Koivisto, J., & Maedche, A. (2017). Gamified crowdsourcing: Conceptualization, literature review, and future agenda. International Journal of Human-Computer Studies, 106, 26–43. doi: 10.1016/j.ijhcs.2017.04.005 DOI: https://doi.org/10.1016/j.ijhcs.2017.04.005
Nicolas, L., Lyding, V., Borg, C., Forăscu, C., Fort, K., Zdravkova, K., Kosem, I., …, & HaCohen-Kerner, Y. (2020). Creating expert knowledge by relying on language learners: a generic approach for mass-producing language resources by combining implicit crowdsourcing and language learning. Proceedings of the 12th language resources and evaluation conference (pp. 268–278). Retrieved from https://aclanthology.org/2020.lrec-1.34
Osborne, J. (2004). Top-down and bottom-up approaches to corpora in language teaching. language and computers. In U. Connor, T. A. Upton (Eds.), Applied Corpus Linguistics. A Multidimensional Perspective (pp. 251–265). Brill. doi: 10.1163/9789004333772_015 DOI: https://doi.org/10.1163/9789004333772_015
Pe-Than, E. P. P., Goh, D. H. L., & Lee, C. S. (2015). A typology of human computation games: an analysis and a review of current games. Behaviour & Information Technology, 34(8), 809–824. doi: 10.1080/0144929X.2013.862304 DOI: https://doi.org/10.1080/0144929X.2013.862304
Pilán, I., Vajjala, S., & Volodina, E. (2016). A readable read: Automatic assessment of language learning materials based on linguistic complexity. ArXiv. doi: 10.48550/arXiv.1603.08868
Pilán, I., Volodina, E., & Johansson, R. (2013). Automatic selection of suitable sentences for language learning exercises. 20 Years of EUROCALL: Learning from the past, looking to the future: 2013 EUROCALL Conference Proceedings (pp. 218–225). Retrieved from https://aclanthology.org/W14-1821.pdf DOI: https://doi.org/10.14705/rpnet.2013.000164
Pilán, I., Volodina, E., & Johansson, R. (2014). Rule-based and machine learning approaches for second language sentence-level readability. Proceedings of the ninth workshop on innovative use of NLP for building educational applications (pp. 174–184). Retrieved from https://aclanthology.org/W14-1821 DOI: https://doi.org/10.3115/v1/W14-1821
Poletto, F., Basile, V., Sanguinetti, M., Bosco, C., & Patti, V. (2021). Resources and benchmark corpora for hate speech detection: a systematic review. Language Resources & Evaluation, 55(2), 477–523. doi: 10.1007/s10579-020-09502-8 DOI: https://doi.org/10.1007/s10579-020-09502-8
Prahalad, C. K., & Ramaswamy, V. (2000). Co-opting customer competence. Harvard Business Review. Retrieved from https://hbr.org/2000/01/co-opting-customer-competence
Preist, C., Massung, E., & Coyle, D. (2014). Competing or aiming to be average? Normification as a means of engaging digital volunteers. Proceedings of the 17th ACM conference on computer supported cooperative work & social computing (CSCW ‘14) (pp. 1222–1233). doi: 10.1145/2531602.2531615 DOI: https://doi.org/10.1145/2531602.2531615
Reynaert, M. (2006). Corpus-induced corpus clean-up. Proceedings of the fifth international conference on language resources and evaluation (pp. 87–92). Retrieved from http://www.lrec-conf.org/proceedings/lrec2006/pdf/229_pdf.pdf
Römer, U. (2009). Using general and specialised corpora in language teaching: Past, present and future. In M. C. Campoy, B. Belles-Fortuno & M. L. Gea-Valor (Eds.), Corpus-based approaches to English language teaching (pp.18–35). Continuum Publishing Corporation.
Šandrih Todorović, B. (2020). Impact of text classification on natural language processing applications. [Универзитет у Београду].
Schmidt, A., & Wiegand, M. (2017). A survey on hate speech detection using natural language processing. Proceedings of the fifth international workshop on natural language processing for social media (pp. 1–10). doi: 10.18653/v1/W17-1101 DOI: https://doi.org/10.18653/v1/W17-1101
Seemakurty, N., Chu, J., von Ahn, L., & Tomasic, A. (2010). Word sense disambiguation via human computation. Proceedings of the ACM SIGKDD workshop on human computation (pp. 60–63). doi: 10.1145/1837885.1837905 DOI: https://doi.org/10.1145/1837885.1837905
Simpson, R., Page, K. R., & De Roure, D. (2014). Zooniverse: observing the world’s largest citizen science platform. Proceedings of the 23rd international conference on world wide web, 1049–1054. doi: 10.1145/2567948.2579215 DOI: https://doi.org/10.1145/2567948.2579215
Sinclair, J. (2005). Corpus and text - basic principles. In M. Wynne (Ed.), Developing linguistic corpora: A guide to good practice (pp. 1–16). Oxbow Books. Retrieved from https://users.ox.ac.uk/~martinw/dlc/chapter1.htm
Stanković, R., Šandrih, B., Stijović, R., Krstev, C., Vitas, D., & Marković, A. (2019). SASA dictionary as the gold standard for good dictionary examples for Serbian. Proceedings of the eLex 2019 conference (pp. 248–269). Retrieved from https://elex.link/elex2019/wp-content/uploads/2019/09/eLex_2019_14.pdf
Trampuš, M., & Novak, B. (2012). The internals of an aggregated web news feed. Proceedings of 15th multiconference on information society 2012 (IS-2012). Retrieved from http://ailab.ijs.si/dunja/SiKDD2012/Papers/Trampus_Newsfeed.pdf
Vajjala, S. (2022). Trends, limitations and open challenges in automatic readability assessment research. Proceedings of the thirteenth language resources and evaluation conference (pp. 5366–5377). Retrieved from https://aclanthology.org/2022.lrec-1.574
Vidgen, B., & Derczynski, L. (2020). Directions in abusive language training data, a systematic review: Garbage in, garbage out. PLoS ONE, 15(12): e0243300. doi: 10.1371/journal.pone.0243300 DOI: https://doi.org/10.1371/journal.pone.0243300
von Ahn, L. (2006). Games with a purpose. Computer, 39(6), 92–94. Retrieved from https://www.cs.cmu.edu/~biglou/ieee-gwap.pdf DOI: https://doi.org/10.1109/MC.2006.196
von Ahn, L., & Dabbish, L. (2008). Designing games with a purpose. Communications of the ACM, 51(8), 58–67. doi: 10.1145/1378704.1378719 DOI: https://doi.org/10.1145/1378704.1378719
Von Hippel, E., & Katz, R. (2002). Shifting innovation to users via toolkits. Management science, 48(7), 821–833. DOI: https://doi.org/10.1287/mnsc.48.7.821.2817
Vyatkina, N., & Boulton, A. (2017). Corpora in language teaching and learning. Language Learning and Technology, 21(3), 1–8.
Xu, L., & Chamberlain, J. (2020). Cipher: a prototype game-with-a-purpose for detecting errors in text. Workshop games and natural language processing (pp. 17–25). Retrieved from https://aclanthology.org/2020.gamnlp-1.3
Zampieri, M., Malmasi, S., Nakov, P., Rosenthal, S., Farra, N., & Kumar, R. (2019). SemEval-2019 task 6: Identifying and categorizing offensive language in social media (OffensEval). Proceedings of the 13th international workshop on semantic evaluation (SemEval-2019) (pp. 75–86). doi: 10.18653/v1/S19-2010 DOI: https://doi.org/10.18653/v1/S19-2010
Zampieri, M., Nakov, P., Rosenthal, S., Atanasova, P., Karadzhov, G., Mubarak, H., Derczynski, L., Pitenis, Z., & Çöltekin, C. (2020). SemEval-2020 task 12: Multilingual offensive language identification in social media (OffensEval 2020). Proceedings of the 14th international workshop on semantic evaluation. Retrieved from https://arxiv.org/abs/2006.07235 DOI: https://doi.org/10.18653/v1/2020.semeval-1.188
Zviel-Girshin, R., Kuhn, T. Z., Luís, A. R., Koppel, K., Šandrih Todorović, B., Holdt, Š. A., Tiberius, C., & Kosem, I. (2021). Developing pedagogically appropriate language corpora through crowdsourcing and gamification. In N. Zoghlami, C. Brudermann, C. Sarré, M. Grosbois, L. Bradley, & S. Thouësny (Eds), CALL and professionalisation: short papers from EUROCALL 2021 (pp. 312–317). doi: 10.14705/rpnet.2021.54.1352 DOI: https://doi.org/10.14705/rpnet.2021.54.1352
Prenosi
Objavljeno
Številka
Rubrika
Licenca
Avtorske pravice (c) 2022 Tanara Zingano Kuhn, Špela Arhar Holdt, Iztok Kosem, Carole Tiberius, Kristina Koppel, Rina Zviel-Girshin
To delo je licencirano pod Creative Commons Priznanje avtorstva-Deljenje pod enakimi pogoji 4.0 mednarodno licenco.
Kako citirati
