AN EFFICIENT METHOD TO EVALUATE AND ENHANCE SPORT JUDGES’ PERFORMANCE DURING COMPETITION: A CASE STUDY IN ACROBATIC GYMNASTICS
DOI:
https://doi.org/10.52165/sgj.16.3.387-400Keywords:
excel-based tool, acrobatic gymnastics, judges, evaluation, ACROAJIR®Abstract
Who judges the gymnastics judges? How do we measure their accuracy and concordance? How do we know if the judging process is fair? The superior jury has this responsibility. However, they normally lack time to provide effective feedback during competitions.
Using macro functions to process statistical and mathematical statements, we designed and validated an automated Excel-based tool called the Automatic Acrobatic Gymnastics Judges Individual Report Tool to evaluate judges’ performances quickly and easily during competition, automatically creating and exporting individual reports showing each judge’s accuracy and concordance performance on a daily basis, rather than after the competition is ended.
We present empirical data for 76 experienced international judges evaluating acrobatic gymnastics routines in four major official events. A total of 1240 individual reports were analyzed and sent confidentially to the judges during the competition, and 952 were analyzed to evaluate whether this feedback was effective in improving judges’ performance during the competition.
The tool provides efficient and easily understood evidence-based feedback on acrobatic gymnastics judges’ performance during competition, quickly and automatically creating, analyzing and sending individualized information to judges, thus helping with specific Technical Committee scoring control tasks during competitions. We suggest that judges' performances remain high or are enhanced after receiving daily evaluation during major competition events.
Downloads
References
Akoglu, H. (2018). User’s guide to correlation coefficients. Turkish Journal of Emergency Medicine, 18(3), 91–93. https://doi.org/10.1016/j.tjem.2018.08.001
Anderlucci, L., Lubisco, A., & Mignani, S. (2020). Investigating the Judges Performance in a National Competition of Sport Dance. Social Indicators Research. https://doi.org/10.1007/s11205-019-02256-z
Auweele, Y. Vanden, Boen, F., De Geest, A., & Feys, J. (2004). Judging bias in synchronized swimming: Open feedback leads to nonperformance-based conformity. Journal of Sport and Exercise Psychology, 26(4), 561–571. https://doi.org/10.1123/jsep.26.4.561
Boen, F., Ginis, P., & Smits, T. (2013). Judges in judo conform to the referee because of the reactive feedback system. European Journal of Sport Science, 13(6), 599–604. https://doi.org/10.1080/17461391.2012.756070
Boen, F., van Hoye, K., Auweele, Y. Vanden, Feys, J., & Smits, T. (2008). Open feedback in gymnastic judging causes conformity bias based on informational influencing. Journal of Sports Sciences, 26(6), 621–628. https://doi.org/10.1080/02640410701670393
Boen, F., Vanden Auweele, Y., Claes, E., Feys, J., & De Cuyper, B. (2006). The impact of open feedback on conformity among judges in rope skipping. Psychology of Sport and Exercise, 7(6), 577–590. https://doi.org/10.1016/j.psychsport.2005.12.001
Bučar, M., Čuk, I., Pajek, J., Karacsony, I., & Leskošek, B. (2012). Reliability and validity of judging in women’s artistic gymnastics at University Games 2009. European Journal of Sport Science, 12(3), 207–215. https://doi.org/10.1080/17461391.2010.551416
Dallas, G., Mavidis, A., & Chairopoulou, C. (2011). Influence of angle of view on judges’ evaluations of inverted cross in men’s rings. Perceptual and Motor Skills, 112(1), 109–121. https://doi.org/10.2466/05.22.24.27.PMS.112.1.109-121
Damisch, L., Mussweiler, T., & Plessner, H. (2006). Olympic medals as fruits of comparison? Assimilation and contrast in sequential performance judgments. Journal of Experimental Psychology: Applied, 12(3), 166–178. https://doi.org/10.1037/1076-898X.12.3.166
De Bruin, W. B. (2005). Save the last dance for me: Unwanted serial position effects in jury evaluations. Acta psychologica, 118(3), 245-260. https://doi.org/10.1016/j.actpsy.2004.08.005
Dosseville, F., Laborde, S., & Garncarzyk, C. (2014). Current research in sports officiating and decision-making. In C. Mohiyeddini (Ed.), Contemporary topics and trends in the psychology of sports (pp.13–38). Nova Publishers.
Fasold, F., Memmert, D., & Unkelbach, C. (2012). Extreme judgments depend on the expectation of following judgments: A calibration analysis. Psychology of Sport and Exercise, 13(2), 197-200. https://doi.org/10.1016/j.psychsport.2011.11.004
FIG. (2015). Regulations for the judges’ evaluation programme (JEP) “former fairbrother system” and its’ application. Federation International de Gymnastique.
FIG. (2020). 2017- 2020 Fig Judges’ Rules Specific Rules for Acrobatic. In Specific Rules for Acrobatic Gymnastics. Federation International de Gymnastique.
FIG. (2017). Appendix to the Codes of Points (COP). Federation International de Gymnastique.
Flessas, K., Mylonas, D., Panagiotaropoulou, G., Tsopani, D., Korda, A., Siettos, C., Di Cagno, A., Evdokimidis, I., & Smyrnis, N. (2015). Judging the judges’ performance in rhythmic gymnastics. Medicine and Science in Sports and Exercise, 47(3), 640–648. https://doi.org/10.1249/MSS.0000000000000425
Gambarelli, G., Iaquinta, G., & Piazza, M. (2012). Anti-collusion indices and averages for the evaluation of performances and judges. Journal of Sports Sciences, 30(4), 411–417. https://doi.org/10.1080/02640414.2011.651153
Heiniger, S., & Mercier, H. (2018). National Bias of International Gymnastics Judges during the 2013-2016 Olympic Cycle. http://arxiv.org/abs/1807.10033
Heiniger, S., & Mercier, H. (2021). Judging the judges: evaluating the accuracy and national bias of international gymnastics judges. Journal of Quantitative Analysis in Sports, 17(4), 289-305. https://doi.org/10.1515/jqas-2019-0113
Kramer RSS. (2017) Sequential effects in Olympic synchronized diving scores. Royal Society Open science. 4: 160812. http://dx.doi.org/10.1098/rsos.160812
Kendall, M. G., & Babington-Smith, B. (1939). The Problem of m Rankings. The Annals of Mathematical Statistics, 10(3), 275–287. https://doi.org/http://dx.doi.org/10.1214/aoms/1177732186
León-Prados, J. A., & Jemni, M. (2022). Reliability and agreement in technical and artistic scores during real-time judging in two European acrobatic gymnastic events. International Journal of Performance Analysis in Sport, 22(1), 132–148. https://doi.org/10.1080/24748668.2021.1996913
León-Prados, J. A., & Rosales, A. (2019). ACROAJIR®; Automatic ACRO Judges Individual Report Tool. Universidad Pablo de Olavide.
Leskošek, B., Čuk, I., & Peixoto, C. J. D. (2018). Inter-rater reliability and validity of scoring men’s individual trampoline routines at European championships 2014. Science of Gymnastics Journal, 10(1), 69–79.
Lin, L. I. (1989). A concordance correlation coefficient to evaluate reproducibility. Biometrics, 45(1), 255–265.
McBride, G. (2005). A proposal for strength-of-agreement criteria for Lin’s Concordance Correlation Coefficient. NIWA Client Report, HAM2005-062 https://www.medcalc.org/download/pdf/McBride2005.pdf
Mercier, H., & Heiniger, S. (2018). Judging the Judges: Evaluating the Performance of International Gymnastics Judges. http://arxiv.org/abs/1807.10021
Mercier, H., & Klahn, C. (2017). Judging the judges : Evaluating the performance of international gymnastics judges. MIT Sloan Sports Analytics. https://arxiv.org/pdf/1807.10021.pdf
Plessner, H., & Schallies, E. (2005). Judging the cross on rings: A matter of achieving shape constancy. Applied Cognitive Psychology, 19(9), 1145–1156. https://doi.org/10.1002/acp.1136
Premelč, J., Vučković, G., James, N., & Leskošek, B. (2019). Reliability of judging in DanceSport. Frontiers in Psychology, 10. 1001. https://doi.org/10.3389/fpsyg.2019.01001
Ste-Marie, D. M. (2000). Expertise in women’s gymnastics judging: an observational approach. Perceptual and Motor Skills, 90(2), 543–546. https://doi.org/10.2466/pms.2000.90.2.543
Van Bokhorst, L. G., Knapová, L., Majoranc, K., Szebeni, Z. K., Táborský, A., Tomić, D., & Cañadas, E. (2016). “It’s always the judge’s fault”: Attention, emotion recognition, and expertise in rhythmic gymnastics assessment. Frontiers in Psychology, 7(JUL). https://doi.org/10.3389/fpsyg.2016.01008
Wallis, W. A. (1939). The Correlation Ratio for Ranked Data. Journal of the American Statistical Association, 34(207), 533–538. https://doi.org/10.1080/01621459.1939.10503552
Downloads
Published
Issue
Section
License
Copyright (c) 2024 Juan Antonio Leon - Prados, Carmen Ramos
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
How to Cite
