The Challenge of Assessing Content and Coherence
DOI:
https://doi.org/10.4312/vestnik.14.275-292Keywords:
writing assessment, analytic scoring, content and coherence, scoring scales, descriptorsAbstract
Content and coherence are the categories most difficult to evaluate fairly when raters use analytic scoring scales. Readers inevitably interpret texts in their own idiosyncratic ways, depending on their knowledge, experience, ethical considerations, and other personal biases that they cannot completely set aside when grading a text. This is also true for descriptors, which are themselves short texts. To make matters worse, due to the very nature of writing but also the lack of consensus among experts in discourse research, writing theory, and writing assessment, descriptors are categorized vaguely and inconsistently. As a result, raters seeking useful evaluation criteria are confronted with descriptors that cover the same concept, such as “relevance”, being categorized in one set of criteria as relating to the content of the written text and in another as belonging to the category of coherence. Nevertheless, the objectivity of the evaluation of written work can be increased. The article examines the relationship between content and coherence, which is reflected in the way the two concepts are defined in the relevant literature, as well as in some descriptors used in two grading scales used in Slovenia. The empirical part of the paper presents a case study involving 46 secondary school teachers, whose responses to a questionnaire confirm the subjectivity of the understanding of individual descriptors and the need for adequate training of teachers in the use of analytic scoring scales, regular standardization in the schools where they work, evaluation of the assessment scales they use and their possible adaptation.
Downloads
References
ALARO, Abebayehu Anjulo (2020) An Assessment of Cohesion and Coherence in Students’ Descriptive and Narrative Essays. Journal of Literature, Languages and Linguistics 64, 41–46.
BACHA, Nahla (2001) Writing evaluation: what can analytic versus holistic essay scoring tell us? System 29, 371–383. DOI:10.1016/S0346-251X(01)00025-2. DOI: https://doi.org/10.1016/S0346-251X(01)00025-2
BAE, Jungok (2001) Cohesion and Coherence in Children’s Written English: Immersion and English-Only Classes. Issues in Applied Linguistics 12 (1), 51–88. https://doi.org/10.5070/L4121005043. DOI: https://doi.org/10.5070/L4121005043
BAŠ, Ivica/Saša BENULIČ/Margaret DALRYMPLE/Vineta ERŽEN/Soča FIDLER,/Majda GRABAR/Meta GROSMAN/Aleša JUVANC/Smiljana KOMAR/Cvetka SOKOLOV/ Rastislav ŠUŠTARŠIČ (1996) Angleščina pri maturi: Kako se uspešno pripravimo na preizkus znanja iz angleškega jezika. Ljubljana: Državni izpitni center.
BEAN, John C. (2011) Engaging Ideas: The Professor’s Guide to Integrating Writing, Critical Thinking and Active Learning in the Classroom. 2nd edn. San Francisco: Jossey-Bass.
BRIESMASTER, Mark/Paulo ETCHEGARAY (2017) Coherence and cohesion in EFL students’ writing production: The impact of a metacognition-based intervention. Íkala, Revista de Lenguaje y Cultura 22 (2), 183 –202. DOI: 10.17533/udea.ikala.v22n02a02. DOI: https://doi.org/10.17533/udea.ikala.v22n02a02
BROWN, H. Douglas (2004) Language Assessment: Principles and Classroom Practices. New York: Longman.
BROWN, H. Douglas/Priyanvada ABEYWICKRAMA (2019) Language Assessment: Principles and Classroom Practices. 3rd edn. Hoboken: Pearson Education ESL.
BROWN, Gillian/George YULE (1983) Discourse Analysis. Cambridge: Cambridge University Press. DOI: https://doi.org/10.1017/CBO9780511805226
CARREL, Patricia L (1982) Cohesion is not coherence. TESOL Quarterly 16 (4), 479–488. https://doi.org/10.2307/3586466. DOI: https://doi.org/10.2307/3586466
CARTER, Ronald (1993) Introducing Applied Linguistics. Harmondsworth, Middlesex: Penguin.
CUSHING WEIGLE, Sara (1998) Using FACETS to model rater training effects. Language Testing 15, 263–287. https://doi.org/10.1177/026553229801500205. DOI: https://doi.org/10.1177/026553229801500205
CUSHING WEIGLE, Sara (2010) Assessing Writing. Cambridge: Cambridge University Press.
DOUGLAS, Yellowlees (2015) The Reader’s Brain: How Neuroscience Can Make You a Better Writer. Cambridge: Cambridge University Press. DOI: https://doi.org/10.1017/CBO9781316178942
ECKES, Thomas (2008) Rater types in writing performance assessments: A classification approach to rater variability. Language Testing 25 (2), 155–185. https://doi.org/10.1177/0265532207086780. DOI: https://doi.org/10.1177/0265532207086780
ELBOW, Peter (1996) Writing Assessment: Do It Better, Do It Less. E. M. White/W. D. Lutz/S. Kamusikiri (eds.), Assessment of Writing: Politics, Policies, Practices. New York: The Modern Language Association of America, 120–134.
ENKVIST, Nils Erik (1990) Seven Problems in the Study of Coherence and Interpretability. U. Connor/A. M. Johns (eds.), Coherence in Writing: Research and Pedagogical Perspectives. Alexandria: TESOL, 11–28.
GLENN, Cheryl/Melissa A. GOLDTHWAITE (2014) The St. Martin’s Guide to Teaching Writing. 3rd edn. Boston/New York: St. Martin’s.
HALLIDAY, M. A. K./Ruqaiya HASAN (1976) Cohesion in English. London/New York: Longman.
HOEY, Michael (1991) Patterns of Lexis in Text. Oxford: Oxford University Press.
HOLDSTEIN, Deborah H. (1996) Gender, Feminism, and Institution-Wide Assessment Programs. E. M. White/W, D. Lutz/S. Kamusikiri (eds.), Assessment of Writing: Politics, Policies, Practices. New York: The Modern Language Association of America, 204–225.
HUGHES, Arthur (1989/2003) Testing for Language Teachers. 2nd edn. Cambridge: Cambridge University Press.
HYLAND, Ken (2002) Teaching and Researching Writing. Harlow/London/New York: Longman.
HYLAND, Ken (2003) Second Language Writing. Cambridge: Cambridge University Press. DOI: https://doi.org/10.1017/CBO9780511667251
IB (1992) Workshops in English B for New Teachers: Working Materials. Cardiff: International Baccalaureate Europe.
ILC, Gašper/Alenka KETIŠ/Aleksandra KOMADINA/Ana LIKAR/Simona MEGLIČ/Irena ZORKO NOVAK (2018) Angleščina: Predmetni izpitni katalog za splošno maturo. Ljubljana: Državni izpitni center.
JACOBS, Holly L./Stephen A. ZINKGRAF/Deanna R. WORMUTH/V. Faye HARTFIEL/ Jane B. HUGHEY (1981) Testing ESL composition: A practical approach. Rowley, MA: Newbury House.
KNOCH, Ute (2007) ‘Little coherence, considerable strain for reader’: A comparison between two rating scales for the assessment of coherence. Assessing Writing 12 (2), 108–128. https://doi.org/10.1016/j.asw.2007.07.002. DOI: https://doi.org/10.1016/j.asw.2007.07.002
McNAMARA, Tim (2000) Language Testing. Oxford: Oxford University Press.
NUNAN, David (1993) Introducing Discourse Analysis. Harmondsworth: Penguin.
PARKS, A. Franklin/James A. LEVERNIER/Ida MASTERS HOLLOWELL (1991) Structuring Paragraphs: A Guide to Effective Writing. 3rd edn. New York: St. Martin’s Press.
RICHARDS, Jack C./John TALBOT PLATT/Heidi PLATT (1992) Dictionary of Language Teaching and Applied Linguistics. 2nd edn. Harlow: Longman.
RICHARDS, Keith (2011) Case Study. E. Hinkel (ed.), Handbook of Research in Second Language Teaching and Learning. 2nd edn. New York/London: Routledge, 207–221.
SHAW, Stuart D./Cyril Weir (2007) Examining Writing. Cambridge: Cambridge University Press.
SKOUFAKI, Sophia (2020) Rhetorical Structure Theory and coherence break identification. Text & Talk 40 (1), 99–124. https://doi-org.nukweb.nuk.uni-lj.si/10.1515/text-2019-2050. DOI: https://doi.org/10.1515/text-2019-2050
SOKOLOV, Cvetka (1999) Pisni sestavek pri študentih angleščine (MA Thesis). Ljubljana: Filozofska fakulteta.
SOKOLOV, Cvetka (2000) Pisna zmožnost – motnje v koherenci. In: Meta GROSMAN (ed.) Angleščina – prenovi na pot. Ljubljana: Zavod RS za šolstvo, 97–136.
SOKOLOV, Cvetka (2013) Pomen standardizacije ocenjevanja pisni sestavkov pri poučevanju angleščine kot tujega jezika/The Role of Standardization in Assessing Writing in Teaching English as a Foreign Language (PhD Thesis). Ljubljana: Filozofska fakulteta.
SOKOLOV, Cvetka et al. (2014) Written Composition: Analytic Scoring Scales. Unpublished. Ljubljana: Filozofska fakulteta.
SOKOLOV, Cvetka (2018) Analiza zgledov meril za razčlenjevalno ocenjevanje raz-
lagalnih in utemeljevalnih pisnih sestavkov. Vestnik 10 (1), 169–186. DOI: 10.4312/vestnik.10.169-186. DOI: https://doi.org/10.4312/vestnik.10.169-186
SPANDEL, Vicki/Richard J. STIGGINS (1990) Creating Writers: Linking Assessment and Writing Instruction. New York/London: Longman.
VOGRINC, Janez (2008) Kvalitativno raziskovanje na pedagoškem področju. Ljubljana: Pedagoška fakulteta.
WEIR, Cyril/Jon ROBERTS (1994) Evaluation in ELT. Oxford UK/Cambridge USA: Blackwell.
WHITE, Edward M (1996) Power and Agenda Setting in Writing Assessment. E. M.
WHITE, W. D. Lutz/S. Kamusikiri (eds.), Assessment of Writing: Politics, Policies, Practices. New York: The Modern Language Association of America, 9–24.
WIDDOWSON, Henry (1978) Teaching Language as Communication. Oxford: Oxford University Press.
WIKBORG, Eleanor (1990) Types of Coherence Breaks in Swedish Student Writing: Misleading Paragraph Division. U. Connor /A. M. Johns (eds.), Coherence in Writing: Research and Pedagogical Perspectives. Alexandria: TESOL, 133–148.
WILSON, Gerald L./Michael S. HANNA (1993) Groups in Context: Leadership and Participation in Small Groups. 3rd edn. New York: McGraw-Hill.
YU, Guoxing (2007) Student’s Voices in the Evaluation of Written Summaries. Language Testing 24 (4), 239–572. https://doi.org/10.1177/0265532207080780. DOI: https://doi.org/10.1177/0265532207080780
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2022 Cvetka Sokolov
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors are confirming that they are the authors of the submitting article, which will be published (print and online) in journal Journal for Foreign Languages by Znanstvena založba Filozofske fakultete Univerze v Ljubljani (University of Ljubljana, Faculty of Arts, Aškerčeva 2, 1000 Ljubljana, Slovenia). Author’s name will be evident in the article in journal. All decisions regarding layout and distribution of the work are in hands of the publisher.
- Authors guarantee that the work is their own original creation and does not infringe any statutory or common-law copyright or any proprietary right of any third party. In case of claims by third parties, authors commit their self to defend the interests of the publisher, and shall cover any potential costs.
- Authors retain copyright and grant the publisher right of first publication with the work simultaneously licensed under a Creative Commons Attribution-ShareAlike 4.0 International License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.
