The Challenge of Assessing Content and Coherence

Authors

  • Cvetka Sokolov University of Ljubljana, Faculty of Arts, Slovenia

DOI:

https://doi.org/10.4312/vestnik.14.275-292

Keywords:

writing assessment, analytic scoring, content and coherence, scoring scales, descriptors

Abstract

Content and coherence are the categories most difficult to evaluate fairly when raters use analytic scoring scales. Readers inevitably interpret texts in their own idiosyncratic ways, depending on their knowledge, experience, ethical considerations, and other personal biases that they cannot completely set aside when grading a text. This is also true for descriptors, which are themselves short texts. To make matters worse, due to the very nature of writing but also the lack of consensus among experts in discourse research, writing theory, and writing assessment, descriptors are categorized vaguely and inconsistently. As a result, raters seeking useful evaluation criteria are confronted with descriptors that cover the same concept, such as “relevance”, being categorized in one set of criteria as relating to the content of the written text and in another as belonging to the category of coherence. Nevertheless, the objectivity of the evaluation of written work can be increased. The article examines the relationship between content and coherence, which is reflected in the way the two concepts are defined in the relevant literature, as well as in some descriptors used in two grading scales used in Slovenia. The empirical part of the paper presents a case study involving 46 secondary school teachers, whose responses to a questionnaire confirm the subjectivity of the understanding of individual descriptors and the need for adequate training of teachers in the use of analytic scoring scales, regular standardization in the schools where they work, evaluation of the assessment scales they use and their possible adaptation.

Downloads

Download data is not yet available.

References

ALARO, Abebayehu Anjulo (2020) An Assessment of Cohesion and Coherence in Students’ Descriptive and Narrative Essays. Journal of Literature, Languages and Linguistics 64, 41–46.

BACHA, Nahla (2001) Writing evaluation: what can analytic versus holistic essay scoring tell us? System 29, 371–383. DOI:10.1016/S0346-251X(01)00025-2. DOI: https://doi.org/10.1016/S0346-251X(01)00025-2

BAE, Jungok (2001) Cohesion and Coherence in Children’s Written English: Immersion and English-Only Classes. Issues in Applied Linguistics 12 (1), 51–88. https://doi.org/10.5070/L4121005043. DOI: https://doi.org/10.5070/L4121005043

BAŠ, Ivica/Saša BENULIČ/Margaret DALRYMPLE/Vineta ERŽEN/Soča FIDLER,/Majda GRABAR/Meta GROSMAN/Aleša JUVANC/Smiljana KOMAR/Cvetka SOKOLOV/ Rastislav ŠUŠTARŠIČ (1996) Angleščina pri maturi: Kako se uspešno pripravimo na preizkus znanja iz angleškega jezika. Ljubljana: Državni izpitni center.

BEAN, John C. (2011) Engaging Ideas: The Professor’s Guide to Integrating Writing, Critical Thinking and Active Learning in the Classroom. 2nd edn. San Francisco: Jossey-Bass.

BRIESMASTER, Mark/Paulo ETCHEGARAY (2017) Coherence and cohesion in EFL students’ writing production: The impact of a metacognition-based intervention. Íkala, Revista de Lenguaje y Cultura 22 (2), 183 –202. DOI: 10.17533/udea.ikala.v22n02a02. DOI: https://doi.org/10.17533/udea.ikala.v22n02a02

BROWN, H. Douglas (2004) Language Assessment: Principles and Classroom Practices. New York: Longman.

BROWN, H. Douglas/Priyanvada ABEYWICKRAMA (2019) Language Assessment: Principles and Classroom Practices. 3rd edn. Hoboken: Pearson Education ESL.

BROWN, Gillian/George YULE (1983) Discourse Analysis. Cambridge: Cambridge University Press. DOI: https://doi.org/10.1017/CBO9780511805226

CARREL, Patricia L (1982) Cohesion is not coherence. TESOL Quarterly 16 (4), 479–488. https://doi.org/10.2307/3586466. DOI: https://doi.org/10.2307/3586466

CARTER, Ronald (1993) Introducing Applied Linguistics. Harmondsworth, Middlesex: Penguin.

CUSHING WEIGLE, Sara (1998) Using FACETS to model rater training effects. Language Testing 15, 263–287. https://doi.org/10.1177/026553229801500205. DOI: https://doi.org/10.1177/026553229801500205

CUSHING WEIGLE, Sara (2010) Assessing Writing. Cambridge: Cambridge University Press.

DOUGLAS, Yellowlees (2015) The Reader’s Brain: How Neuroscience Can Make You a Better Writer. Cambridge: Cambridge University Press. DOI: https://doi.org/10.1017/CBO9781316178942

ECKES, Thomas (2008) Rater types in writing performance assessments: A classification approach to rater variability. Language Testing 25 (2), 155–185. https://doi.org/10.1177/0265532207086780. DOI: https://doi.org/10.1177/0265532207086780

ELBOW, Peter (1996) Writing Assessment: Do It Better, Do It Less. E. M. White/W. D. Lutz/S. Kamusikiri (eds.), Assessment of Writing: Politics, Policies, Practices. New York: The Modern Language Association of America, 120–134.

ENKVIST, Nils Erik (1990) Seven Problems in the Study of Coherence and Interpretability. U. Connor/A. M. Johns (eds.), Coherence in Writing: Research and Pedagogical Perspectives. Alexandria: TESOL, 11–28.

GLENN, Cheryl/Melissa A. GOLDTHWAITE (2014) The St. Martin’s Guide to Teaching Writing. 3rd edn. Boston/New York: St. Martin’s.

HALLIDAY, M. A. K./Ruqaiya HASAN (1976) Cohesion in English. London/New York: Longman.

HOEY, Michael (1991) Patterns of Lexis in Text. Oxford: Oxford University Press.

HOLDSTEIN, Deborah H. (1996) Gender, Feminism, and Institution-Wide Assessment Programs. E. M. White/W, D. Lutz/S. Kamusikiri (eds.), Assessment of Writing: Politics, Policies, Practices. New York: The Modern Language Association of America, 204–225.

HUGHES, Arthur (1989/2003) Testing for Language Teachers. 2nd edn. Cambridge: Cambridge University Press.

HYLAND, Ken (2002) Teaching and Researching Writing. Harlow/London/New York: Longman.

HYLAND, Ken (2003) Second Language Writing. Cambridge: Cambridge University Press. DOI: https://doi.org/10.1017/CBO9780511667251

IB (1992) Workshops in English B for New Teachers: Working Materials. Cardiff: International Baccalaureate Europe.

ILC, Gašper/Alenka KETIŠ/Aleksandra KOMADINA/Ana LIKAR/Simona MEGLIČ/Irena ZORKO NOVAK (2018) Angleščina: Predmetni izpitni katalog za splošno maturo. Ljubljana: Državni izpitni center.

JACOBS, Holly L./Stephen A. ZINKGRAF/Deanna R. WORMUTH/V. Faye HARTFIEL/ Jane B. HUGHEY (1981) Testing ESL composition: A practical approach. Rowley, MA: Newbury House.

KNOCH, Ute (2007) ‘Little coherence, considerable strain for reader’: A comparison between two rating scales for the assessment of coherence. Assessing Writing 12 (2), 108–128. https://doi.org/10.1016/j.asw.2007.07.002. DOI: https://doi.org/10.1016/j.asw.2007.07.002

McNAMARA, Tim (2000) Language Testing. Oxford: Oxford University Press.

NUNAN, David (1993) Introducing Discourse Analysis. Harmondsworth: Penguin.

PARKS, A. Franklin/James A. LEVERNIER/Ida MASTERS HOLLOWELL (1991) Structuring Paragraphs: A Guide to Effective Writing. 3rd edn. New York: St. Martin’s Press.

RICHARDS, Jack C./John TALBOT PLATT/Heidi PLATT (1992) Dictionary of Language Teaching and Applied Linguistics. 2nd edn. Harlow: Longman.

RICHARDS, Keith (2011) Case Study. E. Hinkel (ed.), Handbook of Research in Second Language Teaching and Learning. 2nd edn. New York/London: Routledge, 207–221.

SHAW, Stuart D./Cyril Weir (2007) Examining Writing. Cambridge: Cambridge University Press.

SKOUFAKI, Sophia (2020) Rhetorical Structure Theory and coherence break identification. Text & Talk 40 (1), 99–124. https://doi-org.nukweb.nuk.uni-lj.si/10.1515/text-2019-2050. DOI: https://doi.org/10.1515/text-2019-2050

SOKOLOV, Cvetka (1999) Pisni sestavek pri študentih angleščine (MA Thesis). Ljubljana: Filozofska fakulteta.

SOKOLOV, Cvetka (2000) Pisna zmožnost – motnje v koherenci. In: Meta GROSMAN (ed.) Angleščina – prenovi na pot. Ljubljana: Zavod RS za šolstvo, 97–136.

SOKOLOV, Cvetka (2013) Pomen standardizacije ocenjevanja pisni sestavkov pri poučevanju angleščine kot tujega jezika/The Role of Standardization in Assessing Writing in Teaching English as a Foreign Language (PhD Thesis). Ljubljana: Filozofska fakulteta.

SOKOLOV, Cvetka et al. (2014) Written Composition: Analytic Scoring Scales. Unpublished. Ljubljana: Filozofska fakulteta.

SOKOLOV, Cvetka (2018) Analiza zgledov meril za razčlenjevalno ocenjevanje raz-

lagalnih in utemeljevalnih pisnih sestavkov. Vestnik 10 (1), 169–186. DOI: 10.4312/vestnik.10.169-186. DOI: https://doi.org/10.4312/vestnik.10.169-186

SPANDEL, Vicki/Richard J. STIGGINS (1990) Creating Writers: Linking Assessment and Writing Instruction. New York/London: Longman.

VOGRINC, Janez (2008) Kvalitativno raziskovanje na pedagoškem področju. Ljubljana: Pedagoška fakulteta.

WEIR, Cyril/Jon ROBERTS (1994) Evaluation in ELT. Oxford UK/Cambridge USA: Blackwell.

WHITE, Edward M (1996) Power and Agenda Setting in Writing Assessment. E. M.

WHITE, W. D. Lutz/S. Kamusikiri (eds.), Assessment of Writing: Politics, Policies, Practices. New York: The Modern Language Association of America, 9–24.

WIDDOWSON, Henry (1978) Teaching Language as Communication. Oxford: Oxford University Press.

WIKBORG, Eleanor (1990) Types of Coherence Breaks in Swedish Student Writing: Misleading Paragraph Division. U. Connor /A. M. Johns (eds.), Coherence in Writing: Research and Pedagogical Perspectives. Alexandria: TESOL, 133–148.

WILSON, Gerald L./Michael S. HANNA (1993) Groups in Context: Leadership and Participation in Small Groups. 3rd edn. New York: McGraw-Hill.

YU, Guoxing (2007) Student’s Voices in the Evaluation of Written Summaries. Language Testing 24 (4), 239–572. https://doi.org/10.1177/0265532207080780. DOI: https://doi.org/10.1177/0265532207080780

Downloads

Published

23.12.2022

How to Cite

Sokolov, C. (2022). The Challenge of Assessing Content and Coherence. Journal for Foreign Languages, 14(1), 275–292. https://doi.org/10.4312/vestnik.14.275-292

Issue

Section

Articles