Evaluation and enhancement of practicality in educational assessments: a conceptual analysis and three cases

Abstract

Educational assessments are usually evaluated in terms of their validity and impact. Practicality has also been presented as an additional evaluative criterion in some accounts (e.g. Bachman & Palmer, 1996; Cambridge English Language Assessment, 2013; Newton & Shaw, 2014), but the concept has received little further theoretical or empirical study. The present portfolio thesis, consisting of a theoretical overview and three empirical research articles, addresses this gap. The theoretical overview presents an expanded definition of practicality: as a continuous (rather than binary) variable describing the extent to which all kinds of resource (e.g. material, financial, knowledge-related, political) allow an assessment design to be administered across a range of contexts and time-frames, while achieving the intended assessment purposes. Using this expanded definition, three major strategies for enhancing practicality are presented, and these strategies are exemplified in the three empirical research articles. Article 1 shows that school-leaving assessments in China (‘gaokao’) have a simpler, more centralisable design than those in England (A levels), which is more suited to the economic and logistical limitations of the Chinese context. This exemplifies the first practicality-enhancing strategy: reducing the resources required by the assessment design. Article 2 shows that, for subjectively-scored assessments in higher education, while the think-aloud ‘rater workshop’ method of rating scale development recommended by large assessment providers may be beyond the resources available in HE, universities can achieve largely the same effect by re-purposing an already existing resource – the tutor-to-student feedback comments from previous assessments. This exemplifies the second practicality-enhancing strategy: innovative design to deploy available resources more efficiently. Article 3 shows that, in international assessments of English proficiency, one reason for IELTS’ greater popularity compared to TOEFL may be that certain features of its design (e.g. live interaction with a speaking examiner) are intrinsically more attractive to test-takers. This exemplifies the third practicality-enhancing strategy: assessment design which attracts greater external investment of resource. It is shown that the findings of the three articles can be accommodated using the expanded definition of practicality. This more developed account of assessment practicality can thus be applied by assessment designers across a wide range of educational contexts, both when evaluating the practicality of an assessment design, and when seeking to enhance it.

Date of Award	24 Aug 2018
Original language	English
Awarding Institution	Univerisity of Nottingham
Supervisor	John Lowe (Supervisor)