Moss, Pamela. “Can There Be Validity without Reliability?” Educational Researcher 23.4(1994): 5-12. In Assessing Writing. Eds. Huot and O’Neill. Boston: Bedford/St. Martin’s, 2009. 81-96.
Moss challenges the primacy of reliability in assessment practices, arguing for the value of contextual, hermeneutic alternative assessments that can more accurately reflect the complex nature of writing tasks, knowledge, and performances. She describes the difference between hermeneutic and pyschometric evaluation, the latter which uses outside scorers or readers that do not know the context of the task, curriculum or the student, as teachers would. Pointing out that many high-stakes assessments are not standardized or generalizable (like tenure, granting graduate degrees), she argues that the warrant that writing assessment scholars use in the argument of generalizability, the warrant of standardization, needs to be re-evaluated and rearticulated from a hermeneutic perspective. By making reliability (meaning standardization, I think) an option rather than a requirement, assessment practices can be opened up that reflect more of a range of educational goals.
Hermeneutic: “an ethic of disciplined, collaborative inquiry that encourages challenges and revisions to initial interpretations; and the transparency of the trail of evidence leading to the interpretations, which allows users to evaluate the conclusions for themselves” (87).
“There are certain intellectual activities that standardized assessments can neither document nor promote” (84).
“potential of a hermeneutic approach to drawing and warranting interpretations of human products or performances” (85).
some hermeneutic assessment practices: allowing studnets to choose the products they feel best represent them (not just the same tasks for all) – fair, ethical, and places agency in the student; alos critical discussion and debate during assessment, disagreement does not equate invalidity, the importance of a dialogic perspective of a community (what Broad and Huot draw on)
detached, impartial scorers silence the teachers, those who know students and curriculum best
look @ public education accountability movement