Revolution Lullabye

April 28, 2009

Moss, Can There Be Validity without Reliability?

Moss, Pamela. “Can There Be Validity without Reliability?” Educational Researcher 23.4(1994): 5-12. In Assessing Writing. Eds. Huot and O’Neill. Boston: Bedford/St. Martin’s, 2009. 81-96.

Moss challenges the primacy of reliability in assessment practices, arguing for the value of contextual, hermeneutic alternative assessments that can more accurately reflect the complex nature of writing tasks, knowledge, and performances. She describes the difference between hermeneutic and pyschometric evaluation, the latter which uses outside scorers or readers that do not know the context of the task, curriculumĀ or the student, as teachers would. Pointing out that many high-stakes assessments are not standardized or generalizable (like tenure, granting graduate degrees), she argues that the warrant that writing assessment scholars use in the argument of generalizability, the warrant of standardization, needs to be re-evaluated and rearticulated from a hermeneutic perspective. By making reliability (meaning standardization, I think) an option rather than a requirement, assessment practices can be opened up that reflect more of a range of educational goals.

Quotable Quotes

Hermeneutic: “an ethic of disciplined, collaborative inquiry that encourages challenges and revisions to initial interpretations; and the transparency of the trail of evidence leading to the interpretations, which allows users to evaluate the conclusions for themselves” (87).

“There are certain intellectual activities that standardized assessments can neither document nor promote” (84).

“potential of a hermeneutic approach to drawing and warranting interpretations of human products or performances” (85).

Notable Notes

some hermeneutic assessment practices: allowing studnets to choose the products they feel best represent them (not just the same tasks for all) – fair, ethical, and places agency in the student; alos critical discussion and debate during assessment, disagreement does not equate invalidity, the importance of a dialogic perspective of a community (what Broad and Huot draw on)

detached, impartial scorers silence the teachers, those who know students and curriculum best

look @ public education accountability movement

April 15, 2009

Broad, What We Really Value

Broad, Bob. What We Really Value: Beyond Rubrics in Teaching and Assessing Writing. Logan: Utah State UP, 2003.

Broad introduces the practice of dynamic criteria mapping (DCM) as an inquiry-driven alternative to static, traditional rubrics that have a normative rather than descriptive function, not even addressing many of the things are taught in writing classes (therefore not a valid assessment). His book is a case study of the use of DCM at “City University,” a university with 4000 students in a 3-course English sequence that is assessed through portfolios, graded collectively by 3-teacher teams. Instead of starting with certain textual features to check off, DCM asks teachers and assessors to describe what they see in a text (good and bad.) Together, the instructors find synonyms and antonyms for what they notice, categorize similar ones, and create a visual map that illustrates the values about good writing that the program’s teachers hold collectively. This method, though time-consuming and messy, better articulates the complex processes and ideas that students are showing in their writing. The process is locally, site-baed: though the method of DCM can be used, individual maps cannot be transported across institutions or even across years; it should be a conversation about values that happens continually.

Quotable Quotes

“We can now face the truth equipped with tools (qualitative inquiry) and attitudes (hermeneutics) that help us tap the energy of apparant chaos without being consumed by it. We can embrace the life of things” (137).

“In their rush toward clarity, simplicity, brevity, and authority, traditional scoring guides make substantial knowledge claims based on inadequate research” (3)

“In pursuit of their normative and formative purposes, traditional rubrics surrender thier descriptive and informative potential: responsiveness, detail, and complexity in accounting for how writing is actually evaluated” (2).

“The age of the rubric has passed” (4)

Notable Notes

Vinland map – not appropriate now

move to validity(not the same as reliability)

the DCM finds textual criteria and contextual criteria (things not found in text but have an impact on assessing, before DCM these have not been visible)

benefits of DCM: 1. student learning (shows writing is more complex, they have a better understanding of what they’re doing well andĀ  what teachers are looking for); 2. professional development and community; 3. program development and evaluation; 4. more valid assessment; 5. better relations with the public (values are made public, written down)

drawbacks? time-consuming and needs constant reflection and revisiting

must happen in communal writing assessment so there will be debate, disagreement, and discussion of values.

once the values are visible, you can start having conversations about whether you should value what you do.

a search for truth through hermeneutics, not psychometrics

Create a free website or blog at