Revolution Lullabye

May 1, 2009

Veal and Hudson, Direct and Indirect Measures for Large-Scale Evaluation of Writing

Veal, L. Ramon and Sally Ann Hudson. “Direct and Indirect Measures for Large-Scale Evaluation of Writing.” Research in the Teaching of English. 17 (1983): 290-296. In Assessing Writing. Eds. Huot and O’Neill. Boston: Bedford/St. Martin’s, 2009. 13-18.

Using data collected from over 2400 students attending 24 geographically, ethnically, and culturally diverse Georgia high schools, Veal and Hudson argue that holistically-scored essays allow for the best balance of validity, reliability, and cost-effectiveness in writing assessments. The students took different kinds of national achievement tests which scored their essays using holistic, analytic, primary-trait, or mechanics-based assessments, and the researchers detailed the validity, reliability, and costs associated with the different measures.

Quotable Quotes

“The consideration of the user at this point becomes whether the cost or the face validity of a direct assessment of writing is more critical” (18)

Notable Notes

1983 publication, before portfolios came on the scene, doesn’t address larger questions of ethics or the nature of validity

April 29, 2009

Williamson, The Worship of Efficiency

Williamson, Michael. “The Worship of Efficiency: Untangling Theoretical and Practical Considerations in Writing Assessment.” Assessing Writing 1(1994): 147-174. In Assessing Writing. Eds. Huot and O’Neill. Boston: Bedford/St. Martin’s, 2009. 57-80.

Williamson argues that educators must adopt a different educational model – that of the craft workshop – in order to create an assessment theory and practice that breaks the hold of the god-terms of efficiency, fairness, and reliability. Williamson traces how the concept of efficiency led assessment and educational practices during much of the 20th century, resulting in invalid assessments only based on one data point, assessments grounded in standardized tests that allowed for the development and dominance of factory and bureaucratic educational models. He points to other assessment practices, like those in France, rely on interviews and non-standardized assessments given by the teacher, who knows the curriculum and students best. If teachers are to be treated as the professoinals that they are, Williamson argues, they should be given the right and the responsibility to develop and give assessments to their students.

Quotable Quotes

“we will need to begin to trust teachers” (78).

“the privilege of true professionalism” (79).

“For the most part, students are assessed, labeled, and placed in school curricula on the basis of their scores on succeeding standardized tests…these tests remain one of the single most important indicators of a child’s future” (67).

“efficiency has governed both the theoretical and practical developments in assessment” (69).

Notable Notes

development of psychometrics to allow for an objective, outside scorer – this is reversed in the craft workshop model with teacher in charge

child-centered assessment v. system-centered assessment

libertarian assessment

history of shift from oral exams to written exams to multiple-choice testing (Arthur Otis)

efficiency is a key American cultural and social force

craft workshop model (Shedd and Bacharach; Schon’s reflective practicioner)

assessment as a contextual, dynamic, continuous, reflective process

assessments with multiple data points converging = valid

April 28, 2009

Yancey, Looking Back as We Look Forward

Yancey, Kathleen Blake. “Looking Back as We Look Forward: Historicizing Writing Assessment.” CCC 50 (1999): 483-503. Reprinted in Assessing Writing: A Critical Sourcebook. Eds. Huot and O’Neill. Boston: Bedford/St. Martin’s, 2009. 131-149.

Yancey divides the history of writing assessment into three “waves.” The first wave (1950-1970) focused on objective, non-essay testing that prioritized efficiency and reliability. The second wave (1970-1986) moved towards holistic scoring of essays, based on rubrics and scoring guides first developed through ETS and AP. The third wave (1986-present) expanded assessment to include portfolios (consisting of multiple examples of student writing) and larger, programmatic assessments. She looks at these waves from several perspectives: at how the concepts of reliability and validity are negotiated and valued; at the struggle between the knowledge of the assessment expert (and psychometrics) and the contextual, local knowledge of the non-expert teacher (and hermeneutics); and the move of assessment from outside and before the classroom to within and after the class. She voices concerns and directions for further scholarship and practice in writing assessment, challenging the field to look for ways to use assessment rhetorically and ethically to help students and programs develop and to produce scholarly knowledge.

Quotable Quotes

“It is the self we want to teach, that we hope will learn, but that we are often loathe to evaluate. What is the role of the person/al in any writing assessment?” (132).

Notable Notes

The role of the classroom teacher moving into writing assessment: in the 1st wave, testing specialists evaluated, but through the 2nd and 3rd waves, the roles of teacher and evaluator overlapped into the new discipline of writing assessment.

Questions: Who is authorized to speak about assessment? What is the purpose of writing assessment in education? Who shuold hold hte power?

the use of portfolios shifts the purpose and goals of the assessment: using pass/fail instead of scoring and communal grading moves more towards assessing a program and establishing community values than individual student assessment. Use different stakeholders to read?

waves fall into each other, aren’t strict lines that categorize what’s happening.

Moss, Can There Be Validity without Reliability?

Moss, Pamela. “Can There Be Validity without Reliability?” Educational Researcher 23.4(1994): 5-12. In Assessing Writing. Eds. Huot and O’Neill. Boston: Bedford/St. Martin’s, 2009. 81-96.

Moss challenges the primacy of reliability in assessment practices, arguing for the value of contextual, hermeneutic alternative assessments that can more accurately reflect the complex nature of writing tasks, knowledge, and performances. She describes the difference between hermeneutic and pyschometric evaluation, the latter which uses outside scorers or readers that do not know the context of the task, curriculum or the student, as teachers would. Pointing out that many high-stakes assessments are not standardized or generalizable (like tenure, granting graduate degrees), she argues that the warrant that writing assessment scholars use in the argument of generalizability, the warrant of standardization, needs to be re-evaluated and rearticulated from a hermeneutic perspective. By making reliability (meaning standardization, I think) an option rather than a requirement, assessment practices can be opened up that reflect more of a range of educational goals.

Quotable Quotes

Hermeneutic: “an ethic of disciplined, collaborative inquiry that encourages challenges and revisions to initial interpretations; and the transparency of the trail of evidence leading to the interpretations, which allows users to evaluate the conclusions for themselves” (87).

“There are certain intellectual activities that standardized assessments can neither document nor promote” (84).

“potential of a hermeneutic approach to drawing and warranting interpretations of human products or performances” (85).

Notable Notes

some hermeneutic assessment practices: allowing studnets to choose the products they feel best represent them (not just the same tasks for all) – fair, ethical, and places agency in the student; alos critical discussion and debate during assessment, disagreement does not equate invalidity, the importance of a dialogic perspective of a community (what Broad and Huot draw on)

detached, impartial scorers silence the teachers, those who know students and curriculum best

look @ public education accountability movement

White, Holisticism

White, Edward M. “Holisticism.” CCC 35 (1985): 400-409. In Assessing Writing. Eds. Huot and O’Neil, 19-28.

White contends that assessment must be grounded in humanism, seeing writing and reading as whole-body and whole-mind processes, and in his essay, explains the argument, method, and limitations of holistic assessment of student writing. The first investigations into a cost-effective, efficient holistic scoring process began at ETS in the 1970s, which transformed “general impression” scoring into holistic scoring by introducing constraints that made these assessment reliable. These constraints included things like controlled essay reading, using rubrics and scoring guides in tandem with anchor papers, and multiple independent scoring. Holistic scoring tries to assess more completely the complicated task of writing better than objective tests, but still it only succeeds in ranking student (not giving any other educationally-relevant information) and is not entirely reliable due to the shifting nature of the writing prompts and the variability in scorers.

Quotable Quotes

“Writing must be seen as a whole, and that the evaluating of writing cannot be split into a sequence of objective activities” (28). – no analytic scoring

April 27, 2009

Huot and O’Neill, Assessing Writing

Huot, Brian and Peggy O’Neill, eds. Assessing Writing: A Critical Sourcebook. Boston: Bedford/St. Martin’s, 2009.

This edited collection, divided into three sections – Foundations, Models, and Issues – focus on writing assessment that takes place outside of an individual classroom, namely placement and exit exams and programmatic evalutions. It draws on scholarship within the field of composition and rhetoric as well as that from educational evaluation, K-12 education, and measurement and testing. Huot and O’Neill see much of the scholarship written in writing assessment, starting in the 1940s, as negotiating the tension between reliable evaluation and valid evaluation, and argue that writing assessment needs to be taken up critically and reflectively by comp/rhet scholars as a positive and productive force (not punitive).

I’ve surveyed this volume (it contains 24 essays along with a selected bibliography) and have read the selections that I’ve seen cited in other scholarship about assessment as well as those that seem particularly helpful for WPAs.

Quotable Quotes

“Writing assessment is an activity – a practice – as well as a theoretically rich scholarly field” (6).

February 1, 2009

Miller, Expertise and Agency

Miller, Carolyn R. “‘Expertise and Agency’: Transformations of Ethos in Human-Computer Interaction.” In The Ethos of Rhetoric. Ed. Michael J. Hyde, U of South Carolina P, 2004. 197-218.
 
Miller explores the two complementary modes of human-computer interaction in the post-Cold War era: expert systems and intelligent agents. Using a grounding in twentieth century US history and an understanding in the computer systems and programs developed from the 1950s onward, Miller shows how the ethos in the human-computer interaction changes from an ethos interested in rational reliability (phronesis) to one concerned with interaction with the user (eunoia.) Ethos is not just a normative function; it is descriptive as well and can shed light on the kind of rhetorical community participants belong to. The shift from a rhetoric of domain-specific expert systems to one of intelligent agents happened in the 1970s and 1980s, when public trust in institutions and authority figures took a nose dive (Vietnam, Watergate) and when rapidly evolving technologies allowed for the development of a more complex, more diverse (in markets and knowledge), and more distributed world. Intelligent agents, as opposed to expert systems that are concerned with the accumulation of one domain of knowledge, are semi-autonomous, have choices, and interact with the environment. Their “expertise” and knowledge is collected in a distributed fashion. Miller argues that the ethos of rational reliability and that of sympathy are on two ends of a pendulum and must be balanced with virtues and moral reasoning (arete.)
 
Quotable Quotes
 
“A discourse…delinieates a rhetorical community and consequently an ethos – a sensus communis and a locus communis – a place where interlocutors abide, about which they contest, and from which they draw appeals. Those who dwell within a rhetorical community acquire their character as rhetorical participants from it, as it educates and socializes them. The community does this in part by supplying the Aristotelian components of ethos – the judgment (phronesis), values (arete), and feelings (eonoia) that make a rhetor persuasive to other members of the community” (198)
 
This is important because it’s about how the discourse that we create in turns creates the community – the hive. The writing that happens in blogs, del.ic.ious, GoogleDocs, tagging, etc. – imparts both a community and a shared ethos that is carried out in different projects.
 
Looking at ethos “can help us determine aspects of our community and our communal character” (198)
 
Notable Notes
 
Distrust in authority is historically and culturally grounded, leads to these bottom-up, more user-need sensitive human-computer interaction systems. People aren’t (usually) crazy domain experts. Their expertise is distributed and can be documented and used most efficiently in more of an intelligent agent system.
Ethos is normative and descriptive.

Create a free website or blog at WordPress.com.