Revolution Lullabye

April 29, 2009

Williamson, The Worship of Efficiency

Williamson, Michael. “The Worship of Efficiency: Untangling Theoretical and Practical Considerations in Writing Assessment.” Assessing Writing 1(1994): 147-174. In Assessing Writing. Eds. Huot and O’Neill. Boston: Bedford/St. Martin’s, 2009. 57-80.

Williamson argues that educators must adopt a different educational model – that of the craft workshop – in order to create an assessment theory and practice that breaks the hold of the god-terms of efficiency, fairness, and reliability. Williamson traces how the concept of efficiency led assessment and educational practices during much of the 20th century, resulting in invalid assessments only based on one data point, assessments grounded in standardized tests that allowed for the development and dominance of factory and bureaucratic educational models. He points to other assessment practices, like those in France, rely on interviews and non-standardized assessments given by the teacher, who knows the curriculum and students best. If teachers are to be treated as the professoinals that they are, Williamson argues, they should be given the right and the responsibility to develop and give assessments to their students.

Quotable Quotes

“we will need to begin to trust teachers” (78).

“the privilege of true professionalism” (79).

“For the most part, students are assessed, labeled, and placed in school curricula on the basis of their scores on succeeding standardized tests…these tests remain one of the single most important indicators of a child’s future” (67).

“efficiency has governed both the theoretical and practical developments in assessment” (69).

Notable Notes

development of psychometrics to allow for an objective, outside scorer – this is reversed in the craft workshop model with teacher in charge

child-centered assessment v. system-centered assessment

libertarian assessment

history of shift from oral exams to written exams to multiple-choice testing (Arthur Otis)

efficiency is a key American cultural and social force

craft workshop model (Shedd and Bacharach; Schon’s reflective practicioner)

assessment as a contextual, dynamic, continuous, reflective process

assessments with multiple data points converging = valid

Advertisements

Royer and Gilles, Directed Self-Placement

Royer, Daniel J. and Roger Gilles. “Directed Self-Placement: An Attitude of Orientation.” CCC 50 (1998): 54-70. In Assessing Writing. Eds. Huot and O’Neill. Boston: Bedford/St. Martin’s, 2009. 233-248.

Directed self-placement is an assessment practice that shifts the responsibilty of placing students in the right first-year composition section from the teachers/WPA/administration to the students themselves. Gilles and Royer describe how they developed the idea and explain its benefits: cost-effectiveness, efficiency, a decrease in complaints by students and teachers, positive attitudes in basic writing and first-year courses, and, most importantly, a sense of “rightness,” telling and showing students that they can be entrusted, with guidance, to making decisions about their own education. They argue that directed self-placement is as (or more) valid and reliable than placing students into sections based on their standardized test scores or the score on a timed essay. Directed self-placement is grounded in pragmatic (Dewey) educational philosophy and looks inward, to the needs of students, giving them power and control and starting a culture of communication from the first day on campus..

Quotable Quotes

“Our placement program thus relies on honest student inquiry and interactive participation” (246).

“Normally, the placement universe revolves around teachers; we choose the methods, we score the essays, we tell students what courses to take. Now we began to envision students at the center” (239).

Notable Notes

In the first few years that their writing program implemented directed-self placement (explained and conducted at freshman orientation), 22% of incoming freshman self-placed themselves in basic writing.

simplicity and elegance, honesty about directed self-placement

narrative at beginning about how students are introduced and guided through directed self-placement at orientation

placement tests should be future-directed, about a student’s education, not focused on what teachers might learn about students from one decontextualized sit-down writing prompt

April 28, 2009

Yancey, Looking Back as We Look Forward

Yancey, Kathleen Blake. “Looking Back as We Look Forward: Historicizing Writing Assessment.” CCC 50 (1999): 483-503. Reprinted in Assessing Writing: A Critical Sourcebook. Eds. Huot and O’Neill. Boston: Bedford/St. Martin’s, 2009. 131-149.

Yancey divides the history of writing assessment into three “waves.” The first wave (1950-1970) focused on objective, non-essay testing that prioritized efficiency and reliability. The second wave (1970-1986) moved towards holistic scoring of essays, based on rubrics and scoring guides first developed through ETS and AP. The third wave (1986-present) expanded assessment to include portfolios (consisting of multiple examples of student writing) and larger, programmatic assessments. She looks at these waves from several perspectives: at how the concepts of reliability and validity are negotiated and valued; at the struggle between the knowledge of the assessment expert (and psychometrics) and the contextual, local knowledge of the non-expert teacher (and hermeneutics); and the move of assessment from outside and before the classroom to within and after the class. She voices concerns and directions for further scholarship and practice in writing assessment, challenging the field to look for ways to use assessment rhetorically and ethically to help students and programs develop and to produce scholarly knowledge.

Quotable Quotes

“It is the self we want to teach, that we hope will learn, but that we are often loathe to evaluate. What is the role of the person/al in any writing assessment?” (132).

Notable Notes

The role of the classroom teacher moving into writing assessment: in the 1st wave, testing specialists evaluated, but through the 2nd and 3rd waves, the roles of teacher and evaluator overlapped into the new discipline of writing assessment.

Questions: Who is authorized to speak about assessment? What is the purpose of writing assessment in education? Who shuold hold hte power?

the use of portfolios shifts the purpose and goals of the assessment: using pass/fail instead of scoring and communal grading moves more towards assessing a program and establishing community values than individual student assessment. Use different stakeholders to read?

waves fall into each other, aren’t strict lines that categorize what’s happening.

Moss, Can There Be Validity without Reliability?

Moss, Pamela. “Can There Be Validity without Reliability?” Educational Researcher 23.4(1994): 5-12. In Assessing Writing. Eds. Huot and O’Neill. Boston: Bedford/St. Martin’s, 2009. 81-96.

Moss challenges the primacy of reliability in assessment practices, arguing for the value of contextual, hermeneutic alternative assessments that can more accurately reflect the complex nature of writing tasks, knowledge, and performances. She describes the difference between hermeneutic and pyschometric evaluation, the latter which uses outside scorers or readers that do not know the context of the task, curriculum or the student, as teachers would. Pointing out that many high-stakes assessments are not standardized or generalizable (like tenure, granting graduate degrees), she argues that the warrant that writing assessment scholars use in the argument of generalizability, the warrant of standardization, needs to be re-evaluated and rearticulated from a hermeneutic perspective. By making reliability (meaning standardization, I think) an option rather than a requirement, assessment practices can be opened up that reflect more of a range of educational goals.

Quotable Quotes

Hermeneutic: “an ethic of disciplined, collaborative inquiry that encourages challenges and revisions to initial interpretations; and the transparency of the trail of evidence leading to the interpretations, which allows users to evaluate the conclusions for themselves” (87).

“There are certain intellectual activities that standardized assessments can neither document nor promote” (84).

“potential of a hermeneutic approach to drawing and warranting interpretations of human products or performances” (85).

Notable Notes

some hermeneutic assessment practices: allowing studnets to choose the products they feel best represent them (not just the same tasks for all) – fair, ethical, and places agency in the student; alos critical discussion and debate during assessment, disagreement does not equate invalidity, the importance of a dialogic perspective of a community (what Broad and Huot draw on)

detached, impartial scorers silence the teachers, those who know students and curriculum best

look @ public education accountability movement

White, Holisticism

White, Edward M. “Holisticism.” CCC 35 (1985): 400-409. In Assessing Writing. Eds. Huot and O’Neil, 19-28.

White contends that assessment must be grounded in humanism, seeing writing and reading as whole-body and whole-mind processes, and in his essay, explains the argument, method, and limitations of holistic assessment of student writing. The first investigations into a cost-effective, efficient holistic scoring process began at ETS in the 1970s, which transformed “general impression” scoring into holistic scoring by introducing constraints that made these assessment reliable. These constraints included things like controlled essay reading, using rubrics and scoring guides in tandem with anchor papers, and multiple independent scoring. Holistic scoring tries to assess more completely the complicated task of writing better than objective tests, but still it only succeeds in ranking student (not giving any other educationally-relevant information) and is not entirely reliable due to the shifting nature of the writing prompts and the variability in scorers.

Quotable Quotes

“Writing must be seen as a whole, and that the evaluating of writing cannot be split into a sequence of objective activities” (28). – no analytic scoring

April 27, 2009

Huot and O’Neill, Assessing Writing

Huot, Brian and Peggy O’Neill, eds. Assessing Writing: A Critical Sourcebook. Boston: Bedford/St. Martin’s, 2009.

This edited collection, divided into three sections – Foundations, Models, and Issues – focus on writing assessment that takes place outside of an individual classroom, namely placement and exit exams and programmatic evalutions. It draws on scholarship within the field of composition and rhetoric as well as that from educational evaluation, K-12 education, and measurement and testing. Huot and O’Neill see much of the scholarship written in writing assessment, starting in the 1940s, as negotiating the tension between reliable evaluation and valid evaluation, and argue that writing assessment needs to be taken up critically and reflectively by comp/rhet scholars as a positive and productive force (not punitive).

I’ve surveyed this volume (it contains 24 essays along with a selected bibliography) and have read the selections that I’ve seen cited in other scholarship about assessment as well as those that seem particularly helpful for WPAs.

Quotable Quotes

“Writing assessment is an activity – a practice – as well as a theoretically rich scholarly field” (6).

April 25, 2009

Hawk, A Counter-History of Composition

Hawk, Byron. A Counter-History of Composition: Toward Methodoligies of Complexity. Pittsburgh: U of Pittsburgh P, 2007.

Hawk argues that in modern composition, vitalism (equated with romanticism) is seen in opposition to rhetoric, especially in terms of how composition scholars and teacher talk about and teach invention. He centers on 1980 as a pivotal year, analyzing three articles published that year (Richard Young, James Berlin, and Paul Kameen) to show how they positioned the field to take an oppositional approach to vitalism. He argues that vitalism is a powerful, important philosophy with roots in Aristotle and developed in science and philosophy over centuries. It is at the root of complexity theory, which is an increasingly relevant and important theory today, as digital technologies are rapidly changing the cultural context, showing the inadequacy of methods and techniques rooted only in mind-driven logic. He argues for vitalism to take a central role in reconfiguring composition and rhetoric scholarship and pedagogy, because only through vitalism is the body and experience brought together in concert with the mind. Vitalism also prevents teachers from having a set agenda, a set desire for their students to fulfill, placing instead the onus on the students to develop and find their own relations and metaphors, drawing on all possible means and resources in our complex, dynamic, and ever-changing ecology.

Quotable Quotes

“Composition theorists should be striving to develop methods for situating bodies within ecological contexts in ways that reveal the potential for invention, especially the invention of new techniques, that in turn reveal new models for action within those specific rhetorical ecologies” (206).

“An ethical goal for pedagogy, then, would be to design occassions in which students are more likely to create compositions rather than decompositions. A pedagogical act would be evaluated based upon the relationships it fosters and the relationships it serves – on its ability to increase rather than decrease a student’s agency, power, or capacity to produce new productive relations” (256).

“To desire an outcome for them [students] is to commit a certain violence to them” (257).

“Heuristics do not function in a vacuum; they function within complex and specific rhetorical situations. Importantly, the body is the critical, epistemological link between situation and invention. It is the interface.” (120)

Notable Notes

a counterhistory (drawing on Feyerabend) – “a counter-history is an additive paratactic aggregate rather than a recuperative manuever” (123)

distinguishes between 3 forms of vitalism: oppositional (electronmagnetic forces); investigative (scales of influence and organization); complex (events, cooperation)

dissoi logoi – new ways to group texts and to read them

Young – concerned with disciplinarity, so rejects vitalism

Berlin – concerned with his own political Marxist agenda and can’t see anything else, and so rejects vitalism

all the work in comp/rhet on vitalism seems to stem from one dissertation, Hal Rivers Weidner “Three Models of Rhetoric: Traditional, Mechanical, and Vital” (2)

vitalism became the scapegoat term

Varnum, Fencing with Words

Varnum, Robin. Fencing with Words: A History of Writing Instruction at Amherst College during the Era of Theodore Baird, 1938-1966. Urbana: NCTE, 1996.

Varnum presents a counternarrative to the mainstream history of 20th century composition instruction through her archival investigation of the Amherst freshman writing program directed by Theodore Baird from 1938 to 1966. Her history, unlike other histories written in the field, has three distinct differences: first, she spends much time looking at how outside social and political forces (institutional, national, and global) affected the pedagogy in Amherst’s writing courses; second, she depends on the archives of the program (assignments, student papers, memos, reports, as well as interviews) instead of textbooks and journals to sketch a picture of what was happening in the classrooms; and third, she brings to light a much-misunderstood era of American composition, one that is usually conflated and simplified to be “current-traditional.” Baird’s first-year writing courses were designed by the entire team of teachers (everyone used the same collaboratively-written assignment sequence), and student writing, not textbooks, were the centerpiece of the course. Baird believed that writing was a process of self-discovery, a process through which unexpressible (and unknown) truths could be expressed. Varnum’s history does not sugarcoat or reify Baird’s administration or pedagogy, pointing out that his high-priest attitude was decidedly masculine and top-down, perceived by some students and fellow faculty to be a bully who ran a “boot camp” course.

Quotable Quotes

“The tendancy among composition historians has been to look at practice in the classroom, or at the materials and ideas presented there, without acknowledging the larger forces that created the classroom itself” (7).

Notable Notes

great model for dissertation

Baird’s constant metaphor of running orders through chaos (taken from The Education of Henry Adams and science, philosophy, Burke, Richards that he read)

the Amherst course was taken by all freshman at the same time with the same assignments so that each assignment was a campus-wide event.

focused on conflict, constant questioning and revision

saw student writer as individual who possessed his own voice, the goal was to free that voice and the imagination; break them of writing what he dubbed “the Perfect Theme” (41) influenced heavily by The Education of Henry Adams

Baird’s ideal was to create a community of teachers

all the work that went into teaching and planning the course (assignments were re-invented each year) took away the time the instructors and professors could do their own research

how do policies and politics outside the classroom affect what is being taught?

impact of WWII, GI Bill, co-education, change in American university system, Civil Rights, move towards general education requirements

Varnum interviews professors who taught with Baird, 7 alumni of the program, looks at student papers, essay contest winners, uses letters she writes to Baird and recieves from him, assignment sequences (in appendix)

April 21, 2009

Mentkowski, Learning That Lasts

Mentkowski, Marcia and Associates. Learning That Lasts: Integrating Learning, Development, and Performance in College and Beyond. San Francisco: Jossey-Bass, 2000.

This book explains an educational theory of learning, growth, and performance that was developed through 24 years of research into the abilities and activities of Alverno College students and alumna. The goal of college education, they argue, should be learning that is lasting, and the results of that kind of learning take time to emerge, but continue past graduation. Alverno College’s entire curriculum and assessment is built around the idea of learning that lasts; students are evaluated based on their proficiencies in eight compentency areas. Students are not given traditional grades; rather, their assessment is activity-based through continuous observation, judgment, self-reflection, and feedback. Their educational theory places a large emphasis on performance: real learning does not take place until it is performed through an integration of doing and knowing, both during college and after.

Quotable Quotes

Performance – “The integration of knowing and doing – in class and off campus.” (228) not just application of knowledge

“The way graduates solve problems, interact with others, communicate, and express their values in actin tell us about hte quality of their education and how that education counts in their lives and the lives of those they touch” (175).

“From a systems point of view, the true outcomes of college occur in the interaction of the college experience with the postcollege environment” (175).

“Learning that is lasting – that is, mindful and emotional, intellectual and committed – characterizes the lifelong learner who becomes a seeker, a pligrim, a pathfinder to integrity” (1)

Notable Notes

8 competencies measured along six developmental levels

Communication; Analysis; Problem Solving; Valuing in Decision-Making; Social Interaction; Global Perspectives; Effective Citizenship; Aesthetic Responsiveness

Alverno is all-women

how do students construct meaning out of their education? sustain this meaning?

the ability to self-assess, to reflect and evaluate one’s work is a mark of intellectual maturity, be a performer and contributor in work, civic, and personal lives

move to validate the scholarship of teaching, establish a college culture and faculty that value teaching, create an environment with a constant awareness of reflection and assessment (266)

Principles of learning that lasts  – Chapter 7

four domains of learning that lasts: reasoning, performance, development, self-reflection

April 15, 2009

Broad, What We Really Value

Broad, Bob. What We Really Value: Beyond Rubrics in Teaching and Assessing Writing. Logan: Utah State UP, 2003.

Broad introduces the practice of dynamic criteria mapping (DCM) as an inquiry-driven alternative to static, traditional rubrics that have a normative rather than descriptive function, not even addressing many of the things are taught in writing classes (therefore not a valid assessment). His book is a case study of the use of DCM at “City University,” a university with 4000 students in a 3-course English sequence that is assessed through portfolios, graded collectively by 3-teacher teams. Instead of starting with certain textual features to check off, DCM asks teachers and assessors to describe what they see in a text (good and bad.) Together, the instructors find synonyms and antonyms for what they notice, categorize similar ones, and create a visual map that illustrates the values about good writing that the program’s teachers hold collectively. This method, though time-consuming and messy, better articulates the complex processes and ideas that students are showing in their writing. The process is locally, site-baed: though the method of DCM can be used, individual maps cannot be transported across institutions or even across years; it should be a conversation about values that happens continually.

Quotable Quotes

“We can now face the truth equipped with tools (qualitative inquiry) and attitudes (hermeneutics) that help us tap the energy of apparant chaos without being consumed by it. We can embrace the life of things” (137).

“In their rush toward clarity, simplicity, brevity, and authority, traditional scoring guides make substantial knowledge claims based on inadequate research” (3)

“In pursuit of their normative and formative purposes, traditional rubrics surrender thier descriptive and informative potential: responsiveness, detail, and complexity in accounting for how writing is actually evaluated” (2).

“The age of the rubric has passed” (4)

Notable Notes

Vinland map – not appropriate now

move to validity(not the same as reliability)

the DCM finds textual criteria and contextual criteria (things not found in text but have an impact on assessing, before DCM these have not been visible)

benefits of DCM: 1. student learning (shows writing is more complex, they have a better understanding of what they’re doing well and  what teachers are looking for); 2. professional development and community; 3. program development and evaluation; 4. more valid assessment; 5. better relations with the public (values are made public, written down)

drawbacks? time-consuming and needs constant reflection and revisiting

must happen in communal writing assessment so there will be debate, disagreement, and discussion of values.

once the values are visible, you can start having conversations about whether you should value what you do.

a search for truth through hermeneutics, not psychometrics

Next Page »

Create a free website or blog at WordPress.com.