Revolution Lullabye

August 27, 2014

Newton, Value-added Modeling of Teacher Effectiveness

Newton, Xiaoxia A, et al. “Value-added Modeling of Teacher Effectiveness: Exploration of Stability across Models and Contexts.” Educational Policy Analysis Archives, 18 (23). 2010. Print.

Newton et al investigate measures of teacher effectiveness based on VAM (value-added modeling) to show that these measures, based on in large part on measured student learning gains, are not stable and can vary significantly across years, classes, and contexts. The study focused on 250 mathematics and ELA teachers and approximately 3500 students they taught at six high schools in the San Francisco Bay Area. The researchers argue that measures of teacher effectiveness based solely on student performance scores (those measures that don’t take into account student demographics and other differences) cannot be relied on to get a true understanding of a teacher’s effectiveness because so many other unstable variables impact those student test scores. Models of teacher evaluation that rely heavily on student performance scores can negatively impact teachers who teach in high-need areas. This is especially true with teachers who teach disadvantaged students or students with limited English proficiency.

Quotable Quotes

“Growing interest in tying student learning to educational accountability has stimulated unprecedented efforts to use high-stakes tests in the evaluation of individual teachers and schools. In the current policy climate, pupil learning is increasingly conceptualized as standardized test score gains, and methods to assess teacher effectiveness are increasingly grounded in what is broadly called value-added analysis. The inferences about individual teacher effects many policymakers would like to draw from such value-added analyses rely on very strong and often untestable statistical assumptions about the roles of schools, multiple teachers, student aptitudes and efforts, homes and families in producing measured student learning gains. These inferences also depend on sometimes problematic conceptualizations of learning embodied in assessments used to evaluate gains. Despite the statistical and measurement challenges, value-added models for estimating teacher effects have gained increasing attention among policy makers due to their conceptual and methodological appeal” (3).

Differences in teacher effectiveness in different classes: “An implicit assumption in the value-added literature is that measured teacher effects are stable across courses and time. Previous studies have found that this assumption is not generally met for estimates across different years. There has been less attention to the question of teacher effects across courses. One might expect that teacher effects could vary across courses for any number of reasons. For instance, a mathematics teacher might be better at teaching algebra than geometry, or an English teacher might be better at teaching literature than composition. Teachers may also be differentially adept at teaching new English learners, for example, or 2nd graders rather than 5th graders. It is also possible that, since tracking practices are common, especially at the secondary level, different classes might imply different student compositions, which can impact a teacher’s value-added rankings, as we saw in the previous section.” (12)

“the analyses suggested that teachers’ rankings were higher for courses with “high-track” students than for untracked classes” (13).

“These examples and our general findings highlight the challenge inherent in developing a value-added model that adequately captures teacher effectiveness, when teacher effectiveness itself is a variable with high levels of instability across contexts (i.e., types of courses, types of students, and year) as well as statistical models that make different assumptions about what exogenous influences should be controlled. Further, the contexts associated with instability are themselves highly relevant to the notion of teacher effectiveness” (16).

“The default assumption in the value-added literature is that teacher effects are a fixed construct that is independent of the context of teaching (e.g., types of courses, student demographic compositions in a class, and so on) and stable across time. Our empirical exploration of teacher effectiveness rankings across different courses and years suggested that this assumption is not consistent with reality. In particular, the fact that an individual student’s learning gain is heavily dependent upon who else is in his or her class, apart from the teacher, raises questions about our ability to isolate a teacher’s effect on an individual student’s learning, no matter how sophisticated the statistical model might be” (18).

“Our correlations indicate that even in the most complex models, a substantial portion of the variation in teacher rankings is attributable to selected student characteristics, which is troubling given the momentum gathering around VAM as a policy proposal. Even more troubling is the possibility that policies that rely primarily on student test score gains to evaluate teachers – especially when student characteristics are not taken into account at all (as in some widely used models) — could create disincentives for teachers to want to work with those students with the greatest needs” (18).

“Our conclusion is NOT that teachers do not matter. Rather, our findings suggest that we simply cannot measure precisely how much individual teachers contribute to student learning, given the other factors involved in the learning process, the current limitations of tests and methods, and the current state of our educational system” (20). 

Notable Notes

The problem of variables impacting the calculation of teacher effectiveness: the students’ background (socioeconomic, cultural, disability, language diversity), the effects of the school environment, how teachers perform year-to-year, the curriculum

VAM makes assumptions that schools, teachers, students, parents, curriculum, class sizes, school resources, and communities are similar.

The variables the researchers collected and measured included CST math or ELA scaled test scores, students’ prior test scores for both average and accelerated students, students’ race/ethnicity, gender, and ELL status, students’ parents’ educational level and participation in free or reduced school lunch, and individual school differences. Tries to look at the issue longitudinally by looking at student prior achievement (7). They were able to link students to teachers (8).

Advertisements

Darling-Hammond, Creating a Comprehensive System for Evaluating and Supporting Effective Teaching

Darling-Hammond, Linda. Creating a Comprehensive System for Evaluating and Supporting Effective Teaching. Stanford, CA: Stanford Center for Opportunity Policy in Education. 2012. Print.

This report argues for the development of an aligned, comprehensive K-12 teacher evaluation system that supports students, teachers, curriculum, schools, and communities by being an integral part of a larger teaching and learning system. The report outlines seven “best practices” for creating teacher evaluation systems. Teacher evaluation systems, the report argued, should serve teachers at all stages of their careers and be used for critical decisions at the licensing, hiring, and granting tenure/merit stages. Teacher evaluation systems need to be directly connected to ongoing teacher professional development and encourage collaboration among teachers, not competition. The report makes a distinction between “teacher quality” and “teaching quality,” arguing that helping teachers improve their teaching practices across different kinds of students, contexts, and curriculum will result in better teaching and better student learning. The report includes examples of district and state evaluation systems and procedures that it believes serves as models and starting points for creating a comprehensive teacher evaluation system.

 

Quotable Quotes

“Today, much attention is focused on identifying and removing poor teachers. But what we really need is a conception of teacher evaluation as part of a teaching and learning system that supports continuous improvement, both for individual teachers and for the profession as a whole. Such a system should enhance teacher learning and skill, while at the same time ensuring that teachers who are retained and tenured can effectively support student learning throughout their careers” (1-2)

The problem: “Virtually everyone agrees that teacher evaluation in the United States needs an overhaul. Existing systems rarely help teachers improve or clearly distinguish those who are succeeding from those who are struggling. The tools that are used do not always represent the important features of good teaching. Criteria and methods for evaluating teachers vary substantially across districts and at key career milestones—when teachers complete pre-service teacher education, become initially licensed, are considered for tenure, and receive a professional license.

A comprehensive system should address these purposes in a coherent way and provide support for supervision and professional learning, identify teachers who need additional assistance and—in some cases—a change of career, and recognize expert teachers who can contribute to the learning of their peers.” (i)

Distinction between teacher quality and teaching quality: “Teacher quality might be thought of as the bundle of personal traits, skills, and understandings an individual brings to teaching, including dispositions to behave in certain ways. Teaching quality refers to strong instruction that enables a wide range of students to learn. Teaching quality is in part a function of teacher quality— teachers’ knowledge, skills, and dispositions—but it is also strongly influenced by the context of instruction: the curriculum and assessment system; the “fit” between teachers’ qualifications and what they are asked to teach; and teaching conditions, such as time, class size, facilities, and materials. If teaching is to be effective, policymakers must address the teaching and learning environment as well as the capacity of individual teachers” (i).

Five elements to this teacher evaluation system, as part of a larger teaching and learning system:

  1. “Common statewide standards for teaching that are related to meaningful student learning and are shared across the profession.” These should help direct the preparation of teachers and ongoing professional development (i)
  2. “Performance assessments, based on statewide standards, guiding state function such as teacher preparation, licensure, and advanced certification” – there should be multiple assessments for different points in the profession (initial, mid, advanced) that look at how well teachers can “plan, teach, and assess learning” (ii)
  3. “Local evaluation systems aligned to the same standards, which asses on-the-job teaching based on multiple measures of teaching practice and student learning.” – things like observations, teaching artifacts like lessons plans/assignments, “evidence” of how teachers contribute to their colleagues’ work and student learning (ii) (example on page 11)
  4. “Support structures to ensure trained evaluators, mentoring for teachers who need additional assistance, and fair decisions about personnel actions” – including access to master teacher mentors, fair “governance structures,” and continued resources to maintain the system (ii)
  5. “Aligned professional learning opportunities that support the improvement of teachers and teaching quality” – all kinds of professional development (formal, embedded) that “trigger continuous goal-setting” and “opportunities to share expertise” (ii)

 “To transform systems, incentives should be structured to promote collaboration and knowledge sharing, rather than competition, across organizations” (ii)

“Criteria for an Effective Teacher Evaluation System

“In conclusion, research on successful approaches to teacher evaluation suggests that:

  1. “Teacher evaluation should be based on professional teaching standards and should be sophisticated enough to assess teaching quality across the continuum of development from novice to expert teacher.
  2. “Evaluations should include multi-faceted evidence of teacher practice, student learning, and professional contributions that are considered in an integrated fashion, in relation to one another and to the teaching context. Any assessments used to make judgments about students’ progress should be appropriate for the specific curriculum and students the teacher teaches.
  3. “Evaluators should be knowledgeable about instruction and well trained in the evaluation system, including the process of how to give productive feedback and how to support ongoing learning for teachers. As often as possible, and always at critical decision-making junctures (e.g., tenure or renew- al), the evaluation team should include experts in the specific teaching field.
  4. “Evaluation should be accompanied by useful feedback, and connected to professional development opportunities that are relevant to teachers’ goals and needs, including both formal learning opportunities and peer collaboration, observation, and coaching.
  5. “The evaluation system should value and encourage teacher collaboration, both in the standards and criteria that are used to assess teachers’ work, and in the way results are used to shape professional learning opportunities.
  6. “Expert teachers should be part of the assistance and review process for new teachers and for teachers needing extra assistance. They can provide the additional subject-specific expertise and person-power needed to ensure that intensive and effective assistance is offered and that decisions about tenure and continuation are well grounded.
  7. “Panels of teachers and administrators should oversee the evaluation process to ensure that it is thorough and of high quality, as well as fair and reliable. Such panels have been shown to facilitate more timely and well- grounded personnel decisions that avoid grievances and litigation. Teachers and school leaders should be involved in developing, implementing, and monitoring the system to ensure that it reflects good teaching well, that it operates effectively, that it is tied to useful learning opportunities for teachers, and that it produces valid results.

“Initiatives to measure and improve teaching effectiveness will have the greatest payoff if they stimulate practices known to support student learning and are embedded in systems that also develop greater teaching competence. In this way, policies that create increasingly valid measures of teaching effectiveness—and that create innovative systems for recognizing, developing and utilizing expert teachers—can ultimately help to create a more effective teaching profession” (iii-iv).

 

“Good systems must be designed so that teachers are not penalized for teaching the students who have the greatest educational needs. Rather, they should explicitly seek to provide incentives that recognize and reward teachers who work with challenging students” (24)

Notable Notes 

Need to create a system for evaluating teachers (and developing teaching) that takes into account all the stakeholders at local/state/national levels as well as the curriculum and standards.

The problem with relying on student performance scores to evaluate teaching: a teachers’ scores vary considerably from class-to-class and year-to-year, are affected by and tied directly to the type of students in the classroom (student differences), and the scores themselves are flattened – it’s impossible to discern what exactly impacted student learning: the teacher, the curriculum, the school environment, the home environment? (iii)

Student learning scores can be used in determining teacher effectiveness, but they can’t be the sole indicator and if used, they must be “appropriate for the curriculum and the students being taught” (iii)

Good graphic for representing the three tiers of a teacher career (and the argument to assess and evaluate teachers along these three tiers): initial, professional licensure, experienced/master teacher (7) and an example of New Mexico’s standards-based teacher evaluation system that evaluates teachers at these three tiers (8-9)

discussion of peer-based review of teachers, examples of systems using peer review (28-35)

May 25, 2011

Hauser, Teaching Rhetoric

Hauser, Gerald A. “Teaching Rhetoric: Or Why Rhetoric Isn’t Just Another Kind of Philosophy or Literary Criticism.” Rhetoric Society Quarterly 34.3 (Summer 2004): 39-53

The Association for Rhetoric Societies’ 2003 conference in Evanston led to an alliance among rhetoric scholars to promote the centrality of rhetorical education in civic education. This article lists the five areas where Rhetoric Studies needs sustainable structures in order to reinvigorate rhetoric into the curriculum.

The scholars underscored that rhetoric is inherently tied to teaching: there is no rhetoric without teaching. What has happened in the modern academy, one that values theory and knowledge over praxis, is a divorce of rhetoric from the public and civic sphere, which rhetoric depends on. Hauser and those at the conference call for rhetoric to be reunited to the concerns of the public civic sphere, of preparing citizens and leaders. The Association for Rhetoric Scholars, through Hauser’s article, argues for a manifesto about rhetorical education that can be adopted by institutions, a forum to share rhetorical pedagogy material between scholars, and a way for individual institutions to circumvent the balkanization that happens with rhetorical pedagogy, coordinating it into one collective pursuit.

Notes and Quotes

“Free societies require rhetorically competent citizens. Without rhetorical competence, citizens are disabled in the public arenas of citizen exchange—the marketplace, the representative assembly, the court, and public institutions— and democracy turns into a ruse disguising the reality of oligarchic power.” (52)

Rhetoric has always been a central part in educating future leaders and citizens. Rhetoric is practical, is human, is considered with the right time and right place (kairos.) It seeks to give students a way to pursue and articulate knowedge, not a set content.

Rhetoric is about seeking truth and excellence (aerte), questioning, reflection, learning about values and beliefs, and moving to action. Very similar to Ignatian pedagogy

“Rhetoric is a practical discipline; it has a strong tradition that merges theory and praxis in the concrete conditions of performance, especially as these are realized in democratic societies.” (42)

Students need rhetoric – need to learn how to present their ideas, understand their audience, evaluate their sources and claims, negotiate between different perspectives, see the connection between ethics and action. Rhetoric is needed in a democratic society (so a small elite does not take over power.)

Ideas for the assessment of a first-year writing and speaking course: students develop analytical skills, performance skills (written and spoken), invention skills, an awareness of language, civic skills, consequences of rhetoric

call for K-12 and university educators to come together in the Association for Rhetoric Scholars to talk about rhetorical education, collaborate, work together

December 2, 2010

Lloyd-Jones, Who We Were, Who We Should Become

Lloyd-Jones, Richard. “Who We Were, Who We Should Become.” College Composition and Communication 43.4 (Dec 1992): 486-496.

Lloyd-Jones discusses the tensions in the field’s identity and how the field is implicated in current problems in higher education, and emphasizes that those in composition need to take more leadership in their institutions and insist on better treatment of its teachers, expand connections with K-12 schools through NCTE and the Writing Project, expand its research, and avoid over-specialization. He warns that the increased dependence on contingent faculty is a huge problem for both the field and higher education, as the labor structure has used band-aid solutions to account for a massive change in student population and the purpose of college education in the 20th century. He uses the history of CCCC and a survey in the contents of CCC to explain the 20th century history of the field, how those in the discipline progressed from problem-solving teachers and administrators to beginning to study and theorize language and writing, moving into the academic culture and expectations of the academy. The shifting profile of American college students from post-WWII (the veteran) to the 1960s and 1970s (huge increase in community college students as the high school degree was no longer enough to get a job) constantly challenged the theories and ideas of those in the field, pushing the field to be more progressive, more ethically, socially, and practically oriented than other traditional departments like literature.

Notes and Quotes

College enrollments doubled in 1946 (GI Bill). The first CCCC in 1948 was a practical matter, English college professors and administrators trying to deal with the practical problems of the huge influx of students – placing them, teaching them, assessing them.

The discipline has just as much to do with national, international politics, economics, and demographics as it does with the creation and production of theory. Our focus on social justice and ethics in concern of our students naturally leads to a concern for our teachers (me)

The rapidly growing two-year colleges turned to cheap adjunct labor to teach composition.

The teaching of composition is a profit-generating enterprise. Why don’t we argue this point to increase working conditions for the teachers of writing?

The two-tiered college system, with tenured managers and untenured workers, “offer dark images for the future” (491). “I worry that we may ease toward a situation in composiiton where most of the people who do the actual teaching are disenfranchised. Or worse, a situation in which decisions are given over to people who have relatively little training in an extremely complex field” (491).

“I am not calling to separate composition from English; that question is passe…Literature is a form of rhetoric” (493).

April 15, 2009

Huot, (Re)Articulating Writing Assessment

Huot, Brian. (Re)Articulating Writing Assessment for Teaching and Learning. Logan, Utah: Utah State UP, 2002.

Assessment needs to be rearticulated by composition and rhetoric scholars as an important, necessary part of writing scholarship and teaching. Huot addresses assessment in a different way in each chapter (highlighting its connection to student response, teaching students self-assessment, need to create a field of writing assessment, and a history of writing assessment practices), but all of his studies and discussion point to central principles for his new theory and practice of writing assessment. Assessment must be site-based, locally controlled, context-sensitive, rhetorically-based, and accessible (to students, public, teachers, adminstrators.) Composition and rhetoric scholars and teachers are doing themselves no favors by abdicating assessment to education or to self-appointed writing assessment specialists; assessment is an issue that must be taken up by every WPA and teacher.

Quotable Quotes

“Instead of envisioning assessment as a way to enforce certain culturally positioned standards and refuse entrance to certain people and groups of people, we need to use our assessments to aid the learning environment for both teachers and students” (8).

“People who write well have hte ability to assess their own writing, and if we are to teach students to write successfully, then we have to teach them to assess their own writing” (10)

Notable Notes

assessment is articulating what we value; it marks our identities as teachers, programs, and a field; how do our judgments get articulated into our assessments?

Chapter 2 – need to connect comp/rhet with K-12 assessment to create  a writing assessment subfield, pooling knowledge and methods, talk about validity

Chapter 3 – need to teach students how to assess their own writing; writing as reflective judgment; use portfolios to full advantage

Chapter 4 – history of assessment practices

Chapter 5 – teacher response to student writing (draw on Phelps) and the contraint inherent in the act of reading

Chapter 6 – writing assessment is treated like a technology. It needs to be reimagined as research. This changes the role and activity of the assessors (151)

Chapter 7 – the practice of writing assessment needs to be reflective, conscious, theoretical, and instructive. Assessment can be social action, something that the field claims again, led by WPAs and teachers. (175)

movement away from objective rubric-like assessments, more based on community questions, inquiry, research, and practice

technocentric argument (Hawshier) – the tool of the assessment should not drive the practice

Create a free website or blog at WordPress.com.