Online Discussion: Teacher Evaluation, How Observation Fits In

Elena Silva: Hello, everyone, and thanks in advance for your willingness to take part in our online discussion about teacher observation models.

To begin, I wonder if each of you could respondto the following two-part question:

1. Observation can mean many things and take many forms. Each of you has experience with a specific observation model, from the traditional administrator-led approach to external evaluators, peer evaluators, or videos. Describe the observational model you are working (or have worked) with, how long the model has been in use (in your district or overall), and what your role is within this.
2. Secondly, some say the evaluation of teacher performance should function separately from the support and development of teachers, while others argue that they are one and the same. Is the model you’re working with intended for assessment purposes, for support and development purposes, or both? If you say “both” explain how the model works to do both and where it might falter on this.

Mark Simon: I helped to develop the Peer Assistance and Review model in Montgomery County in the late 90’s as the president of the teachers’ union. It went into operation in 2000 and even the doubters agree it’s been a huge success. It has credibility with teachers and school level administrators. It is tough but respectful and supportive. The old system relied on principals, assistant principals and central office specialists to evaluate teachers. We looked at Peer Assistance and Review as practiced in Cincinnati, and adapted it to our system. The whole system was designed, implemented and continues to be run by the teachers and principals unions.

“Consulting teachers" (CTs) identified not only as high performing teachers but also excellent at working with adults, leave the classroom for three years to work intensively with teachers either new to teaching or identified as having difficulty by their principals. They must then return to the classroom, guaranteeing that they bring a teacher-head to the work, not that this is a stepping-stone to becoming administrators or central office bureaucrat. Each consulting teacher has a caseload of 16 to 1. The CTs are hired, supervised and themselves evaluated by teachers and principals who serve on the PAR governing panel, recommended by the principals and teachers unions, and appointed by the superintendent. The involvement of the unions ensures that the details of how the program works maintains the integrity that teachers and principals will respect.

The heart of the model is the intensive training in “skillful teaching” and “observing and analyzing teaching” provided consulting teachers, principals and all evaluators, and that is available to all teachers. The training is the equivalent of 6 to 18 credit hours of coursework.

It has served to elevate the level of understanding of what good teaching is throughout the workforce—creating a language of how to talk about the craft of teaching. The respectful professional learning culture may be the most important consequence of the model.

Principals, APs, and department chairpeople continue to play a role, but we've analyzed the quality of the narrative feedback provided to teachers by principals and that provided by Consulting Teachers and there is no comparison. CTs do a better job of helping other teachers understand how to improve. Their write-ups are superior.

In Montgomery County we specifically integrated our evaluation, support and development systems. We call the whole thing our "Professional Growth System." It is the more respectful approach from a teacher perspective. The problem with stand-alone evaluation models using drive-by observation with or without test score metrics, in which evaluators simply pass judgment is that you don't have to be very good yourself to pass judgment. It becomes an exercise of sometimes arbitrary power. It may or may not be accurate. It may or may not reflect deep understanding of the craft of teaching. It reflects a factory-era management model more than a professional approach.

Ultimately it risks widespread resentment and breeds cynicism. That may sound harsh, but I think that the construction of a system that has credibility with the workforce and really adds value to the work is no easy task. It’s much easier to make a hash out of it.

K.K. Owen: Our new teacher evaluation system is based on the Charlotte Danielson Framework for Teaching.  While principals and assistant principals will be using observation to evaluate experienced teachers, we have consulting teachers who will be doing peer evaluations with our novice teachers. We will be using the Teachscape Reflect video capture system along with live observations for the novice teachers. We have several reasons for doing it this way-we feel it would be a valuable resource for self-reflection and faster growth for new teachers, consulting teachers would have the opportunity to view lessons the new teachers may be teaching on days when the consulting teacher is at another campus, and finally, we hope that new and veteran teachers will become interested in sharing their videos with colleagues which could lead to increased collaboration and a more dynamic professional learning community.

We have modeled our START Plan (Successful Teachers Assisting Rising Teachers) after the very successful Toledo Plan.  In Toledo and in our plan, the consulting teachers will be both coach/mentor and peer evaluator.  We have matched consulting teachers carefully with new teachers so that the consulting teacher is an expert in the area the new teacher is assigned (we assign experienced primary teachers to new teachers in the primary grades for example). This will be a much better relationship than perhaps a principal who was a middle school math teacher attempting to coach and evaluate a kindergarten teacher, for example. We believe there is potential for peer evaluation and would like to expand our program to assist all teachers who are struggling in some area by assigning a peer mentor and evaluator. In the future, we hope to add a peer evaluation element to our teacher evaluation system for all teachers in an effort to increase collaboration between teachers and to make teaching more transparent.

Silva:I wonder what the major differences are—either practically or theoretically—between Montgomery County's consulting teachers and DC's master educators? At least one difference seems to be that CTs are only for new and struggling teachers in Montgomery County, no? But there are similarities too, right? Both are expert educators who bring a degree of objectivity (as they are not current teachers in the schools they are working) and provide a way to observe teacher classroom performance with ostensibly more time and attention than a typical principal or AP.

Simon:One big difference between Montgomery County and DCPS is that Master Educators (MEs) in DC have caseloads between 85 and 100 clients. That means that no intensive work can be done with any teachers. It is not development or support work. It is simply ranking and rating, with a narrative hastily written according to a rubric. I worry that no matter how good the ME, they don't have the time to take care to really understand the teaching context or to offer tailored insights. I have watched both systems up close and although the verbiage and appearances seem similar the effect is very different. I believe this is by design.

When Rhee/Henderson/Kamras designed IMPACT the goal was to ID the worst teachers for firing and the best for bonuses. It was not intended to improve teaching.

Matt Radigan:I've heard there are people who feel this way, which is interesting.  While the majority of a Master Educator's role is to observe teachers, it's definitely not all we do!  After each observation, Master Educators write detailed reports and then conduct a conference where we give tailored feedback to each teacher based on DCPS' Teaching and Learning Framework. During this conference we also discuss best practices and ways in which the teacher can implement them.  Each Master Educator works with a subset of DCPS teachers; however the number of teachers we each work with in no way diminishes the amount or type of support we are able to provide.  In fact, many Master Educators go above and beyond to help teachers—just this past Sunday I met with a teacher at a coffee shop for 1.5 hours to discuss her upcoming year and coach her through best classroom management practices.  Teachers all across DCPS have reached out to their Master Educator, thanking them for the additional support or for the ideas that were given to them during the conference.

IMPACT is an evaluation system that allows us to identify the effectiveness of ALL teachers. By identifying where teachers are performing, we are able to leverage and celebrate the most effective teachers and help improve the effectiveness of teachers who need additional support.  Ultimately, the students of DCPS deserve teachers who can ensure they achieve at high levels!

Simon: That's great, Matt. Sounds like you're really going the extra mile and connecting with your clients. Is it true that you have between 85 and 100 clients at a time? I'm curious about the nature of the training you receive. It seems to be one of the big differences between DCPS and MCPS. In MCPS CTs study the Research for Better Teaching approach to "Studying Skillful Teaching" and "Observing and Analyzing Teaching."  How much of your training is on the theory of what good teaching is and the variety of ways it can be practiced, and how much of it is on getting the scoring right? Do you think there is too much attention to a rigid rubric?

Susan Headden:Montgomery County matches Consulting Teachers and teachers by content when possible, but—unlike DC—not always. The theory, according to Mark, is that "the craft of teaching has a set of skills and a knowledge base that transcends subject, grade, or discipline." Mark reports some tension over this in MCPS, with high school teachers pushing back a bit. But he says it has forced particularly high school teachers "to consider the craft of teaching practiced and not just the content of what's taught." How important do you think it is to match content areas between the observer and the observed?

Owen:I can see both sides of this argument.  Since we are implementing the Danielson Framework as the basis for our evaluation, it is clearly focused on instructional strategies that are the same Pre-K through adult in any content area.  That would lead us to believe that anyone with a deep knowledge of the instructional strategies can effectively coach and evaluate a teacher in any content area-even one that consulting teacher is not familiar with...BUT I think there is a level of comfort on the part of evaluator and new teacher when they are closely matched for subject and level.

Radigan: I think it is very important to match content and/or grade levels to teachers. While I do believe that ALL teachers are teachers of literacy, elementary teachers have to know how to use guided reading techniques, shared reading techniques, read aloud techniques, etc. In my experience, I've found that many Science or Math High School teachers are less  familiar with these techniques than an elementary teacher. I think this is especially true when one thinks about phonemic awareness and/or phonics. These two subjects require a very specific knowledge and a struggling teacher might benefit more by receiving help from a content specialist.

Owen:I agree with you completely regarding subjects such as reading in elementary school (particularly the primary grades) and the idea that instructional techniques are very different for students learning the "language of literacy" as they learn to read.

That was one of the driving factors in our decision to match by level our consulting teachers working with our novice teachers.  Our hope as we grow our START Plan to include not only new teachers but struggling teachers, we will use this same matching technique as much as possible and maintain consulting teachers matched by level to all teachers they work with in the program.  This same idea of matching levels holds true for other initiatives as well.  As we grow our lesson study plan using the video capture system, Teachscape Reflect, we think it will continue to be important to have level-alike conversations during the lesson study process of observing the videos together and improving the instructional practices of the group as a whole.

Simon: Since you asked about differences between Montgomery and DC, Elena...

There are several other important differences between Montgomery County's Evaluation system and the IMPACT system implemented in the past two years in DCPS.

First, as was reported in the NY Times, Montgomery County developed their system with the teachers and their union. DC, on the other hand, took advantage of a federal law that applies only to the District and developed the evaluation system unilaterally, without consulting with the union. It was done to teachers.

Secondly, DC's Master Educators received almost no training before evaluating teachers in 2009. They implemented IMPACT in the first year by having the MEs use a rubric, matching what they saw in the classroom to the rubric to determine the score. They back-filled with extensive training in the summer before the second year, but it was all training aimed at getting the scoring right—what they call inter-relator reliability. This is not deep training about the theory of what constitutes good teaching. DCPS assumed that if you hire good teachers into the role of being "master educators" they must know what good teaching is even if they did their good teaching in a variety of other school districts. In Montgomery, the deep training of Consulting Teachers in the language of how to respectfully communicate with teachers about the craft of teaching was part of the point.

Thirdly, Montgomery County built in due-process safeguards and appeals to an impartial PAR panel for both teachers and principals. In DC there is no check on the judgments of the ME and the principal except to go to the Associate Superintendent in charge of the system, Jason Kamras. So DC's ends up being an authoritarian and paternal system without checks and balances to the top-down power to decide the fate of teachers. This kind of system, I would argue, breeds cynicism. It also led to legal action and challenges to the firing of teachers.  In Montgomery the number of teachers exited from the system for performance reasons is significant—almost 500 over ten years—but there are very few appeals or challenges. There is a sense of legitimacy in the process.

Fourth, DCPS subjects every teacher to 5 observations by MEs and principals every year. Every teacher gets an IMPACT score every year. Two side effects of this are that excellent teachers are subject to the same drive-by evaluations that some—particularly the most accomplished teachers—find humiliating, and the system waters down their observations to apply this process to every teacher equally. In Montgomery County, most teachers deemed to be doing a good job get observed much less frequently (once a year by the principal only with an intensive evaluation only every 4 years) and they rather engage in professional growth activities based on their own learning goals. The intensive intervention of CTs are reserved for teachers having difficulty and those brand new to teaching, so approximately 50 CTs are focused on teachers who need help.  I believe this is a better use of limited resources and has the potential to actually improve the quality of teaching, not just ID good and bad teachers.

Lastly, DCPS has designed a system that allows standardized test scores to determine 55% or a teacher's evaluation score. This is problematic because test scores have been shown to be unreliable measures, particularly when students are not randomly assigned. Test scores are constantly being compared with the average, so test prep and other measures taken in some schools and classes can throw off that average.

Excellent teachers may get student scores that are below average because of other factors that affect their students and poor teachers may get above average scores due to factors other than the quality of their work. In Montgomery County, test scores never speak for themselves and constitute no unfiltered percentage of a teachers' rating. Test scores and other data are used significantly by evaluators, but they always inform evaluator judgment. I believe that the over-reliance on standardized test scores in DCPS is a time bomb waiting to go off. To fully realize the model, the number of standardized tests will have to increase many-fold in a system already crying out under too much testing.

There are other differences, but these are pretty significant.

Owen:Thank you for this information explaining the differences in the two systems. In Escambia County, since we modeled our plan after the Toledo Plan, we have a Board of Review that is (like Toledo) made up of half union reps and half district reps.  This board has developed a series of board reviews for our new teachers and the final review (in May) will be specifically for appeals from teachers who wish to appeal their evaluation results.

We, also, have standardized test scores as 50% of the teacher evaluation. That was not our intent as we built our new evaluation system, but the legislature intervened and passed legislation this past spring requiring that every teacher and every principal and assistant principal have 50% of his/her evaluation based on test scores.  We have only the FCAT at this point (Florida Comprehensive Assessment Test), so that becomes difficult and somewhat unfair to music teachers, Career and Technical Ed teachers, Pre-K teachers, etc. The state is in the process of developing more appropriate assessments for non-FCAT subject areas that should be in place within three years.

The final comment I will make is to emphasize that systems such as these new evaluations for teachers MUST be built on trust and the trust between union and district leadership is critical to the success of this process.  We are just completing our teacher and evaluator training on our new instrument and the fact that union leaders were right up in front of the groups training alongside those of us from the district side was important. Teachers and evaluators see us training together, they know that we built the system together, and so the culture improves and there is more trust out there among the ranks.

Radigan: Master educators have received extensive training on the rubric and in how to respectfully communicate with teachers, including training by Laura Lipton, a recognized expert in the field.   Specifically, Master Educators are trained in active listening skills, invitational language, and consciousness.  The selection process for Master Educators also specifically looks for individuals with strong communication skills in addition to being a highly effective teacher.

Simon:This issue of the content of the training is an important one. It is one thing to be trained in how to communicate with teachers—dealing with different types of personalities and levels of resistance and anger, for example. It is quite another to be trained in what constitutes good teaching in a way that respects the complexity of the craft. Jonathon Saphier, the founder of Research for Better Teaching (RBT) who designed the MCPS training was fond of saying that there are 50 different ways to teach the same group of students well. Do you think the application of a rubric too narrowly can be a problem, making ME's look for specific activities in every class lesson rather than being open to very different ways of approaching a lesson?

Radigan: This is a great question! One way we deal with the rubric and the complexity of the craft of teaching is through matching Master Educators to grade level and content.  During training and throughout the year we meet as a whole group, but also break out to meet in cohorts organized by grade and content.  During these cohort meetings we discuss what effective teaching is in elementary lessons, reading lessons, content lessons, etc.  This helps us push our thinking and honor both the art and science of teaching.

Silva:In response to KK’s previous comment (on the question of content-specific evaluation) “I can see both sides of this argument.  Since we are implementing the Danielson Framework as the basis for our evaluation, it is clearly focused on instructional strategies that are the same Pre-K through adult in any content area.  That would lead us to believe that anyone with a deep knowledge of the instructional strategies can effectively coach and evaluate a teacher in any content area-even one that consulting teacher is not familiar with...BUT I think there is a level of comfort on the part of evaluator and new teacher when they are closely matched for subject and level":

It seems to me that there would be a great cost to matching every evaluator and teacher by content (and presumably grade level), so that leads me to think it’s more practical to design a system that doesn’t guarantee that. The tradeoff, though, as KK suggests, is that teachers won’t trust that the evaluator fully understands the performance that he/she is evaluating. If an external evaluator—let’s say some policy wonk or researcher who doesn’t work in education—came in to evaluate me, I’m sure I could learn a lot from him or her about general research and policy skills, but if they didn’t know anything about the issues faced by schools or in education more broadly, I wouldn’t take it as a very thorough assessment of my performance. And, with all of the teacher evaluation systems under development, one early lesson seems to be that no system will work if teachers don’t understand and trust it. They will game it, ignore it, and eventually it will fail.

This leads me to the use of video as an observation tool. Big costs to begin with but in the long term this would be the less expensive option, and might be a very effective one if it's endorsed by teachers. This would likely mean that its use is primarily for teacher development (as it is in some preparation programs), and not just for assessment. KK, are teachers in Escambia concerned about the use of videos as part of their evaluation?

Owen:Yes! Our long range plans in Escambia definitely include using the video capture system for teacher development. The ultimate goal is for all teachers to use this system for the purpose of lesson study (rather than using lots of substitutes and actually having teachers observe each other face to face).  We are beginning with our novice teachers and making it part of our teacher induction system for exactly the reason you mention-teacher concern over being videotaped.  Our thought is that new teachers are already doing everything new and this will just be another new tool for them. Our hope is that other veteran teachers will become more comfortable with the idea as they watch others in their schools using the system and perhaps even being involved with a new teacher as a school-based mentor. We haven't heard any pushback from veteran teachers, but we are starting with new teachers only until it becomes a part of our practice and a part of our culture.  It has infinite possibilities for teacher development, professional learning communities, lesson study, etc. as well as for self-reflection by teachers and peer evaluation.

Simon:I'm curious about what it’s like to be a Master Educator (ME) in DCPS. From talking with some MEs I know that there is some frustration with the large number of teachers on their caseloads (85 to 100) that prevents professional development with teachers from taking place. My sense is that MEs would like to spend more time with teachers, mentoring and coaching, doing demonstration lessons, etc. but that this is not part of their job description. There is frustration from accomplished teachers being evaluated that the MEs are required to rigidly apply a rubric for what satisfactory and exemplary classes look like that disadvantages classes with students who are behavior problems or significantly behind grade level as is the case in higher poverty communities. So, truly outstanding teachers working with disadvantaged kids get evaluated as mediocre. It must be very frustrating to both the teacher and to the ME required to apply a rubric rigidly. In general I think that teachers would like more attention by MEs to the context in each class and to the intention of each teacher being observed, rather than a fixed model rubric of what good teaching is supposed to look like. Do I have these concerns right? These seem like interesting and complicated issues. I also have a sense that MEs not toeing the line get summarily dismissed, so there must be some sense of trepidation about making suggestions about how things might be done differently.  The details of how the program operates are very important and getting them right is tough.

Headden:Great questions, Mark. I heard lots of concerns from teachers about not being able to put things in context for the MEs.

Silva:We're getting a lot of good questions from our readers. Here's one:

What about teachers in non-tested grades, like pre-K, K, and music? When observation is used to evaluate them, do the “scores” from their observations carry more weight than the observation scores of tested teachers who are also being evaluated using student testing data? How standardized are their observations, and have the rubrics for observation been validated?

I'll begin by saying that many of the systems that I'm familiar with use some combination of student outcome data and observation data for teachers in tested subjects and grades, and some are doing this even for those in non-tested subjects. For example, TAP uses the school-wide average for a value-added score and combines this with observation scores (all teachers are observed in their classrooms multiple times a year—6 I believe—by a combination of administrator and trained teacher evaluators). The Danielson observation rubric is borrowed by many, including TAP. IMPACT too has a detailed rubric for observation, as does TFA. In Memphis, where there is tremendous activity around (and $90 million to support) teacher effectiveness efforts, a teacher evaluation working group looked at a bunch of rubrics, settling on TAP, IMPACT and  a revised version of Memphis's existing observation component for field testing. They picked IMPACT—altered somewhat for Memphis context. Observations feed into the overall evaluation system, what they call the Teacher Evaluation Measure (TEM).

Owen: Great question! In the Escambia County teacher evaluation plan, and as required in Florida, 50% of every teacher's evaluation must be based on student growth (not proficiency).  We will begin this year by using FCAT as the sole measure (although that only measures math and reading gains) since we do not yet have adequate fair reliable measurements for subjects like music, art, etc.  Our plan will use FCAT schoolwide scores and district wide scores as the evaluation scores for teachers of non-FCAT subjects and grades (like K and Pre-K) until such time as the state develops and approves for teacher evaluation appropriate pre and post assessments for those non-FCAT subjects and grade levels.  As the new assessments come on line, we will switch out the FCAT scores for non-FCAT teachers and add the new, subject-specific scores for those teachers. The new assessments will all be in place by the time performance pay kicks in in three years.

Radigan: I know I responded to some posts earlier, but I have two questions I would love the group's thoughts on. Often I hear a lot of feedback about what is wrong with IMPACT or the Master Educator program. My questions to the group are:

1. What has DCPS "got right" with IMPACT or the Master Educator Program?
2. What would be your "must have" or "non-negotiable" in a teacher assessment program? (I often ask teachers this type of question during a conference so I can share their collective voice and hear their feedback.)

Thanks for all the great comments so far. It has been an enriching discussion!

Simon: Good questions, Matt. In terms of what IMPACT gets right, I'd say it is the focus on the quality of teaching. This is absolutely what the system needed. As a longtime observer of the DCPS system I think that in the past downtown added very little value to nurturing good teaching. Great teachers in DCPS were great in spite of the system not because of it. (And let me stipulate here that the very best teacher I have ever seen is in DCPS.) Even the Washington Teachers Union acknowledged that the evaluation system needed to be improved. However, having said that, I think the opportunity was squandered when what was designed came across as a gotcha, something being done to teachers rather than with them. So the challenge now is to implement mid-course corrections that regain the trust and respect of teachers, particularly the most accomplished ones. I would suspend the rating of teachers for a year, except where a teacher is seriously incompetent, and focus on the conversations with teachers about how to strengthen what they are doing. I would do a lot more listening to teachers and a lot less presuming to know. I would make the process much more of a dialogue between practitioners. The "must have" at this point is the re-gaining of the respect and credibility among the workforce.

Quite frankly, I think the study that Susan Headden did for Ed Sector that is referenced in this Webinar draws some of the same conclusions — that IMPACT needs to be re-focused on development and support and away from what is perceived to be a model aimed at firings and bonuses. Changing that perception us the urgent task and will take some fairly dramatic mid-course correction.

Radigan:Thanks for a great response Mark. People's perception is often their reality, and it is so helpful to hear about what is going well so we can capitalize on that!

Silva: Another question from a reader:

It seems that it will require considerable time for teachers to learn how to use video technology and to look critically at their work. This aspect of teacher self-critique or analysis isn't simple or easy to implement. Many teacher education programs are gradually moving to use video as a vehicle to critique pre-service teachers but it has been a steep learning curve for some faculty. Is this equally true for administrators and even the most experienced teachers?

Owen: We haven't had our training yet on the video capture systems, but I know that many of our new teachers have participated in using videos of their teaching in their university teacher preparation programs.  Also, many of our current teachers have experience with videos of their teaching from the National Board Certification process.  We plan to train our consulting teachers and place the responsibility of actually setting up the systems and capturing the videos on them and they will guide our new teachers in their implementation of using the video capture system.  Our hope is that people will quickly assimilate this new tool into their practice and that it will become a way to eliminate some of the isolation of the teaching professional and increase the ability to collaborate and share best practices.

Silva: Thanks, everyone, for taking time out to generate such a rich and engaging discussion. Many readers have already reached out to thank us for hosting a discussion on this timely topic, so I pass that thanks on to all of you. We could go on for much longer I’m sure (there were many more questions we couldn't get to — thank you readers!) but it’s time to draw this to a close.

This discussion has concluded. Join us for another online discussion soon!