Current policy promotes more rigorous evaluations of teaching as a key lever in measuring and promoting teacher quality. As school systems implement new evaluation systems, Kane and Staiger (2012) argue that school systems must ensure that principals are capable of implementing the evaluations well before they use the results to make high-stakes decisions about teachers' careers. Additionally, this lever of reform rests on principals' abilities to diagnose and evaluate teacher performance well. In this exploratory study, I summarize the results of one organization's efforts to assess principals' evaluation skills just as Kane and Staiger suggest. In 2011, 125 charter-school principals completed three video-based calibration assessments over six months to determine their skills in rating teacher performance. I use the data to describe principals' ability to provide two types of feedback in their ratings of teacher performance. First, I summarize how well principals provided teachers with diagnostic feedback that could be used immediately to improve teacher performance on specific teaching behaviors. Second, I summarize how well principals provided the organization with comparative feedback that could be used annually to distinguish the "better" teachers from the "worse." Third, I compared the relationship between their diagnostic and comparative feedback. Finally, I estimated the reliability of these assessments for evaluating principals' skills in rating teacher performance.

I found that principals' quality of diagnostic feedback was moderately high and did not improve across the occasions of assessment, except when principals assessed teachers' performance in managing the classroom environment. Also, I found that principals' quality of comparative feedback did not improve across the occasions of assessment and was easily affected by the "likeability" of the teacher being evaluated. Third, I found that the quality of the principals' diagnostic and comparative feedback were closely related; but, not interchangeable. Fourth, I found that the assessments did not provide a highly reliable method for assessing principals' evaluation skills. Lastly, I provide recommendations for school systems that may also choose to assess principals' performance in evaluating teachers. Included in these recommendations are suggestions for designing the assessments, providing supports to principals and monitoring evaluations done in the field.