Donate
Independent, objective, nonpartisan research

R 709EL2R

Authors

R 709EL2R

Tagged with:

Publication PDFs

Database

This is the content currently stored in the post and postmeta tables.

View live version

object(Timber\Post)#3742 (44) { ["ImageClass"]=> string(12) "Timber\Image" ["PostClass"]=> string(11) "Timber\Post" ["TermClass"]=> string(11) "Timber\Term" ["object_type"]=> string(4) "post" ["custom"]=> array(5) { ["_wp_attached_file"]=> string(13) "R_709EL2R.pdf" ["wpmf_size"]=> string(6) "331799" ["wpmf_filetype"]=> string(3) "pdf" ["wpmf_order"]=> string(1) "0" ["searchwp_content"]=> string(78661) "School Accountability and Administrator Incentives in California S. Eric Larsen with research support from Karina Jaquet Supported with funding from The William and Flora Hewlett Foundation July 2009 The Public Policy Institute of California is dedicated to informing and improving public policy in California through independent, objective, nonpartisan research on major economic, social, and political issues. The institute’s goal is to raise public awareness and to give elected representatives and other decisionmakers a more informed basis for developing policies and programs. The institute’s research focuses on the underlying forces shaping California's future, cutting across a wide range of public policy concerns, including economic development, education, environment and resources, governance, population, public finance, and social and health policy. PPIC is a private operating foundation. It does not take or support positions on any ballot measures or on any local, state, or federal legislation, nor does it endorse, support, or oppose any political parties or candidates for public office. PPIC was established in 1994 with an endowment from William R. Hewlett. Mark Baldassare is President and Chief Executive Officer of PPIC. Walter B. Hewlett is Chair of the Board of Directors. Copyright © 2009 by Public Policy Institute of California All rights reserved San Francisco, CA Short sections of text, not to exceed three paragraphs, may be quoted without written permission provided that full attribution is given to the source and the above copyright notice is included. Research publications reflect the views of the authors and do not necessarily reflect the views of the staff, officers, or Board of Directors of the Public Policy Institute of California. 3 Contents Summary 4 Acknowledgments 6 INTRODUCTION 7 SCHOOL ACCOUNTABILITY IN CALIFORNIA 8 THE EFFECTS OF SCHOOL ACCOUNTABILITY 11 SCHOOL ACCOUNTABILITY AND ADMINISTRATOR INCENTIVES IN CALIFORNIA 16 Study Data 20 School Board Accountability and Student Achievement 22 Superintendent Accountability and Student Achievement 23 Principal Accountability and Student Achievement 24 POLICY IMPLICATIONS 26 How Might NCLB Be Improved? 27 References 30 About the Author 32 4 Summary There is only limited evidence tha t differences in resources available to public schools can explain differences in student achievement among these schools. Many critics have argued that there is little relationship between resources and achievement because schools do not use additional re sources efficiently. B ecause most public schools hold a local monopoly on the provision of public education it has been argued that they are not held accountable for using resources efficiently. Beginning in the mid -1990s and culminating with the federal No Child Left Behin d Act of 2001 (NCLB), school districts , states, and the federal government have introduced school accountability programs. The main objective of these programs is to improve student achievement by defining the primary goals of schools and districts and by holding schools and districts accountable for meeting those goals. However, i t is not clear that either the Public Schools Accountability Act of 1999 ( PSAA) or NCLB has had a significant effect on the accountability of teachers, principals, superinten - dents, or school board members. Individual teachers and administrators are not evaluated under either of the plans, and neither plan provides explicit financial incentives that might motivate improvement. A lthough the school - and district -based sanctions of the laws may affect individuals, the sanctions may have little effect on efficiency if there is uncertainty about the sanctions or if individuals do not expect to be greatly affected by the sanctions. This study measures one set of incentives directly affecting school board members, superintendents, and principals in California before and after PSAA and NCLB. These i ncentives include the retention, promotion, and salaries of principals, the retention and salaries of superintendents, and the re-election of incumbent school board members. Of course, t hese are not the only incentives motivating these individuals . I believe that most administrators and school board members are driven by a strong desire to help children maximize their potential. However, PSAA and NCLB were intended to introduce an incentive system that would hold schools accountable in new ways . I find evidence that NCLB increased the accountability of school board members: i ncumbent school bo ard members in districts that meet the NCLB Adequate Yearly Progress (AYP ) requirements are more likely to be re-elected than would have been the case before NCLB. I also find some evidence that NCLB’s sanctions against low -performing schools changed the accountability of principals: principals of schools identified for Program Improvement (PI) after they fail to meet their AYP requirements are more likely to be demoted than would have been the case before NCLB . However, I find no evidence of relationships between student achievement and principal salaries, student achievement and superintendent salaries , or student achievement and superintendent retention rates before PSAA, after PSAA, or after NCLB. Together, these results suggest that accountability prog rams have had a limited effect on school officials. For a s long as public schools have existed in California, the public has been able to vote ineffective school boards out of office, and district leaders have been able to demote ineffective principals. It is not entirely clear why NCLB increased the likelihood of voters holding school boards accountable and the likelihood of district leaders holding principals accountable . One possibility is that NCLB gives voters clear, simple information about student ac hievement in the form of AYP requirements. Another possibility , which does not preclude the first, is that 5 because of the potential school and district sanctions of NCLB, local voters began to share the state’s priorities for schools and districts—for example, student achievement in English language arts and mathematics . Understanding the extent to which each of these two mechanisms increased accountability in California could have important policy implications . This study recommends the following changes to NCLB when it is reauthorized : • To help voters evaluate school board members, policymakers should improve the information available to voters. A district’s level of achievement is a reflection of both the students residing within the district and the effec tiveness of the school board members and administrators running that district. A school accountability system based on growth in student achievement, rather than levels of student achievement, would provide voters with much better information about the effectiveness of the district’s governing board and administrators. • School boards are able to hold administrators accountable for student achievement, and evidence presented here suggests that voters hold school boards accountable for student achievement. The refore, many NCLB sanctions directed at schools and districts, rather than school boards, may be misdirected. It is likely that school boards have a better understanding of conditions in their district than do state and federal policymakers, and can make b etter decisions about the sanctions most appropriate for the schools and administrators in their district. All of NCLB’s sanctions require resources, and school boards may have more information than state and federal policymakers about where resources can best be put to use to improve student achievement in their districts. • When NCLB is reauthorized, its sanctions should focus solely on governance, to ensure that school boards are held accountable for student achievement. Sanctions affecting the governance of a district would hold voters accountable for making good choices. However, the reauthorization of NCLB should not include governance -based sanctions unless it also gives voters better information on school and district effectiveness. A school accountabi lity system based on growth in student achievement, rather than levels, could help voters make better decisions. • Finally, the reauthorized NCLB should help school boards determine the best way to improve student achievement. There are very few intervention s that have been rigorously evaluated and shown to have a positive effect on student outcomes. The reauthorized NCLB should provide much greater support to efforts to collect information about promising interventions, pilot these interventions at a small n umber of schools and districts across the country , rigorously evaluate the effectiveness of these programs (ideally by randomly implementing the programs at some schools but not at others), and provide school boards with information about effective interventions. All technical appendices to this paper are available on the PPIC website: http://www.ppic.org/content/pubs/other/709EL2R_appendix.pdf 6 Acknowledgments I would like to thank Sue Burr, Gary Hart, John Mockler, Ray Reinhard, Paul Warren, and the members of PPIC’s Education Advisory Committee, all of who m shared important insights about the issues discussed in this report. Matthew Springer provided invaluab le comments and suggestions on an earlier draft of this paper. Additional thanks are due to Michael Addonizio, who commented on a draft of this report at the 33 rd Annual Conference of the America n Education Finance Association, and also to Amanda Datnow. M y colleague s Gary Bjork, Jill Cannon, Richard Greene, Ellen Hanak, Karina Jaquet, Hans Johnson, Dave Lesher, Steve Lipscomb, Magnus Lofstrom, Max Neiman, David Neumark, Debbie Reed, Heather Rose, Ria Sengupta, Jon Sonstelie, Lynette Ubois, and Maggie Westo n offered valuable feedback on this project throughout the various stages of its development. Karina Jaquet provided outstanding research support, and Ria Sengupta helped with California’s Standardized Account Code Structure (SACS) data . Any errors , omissions, or other shortcomings of this report are entirely my own. PPIC’s r esearch publications reflect the views of the authors and do not necessarily reflect the views of the institute’s staff, officers, or Board of Directors. 7 Introduct ion The No Child Left Behind Act of 2001 (NCLB) requires that every student be proficient in math and English-language arts by 2013 –14. In California, student proficiency rates have climbed steadily for several years. Yet California schools and districts will soon miss the mark , because NCLB’s timetable for proficiency growth is now climbing rapidl y toward the 100 percent goal. Although average student test scores in California are at all -time highs in many subjects, 1 Fortunately, the United States Congress will reauthorize NCLB in 2010 and will thus have an opportunity to revise and improve the law. While revising the law, members of C ongress should reflect on the extent to which NCLB has achieved its primary goal: “To close the achievement gap with accountability, flexibility, and choice, so that no child is left behind .” nearly every school and school distric t in California will be labeled a failure within the next five years. 2 The implicit goal of NCLB is to give schools and districts incentives simila r to those of firms in a competitive market. Under NCLB, unsuccessful schools must allow their students to attend other, more successful schools in the district. U nsuccessful schools may lose all of their c u stomers— their students—and go out of business, wh ile other schools may respond to the market-like pressures of NCLB and learn to use resources as efficiently as their successful competitors. Of course, it is not possible for a school to be held accountable, but the individual “managers” of that school —school board members, administrators, and teachers, for example— can be held accountable. The goal of NCLB is to improve student achievement by changing the incentives of school board members, school administrators, teachers, and other responsi ble individuals —i.e., to increase accountability . For reasons I discuss below, it may not be possible to determine if NCLB has improved student achievement. However, it is possible to determine if this legislation has improved accountability . If NCLB has n ot improved accountability , then it has not achieved its goal of improving student achievement by improving accountability. Th is report presents evidence that NCLB has increased the accountability of school board members and school principals. However, it is not entirely clear how the legislation has achieved this effect: e ven b efore NCLB came into being , voters were able to hold school board members accountable, and district leaders were able to demote ineffective principals. Th is study explores these issu es and concludes with a discussion of the possible mechanisms through which NCLB may have affected these relationships. I mproving these mechanisms could increase the effect of NCLB on accountability, and in doing so, increase NCLB’s capacity to improve stu dent achievement. 1 National Assessment of Educational Progress, http://nces.ed.gov/nationsreportcard/. 2 Public Law 107-110, the No Child Left Behind Act of 2001, Section 1. 8 School Accountability in California During the 1990s, many states and school districts enacted policies designed to improve student achievement by increasing school efficiency. California’s efforts in this direction began with Senate Bill 1570, introduced by Senator Leroy F. Greene in February 1996 and signed into law by Governor Pete Wilson in September of that year. This law required “the State Board of Education to transmit to the Governor and the Legislature a plan …for the establishment of incentives for the improvement of pupil academic achievement.” Three years later, Governor Gray Davis made “ increased accountability for public schools ” the primary focus of his first three months in office. During his first State of the State speech, he said that education was his “first, second, and third priority,” 3 On April 5, 1999, Davis signed the Public Schoo ls Accountability Act of 1999 (PSAA) into law . In th is bill, the legislature states its intent to increase student achievement not by increasing resources available to schools, but by giving schools incentives to use these resources more efficiently: “The state is in need of an immediate and comprehensive accountability system to hold each of the state’s public schools accountable for the academic progress and achievement of its pupils within the resources available to schools” and he announced a $42 million plan to increase school accountability, stating that “d eficient public schools must be held to task.” 4 [italics mine]. In the same section of the bill, the legislature explicitly states its intent to change schools’ incentives: “The statewide school accountability system must include rewards that recognize high achieving schools as wel l as…sanctions for schools that are continuously l ow performing.” 5 Despite PSAA’s stated intent of improving student achievement “within the resources available to schools,” this legislation also allocated $96 million to assist low -performing schools d uring the 1999 –2000 school year. Because it contained provisions to increase both accountability and spending, PSAA appealed to both those on the right and those on the left of the political spectrum. Since at least the 1950s, national arguments about the best way to improve student achievement have often sp lit along political or ideological lines. Drawing on Milton Frie dman’s article “The Role of Government in Education,” those on the right have often argued that the introduction of market forces in the K –12 education sector would improve the economic efficiency of schools , while those on the left have argued for increases in spending . 6 By incorporating into PSAA themes from both the left and the right, Davis made good on his promise to “govern from the middle. ” 7 PSAA was supplanted by NCLB, which was signed into law by President George W. Bush in January 2002. The political debate preceding NCLB continued the ideological split surrounding education policy . For example, during the U.S. Senate Committee on Health, Education, Labor, and Pension’s February 2001 hearings on NCLB, Democratic Senator John 3 Mike Lewis, “Davis Pushes Education Plan: Governor Urges $444 Million in State of the State Address,” The Fresno Bee, January 7, 1999. 4 California Education Code Section 52050.5.d 5 California Education Code Section 52050.5.i 6 For example, the Elementar y and Secondary Education Act of 1965, part of President Lyndon B. Johnson’s set of “Great Society” domestic programs, more than doubled federal spending on elementary and secondary education. 7 Dan Smith, “Davis, Democrats Set Sail: He Pledges an End to t he ‘Politics of Division,’” The Sacramento Bee, January 5, 1999. 9 Edwards emphasized the need to increase the resources av ailable for schools serving low - achieving students, while Roderick Paige, U.S. Education Secretary under Republican President George W. Bush , stressed a need f or greater efficiency . 8 Democratic Senator Edward Kennedy , who was one of the original authors of NCLB, stressed that NCLB would make additional resources available for schools: “A key component of the No Child Left Behind Act was not just requiring more of schools, but giving them the resources needed to put these reforms in place. ” 9 And i n 2008 , he criticized Bush for failing to “invest sufficiently in public schools .” 10 NCLB was criticized by those on the right as well. In 2002, George F . Will criticized NCLB for not giving low -achieving students sufficient choice in the market for public education, noting that “[b]ecause failing schools frequently are in failing districts, [NCLB’s] ‘choice’ provisions are derisory .” 11 Nonetheless, NCLB included elements d esigned to appeal to both those on the left and those on the right: i n addition to provisions designed to increase school accountability, the legislation dramatically increased the size of grants to districts under Title I, 12 which directs funds to schools and districts serving students from low-income families. B etween 1996 and 2001 , Title I grants to districts averaged $9.5 billion per year, 13,14 but between 2002 and 2007 Title I grants to districts averaged $13 billion per year . 15 It is likely that the push to hold schools and districts accountable for maximizing student achievement has resulted, in part, from research on the relationship between school resources and student achievement. O ther than a few notable exceptions, 16 8 Federal Document Clearing House, “U.S. Senate Committee on Health, Education, Labor, and Pensions Holds Hearing on Education,” February 15, 2001. there i s little evidence that differences in resources available to schools explain differences in student achievement among schools (Hanushek, 2003 ). Why have researchers been unable to identify a positive relationship between resources and student achievement? Some studies on the relationship between resources and achievement have compared changes in resources over long periods of time with changes in achievement over the same period. Others have compared the relationship between resources and achievement across countries at a given point in time. However, there may be differences in the prices of inputs to education or differences in other factors affecting student 9 http://kennedy.senate.gov/issues_and_agenda/issue.cfm?id=076416df -2ae1-42e3-a840-df17d59eccc6, retrieved June 19, 2008. 10 Ibid. 11 George F. Will, “The Learning Curve vs. the Spending Curve, ” The Plain Dealer, January 6, 2002. 12 Title I of NCLB provides federal funds to improve the academic achievement of disadvantaged students. 13 http://www.ed.gov/about/overview/budget/history/, retrieved June 30, 2008. 14 All statistics in this paragraph are reported in inflation -adjusted 2007 dollars. 15 Title I funding per student enrolled in public elementary schools in the U.S. averaged $204 per year between 1996 and 2001 and $266 per year between 2002 and 2007. 16 The best evidence of a significant relationship between school resources and student achi evement comes from Tennessee’s Project STAR. In this experiment, students were randomly assigned to classes with either small (12 –20 students) or large (16 – 30 students) pupil -teacher ratios during their first year of school, which was either kindergarten or first grade. Students remained in either large or small classes until the end of third grade. Each year, these students were tested using the Stanford Achievement Test, an assessment of students’ reading, word - recognition, and math ability. Krueger (199 9) estimates that following the first year of assignment to a small classroom, the test-score gains of students assigned to a small class were 4 percentile points greater, on average, than those of students assigned to a large class. No differences in test-score gains are found during years following the first year of assignment. However, Schanzenbach (2006) estimates that differences in achievement persisted through at least sixth grade, that white females assigned to small classrooms were less likely to h ave given birth before graduating from high school, and that students assigned to small classrooms were less likely to be arrested for property crimes before age 21. 10 achievement over time or across countries. And s uch differences would confound estimates of the relationship between resources and achievement and could explain why these types of studies often fail to find a relationship between resources and achievement. Other studies have looked at the relationship between resources and achievement across schools wi thin the United States at a given point in time; these studies often find no relationship between resources and achievement. Such studies may not effectively take into account differences in local prices that schools face or unobservable differences in stu dents across schools . Moreover, if the differences in resources across schools are small, it may be difficult to identify a relationship between resources and achievement. There may be little measurable relationship between resources and student achievement because we have reached a point where marginal inputs to education have very little effect on student achievement, even if those inputs are used as efficiently as possible. On the other hand, it is also possible that marginal inputs can have a positive effect on student achievement but that schools do not always use the additional resources efficiently. This inefficiency may result from either of two sources. First, teachers and administrators may have allocated additional resources in ways they thought would improve student achievement but which , in fact , had little effect. Or second , teachers and administrators may not have had sufficient incentives to maximize student achievement, given the resources allocated to schools (Hanushek, 1986) . Hanushek (2003) notes that “Pay, promotion, retention in a job, and the like appear to be little different for high quality teachers and low quality teachers. Similarly , jobs for school principals or other administrative and support personnel do no t seem closely related to any student outcomes” (p. 93). This idea—that teachers and administrators may have few , if any, financial incentives to improve student achievement—led to the development of school accountability programs in the United States. 11 The Effects of School Accountability It may not be possible to determine the effect of state and federal school accountability programs on the achieve ment of California students. D ata on the achievement of California students were not collected before the mid -1990s, and thus it is im possible to distinguish between longer -term trends in student achievement and the effects of PSAA and NCLB. Although average student reading and math scores on the Stanford Achievement Test, Ninth Edition (S tanford 9) , increased following the implementation of P SAA, these scores were also increas ing before the implementation of PSAA ( Figures 1–3 ). Similarly, average student test scores on the California Achieve ment Test, Sixth Edition (CAT/6) (Figures 1 –3) have for the most part trended upward both before and after the implementation of NCLB. Figure 1 12 Figure 2 Figure 3 13 Other programs affecting children that were implemented simultaneously with PSAA or NCLB also complicate efforts to identify the effect of school accountability programs on student achievement. And even if it were possible to separate the effects of PSAA or NCLB from long - term trends or the effects of other programs, it would remain unclear whether the accountability co mponents, the increased spending components, or both affected student achievement . Difficulties such as these have led the Center on Education Policy (2007) to conclude that it is “nearly impossible” to estimate whether NCLB has had an effect on student ac hievement. To improve student achievement, accountability programs must give teachers, administrators, and school boards stronger incentives to focus on the student outcomes emphasized by the accountability program and to use resources more intensively an d efficiently to improve those outcomes. It is not cle ar that either PSAA or NCLB has significant ly affect ed the incentives of teachers or the several layers of sch ool system management: school board members, superintendents, and principals. Individual teachers or administrators are not evaluated under either of the plans, and except for the group performance awards of PSAA (funded for only two years ), neither plan provides explicit financial incentives for individual administrators. The sanctions of PSA A and NCLB, described in greater detail below, target schools and districts rather than individual administrators, and b ecause these sanctions affect organizations rather than individuals, it is not clear whether the sanctions can motivate i ndividual administrators. Moreover, unless the sanctions involve severe consequences, it is un likely they will have much effect on productivity (Hanushek and Raymond, 2002). A lthough there is reason to question the effectiveness of PSAA and NCLB in changing individual be havior, a sizable body of research on the effect of other school accountability programs makes it clear that collective rewards and sanctions can be effective. Carnoy and Loeb (2002) and Hanushek and Raymond (2005) compare states whose school accountability systems include consequences for districts, schools, and/or individuals —consequential accountability systems—with states whose accountability systems reported school achievement scores to the public but did not include consequences —“report card ” accountability systems. The authors estimate that between 1996 and 2000, the NAEP math achievement gains of white, African American , and Latino students in grade 8 were signi ficantly larger in states with consequential accountabili ty systems than in state s with report card accountability systems. Carnoy and Loeb also estimate positive math achievement gains among African American (but not white and Latino) students in grade 4 in states with consequential accountability programs . Studies of the effect of ac countability systems introduced in Dallas in 1991 (Ladd, 1999) , in Chicago in 1996 (Jacob, 2005; Neal & Schanzenbach, 2007), and in Florida in 1999 (Figlio and Rouse, 2006; Rouse et al., 2007) also estimate a positive effect on student test sc ores. With the exception of the Dallas program, these accountability systems provide rewards and sanctions to schools and districts , rather than individuals. School accountability programs also change the way in which resources are allocated within a school. There is evidence that schools facing accountability pressure may change the focus of the curriculum to help students acquire the skills necessary for successful test performance (Figlio and Rouse, 2006; Jacob, 2007). As a result, resources may be directed away from “low stakes” curricula in science and social studies (Jacob, 2005) or from difficult advanced topics and remedial topics that benefit students far above or far below the accountability system’s 14 profic iency threshold (Donovan et al., 2006; Neal & Schanzenbach, 2007). Similarly, Figlio and Rouse (2006) estimate that the effects of accountability pressure on student math achievement are limited to the grade being tested. This suggests that r esources were directed toward students in this “high stakes ” grade, although it does not suggest that this occurred at the expense of the math achievement of other students. Springer (2008) also finds evidence that accountability pressures can cause districts to focus o n increasing the achievement of failing students, and that this focus does not necessarily come at the expense of other students’ achievement. Teachers in California report that their instruction has changed in reaction to state accountability tests. A maj ority of California elementary teachers and middle school math teachers surveyed in 2003 –2004 reported that they focused more on content included in the state standards, spent more time teaching test-taking strategies, and/or searched for more effective teaching methods as a result of California’s standards -based assessments (Hamilton et al., 2005) . A majority of these teachers also reported that pressure to meet the state’s Adequate Yearly Progress (AYP ) target led them to focus more on improving student a chievement at their schools. Similarly, Russell and McCombs (2006) find that 77 percent of California teachers they surveyed reported that they are focusing more on improving student achievement at their schools as a result of pressure to meet the AYP targ et. (A larger share of teachers at Title I schools , which are subject to sanctions under NCLB , than at no n-Title I schools reported that they were focusing more on improving student achievement. ) However , a majority of teachers who report ed that they were focusing more on student achievement also reported that it is not the possibility of school rewards or sanctions that has caused them to change their behavior. The source of motivation among these teachers is not identified. School accountability programs may also change individual behavior in ways that those who designe d the programs may not have intended. Research suggests that administrators have responded to accountability pressure in Texas (Cullen & Reback, 2006), Florida (Figlio and Getzler, 2002) , and Chicago (Jacob, 2005) by strategically labeling students “ disabled” and thereby exempting them from testing. Sims (2008) presents evidence that accountability pressures in Wisconsin led low-scoring districts to move up their school start-dates to earlie r in the year , without changing the total number of school days, to provide more time to prepare students for high -stakes testing. He finds that the extra days are associated with small increases in math scores of 4 th graders, but finds no association between earlier start-dates and the reading and language scores of 4 th graders. Finally, schools in Virginia threatened with the pressure of accountability may have increased the caloric content of school lunches on testing days to boost students’ short-term cognitive performance (Figlio and Winicki, 2005). It is unlikely that these are the behaviors the accountability programs were designed to change. Previous research on accountability programs presents strong evidence that group incentives can change indi vidual behavior in many ways, including those that result in student test score gains. However, no published study 17 17 A working paper by Julie Berry Cullen at the University of California, San Diego, and M ichael J. Mazzeo at Northwestern University explores the relationships between student achievement and administrator incentives in Texas, and a working paper by Steve Machin and Tim Besley at the London School of Economics explores the relationships between student achievement and administrator incentives in the United Kingdom. has attempted to measure the effects of changes in the incentives faced by school administrators, and no study has focused on California’s a ccountability programs , as implemented under PSAA and NCLB. The following section 15 explores these issues, examin ing incentives and accountability in California . Although a number of possible incentives may influence the behavior of school administrators, th is study focuses on those that are easily measurable: re -election rates of school -board members and salaries and turnover rates of principals and superintendents. 16 School Accountability and Administrator Incentives in California PSAA legislated the creation of the Academic Performance Index (API), which is used to measure growth of student achievement at each school. Scores from standardized tests given to students in four content areas—English language arts, math, science, and hi story and social studies—are weighted to construct a single API score. Only the scores of students who have been continuously enrolled in a school or local educational agency (LEA) 18 PSAA was supplanted by NCLB in 2002. An important difference between PSAA and NCLB is that while PSAA assessed schools on the basis of growth in student achievement, NCLB assesses schools on the basis of levels of student achievement. To satisfy NCLB accountability , schools and districts must meet the following AYP targets: since October of a given year are counted toward th e school’s or district’ s API. Because demographic changes in a school’s student population may invalidate year -to-year comparisons of a school’s API, schools experiencing significant demographic change, as determined by the superintendent of public instruction, do not receive an API score. 19 • 95% of all students school -wide or district-wide and within each numerically significant subgroup must be tested . 20 • The share of students school -wide or district-wide and within each numerically significant subgroup who achieve the level of “proficient” or above must meet the year’s target in English language arts (44.5% in 2009). • The share of students school -wide or district-wide and within each numerically significa nt subgroup who achieve the level of “proficient” or above must meet the year’s target in math ( 32.2% in 2009 ). • Districts must meet the AYP requirements in each content area ( English language arts and math) in each grade span: grades 2 –5, grades 6 –8, and g rade 10. • California included an additional requirement in its NCLB plan: t he API score for a school or district must increase by 1 point or be at or above that year’s minimum score ( 620 in 2009). To meet AYP requirements, high schools and districts serving high school students must also meet minimum graduation rates and must improve their graduation rate by at least 0.1 percentage points over one year or 0.2 percentage points over two years. The proficiency and 18 Most LEAs are school districts. 19 Schools and districts that do not meet AYP targets but that show progress toward meeting one or more of the targets may be designated as satisfyin g AYP requirements under NCLB’s “safe harbor” provision. 20 Since 2003, students with significant cognitive disabilities who cannot participate in NCLB’s general assessment, with or without accommodations, have been assessed using the California Alternate Performance Assessment (CAPA). There is a cap of 1 percent on the share of all students in an LEA whose scores can be counted as proficient based on CAPA. 17 graduation rate targets increase each y ear beginning in 2006–2007. All students must be proficient in English language arts and math by 2013– 2014, and graduation rates must be at least 83.6 percent by 2013 –2014. Schools and districts that fail to meet the AYP requirement on the same measure fo r two consecutive years are targeted for Program Improvement (PI). The sanctions faced by schools and districts increase each year they are in PI. (See text boxes.) A school or district exits PI when it meets for two consecutive years the AYP criteria that caused it to be placed in PI. Sanctions for Schools in Program Improvement All Years: District is required to allow students to attend a school in the district that is not identified for PI. All Years : School must use 10% of Title I funds for staff professional development. Year 2 : District must offer free tutoring outside of the school day to all low -income students attending schools identified for PI. Year 3 : District must do at least one of the following: • Replace the school staff who are relevant to the school’s failure to meet AYP. • Institute and fully implement a new curriculum. • Significantly decrease management authority at the school. • Appoint an outside expert to advise the school on its progress in meeting AYP. • Restructure the internal organiza tion of the school. Year 4 : District must plan for alternative governance for the school, including one or more of the following: • Close the school and reopen it as a charter school . • Replace all or most of the staff who are relevant to the failure to meet AYP. • Contract with an entity with a demonstrated record of effectiveness to operate the school. • Initiate other major restructuring of the school’s governance arrangement. Year 5 : Implement alternative governance plan. 18 The chain of responsibility —and accountability—in California public schools is very clear: local voters elect school board members, school board members hire superintendents to implement their policies and manage the day -to-day operations of the school district, and superintendents, subject to the approval of the school board, hire principals, who manage the day-to -day operations of individual schools. PSAA and NCLB have made the state’s goals for schools and districts ver y clear. What is not clear is the degree to which these goals are shared by local voters, school board members, and district administrators, and whether the rules of school governance are flexible enough to allow the use of rewards and sanctions to affect behavior . Do voters reward school board members within whose district student achievement has improved as measured by PSAA and NCLB? Do school board members reward the superintendents of such districts ? Are principals rewarded for improving student achievement with in individual schools ? Have these relationships between educator incentives and student achievement changed since the implementation of PSAA and NCLB? School boards are the ultimate authority within each district. The California Education Code gives school boards the authority to “initiate and carry on activities and programs, including the expenditure of funds for programs and activities” 21 21 California Education code, Section 35160.1.b as long as these activities or programs do not conflict with state or federal law. School district go verning boards generally include five members, although unified distri cts may include seven members. Voters within a Sanctions for Districts in Program Improvement All Years: District must devote at least 10% of its Title I, Part A, funds to staff professional development. Year 1 : District, in consultation with parents, school staff, and others, must revise its district plan within three months of identification for PI and must implement the plan during the subsequent school year. Year 3 : State must take corrective action, including at least one of the following: • Replace the district personnel relevant to the failure to meet AYP . • Remove schools from the jurisdiction of the district and establish alternative arrangement for governance and supervision of schools. • Appoint a trustee to administer the district. • Abolish or restructure the district. • Authorize pupils to transfer from a school operated by the district to a higher - performing school operated by another district and provide transportation. • Develop and fully implement a new curriculum based on state academic content and achievement standards. • Defer programmatic funds or reduce a dministrative funds. 19 district are able to hold each school board member accountable for student achievement. School board members are elected to four -year terms, and school board elections are held every two years, so that roughly half of the seats on a school board can be contested every two years. School boards hire superintendents to carry out the day -to-day operations of the school district and implement boa rd policies. A superintendents acts as the chief executive officer of a school district, preparing and submitting a budget to the governing school board of a district , assigning all employees (subject to approval of the school board), and implementing scho ol board policies. It is possible for school boards to hold superintendents directly accountable for their performance—district superintendents do not receive tenure in their positions, and they are not paid according to a standardized salary schedule. Le ss flexibility is available for holding principals accountable for their performance. Principals can attain tenure, just as teachers can, but principals do not receive tenure for a specific position. Thus , a successful principal may be promoted to a more -desirable position, while an unsuccessful principal may be demoted to a less-desirable position, including to the position of a classroom teacher. Most principals in California are paid according to a standard salary schedule, which limits the ability of su perintendents or school boards to reward or sanction principals through salaries . Under a standard salary schedule, a principal bears all of the cost of the extra effort he or she expends in increas ing student achievement, but that same principal shares the rewards of this effort with all of the other principals in a district , albeit with fewer of his or her peers than would be the case for an individual teacher. Although teachers can unquestionably play an important role in improving student achievement (Rivkin et al. , 2005), it is unlikely that school accountability programs have changed the incentives for individual teachers in California . Individual public school teachers do not receive performance bonuses, because most—if not all—public school dist ricts pay teachers according to a standardized “step -and-column” salary schedule. Moreover, the California Education Code makes it very difficult to dismiss low-performing tenured teachers, and workplace rules within each district may make it difficult to transfer low-performing teachers to less-desirable assignments. A district may reward or punish all teachers within a district by granting larger or smaller increases to the teacher salary schedule. However, a n individual teacher would bear the full cost of his or her effort to increase student achievement, while the improved achievement of the students would have only a small effect on the district’s achievement scores, and the rewards from that particular teacher’s efforts would be shared among all teac hers in the district. Thus , it is unlikely that such group incentives in the school accountability program would motivate individual teachers unless the potential rewards or penalties were very large. In this study, I use re-election rates of school -board members and salaries and turnover of principals and superintendents as measures of the incentives facing California school administrators. While it is possible that these administrators may also encounter incentives in the form of pressure from parents, teachers, and other individuals within the community, these pressures may be largely ignored if they are not backed by real threats. Moreover, it would be impossible to measure all of the motivational pressures influencing the behavior of school board memb ers, superintendents, and principals. 20 Ideally, one would measure administrator incentives in California by following principals and superintendents over time and across assignments. The ability to follow administrators in this way would allow one to d etermine if successful administrators are promoted to more favorable assignments within the same district, and if successful administrators are lured away to preferable assignments in other districts. Unfortunately, such data are not available , 22 Study Data but it is possible to infer whether a principal or superintendent is retained at the same school or district from one year to the next, and salary data for administrators who stay within the same district are also available. My e stimates of the relation ship between school accountability and educator incentives are based on data from a number of different sources: • School -level measures of student achievement and other student characteristics from California’s Standardized Testing and Reporting (STAR) prog ram for the years 1998– 2004 23, 24 • School - and district -level AYP reports for the years 2003 –2007. . Both the “high- stakes” tests used for accountability purposes and “low- stakes” tests, which are not, are included in the analysis. 25 • Data on individual superintendent salaries and average salaries for elementary, middle, and high school principals within each district in California from the Salary and Benefits Schedule for the Certificated Bargaining Unit (Form J -90) for the years 1998– 2007. 26 • The Californ ia Basic Education Data System (CBEDS) Professional Assignment Information Form (PAIF) for the years 1998 –2007. 27 • School and district enrollment and demographic information and di strict finance information from the Common Core of Data for the years 1998 –2007 . PAIF includes the following information for each principal and superintendent in California public schools: type of assignment ( e.g., superint endent, principal, or assistant principal), gender, ethnicity, education level, years of educational service, years of employment in the district, and type of credential. Although the data do not include identifiers that allow one to follow an individual a cross districts or across years, there is sufficient information included in PAIF to allow one to infer whether an administrator has remained within the district across two (or more) years. 28 22 The California Longitudinal Teacher Integrated Data Education System (CALTIDES) will begin collecting longitudinal educator data during the 2009– 2010 school year. 23 After 2004, only students in 3rd and 7th grade were tested. 24 http://www.cde.ca.gov/ta/tg/sr/ 25 http://www.cde.ca.gov/ta/ac/ar/ 26 http://www.cde.ca.gov/ds/fd/cs/ 27 http://www.cde.ca.gov/ds/ss/cb/filespaif.asp 28 http://nces.ed.gov/ccd/ 21 • School board election resu lts for the years 1998–2006 from the California Elections Data Archive (CEDA) . 29 Summary statistics for principals, superintendents, schools, distri cts, and school board elections are included in Te chnical Appendix A. If student achievement within a school or district varies little over time, it may not be possible to identify the relationship between administrator incentives and within -school or within -district changes in student achievement. Fortunately, a fair amount of within -school and within -district changes in student achievement are observable. Among the subset of schools in my dataset for which a full panel are available between 2003– 2007, 56 percent either meet AYP all five years or missed AYP all five years, while 44 percent of these schools experienced at least one change in AYP status. Among the subset of districts for which a full panel are available, 59 percent experienced at least one change in AYP status. There is a great deal of variation in PI status within schools as well. In 1998, only 3 percent of schools were in PI; by 2007 , 38 percent were in PI. When the sample is limited to schools for which a full panel of data is available for the years 2003 –2007, 30 percent of schools experience a change in PI status. However, there appears to be very little variation in district PI status. Among districts with a full panel of observations, 87 percent either were placed on PI status ever y year or were not placed on PI status every year between 2005 and 2007. Sufficient variation over time in high -stakes and low-stakes test scores occurs within schools and districts as well. 30 Although there is a fair amount of variation in student achievement within schools and districts over time, most of the variation in student achievement occurs across schools and districts. Not surprisingly, a fair amount of this variation is related to student characteristics. In particular, the likelihood that a sch ool meets its AYP is strongly related to the share of low- income children attending that school (Figure 4 ). About half of districts made AYP in a given year. If one were asked to guess whether a given district had ma de AYP, one would be correct about 50 percent of the time. However, if one also knew what share of the students qualified for federally subsidized school meals, one would be correct 68 percent of the time. Measureable student characteristics are not the only determinants of school success, however: some schools with a high share of low -income students do make AYP, and some schools with a low share of low -income students fail to make AYP. 29 http://www.csus.edu/isr/includes/ISRwebsite/CEDAIntro.stm 30 I have calculated the standard deviation within each school in annual measures of the share of students who are proficient in English language arts (ELA); the average standard deviation amon g schools for which I have a full panel of data is 5.6. For math, the average standard deviation is 4.8. The average standard deviations within districts in the share of students who are proficient in ELA and math are 4.5 and 3.9, respectively. The average standard deviation within each school in low -stakes reading and math test scores —which have been normalized each year to have a mean of 0 and a standard deviation of 1 — are 0.22 and 0.17 standard deviations, respectively, and the standard deviation within each district in low- stakes reading and math scores are 0.15 and 0.12 standard deviations, respectively. 22 Figure 4 School Board Accountability and Student Achievement School boards hold the ultimate decision -making authority within a district, and every two years elections provide voters with the opportunity to hold school boards accountable for these decisions. In the typical school board election in my sample, two incumbents and one challenger vie for two or three open school board seats, and one or two of these open seats is won by an incumbent. Information about achievement of students within a district is readily available on district websites, and information on school and district achievement through the years 1998 to the present are available on the California Department of Education website. To the extent that voters value student achievement as measured by standardized test scores, that open school board seats are contested, and that voters have sufficient information about both challengers and incumbents, one would expect school board members to be held accountable for student achievement. In any given year between 2005 and 2007, about one -fifth of the districts in my sample were identified for PI. The share of districts in the sample meeting AYP in any given year increased steadily from 35 percent in 2003 to 62 percent in 2006, but fell to 46 percent in 2007. The decline between 2006 and 2007 in the share of districts in the sample meeting AYP results primarily from a change in proficiency requirements for students with disabilities. I use regression analysis to analyze the relationship between student achievement and the number of school board members re-elected . 31 31 Details of the empirical models and tables of estimates discussed in the following pages are provided in Appendices B and C. I assume that the number of incumbents winning re -election to the school board is a function of the number of open seats, district 0% 20% 40% 60% 80% 100% Share of Schools that Meet AYP Share of Students who Qualify for Subsidized Meals 23 characteristics , 32 I find no evidence that incumbent school board members are less likely to be elected when a district enters PI, and I find very little evidence of a positive relationship between student achievement within a district and the number of incumbents re-elected . and student achievement over the past two years. The models also control for constant differences across distri cts. The relationship between the number of incumbents winning re -election and student achievement is allowed to be different in years that NCLB is in effect. Because a chievement results are released in August and most school board elections occur in Novem ber, student achievement for the current year is assumed to affect school board re -election rates. Thus, the feedback between student achievement and school board elections is potentially quicker than it is for principals and superintendents. 33 Superintendent Accountability and Student Achievement However, it appears incumbent school board members are much more likely to be re-elected when the school met AYP during the previous year. The estimates suggest that an additional incumbent is re -elected in about half of the cases where a district met AYP requirements in the previous year. This in turn implies that in districts that met AYP, approximately 600 incumbents who would not have been re-elected in the absence of NCLB were re-elected. Relative to the other measures of student achievement discussed in this report, AYP gives voters a strong, simple signal of the success of the school district, and voters appear to respond to this signal. I assume that the retention of a superintendent between the current and following year is a function of superintendent and district characteristics , 34 the achievement of students in the district during the previous year, and other unobservable factors affecting the retention of all superintendents in a given year. Districts must inform superintendents before March 15 35 I find no evidence that a relationship exists between superintendent retention and student achievement as measured by high-stakes tests, low -stakes tests, AYP status, or PI whether or not they will be retained, but student achievement data are not released until the end of the following August. I therefore assume that student achievement from the previous year can influence superintendent retention decisions, but that student achievement during the current year does not affect superintendent retention between the current and following year. Moreover, a school board that dismissed its superintendent after receiving student ach ievement data at the end of August would have very little time to hire a new superintendent before the beginning of the school year. I allow the relationship between principal retention and student achievement to differ in years in which PSAA or NCLB are i n effect. 32 Shares of students wh o are African American, Latino, and white; share of students who qualify for subsidized meals; log of enrollment; and number of schools in the district. 33 See Table C1 in Technical Appendix C. 34 Years in education and its square; gender; race; level of education; shares of students who are African American, Latino, and white; share of students who qualify for subsidized meals; log of district enrollment; and number of schools in the district. 35 California Education Code 44951 states, “Unless a certificated employee holding a position requiring an administrative or supervisory credential is sent written notice deposited in the United States registered mail with postage prepaid and addressed to his or her last known address by March 15 that he or she may be released from his or her position for the following school year, or unless the signature of the employee is obtained by March 15 on the written notice that he or she may be released from his or her position for the following year, he or she shall be continu ed in the position.” 24 status. 36 Principal Accountability and Student Achievement I also find no evidence of a relationship between changes in superintendent salaries and high-stakes tests, AYP status, or PI status. Although I find evidence that change in superintendent salaries are related to student achievement on low-stakes reading exams, the magnitude of these estimates suggest the relationship is very small. The estimates suggest that a superintendent at a district where student achievement in reading is 0.2 standard deviations above what is typical for the district—a very large difference, occurring in fewer than 5 percent of all observations —would expect to receive a salary increase only 0.7 percentage points higher than would otherwise be expected. Although there may be a number of criteria upon which superintendent retention decisions and salary increases are based, the eviden ce presented here suggests that changes in student achievement are not among them. My model of the relationship between school -level measures of student achievement and principal retention is analogous to my model of the r elationship between superintendent retention and student achievement. In a typical year of my sample, about two -thirds of all principals remain in their position the following year; three- fourths of all schools meet their API growth target after 1999, and 63 percent of schools meet AYP after 2002. The share of schools in the sample identified for PI, which predates NCLB, grows steadily over time: in 1998, only 2.8 percent of schools in the sample are identified for PI, but by 2007 the share grows to almost 38 percent. My estimates of the relationship between principal retention and student achievement are somewhat inconsistent. On the one hand, principals at schools that meet their API growth targets are more likely to be retained, and principals at schools that are identified for PI are less likely to be retained. This is as one would expect: s uccessful principals are more likely to keep their jobs. On the other hand, the estimates suggest principals are less likely to be retained when student achievement is high on the high -stakes ELA test, the high-stakes math test, or the low- stakes reading test. This set of results may not be self -contradictory if successful principals are offered pref erable positions in other school districts, while unsuccessful principals are demoted or dismissed. My estimates suggest that this is the case: principals are more likely to be demoted when their school is identified for PI. Taken together, the estimate of the relationship between principal retention and PI status and the estimate of the relationship between principal promotion and PI status suggest that each principal who is not retained when his or her school enters PI is demoted one step. The results imply that between 2003 and 2007 a total of approximately 50 principals who otherwise would have retained their positions were demoted as a result of NCLB. I find no evidence that principals are more likely to be promoted when a school meets its API growth target or when student achievement, as measured by the high -stakes ELA or math tests, is high. And this absence of a relationship does not appear to result from my inability to follow principals over time , either. When I limit my sample to those principals for whom I find 36 These estimates are presented in Table C9 of T echnical Appendix C. 25 exactly one match the following year, I also find no relationship between these measures of student achievement and principal promotion. Finally, I find no evidence of a relationship between chang es in principal salaries and the high -stakes ELA or math exam or a school’s API, AYP, or PI status. Although my estimates of the relationship between low-stakes exams and changes in principal salaries are significant in a statistical sense, the coefficients are very small, and are essential ly precise estimates of exactly zero relationship between changes in principal salaries and student achievement on low-stakes exams. 26 Policy Implications An explicit goal of NCLB is to improve student achievement by holding schools and districts accountable for using resources efficiently and , in doing so , to improve student achievement. The fundamental idea behind school accountability programs is that because schools and districts hold a local monop oly on the provision of public education, teachers, principals, superintendents, school board members, and others working in the district are not sufficiently accountable for improving student achievement. This idea is supported by a body of research that finds little evidence that schools effectively use additional resources to improve student achievement. By holding sc hools accountable, school accountability programs attempt to introduce market -like incentives to the public K –12 education sector . 37 This report presents evidence that NCLB increased the accountability of school board members and principals for improving student achievement. Specifically, incumbent school board members in districts that meet AYP are more likely to be re-elected than would have been the case before NCLB, and principals of schools that enter PI are more likely to be demoted than would have been the case before NCLB. However, I find no evidence of a relationship between changes in principal salaries and a number of different stude nt achievement measures; and I find no evidence of a relationship between student achievement and either superintendent salaries or superintendent retention. As long as public schools have existed in California, the public has been able to vote ineffective school boards out of office, and district leaders have been able to demote ineffective principals. Why did voters and school leaders not exercise accountability before NCLB, given that they had the power to do so? One possibility is that NCLB gave voters a simple metric by which to judge schools and districts —they either met AYP or they did not . 38 A second possibility , which is not mutually exclusive with the first, is that the potential sanctions of NCLB caused local voters to share the state’s priorities for schools and districts. B efore NCLB, local voters may have felt that other school outcomes—such as student achievement in science, social studies, arts, citizenship, and athletics—were as important as levels of student achievement in ELA or math. By threatening to take local contr ol of schools and districts away from voters, NCLB may have given voters reason to prioritize levels of student achievement in ELA and math above other school outcomes. This increase in information may have helped voters make informed decisions about school board members and may have given parents additional leverage when pressur ing boards or superintendents to demote unsuccessful principals. Understanding the extent to which each of these two mechanisms increased accountability in California could have important policy implications. In fact, the policy implications of the first mechanism may be diametrically opposed to the policy implications of the second mechanism. If it is the increase in information , rather than the threat of sanctions, 37 Although both PSAA and NCLB also provided additional resources for schools, these resources were not contingent upon improvements in achievement. 38 Figlio and Rouse (2006) conclude that it is the stigma of low school grades, and not the threat of vouchers, that led to improvements in low -performing Florida schools in the years preceding NCLB. 27 that improved incentives, then the resources that schools and districts in PI devote to revising their school or district plans, implementing these plans, and perhaps eventually restructuring their systems, may in fact be resources that are wa sted. On the other hand, the threat of PI and major restructuring may have an important effect on the priorities of local voters. V oters may worry that property values could decline if schools or districts are labeled failing under NCLB. They may worry tha t major restructuring under NCLB could reduce their ability to shape local education policies. Under this scenario, NCLB improves school accountability by bring ing voters’ priorities closer to those of the state. How Might NCLB Be Improved? First, to help voters evaluate school board members, policymakers should improve the information available to voters. A district’s level of achievement is a reflection of both the students residing within the district and the effectiveness of the school board members and administrators running that district. If student achievement in a district is low, voters may not know the extent to which the low achievement is attributable to the low socioeconomic status of the students in the district and the extent to which the low achievement is attributable to the effectiveness of the district’s governing board and administrators. A school accountability system based on growth in student achievement, rather than levels of student achievement, would provide voters with much better i nformation about the effectiveness of the district’s governing board and administrators. School boards are able to hold administrators accountable for student achievement. The fact that incumbent school board members are more likely to be re-elected when their district makes AYP suggests that voters hold school boards accountable for student achievement. Therefore, many NCLB sanctions directed at schools and districts, rather than school boards, are misdirected . It is likely that school boards have a better understanding of conditions in their district than do state and federal policymakers , and can make better decisions about the sanctions most appropriate for the schools and administrators in their district. For example, if few schools in a district make PI, and the schools that do make PI have little room for additional students, the opportunities for students to transfer from a school in the district that is in PI to one that is not may be limited. It may not make sense to offer low-income students free tutoring outside of the regular school day if doing so limits the resources available to serve these students during the regular school day or if few students choose to participate in after -school tutoring programs. If the teachers at a school in PI are all well -trained and highly qualified, requiring that 10 percent of Title I funds be used for staff development may direct funds away from more effective interventions, such as smaller class sizes for low-income students. All of NCLB’s sanctions require resou rces 39 Because school boards have more information about local conditions than do state or federal policymakers, and because voters appear to hold school boards accountable for student achievement, p olicymakers should remove from NCLB any sanction that could be implemented , and school boards may have more information than state and federal policymakers about where resources can best be put to use to improve student achievement. 39 There are even costs to replacing the staff who are relevant to a school’ s failure to make AYP, as the recruitment of new staff requires resources. 28 by a school board in the absence of NCL B. Specifically, the following sanctions should be removed from NCLB: revising a district plan, requiring that a smaller share of funds be spent on administration, allow ing students to transfer between schools in a district, using at least 10 percent of Title I funds for staff development, offeri ng free tutoring outside the school day to all low -income students, instituting and implementing a new curriculum, appoint ing an outside expert to advise a school on its progress toward improving student achievement, restructuring the internal organizational structure of a school, and closing a school and reopen ing it as a charter. If the sanctions listed above are good policies, effective administrators will suggest these policies to school board members, effective school boards will mandate these policies, and effective administrators will implement them. Administrators who do not implement policies effectively can be held accountable by school boards, and if school boards do not hold administrators accountable, evidence suggests that voters will hold scho ol boards accountable. However, school boards might not hold administrators accountable for student achievement if there are few incentives for school boards to do so. Therefore, policymakers should consider maintaining those sanctions that focus on the g overnance of the district. These sanctions include removing schools from the jurisdiction of the district and establishing alternative arrangement for the governance and supervision of these schools, appointing a trustee to administer the district, abolish ing or restructuring the district, authorizing pupils to transfer from a school operated by the district to a higher -performing school operated by another district, and providing transportation for those students who choose to transfer . If these sanctions are not retained when NCLB is reauthorized, the law should nonetheless include sanctions that focus on governance , to ensure that school boards are held accountable for student achievement. If voters do not elect school board members who mandate the policies necessary to improve student achievement and who employ administrators who can effectively administer these policies, the state should b e given the power to intervene. For example, the state could replace the school board members in such dist ricts with individuals chosen by the state. The state- appointed school board could then mandate the policies necessary to improve student achievement and hold administrators accountable for implementing th ose policies. Once student achievement has improved at the district, state-appointed school board members could be replaced with those elected by voters. Sanctions affecting the governance of a district would hold voters accountable for making good choices. However, the reauthorization should not include g overnance-based sanctions unless it also gives voters better information on school and district effectiveness. A school accountability system based on growth in student achievement, rather than levels, could help voters make better decisions. Finally, the reauthorized NCLB should help school boards determine the best way to improve student achievemen t. The U .S. Department of Education’s What Works Clearinghouse (WWC) labels itself as a “central and trusted source of scientific evidence for what works in edu cation .” 40 40 http://ies.ed.gov/ncee/wwc/ The information presented by the WWC suggests that much more can be learned about what works and what does not. For example, the WWC cannot identify an intervention for which there is strong evidence of a positive effect on the reading achievement or English language development of English language learners . The reauthorized NCLB should provide 29 much greater support to efforts to collect information about promising interventions, pilot these interventions at a small number of schools and districts across the country , rigorously evaluate the effectiveness of these programs (ideally by randomly implementing the programs at some schools but not at others), and provide school boards with information about effective interventions. 30 References Carnoy, Martin and Susanna Loeb. 2002. "Does External Accountability Affect Student Outcomes? A Cross-State Analysis." Educational Evaluation and Policy Analysis , 24:4, pp. 305–331. Center on Education Policy. 2007. "Answering the Question That Matters Most: Has Student Achievement Increased Since No Child Left Behind?" Center on Education Policy: Washington, D.C. Cullen, Julie Berry and Randall Reback. 2006. "Tinkering Toward Accolades: School Gaming Under a Performance Accountability System." National Bureau of Economic Research , Working Paper 12286. Donovan, Colleen, David N. Figlio, and Mark Rush. 2006. "Cramming: The Effects of School Accountability on College -Bound Students." National Bureau of Economic Research , Working Paper 12628. Figlio, David N. and Lawrence S. Getzler. 2002. "Accountability, Ability, and Disability: Gaming the System." National Bureau of Economic Research , Working Paper 9307. Figlio, David N. and Cecilia Elena Rouse. 2006. "Do Accountability and Voucher Threats Improve Low-Performing Schools?" Journal of Public Economics , 90:1–2, pp. 239 –255. Figlio, David N. and Joshua Winicki. 2005. "Food for Thought: The Effects of School Accountability Plans on School Nutrition." Journal of Public Economics , 89:2–3, pp. 381– 394. Frie dman, Milton. 1955. "The Role of Government in Education," in Economics and the Public Interest . Robert A. Solo ed. New Brunswick: Rutgers University Press. Hamilton, Laura, Mark Berends, and Brian Stecher. 2005. "Teachers' Responses to Standards - Based Acc ountability." Rand Working Paper, WR –259. Hanushek, Eric and Margaret E. Raymond. 2002. "Sorting Out Accountability Systems," in School Accountability . Williamson M. Evers and Herbert J. Walberg eds. Stanford, CA: Hoover Institution Press, pp. 75 –104. Hanu shek, Eric A. 1986. "The Economics of Schooling: Production and Efficiency in Public Schools." Journal of Economic Literature , 24:September, pp. 1141–1177. Hanushek, Eric A. 2003. "The Failure of Input -Based Schooling Policies." Economic Journal, 113:Febru ary, pp. 64 –98. Hanushek, Eric A. and Margaret E. Raymond. 2005. "Does School Accountability Lead to Improved Student Performance?" Journal of Policy Analysis and Management , 24:2, pp. 286–327. Jacob, Brian A. 2005. "Accountability, Incentives, and Behavio r: The Impact of High-Stakes Testing in the Chicago Public Schools." Journal of Public Economics , 89:5–6, pp. 761 –796. 31 Jacob, Brian A. 2007. "Test -Based Accountability and Student Achievement: An Investigation of Differential Performance on NAEP and State Assessments." National Bureau of Economic Research, Working Paper 12817. Krueger, Alan B. 1999. "Experimental Estimates of Education Production Functions." Quarterly Journal of Economics , 113:1, pp. 34–63. Ladd, Helen F. 1999. "The Dallas School Accountabi lity and Incentive Program: An Evaluation of Its Impacts on Student Outcomes." Economics of Education Review, 18:1, pp. 1 –16. National Assessment of Educational Progress, available at http://nces.ed.gov/n ationsreportcard/ Neal, Derek and Diane Whitmore Schanzenbach. 2007. "Left Behind By Design: Proficiency Counts and Test-Based Accountability." National Bureau of Economic Research, Working Paper 13293. No Child Left Behind Act of 2001, Public Law 107–110, available at http://www.cde.ca.gov/nclb/ Rivkin, Steven G., Eric A. Hanushek, and John F. Kain. 2005. "Teachers, Schools, and Academic Achievement." Econometrica, 73:2, pp. 417 –458. Rouse, Cecilia Elena, Jane Hannaway, Dan Goldhaber, and David Figlio. 2007. "Feeling the Florida Heat? How Low -Performing Schools Respond to Voucher and Accountability Pressure." National Bureau of Economic Research , Working Paper 13681. Russell, Jennifer and Jennifer Sloan McCombs. 2006. "Superintendent, Principal, and Teacher Understanding of and Attitudes about Accountability." Rand Working Paper, WR –376 –1. Schanzenbach, Diane Whitmore. 2006. "What Have Researchers Learned from Project Star?" Brookings P apers on Education Policy : 205–228. Sims, David P. 2008. "Strategic Responses to School Accountability Measures: It's All in the Timing." Economics of Education Review , 27:1, pp. 58–68. Springer, Matthew G. 2008. "The Influence of an NCLB Accountability Pl an on the Distribution of Student Test-Score Gains." Economics of Education Review , 27:5, pp. 556–563. 32 About the Author Eric Larsen is a research fellow at the Public Policy Institute of California, where he focuses on the economics of public education, particularly school accountability programs, the relationship between school resources and student achievement, and labor markets for teachers. He holds a master’s degree in education from UCLA and a Ph.D. in economics from UC Davis. He was also an Englis h teacher at California public middle schools for eight years. . PUBLIC POLICY INSTITUTE OF CALIFORNIA Board of Directors Walter B. Hewlett, Chair Director Center for Computer Assisted Research in the Humanities Mark Baldassare President and Chief Executive Officer Public Policy Institute of California Ruben Barrales President and Chief Executive Officer San Diego Regional Chamber of Commerce John E. Bryson Retired Chairman and CEO Edison International Gary K. Hart For mer State Senator and Secretary of Education State of California Donna Lucas Chief Executive Officer Lucas Public Affairs Ki Suh Park Design and Managing Partner Gruen Associates Constance L. Rice Co -Director The Advancement Project Thomas C. Sutton Reti red Chairman and Chief Executive Officer Pacific Life Insurance Company Raymond L. Watson Vice Chairman of the Board Emeritus The Irvine Company Carol Whiteside President Emeritus Great Valley Center PUBLIC POLICY INSTITUTE OF CALIFORNIA 500 Washington Street, Suite 600 San Francisco, California 94111 p hone: 415. 291.4400 f ax: 415. 291.4401 PPIC SACRAMENTO CENTER Senator Office Building 1121 L Street, Suite 801 Sacramento, California 95814 p hone: 916.440.1120 f ax: 916.440.1121 www.ppic. org" } ["___content":protected]=> string(104) "

R 709EL2R

" ["_permalink":protected]=> string(108) "https://www.ppic.org/publication/school-accountability-and-administrator-incentives-in-california/r_709el2r/" ["_next":protected]=> array(0) { } ["_prev":protected]=> array(0) { } ["_css_class":protected]=> NULL ["id"]=> int(8630) ["ID"]=> int(8630) ["post_author"]=> string(1) "1" ["post_content"]=> string(0) "" ["post_date"]=> string(19) "2017-05-20 02:39:20" ["post_excerpt"]=> string(0) "" ["post_parent"]=> int(3884) ["post_status"]=> string(7) "inherit" ["post_title"]=> string(9) "R 709EL2R" ["post_type"]=> string(10) "attachment" ["slug"]=> string(9) "r_709el2r" ["__type":protected]=> NULL ["_wp_attached_file"]=> string(13) "R_709EL2R.pdf" ["wpmf_size"]=> string(6) "331799" ["wpmf_filetype"]=> string(3) "pdf" ["wpmf_order"]=> string(1) "0" ["searchwp_content"]=> string(78661) "School Accountability and Administrator Incentives in California S. Eric Larsen with research support from Karina Jaquet Supported with funding from The William and Flora Hewlett Foundation July 2009 The Public Policy Institute of California is dedicated to informing and improving public policy in California through independent, objective, nonpartisan research on major economic, social, and political issues. The institute’s goal is to raise public awareness and to give elected representatives and other decisionmakers a more informed basis for developing policies and programs. The institute’s research focuses on the underlying forces shaping California's future, cutting across a wide range of public policy concerns, including economic development, education, environment and resources, governance, population, public finance, and social and health policy. PPIC is a private operating foundation. It does not take or support positions on any ballot measures or on any local, state, or federal legislation, nor does it endorse, support, or oppose any political parties or candidates for public office. PPIC was established in 1994 with an endowment from William R. Hewlett. Mark Baldassare is President and Chief Executive Officer of PPIC. Walter B. Hewlett is Chair of the Board of Directors. Copyright © 2009 by Public Policy Institute of California All rights reserved San Francisco, CA Short sections of text, not to exceed three paragraphs, may be quoted without written permission provided that full attribution is given to the source and the above copyright notice is included. Research publications reflect the views of the authors and do not necessarily reflect the views of the staff, officers, or Board of Directors of the Public Policy Institute of California. 3 Contents Summary 4 Acknowledgments 6 INTRODUCTION 7 SCHOOL ACCOUNTABILITY IN CALIFORNIA 8 THE EFFECTS OF SCHOOL ACCOUNTABILITY 11 SCHOOL ACCOUNTABILITY AND ADMINISTRATOR INCENTIVES IN CALIFORNIA 16 Study Data 20 School Board Accountability and Student Achievement 22 Superintendent Accountability and Student Achievement 23 Principal Accountability and Student Achievement 24 POLICY IMPLICATIONS 26 How Might NCLB Be Improved? 27 References 30 About the Author 32 4 Summary There is only limited evidence tha t differences in resources available to public schools can explain differences in student achievement among these schools. Many critics have argued that there is little relationship between resources and achievement because schools do not use additional re sources efficiently. B ecause most public schools hold a local monopoly on the provision of public education it has been argued that they are not held accountable for using resources efficiently. Beginning in the mid -1990s and culminating with the federal No Child Left Behin d Act of 2001 (NCLB), school districts , states, and the federal government have introduced school accountability programs. The main objective of these programs is to improve student achievement by defining the primary goals of schools and districts and by holding schools and districts accountable for meeting those goals. However, i t is not clear that either the Public Schools Accountability Act of 1999 ( PSAA) or NCLB has had a significant effect on the accountability of teachers, principals, superinten - dents, or school board members. Individual teachers and administrators are not evaluated under either of the plans, and neither plan provides explicit financial incentives that might motivate improvement. A lthough the school - and district -based sanctions of the laws may affect individuals, the sanctions may have little effect on efficiency if there is uncertainty about the sanctions or if individuals do not expect to be greatly affected by the sanctions. This study measures one set of incentives directly affecting school board members, superintendents, and principals in California before and after PSAA and NCLB. These i ncentives include the retention, promotion, and salaries of principals, the retention and salaries of superintendents, and the re-election of incumbent school board members. Of course, t hese are not the only incentives motivating these individuals . I believe that most administrators and school board members are driven by a strong desire to help children maximize their potential. However, PSAA and NCLB were intended to introduce an incentive system that would hold schools accountable in new ways . I find evidence that NCLB increased the accountability of school board members: i ncumbent school bo ard members in districts that meet the NCLB Adequate Yearly Progress (AYP ) requirements are more likely to be re-elected than would have been the case before NCLB. I also find some evidence that NCLB’s sanctions against low -performing schools changed the accountability of principals: principals of schools identified for Program Improvement (PI) after they fail to meet their AYP requirements are more likely to be demoted than would have been the case before NCLB . However, I find no evidence of relationships between student achievement and principal salaries, student achievement and superintendent salaries , or student achievement and superintendent retention rates before PSAA, after PSAA, or after NCLB. Together, these results suggest that accountability prog rams have had a limited effect on school officials. For a s long as public schools have existed in California, the public has been able to vote ineffective school boards out of office, and district leaders have been able to demote ineffective principals. It is not entirely clear why NCLB increased the likelihood of voters holding school boards accountable and the likelihood of district leaders holding principals accountable . One possibility is that NCLB gives voters clear, simple information about student ac hievement in the form of AYP requirements. Another possibility , which does not preclude the first, is that 5 because of the potential school and district sanctions of NCLB, local voters began to share the state’s priorities for schools and districts—for example, student achievement in English language arts and mathematics . Understanding the extent to which each of these two mechanisms increased accountability in California could have important policy implications . This study recommends the following changes to NCLB when it is reauthorized : • To help voters evaluate school board members, policymakers should improve the information available to voters. A district’s level of achievement is a reflection of both the students residing within the district and the effec tiveness of the school board members and administrators running that district. A school accountability system based on growth in student achievement, rather than levels of student achievement, would provide voters with much better information about the effectiveness of the district’s governing board and administrators. • School boards are able to hold administrators accountable for student achievement, and evidence presented here suggests that voters hold school boards accountable for student achievement. The refore, many NCLB sanctions directed at schools and districts, rather than school boards, may be misdirected. It is likely that school boards have a better understanding of conditions in their district than do state and federal policymakers, and can make b etter decisions about the sanctions most appropriate for the schools and administrators in their district. All of NCLB’s sanctions require resources, and school boards may have more information than state and federal policymakers about where resources can best be put to use to improve student achievement in their districts. • When NCLB is reauthorized, its sanctions should focus solely on governance, to ensure that school boards are held accountable for student achievement. Sanctions affecting the governance of a district would hold voters accountable for making good choices. However, the reauthorization of NCLB should not include governance -based sanctions unless it also gives voters better information on school and district effectiveness. A school accountabi lity system based on growth in student achievement, rather than levels, could help voters make better decisions. • Finally, the reauthorized NCLB should help school boards determine the best way to improve student achievement. There are very few intervention s that have been rigorously evaluated and shown to have a positive effect on student outcomes. The reauthorized NCLB should provide much greater support to efforts to collect information about promising interventions, pilot these interventions at a small n umber of schools and districts across the country , rigorously evaluate the effectiveness of these programs (ideally by randomly implementing the programs at some schools but not at others), and provide school boards with information about effective interventions. All technical appendices to this paper are available on the PPIC website: http://www.ppic.org/content/pubs/other/709EL2R_appendix.pdf 6 Acknowledgments I would like to thank Sue Burr, Gary Hart, John Mockler, Ray Reinhard, Paul Warren, and the members of PPIC’s Education Advisory Committee, all of who m shared important insights about the issues discussed in this report. Matthew Springer provided invaluab le comments and suggestions on an earlier draft of this paper. Additional thanks are due to Michael Addonizio, who commented on a draft of this report at the 33 rd Annual Conference of the America n Education Finance Association, and also to Amanda Datnow. M y colleague s Gary Bjork, Jill Cannon, Richard Greene, Ellen Hanak, Karina Jaquet, Hans Johnson, Dave Lesher, Steve Lipscomb, Magnus Lofstrom, Max Neiman, David Neumark, Debbie Reed, Heather Rose, Ria Sengupta, Jon Sonstelie, Lynette Ubois, and Maggie Westo n offered valuable feedback on this project throughout the various stages of its development. Karina Jaquet provided outstanding research support, and Ria Sengupta helped with California’s Standardized Account Code Structure (SACS) data . Any errors , omissions, or other shortcomings of this report are entirely my own. PPIC’s r esearch publications reflect the views of the authors and do not necessarily reflect the views of the institute’s staff, officers, or Board of Directors. 7 Introduct ion The No Child Left Behind Act of 2001 (NCLB) requires that every student be proficient in math and English-language arts by 2013 –14. In California, student proficiency rates have climbed steadily for several years. Yet California schools and districts will soon miss the mark , because NCLB’s timetable for proficiency growth is now climbing rapidl y toward the 100 percent goal. Although average student test scores in California are at all -time highs in many subjects, 1 Fortunately, the United States Congress will reauthorize NCLB in 2010 and will thus have an opportunity to revise and improve the law. While revising the law, members of C ongress should reflect on the extent to which NCLB has achieved its primary goal: “To close the achievement gap with accountability, flexibility, and choice, so that no child is left behind .” nearly every school and school distric t in California will be labeled a failure within the next five years. 2 The implicit goal of NCLB is to give schools and districts incentives simila r to those of firms in a competitive market. Under NCLB, unsuccessful schools must allow their students to attend other, more successful schools in the district. U nsuccessful schools may lose all of their c u stomers— their students—and go out of business, wh ile other schools may respond to the market-like pressures of NCLB and learn to use resources as efficiently as their successful competitors. Of course, it is not possible for a school to be held accountable, but the individual “managers” of that school —school board members, administrators, and teachers, for example— can be held accountable. The goal of NCLB is to improve student achievement by changing the incentives of school board members, school administrators, teachers, and other responsi ble individuals —i.e., to increase accountability . For reasons I discuss below, it may not be possible to determine if NCLB has improved student achievement. However, it is possible to determine if this legislation has improved accountability . If NCLB has n ot improved accountability , then it has not achieved its goal of improving student achievement by improving accountability. Th is report presents evidence that NCLB has increased the accountability of school board members and school principals. However, it is not entirely clear how the legislation has achieved this effect: e ven b efore NCLB came into being , voters were able to hold school board members accountable, and district leaders were able to demote ineffective principals. Th is study explores these issu es and concludes with a discussion of the possible mechanisms through which NCLB may have affected these relationships. I mproving these mechanisms could increase the effect of NCLB on accountability, and in doing so, increase NCLB’s capacity to improve stu dent achievement. 1 National Assessment of Educational Progress, http://nces.ed.gov/nationsreportcard/. 2 Public Law 107-110, the No Child Left Behind Act of 2001, Section 1. 8 School Accountability in California During the 1990s, many states and school districts enacted policies designed to improve student achievement by increasing school efficiency. California’s efforts in this direction began with Senate Bill 1570, introduced by Senator Leroy F. Greene in February 1996 and signed into law by Governor Pete Wilson in September of that year. This law required “the State Board of Education to transmit to the Governor and the Legislature a plan …for the establishment of incentives for the improvement of pupil academic achievement.” Three years later, Governor Gray Davis made “ increased accountability for public schools ” the primary focus of his first three months in office. During his first State of the State speech, he said that education was his “first, second, and third priority,” 3 On April 5, 1999, Davis signed the Public Schoo ls Accountability Act of 1999 (PSAA) into law . In th is bill, the legislature states its intent to increase student achievement not by increasing resources available to schools, but by giving schools incentives to use these resources more efficiently: “The state is in need of an immediate and comprehensive accountability system to hold each of the state’s public schools accountable for the academic progress and achievement of its pupils within the resources available to schools” and he announced a $42 million plan to increase school accountability, stating that “d eficient public schools must be held to task.” 4 [italics mine]. In the same section of the bill, the legislature explicitly states its intent to change schools’ incentives: “The statewide school accountability system must include rewards that recognize high achieving schools as wel l as…sanctions for schools that are continuously l ow performing.” 5 Despite PSAA’s stated intent of improving student achievement “within the resources available to schools,” this legislation also allocated $96 million to assist low -performing schools d uring the 1999 –2000 school year. Because it contained provisions to increase both accountability and spending, PSAA appealed to both those on the right and those on the left of the political spectrum. Since at least the 1950s, national arguments about the best way to improve student achievement have often sp lit along political or ideological lines. Drawing on Milton Frie dman’s article “The Role of Government in Education,” those on the right have often argued that the introduction of market forces in the K –12 education sector would improve the economic efficiency of schools , while those on the left have argued for increases in spending . 6 By incorporating into PSAA themes from both the left and the right, Davis made good on his promise to “govern from the middle. ” 7 PSAA was supplanted by NCLB, which was signed into law by President George W. Bush in January 2002. The political debate preceding NCLB continued the ideological split surrounding education policy . For example, during the U.S. Senate Committee on Health, Education, Labor, and Pension’s February 2001 hearings on NCLB, Democratic Senator John 3 Mike Lewis, “Davis Pushes Education Plan: Governor Urges $444 Million in State of the State Address,” The Fresno Bee, January 7, 1999. 4 California Education Code Section 52050.5.d 5 California Education Code Section 52050.5.i 6 For example, the Elementar y and Secondary Education Act of 1965, part of President Lyndon B. Johnson’s set of “Great Society” domestic programs, more than doubled federal spending on elementary and secondary education. 7 Dan Smith, “Davis, Democrats Set Sail: He Pledges an End to t he ‘Politics of Division,’” The Sacramento Bee, January 5, 1999. 9 Edwards emphasized the need to increase the resources av ailable for schools serving low - achieving students, while Roderick Paige, U.S. Education Secretary under Republican President George W. Bush , stressed a need f or greater efficiency . 8 Democratic Senator Edward Kennedy , who was one of the original authors of NCLB, stressed that NCLB would make additional resources available for schools: “A key component of the No Child Left Behind Act was not just requiring more of schools, but giving them the resources needed to put these reforms in place. ” 9 And i n 2008 , he criticized Bush for failing to “invest sufficiently in public schools .” 10 NCLB was criticized by those on the right as well. In 2002, George F . Will criticized NCLB for not giving low -achieving students sufficient choice in the market for public education, noting that “[b]ecause failing schools frequently are in failing districts, [NCLB’s] ‘choice’ provisions are derisory .” 11 Nonetheless, NCLB included elements d esigned to appeal to both those on the left and those on the right: i n addition to provisions designed to increase school accountability, the legislation dramatically increased the size of grants to districts under Title I, 12 which directs funds to schools and districts serving students from low-income families. B etween 1996 and 2001 , Title I grants to districts averaged $9.5 billion per year, 13,14 but between 2002 and 2007 Title I grants to districts averaged $13 billion per year . 15 It is likely that the push to hold schools and districts accountable for maximizing student achievement has resulted, in part, from research on the relationship between school resources and student achievement. O ther than a few notable exceptions, 16 8 Federal Document Clearing House, “U.S. Senate Committee on Health, Education, Labor, and Pensions Holds Hearing on Education,” February 15, 2001. there i s little evidence that differences in resources available to schools explain differences in student achievement among schools (Hanushek, 2003 ). Why have researchers been unable to identify a positive relationship between resources and student achievement? Some studies on the relationship between resources and achievement have compared changes in resources over long periods of time with changes in achievement over the same period. Others have compared the relationship between resources and achievement across countries at a given point in time. However, there may be differences in the prices of inputs to education or differences in other factors affecting student 9 http://kennedy.senate.gov/issues_and_agenda/issue.cfm?id=076416df -2ae1-42e3-a840-df17d59eccc6, retrieved June 19, 2008. 10 Ibid. 11 George F. Will, “The Learning Curve vs. the Spending Curve, ” The Plain Dealer, January 6, 2002. 12 Title I of NCLB provides federal funds to improve the academic achievement of disadvantaged students. 13 http://www.ed.gov/about/overview/budget/history/, retrieved June 30, 2008. 14 All statistics in this paragraph are reported in inflation -adjusted 2007 dollars. 15 Title I funding per student enrolled in public elementary schools in the U.S. averaged $204 per year between 1996 and 2001 and $266 per year between 2002 and 2007. 16 The best evidence of a significant relationship between school resources and student achi evement comes from Tennessee’s Project STAR. In this experiment, students were randomly assigned to classes with either small (12 –20 students) or large (16 – 30 students) pupil -teacher ratios during their first year of school, which was either kindergarten or first grade. Students remained in either large or small classes until the end of third grade. Each year, these students were tested using the Stanford Achievement Test, an assessment of students’ reading, word - recognition, and math ability. Krueger (199 9) estimates that following the first year of assignment to a small classroom, the test-score gains of students assigned to a small class were 4 percentile points greater, on average, than those of students assigned to a large class. No differences in test-score gains are found during years following the first year of assignment. However, Schanzenbach (2006) estimates that differences in achievement persisted through at least sixth grade, that white females assigned to small classrooms were less likely to h ave given birth before graduating from high school, and that students assigned to small classrooms were less likely to be arrested for property crimes before age 21. 10 achievement over time or across countries. And s uch differences would confound estimates of the relationship between resources and achievement and could explain why these types of studies often fail to find a relationship between resources and achievement. Other studies have looked at the relationship between resources and achievement across schools wi thin the United States at a given point in time; these studies often find no relationship between resources and achievement. Such studies may not effectively take into account differences in local prices that schools face or unobservable differences in stu dents across schools . Moreover, if the differences in resources across schools are small, it may be difficult to identify a relationship between resources and achievement. There may be little measurable relationship between resources and student achievement because we have reached a point where marginal inputs to education have very little effect on student achievement, even if those inputs are used as efficiently as possible. On the other hand, it is also possible that marginal inputs can have a positive effect on student achievement but that schools do not always use the additional resources efficiently. This inefficiency may result from either of two sources. First, teachers and administrators may have allocated additional resources in ways they thought would improve student achievement but which , in fact , had little effect. Or second , teachers and administrators may not have had sufficient incentives to maximize student achievement, given the resources allocated to schools (Hanushek, 1986) . Hanushek (2003) notes that “Pay, promotion, retention in a job, and the like appear to be little different for high quality teachers and low quality teachers. Similarly , jobs for school principals or other administrative and support personnel do no t seem closely related to any student outcomes” (p. 93). This idea—that teachers and administrators may have few , if any, financial incentives to improve student achievement—led to the development of school accountability programs in the United States. 11 The Effects of School Accountability It may not be possible to determine the effect of state and federal school accountability programs on the achieve ment of California students. D ata on the achievement of California students were not collected before the mid -1990s, and thus it is im possible to distinguish between longer -term trends in student achievement and the effects of PSAA and NCLB. Although average student reading and math scores on the Stanford Achievement Test, Ninth Edition (S tanford 9) , increased following the implementation of P SAA, these scores were also increas ing before the implementation of PSAA ( Figures 1–3 ). Similarly, average student test scores on the California Achieve ment Test, Sixth Edition (CAT/6) (Figures 1 –3) have for the most part trended upward both before and after the implementation of NCLB. Figure 1 12 Figure 2 Figure 3 13 Other programs affecting children that were implemented simultaneously with PSAA or NCLB also complicate efforts to identify the effect of school accountability programs on student achievement. And even if it were possible to separate the effects of PSAA or NCLB from long - term trends or the effects of other programs, it would remain unclear whether the accountability co mponents, the increased spending components, or both affected student achievement . Difficulties such as these have led the Center on Education Policy (2007) to conclude that it is “nearly impossible” to estimate whether NCLB has had an effect on student ac hievement. To improve student achievement, accountability programs must give teachers, administrators, and school boards stronger incentives to focus on the student outcomes emphasized by the accountability program and to use resources more intensively an d efficiently to improve those outcomes. It is not cle ar that either PSAA or NCLB has significant ly affect ed the incentives of teachers or the several layers of sch ool system management: school board members, superintendents, and principals. Individual teachers or administrators are not evaluated under either of the plans, and except for the group performance awards of PSAA (funded for only two years ), neither plan provides explicit financial incentives for individual administrators. The sanctions of PSA A and NCLB, described in greater detail below, target schools and districts rather than individual administrators, and b ecause these sanctions affect organizations rather than individuals, it is not clear whether the sanctions can motivate i ndividual administrators. Moreover, unless the sanctions involve severe consequences, it is un likely they will have much effect on productivity (Hanushek and Raymond, 2002). A lthough there is reason to question the effectiveness of PSAA and NCLB in changing individual be havior, a sizable body of research on the effect of other school accountability programs makes it clear that collective rewards and sanctions can be effective. Carnoy and Loeb (2002) and Hanushek and Raymond (2005) compare states whose school accountability systems include consequences for districts, schools, and/or individuals —consequential accountability systems—with states whose accountability systems reported school achievement scores to the public but did not include consequences —“report card ” accountability systems. The authors estimate that between 1996 and 2000, the NAEP math achievement gains of white, African American , and Latino students in grade 8 were signi ficantly larger in states with consequential accountabili ty systems than in state s with report card accountability systems. Carnoy and Loeb also estimate positive math achievement gains among African American (but not white and Latino) students in grade 4 in states with consequential accountability programs . Studies of the effect of ac countability systems introduced in Dallas in 1991 (Ladd, 1999) , in Chicago in 1996 (Jacob, 2005; Neal & Schanzenbach, 2007), and in Florida in 1999 (Figlio and Rouse, 2006; Rouse et al., 2007) also estimate a positive effect on student test sc ores. With the exception of the Dallas program, these accountability systems provide rewards and sanctions to schools and districts , rather than individuals. School accountability programs also change the way in which resources are allocated within a school. There is evidence that schools facing accountability pressure may change the focus of the curriculum to help students acquire the skills necessary for successful test performance (Figlio and Rouse, 2006; Jacob, 2007). As a result, resources may be directed away from “low stakes” curricula in science and social studies (Jacob, 2005) or from difficult advanced topics and remedial topics that benefit students far above or far below the accountability system’s 14 profic iency threshold (Donovan et al., 2006; Neal & Schanzenbach, 2007). Similarly, Figlio and Rouse (2006) estimate that the effects of accountability pressure on student math achievement are limited to the grade being tested. This suggests that r esources were directed toward students in this “high stakes ” grade, although it does not suggest that this occurred at the expense of the math achievement of other students. Springer (2008) also finds evidence that accountability pressures can cause districts to focus o n increasing the achievement of failing students, and that this focus does not necessarily come at the expense of other students’ achievement. Teachers in California report that their instruction has changed in reaction to state accountability tests. A maj ority of California elementary teachers and middle school math teachers surveyed in 2003 –2004 reported that they focused more on content included in the state standards, spent more time teaching test-taking strategies, and/or searched for more effective teaching methods as a result of California’s standards -based assessments (Hamilton et al., 2005) . A majority of these teachers also reported that pressure to meet the state’s Adequate Yearly Progress (AYP ) target led them to focus more on improving student a chievement at their schools. Similarly, Russell and McCombs (2006) find that 77 percent of California teachers they surveyed reported that they are focusing more on improving student achievement at their schools as a result of pressure to meet the AYP targ et. (A larger share of teachers at Title I schools , which are subject to sanctions under NCLB , than at no n-Title I schools reported that they were focusing more on improving student achievement. ) However , a majority of teachers who report ed that they were focusing more on student achievement also reported that it is not the possibility of school rewards or sanctions that has caused them to change their behavior. The source of motivation among these teachers is not identified. School accountability programs may also change individual behavior in ways that those who designe d the programs may not have intended. Research suggests that administrators have responded to accountability pressure in Texas (Cullen & Reback, 2006), Florida (Figlio and Getzler, 2002) , and Chicago (Jacob, 2005) by strategically labeling students “ disabled” and thereby exempting them from testing. Sims (2008) presents evidence that accountability pressures in Wisconsin led low-scoring districts to move up their school start-dates to earlie r in the year , without changing the total number of school days, to provide more time to prepare students for high -stakes testing. He finds that the extra days are associated with small increases in math scores of 4 th graders, but finds no association between earlier start-dates and the reading and language scores of 4 th graders. Finally, schools in Virginia threatened with the pressure of accountability may have increased the caloric content of school lunches on testing days to boost students’ short-term cognitive performance (Figlio and Winicki, 2005). It is unlikely that these are the behaviors the accountability programs were designed to change. Previous research on accountability programs presents strong evidence that group incentives can change indi vidual behavior in many ways, including those that result in student test score gains. However, no published study 17 17 A working paper by Julie Berry Cullen at the University of California, San Diego, and M ichael J. Mazzeo at Northwestern University explores the relationships between student achievement and administrator incentives in Texas, and a working paper by Steve Machin and Tim Besley at the London School of Economics explores the relationships between student achievement and administrator incentives in the United Kingdom. has attempted to measure the effects of changes in the incentives faced by school administrators, and no study has focused on California’s a ccountability programs , as implemented under PSAA and NCLB. The following section 15 explores these issues, examin ing incentives and accountability in California . Although a number of possible incentives may influence the behavior of school administrators, th is study focuses on those that are easily measurable: re -election rates of school -board members and salaries and turnover rates of principals and superintendents. 16 School Accountability and Administrator Incentives in California PSAA legislated the creation of the Academic Performance Index (API), which is used to measure growth of student achievement at each school. Scores from standardized tests given to students in four content areas—English language arts, math, science, and hi story and social studies—are weighted to construct a single API score. Only the scores of students who have been continuously enrolled in a school or local educational agency (LEA) 18 PSAA was supplanted by NCLB in 2002. An important difference between PSAA and NCLB is that while PSAA assessed schools on the basis of growth in student achievement, NCLB assesses schools on the basis of levels of student achievement. To satisfy NCLB accountability , schools and districts must meet the following AYP targets: since October of a given year are counted toward th e school’s or district’ s API. Because demographic changes in a school’s student population may invalidate year -to-year comparisons of a school’s API, schools experiencing significant demographic change, as determined by the superintendent of public instruction, do not receive an API score. 19 • 95% of all students school -wide or district-wide and within each numerically significant subgroup must be tested . 20 • The share of students school -wide or district-wide and within each numerically significant subgroup who achieve the level of “proficient” or above must meet the year’s target in English language arts (44.5% in 2009). • The share of students school -wide or district-wide and within each numerically significa nt subgroup who achieve the level of “proficient” or above must meet the year’s target in math ( 32.2% in 2009 ). • Districts must meet the AYP requirements in each content area ( English language arts and math) in each grade span: grades 2 –5, grades 6 –8, and g rade 10. • California included an additional requirement in its NCLB plan: t he API score for a school or district must increase by 1 point or be at or above that year’s minimum score ( 620 in 2009). To meet AYP requirements, high schools and districts serving high school students must also meet minimum graduation rates and must improve their graduation rate by at least 0.1 percentage points over one year or 0.2 percentage points over two years. The proficiency and 18 Most LEAs are school districts. 19 Schools and districts that do not meet AYP targets but that show progress toward meeting one or more of the targets may be designated as satisfyin g AYP requirements under NCLB’s “safe harbor” provision. 20 Since 2003, students with significant cognitive disabilities who cannot participate in NCLB’s general assessment, with or without accommodations, have been assessed using the California Alternate Performance Assessment (CAPA). There is a cap of 1 percent on the share of all students in an LEA whose scores can be counted as proficient based on CAPA. 17 graduation rate targets increase each y ear beginning in 2006–2007. All students must be proficient in English language arts and math by 2013– 2014, and graduation rates must be at least 83.6 percent by 2013 –2014. Schools and districts that fail to meet the AYP requirement on the same measure fo r two consecutive years are targeted for Program Improvement (PI). The sanctions faced by schools and districts increase each year they are in PI. (See text boxes.) A school or district exits PI when it meets for two consecutive years the AYP criteria that caused it to be placed in PI. Sanctions for Schools in Program Improvement All Years: District is required to allow students to attend a school in the district that is not identified for PI. All Years : School must use 10% of Title I funds for staff professional development. Year 2 : District must offer free tutoring outside of the school day to all low -income students attending schools identified for PI. Year 3 : District must do at least one of the following: • Replace the school staff who are relevant to the school’s failure to meet AYP. • Institute and fully implement a new curriculum. • Significantly decrease management authority at the school. • Appoint an outside expert to advise the school on its progress in meeting AYP. • Restructure the internal organiza tion of the school. Year 4 : District must plan for alternative governance for the school, including one or more of the following: • Close the school and reopen it as a charter school . • Replace all or most of the staff who are relevant to the failure to meet AYP. • Contract with an entity with a demonstrated record of effectiveness to operate the school. • Initiate other major restructuring of the school’s governance arrangement. Year 5 : Implement alternative governance plan. 18 The chain of responsibility —and accountability—in California public schools is very clear: local voters elect school board members, school board members hire superintendents to implement their policies and manage the day -to-day operations of the school district, and superintendents, subject to the approval of the school board, hire principals, who manage the day-to -day operations of individual schools. PSAA and NCLB have made the state’s goals for schools and districts ver y clear. What is not clear is the degree to which these goals are shared by local voters, school board members, and district administrators, and whether the rules of school governance are flexible enough to allow the use of rewards and sanctions to affect behavior . Do voters reward school board members within whose district student achievement has improved as measured by PSAA and NCLB? Do school board members reward the superintendents of such districts ? Are principals rewarded for improving student achievement with in individual schools ? Have these relationships between educator incentives and student achievement changed since the implementation of PSAA and NCLB? School boards are the ultimate authority within each district. The California Education Code gives school boards the authority to “initiate and carry on activities and programs, including the expenditure of funds for programs and activities” 21 21 California Education code, Section 35160.1.b as long as these activities or programs do not conflict with state or federal law. School district go verning boards generally include five members, although unified distri cts may include seven members. Voters within a Sanctions for Districts in Program Improvement All Years: District must devote at least 10% of its Title I, Part A, funds to staff professional development. Year 1 : District, in consultation with parents, school staff, and others, must revise its district plan within three months of identification for PI and must implement the plan during the subsequent school year. Year 3 : State must take corrective action, including at least one of the following: • Replace the district personnel relevant to the failure to meet AYP . • Remove schools from the jurisdiction of the district and establish alternative arrangement for governance and supervision of schools. • Appoint a trustee to administer the district. • Abolish or restructure the district. • Authorize pupils to transfer from a school operated by the district to a higher - performing school operated by another district and provide transportation. • Develop and fully implement a new curriculum based on state academic content and achievement standards. • Defer programmatic funds or reduce a dministrative funds. 19 district are able to hold each school board member accountable for student achievement. School board members are elected to four -year terms, and school board elections are held every two years, so that roughly half of the seats on a school board can be contested every two years. School boards hire superintendents to carry out the day -to-day operations of the school district and implement boa rd policies. A superintendents acts as the chief executive officer of a school district, preparing and submitting a budget to the governing school board of a district , assigning all employees (subject to approval of the school board), and implementing scho ol board policies. It is possible for school boards to hold superintendents directly accountable for their performance—district superintendents do not receive tenure in their positions, and they are not paid according to a standardized salary schedule. Le ss flexibility is available for holding principals accountable for their performance. Principals can attain tenure, just as teachers can, but principals do not receive tenure for a specific position. Thus , a successful principal may be promoted to a more -desirable position, while an unsuccessful principal may be demoted to a less-desirable position, including to the position of a classroom teacher. Most principals in California are paid according to a standard salary schedule, which limits the ability of su perintendents or school boards to reward or sanction principals through salaries . Under a standard salary schedule, a principal bears all of the cost of the extra effort he or she expends in increas ing student achievement, but that same principal shares the rewards of this effort with all of the other principals in a district , albeit with fewer of his or her peers than would be the case for an individual teacher. Although teachers can unquestionably play an important role in improving student achievement (Rivkin et al. , 2005), it is unlikely that school accountability programs have changed the incentives for individual teachers in California . Individual public school teachers do not receive performance bonuses, because most—if not all—public school dist ricts pay teachers according to a standardized “step -and-column” salary schedule. Moreover, the California Education Code makes it very difficult to dismiss low-performing tenured teachers, and workplace rules within each district may make it difficult to transfer low-performing teachers to less-desirable assignments. A district may reward or punish all teachers within a district by granting larger or smaller increases to the teacher salary schedule. However, a n individual teacher would bear the full cost of his or her effort to increase student achievement, while the improved achievement of the students would have only a small effect on the district’s achievement scores, and the rewards from that particular teacher’s efforts would be shared among all teac hers in the district. Thus , it is unlikely that such group incentives in the school accountability program would motivate individual teachers unless the potential rewards or penalties were very large. In this study, I use re-election rates of school -board members and salaries and turnover of principals and superintendents as measures of the incentives facing California school administrators. While it is possible that these administrators may also encounter incentives in the form of pressure from parents, teachers, and other individuals within the community, these pressures may be largely ignored if they are not backed by real threats. Moreover, it would be impossible to measure all of the motivational pressures influencing the behavior of school board memb ers, superintendents, and principals. 20 Ideally, one would measure administrator incentives in California by following principals and superintendents over time and across assignments. The ability to follow administrators in this way would allow one to d etermine if successful administrators are promoted to more favorable assignments within the same district, and if successful administrators are lured away to preferable assignments in other districts. Unfortunately, such data are not available , 22 Study Data but it is possible to infer whether a principal or superintendent is retained at the same school or district from one year to the next, and salary data for administrators who stay within the same district are also available. My e stimates of the relation ship between school accountability and educator incentives are based on data from a number of different sources: • School -level measures of student achievement and other student characteristics from California’s Standardized Testing and Reporting (STAR) prog ram for the years 1998– 2004 23, 24 • School - and district -level AYP reports for the years 2003 –2007. . Both the “high- stakes” tests used for accountability purposes and “low- stakes” tests, which are not, are included in the analysis. 25 • Data on individual superintendent salaries and average salaries for elementary, middle, and high school principals within each district in California from the Salary and Benefits Schedule for the Certificated Bargaining Unit (Form J -90) for the years 1998– 2007. 26 • The Californ ia Basic Education Data System (CBEDS) Professional Assignment Information Form (PAIF) for the years 1998 –2007. 27 • School and district enrollment and demographic information and di strict finance information from the Common Core of Data for the years 1998 –2007 . PAIF includes the following information for each principal and superintendent in California public schools: type of assignment ( e.g., superint endent, principal, or assistant principal), gender, ethnicity, education level, years of educational service, years of employment in the district, and type of credential. Although the data do not include identifiers that allow one to follow an individual a cross districts or across years, there is sufficient information included in PAIF to allow one to infer whether an administrator has remained within the district across two (or more) years. 28 22 The California Longitudinal Teacher Integrated Data Education System (CALTIDES) will begin collecting longitudinal educator data during the 2009– 2010 school year. 23 After 2004, only students in 3rd and 7th grade were tested. 24 http://www.cde.ca.gov/ta/tg/sr/ 25 http://www.cde.ca.gov/ta/ac/ar/ 26 http://www.cde.ca.gov/ds/fd/cs/ 27 http://www.cde.ca.gov/ds/ss/cb/filespaif.asp 28 http://nces.ed.gov/ccd/ 21 • School board election resu lts for the years 1998–2006 from the California Elections Data Archive (CEDA) . 29 Summary statistics for principals, superintendents, schools, distri cts, and school board elections are included in Te chnical Appendix A. If student achievement within a school or district varies little over time, it may not be possible to identify the relationship between administrator incentives and within -school or within -district changes in student achievement. Fortunately, a fair amount of within -school and within -district changes in student achievement are observable. Among the subset of schools in my dataset for which a full panel are available between 2003– 2007, 56 percent either meet AYP all five years or missed AYP all five years, while 44 percent of these schools experienced at least one change in AYP status. Among the subset of districts for which a full panel are available, 59 percent experienced at least one change in AYP status. There is a great deal of variation in PI status within schools as well. In 1998, only 3 percent of schools were in PI; by 2007 , 38 percent were in PI. When the sample is limited to schools for which a full panel of data is available for the years 2003 –2007, 30 percent of schools experience a change in PI status. However, there appears to be very little variation in district PI status. Among districts with a full panel of observations, 87 percent either were placed on PI status ever y year or were not placed on PI status every year between 2005 and 2007. Sufficient variation over time in high -stakes and low-stakes test scores occurs within schools and districts as well. 30 Although there is a fair amount of variation in student achievement within schools and districts over time, most of the variation in student achievement occurs across schools and districts. Not surprisingly, a fair amount of this variation is related to student characteristics. In particular, the likelihood that a sch ool meets its AYP is strongly related to the share of low- income children attending that school (Figure 4 ). About half of districts made AYP in a given year. If one were asked to guess whether a given district had ma de AYP, one would be correct about 50 percent of the time. However, if one also knew what share of the students qualified for federally subsidized school meals, one would be correct 68 percent of the time. Measureable student characteristics are not the only determinants of school success, however: some schools with a high share of low -income students do make AYP, and some schools with a low share of low -income students fail to make AYP. 29 http://www.csus.edu/isr/includes/ISRwebsite/CEDAIntro.stm 30 I have calculated the standard deviation within each school in annual measures of the share of students who are proficient in English language arts (ELA); the average standard deviation amon g schools for which I have a full panel of data is 5.6. For math, the average standard deviation is 4.8. The average standard deviations within districts in the share of students who are proficient in ELA and math are 4.5 and 3.9, respectively. The average standard deviation within each school in low -stakes reading and math test scores —which have been normalized each year to have a mean of 0 and a standard deviation of 1 — are 0.22 and 0.17 standard deviations, respectively, and the standard deviation within each district in low- stakes reading and math scores are 0.15 and 0.12 standard deviations, respectively. 22 Figure 4 School Board Accountability and Student Achievement School boards hold the ultimate decision -making authority within a district, and every two years elections provide voters with the opportunity to hold school boards accountable for these decisions. In the typical school board election in my sample, two incumbents and one challenger vie for two or three open school board seats, and one or two of these open seats is won by an incumbent. Information about achievement of students within a district is readily available on district websites, and information on school and district achievement through the years 1998 to the present are available on the California Department of Education website. To the extent that voters value student achievement as measured by standardized test scores, that open school board seats are contested, and that voters have sufficient information about both challengers and incumbents, one would expect school board members to be held accountable for student achievement. In any given year between 2005 and 2007, about one -fifth of the districts in my sample were identified for PI. The share of districts in the sample meeting AYP in any given year increased steadily from 35 percent in 2003 to 62 percent in 2006, but fell to 46 percent in 2007. The decline between 2006 and 2007 in the share of districts in the sample meeting AYP results primarily from a change in proficiency requirements for students with disabilities. I use regression analysis to analyze the relationship between student achievement and the number of school board members re-elected . 31 31 Details of the empirical models and tables of estimates discussed in the following pages are provided in Appendices B and C. I assume that the number of incumbents winning re -election to the school board is a function of the number of open seats, district 0% 20% 40% 60% 80% 100% Share of Schools that Meet AYP Share of Students who Qualify for Subsidized Meals 23 characteristics , 32 I find no evidence that incumbent school board members are less likely to be elected when a district enters PI, and I find very little evidence of a positive relationship between student achievement within a district and the number of incumbents re-elected . and student achievement over the past two years. The models also control for constant differences across distri cts. The relationship between the number of incumbents winning re -election and student achievement is allowed to be different in years that NCLB is in effect. Because a chievement results are released in August and most school board elections occur in Novem ber, student achievement for the current year is assumed to affect school board re -election rates. Thus, the feedback between student achievement and school board elections is potentially quicker than it is for principals and superintendents. 33 Superintendent Accountability and Student Achievement However, it appears incumbent school board members are much more likely to be re-elected when the school met AYP during the previous year. The estimates suggest that an additional incumbent is re -elected in about half of the cases where a district met AYP requirements in the previous year. This in turn implies that in districts that met AYP, approximately 600 incumbents who would not have been re-elected in the absence of NCLB were re-elected. Relative to the other measures of student achievement discussed in this report, AYP gives voters a strong, simple signal of the success of the school district, and voters appear to respond to this signal. I assume that the retention of a superintendent between the current and following year is a function of superintendent and district characteristics , 34 the achievement of students in the district during the previous year, and other unobservable factors affecting the retention of all superintendents in a given year. Districts must inform superintendents before March 15 35 I find no evidence that a relationship exists between superintendent retention and student achievement as measured by high-stakes tests, low -stakes tests, AYP status, or PI whether or not they will be retained, but student achievement data are not released until the end of the following August. I therefore assume that student achievement from the previous year can influence superintendent retention decisions, but that student achievement during the current year does not affect superintendent retention between the current and following year. Moreover, a school board that dismissed its superintendent after receiving student ach ievement data at the end of August would have very little time to hire a new superintendent before the beginning of the school year. I allow the relationship between principal retention and student achievement to differ in years in which PSAA or NCLB are i n effect. 32 Shares of students wh o are African American, Latino, and white; share of students who qualify for subsidized meals; log of enrollment; and number of schools in the district. 33 See Table C1 in Technical Appendix C. 34 Years in education and its square; gender; race; level of education; shares of students who are African American, Latino, and white; share of students who qualify for subsidized meals; log of district enrollment; and number of schools in the district. 35 California Education Code 44951 states, “Unless a certificated employee holding a position requiring an administrative or supervisory credential is sent written notice deposited in the United States registered mail with postage prepaid and addressed to his or her last known address by March 15 that he or she may be released from his or her position for the following school year, or unless the signature of the employee is obtained by March 15 on the written notice that he or she may be released from his or her position for the following year, he or she shall be continu ed in the position.” 24 status. 36 Principal Accountability and Student Achievement I also find no evidence of a relationship between changes in superintendent salaries and high-stakes tests, AYP status, or PI status. Although I find evidence that change in superintendent salaries are related to student achievement on low-stakes reading exams, the magnitude of these estimates suggest the relationship is very small. The estimates suggest that a superintendent at a district where student achievement in reading is 0.2 standard deviations above what is typical for the district—a very large difference, occurring in fewer than 5 percent of all observations —would expect to receive a salary increase only 0.7 percentage points higher than would otherwise be expected. Although there may be a number of criteria upon which superintendent retention decisions and salary increases are based, the eviden ce presented here suggests that changes in student achievement are not among them. My model of the relationship between school -level measures of student achievement and principal retention is analogous to my model of the r elationship between superintendent retention and student achievement. In a typical year of my sample, about two -thirds of all principals remain in their position the following year; three- fourths of all schools meet their API growth target after 1999, and 63 percent of schools meet AYP after 2002. The share of schools in the sample identified for PI, which predates NCLB, grows steadily over time: in 1998, only 2.8 percent of schools in the sample are identified for PI, but by 2007 the share grows to almost 38 percent. My estimates of the relationship between principal retention and student achievement are somewhat inconsistent. On the one hand, principals at schools that meet their API growth targets are more likely to be retained, and principals at schools that are identified for PI are less likely to be retained. This is as one would expect: s uccessful principals are more likely to keep their jobs. On the other hand, the estimates suggest principals are less likely to be retained when student achievement is high on the high -stakes ELA test, the high-stakes math test, or the low- stakes reading test. This set of results may not be self -contradictory if successful principals are offered pref erable positions in other school districts, while unsuccessful principals are demoted or dismissed. My estimates suggest that this is the case: principals are more likely to be demoted when their school is identified for PI. Taken together, the estimate of the relationship between principal retention and PI status and the estimate of the relationship between principal promotion and PI status suggest that each principal who is not retained when his or her school enters PI is demoted one step. The results imply that between 2003 and 2007 a total of approximately 50 principals who otherwise would have retained their positions were demoted as a result of NCLB. I find no evidence that principals are more likely to be promoted when a school meets its API growth target or when student achievement, as measured by the high -stakes ELA or math tests, is high. And this absence of a relationship does not appear to result from my inability to follow principals over time , either. When I limit my sample to those principals for whom I find 36 These estimates are presented in Table C9 of T echnical Appendix C. 25 exactly one match the following year, I also find no relationship between these measures of student achievement and principal promotion. Finally, I find no evidence of a relationship between chang es in principal salaries and the high -stakes ELA or math exam or a school’s API, AYP, or PI status. Although my estimates of the relationship between low-stakes exams and changes in principal salaries are significant in a statistical sense, the coefficients are very small, and are essential ly precise estimates of exactly zero relationship between changes in principal salaries and student achievement on low-stakes exams. 26 Policy Implications An explicit goal of NCLB is to improve student achievement by holding schools and districts accountable for using resources efficiently and , in doing so , to improve student achievement. The fundamental idea behind school accountability programs is that because schools and districts hold a local monop oly on the provision of public education, teachers, principals, superintendents, school board members, and others working in the district are not sufficiently accountable for improving student achievement. This idea is supported by a body of research that finds little evidence that schools effectively use additional resources to improve student achievement. By holding sc hools accountable, school accountability programs attempt to introduce market -like incentives to the public K –12 education sector . 37 This report presents evidence that NCLB increased the accountability of school board members and principals for improving student achievement. Specifically, incumbent school board members in districts that meet AYP are more likely to be re-elected than would have been the case before NCLB, and principals of schools that enter PI are more likely to be demoted than would have been the case before NCLB. However, I find no evidence of a relationship between changes in principal salaries and a number of different stude nt achievement measures; and I find no evidence of a relationship between student achievement and either superintendent salaries or superintendent retention. As long as public schools have existed in California, the public has been able to vote ineffective school boards out of office, and district leaders have been able to demote ineffective principals. Why did voters and school leaders not exercise accountability before NCLB, given that they had the power to do so? One possibility is that NCLB gave voters a simple metric by which to judge schools and districts —they either met AYP or they did not . 38 A second possibility , which is not mutually exclusive with the first, is that the potential sanctions of NCLB caused local voters to share the state’s priorities for schools and districts. B efore NCLB, local voters may have felt that other school outcomes—such as student achievement in science, social studies, arts, citizenship, and athletics—were as important as levels of student achievement in ELA or math. By threatening to take local contr ol of schools and districts away from voters, NCLB may have given voters reason to prioritize levels of student achievement in ELA and math above other school outcomes. This increase in information may have helped voters make informed decisions about school board members and may have given parents additional leverage when pressur ing boards or superintendents to demote unsuccessful principals. Understanding the extent to which each of these two mechanisms increased accountability in California could have important policy implications. In fact, the policy implications of the first mechanism may be diametrically opposed to the policy implications of the second mechanism. If it is the increase in information , rather than the threat of sanctions, 37 Although both PSAA and NCLB also provided additional resources for schools, these resources were not contingent upon improvements in achievement. 38 Figlio and Rouse (2006) conclude that it is the stigma of low school grades, and not the threat of vouchers, that led to improvements in low -performing Florida schools in the years preceding NCLB. 27 that improved incentives, then the resources that schools and districts in PI devote to revising their school or district plans, implementing these plans, and perhaps eventually restructuring their systems, may in fact be resources that are wa sted. On the other hand, the threat of PI and major restructuring may have an important effect on the priorities of local voters. V oters may worry that property values could decline if schools or districts are labeled failing under NCLB. They may worry tha t major restructuring under NCLB could reduce their ability to shape local education policies. Under this scenario, NCLB improves school accountability by bring ing voters’ priorities closer to those of the state. How Might NCLB Be Improved? First, to help voters evaluate school board members, policymakers should improve the information available to voters. A district’s level of achievement is a reflection of both the students residing within the district and the effectiveness of the school board members and administrators running that district. If student achievement in a district is low, voters may not know the extent to which the low achievement is attributable to the low socioeconomic status of the students in the district and the extent to which the low achievement is attributable to the effectiveness of the district’s governing board and administrators. A school accountability system based on growth in student achievement, rather than levels of student achievement, would provide voters with much better i nformation about the effectiveness of the district’s governing board and administrators. School boards are able to hold administrators accountable for student achievement. The fact that incumbent school board members are more likely to be re-elected when their district makes AYP suggests that voters hold school boards accountable for student achievement. Therefore, many NCLB sanctions directed at schools and districts, rather than school boards, are misdirected . It is likely that school boards have a better understanding of conditions in their district than do state and federal policymakers , and can make better decisions about the sanctions most appropriate for the schools and administrators in their district. For example, if few schools in a district make PI, and the schools that do make PI have little room for additional students, the opportunities for students to transfer from a school in the district that is in PI to one that is not may be limited. It may not make sense to offer low-income students free tutoring outside of the regular school day if doing so limits the resources available to serve these students during the regular school day or if few students choose to participate in after -school tutoring programs. If the teachers at a school in PI are all well -trained and highly qualified, requiring that 10 percent of Title I funds be used for staff development may direct funds away from more effective interventions, such as smaller class sizes for low-income students. All of NCLB’s sanctions require resou rces 39 Because school boards have more information about local conditions than do state or federal policymakers, and because voters appear to hold school boards accountable for student achievement, p olicymakers should remove from NCLB any sanction that could be implemented , and school boards may have more information than state and federal policymakers about where resources can best be put to use to improve student achievement. 39 There are even costs to replacing the staff who are relevant to a school’ s failure to make AYP, as the recruitment of new staff requires resources. 28 by a school board in the absence of NCL B. Specifically, the following sanctions should be removed from NCLB: revising a district plan, requiring that a smaller share of funds be spent on administration, allow ing students to transfer between schools in a district, using at least 10 percent of Title I funds for staff development, offeri ng free tutoring outside the school day to all low -income students, instituting and implementing a new curriculum, appoint ing an outside expert to advise a school on its progress toward improving student achievement, restructuring the internal organizational structure of a school, and closing a school and reopen ing it as a charter. If the sanctions listed above are good policies, effective administrators will suggest these policies to school board members, effective school boards will mandate these policies, and effective administrators will implement them. Administrators who do not implement policies effectively can be held accountable by school boards, and if school boards do not hold administrators accountable, evidence suggests that voters will hold scho ol boards accountable. However, school boards might not hold administrators accountable for student achievement if there are few incentives for school boards to do so. Therefore, policymakers should consider maintaining those sanctions that focus on the g overnance of the district. These sanctions include removing schools from the jurisdiction of the district and establishing alternative arrangement for the governance and supervision of these schools, appointing a trustee to administer the district, abolish ing or restructuring the district, authorizing pupils to transfer from a school operated by the district to a higher -performing school operated by another district, and providing transportation for those students who choose to transfer . If these sanctions are not retained when NCLB is reauthorized, the law should nonetheless include sanctions that focus on governance , to ensure that school boards are held accountable for student achievement. If voters do not elect school board members who mandate the policies necessary to improve student achievement and who employ administrators who can effectively administer these policies, the state should b e given the power to intervene. For example, the state could replace the school board members in such dist ricts with individuals chosen by the state. The state- appointed school board could then mandate the policies necessary to improve student achievement and hold administrators accountable for implementing th ose policies. Once student achievement has improved at the district, state-appointed school board members could be replaced with those elected by voters. Sanctions affecting the governance of a district would hold voters accountable for making good choices. However, the reauthorization should not include g overnance-based sanctions unless it also gives voters better information on school and district effectiveness. A school accountability system based on growth in student achievement, rather than levels, could help voters make better decisions. Finally, the reauthorized NCLB should help school boards determine the best way to improve student achievemen t. The U .S. Department of Education’s What Works Clearinghouse (WWC) labels itself as a “central and trusted source of scientific evidence for what works in edu cation .” 40 40 http://ies.ed.gov/ncee/wwc/ The information presented by the WWC suggests that much more can be learned about what works and what does not. For example, the WWC cannot identify an intervention for which there is strong evidence of a positive effect on the reading achievement or English language development of English language learners . The reauthorized NCLB should provide 29 much greater support to efforts to collect information about promising interventions, pilot these interventions at a small number of schools and districts across the country , rigorously evaluate the effectiveness of these programs (ideally by randomly implementing the programs at some schools but not at others), and provide school boards with information about effective interventions. 30 References Carnoy, Martin and Susanna Loeb. 2002. "Does External Accountability Affect Student Outcomes? A Cross-State Analysis." Educational Evaluation and Policy Analysis , 24:4, pp. 305–331. Center on Education Policy. 2007. "Answering the Question That Matters Most: Has Student Achievement Increased Since No Child Left Behind?" Center on Education Policy: Washington, D.C. Cullen, Julie Berry and Randall Reback. 2006. "Tinkering Toward Accolades: School Gaming Under a Performance Accountability System." National Bureau of Economic Research , Working Paper 12286. Donovan, Colleen, David N. Figlio, and Mark Rush. 2006. "Cramming: The Effects of School Accountability on College -Bound Students." National Bureau of Economic Research , Working Paper 12628. Figlio, David N. and Lawrence S. Getzler. 2002. "Accountability, Ability, and Disability: Gaming the System." National Bureau of Economic Research , Working Paper 9307. Figlio, David N. and Cecilia Elena Rouse. 2006. "Do Accountability and Voucher Threats Improve Low-Performing Schools?" Journal of Public Economics , 90:1–2, pp. 239 –255. Figlio, David N. and Joshua Winicki. 2005. "Food for Thought: The Effects of School Accountability Plans on School Nutrition." Journal of Public Economics , 89:2–3, pp. 381– 394. Frie dman, Milton. 1955. "The Role of Government in Education," in Economics and the Public Interest . Robert A. Solo ed. New Brunswick: Rutgers University Press. Hamilton, Laura, Mark Berends, and Brian Stecher. 2005. "Teachers' Responses to Standards - Based Acc ountability." Rand Working Paper, WR –259. Hanushek, Eric and Margaret E. Raymond. 2002. "Sorting Out Accountability Systems," in School Accountability . Williamson M. Evers and Herbert J. Walberg eds. Stanford, CA: Hoover Institution Press, pp. 75 –104. Hanu shek, Eric A. 1986. "The Economics of Schooling: Production and Efficiency in Public Schools." Journal of Economic Literature , 24:September, pp. 1141–1177. Hanushek, Eric A. 2003. "The Failure of Input -Based Schooling Policies." Economic Journal, 113:Febru ary, pp. 64 –98. Hanushek, Eric A. and Margaret E. Raymond. 2005. "Does School Accountability Lead to Improved Student Performance?" Journal of Policy Analysis and Management , 24:2, pp. 286–327. Jacob, Brian A. 2005. "Accountability, Incentives, and Behavio r: The Impact of High-Stakes Testing in the Chicago Public Schools." Journal of Public Economics , 89:5–6, pp. 761 –796. 31 Jacob, Brian A. 2007. "Test -Based Accountability and Student Achievement: An Investigation of Differential Performance on NAEP and State Assessments." National Bureau of Economic Research, Working Paper 12817. Krueger, Alan B. 1999. "Experimental Estimates of Education Production Functions." Quarterly Journal of Economics , 113:1, pp. 34–63. Ladd, Helen F. 1999. "The Dallas School Accountabi lity and Incentive Program: An Evaluation of Its Impacts on Student Outcomes." Economics of Education Review, 18:1, pp. 1 –16. National Assessment of Educational Progress, available at http://nces.ed.gov/n ationsreportcard/ Neal, Derek and Diane Whitmore Schanzenbach. 2007. "Left Behind By Design: Proficiency Counts and Test-Based Accountability." National Bureau of Economic Research, Working Paper 13293. No Child Left Behind Act of 2001, Public Law 107–110, available at http://www.cde.ca.gov/nclb/ Rivkin, Steven G., Eric A. Hanushek, and John F. Kain. 2005. "Teachers, Schools, and Academic Achievement." Econometrica, 73:2, pp. 417 –458. Rouse, Cecilia Elena, Jane Hannaway, Dan Goldhaber, and David Figlio. 2007. "Feeling the Florida Heat? How Low -Performing Schools Respond to Voucher and Accountability Pressure." National Bureau of Economic Research , Working Paper 13681. Russell, Jennifer and Jennifer Sloan McCombs. 2006. "Superintendent, Principal, and Teacher Understanding of and Attitudes about Accountability." Rand Working Paper, WR –376 –1. Schanzenbach, Diane Whitmore. 2006. "What Have Researchers Learned from Project Star?" Brookings P apers on Education Policy : 205–228. Sims, David P. 2008. "Strategic Responses to School Accountability Measures: It's All in the Timing." Economics of Education Review , 27:1, pp. 58–68. Springer, Matthew G. 2008. "The Influence of an NCLB Accountability Pl an on the Distribution of Student Test-Score Gains." Economics of Education Review , 27:5, pp. 556–563. 32 About the Author Eric Larsen is a research fellow at the Public Policy Institute of California, where he focuses on the economics of public education, particularly school accountability programs, the relationship between school resources and student achievement, and labor markets for teachers. He holds a master’s degree in education from UCLA and a Ph.D. in economics from UC Davis. He was also an Englis h teacher at California public middle schools for eight years. . PUBLIC POLICY INSTITUTE OF CALIFORNIA Board of Directors Walter B. Hewlett, Chair Director Center for Computer Assisted Research in the Humanities Mark Baldassare President and Chief Executive Officer Public Policy Institute of California Ruben Barrales President and Chief Executive Officer San Diego Regional Chamber of Commerce John E. Bryson Retired Chairman and CEO Edison International Gary K. Hart For mer State Senator and Secretary of Education State of California Donna Lucas Chief Executive Officer Lucas Public Affairs Ki Suh Park Design and Managing Partner Gruen Associates Constance L. Rice Co -Director The Advancement Project Thomas C. Sutton Reti red Chairman and Chief Executive Officer Pacific Life Insurance Company Raymond L. Watson Vice Chairman of the Board Emeritus The Irvine Company Carol Whiteside President Emeritus Great Valley Center PUBLIC POLICY INSTITUTE OF CALIFORNIA 500 Washington Street, Suite 600 San Francisco, California 94111 p hone: 415. 291.4400 f ax: 415. 291.4401 PPIC SACRAMENTO CENTER Senator Office Building 1121 L Street, Suite 801 Sacramento, California 95814 p hone: 916.440.1120 f ax: 916.440.1121 www.ppic. org" ["post_date_gmt"]=> string(19) "2017-05-20 09:39:20" ["comment_status"]=> string(4) "open" ["ping_status"]=> string(6) "closed" ["post_password"]=> string(0) "" ["post_name"]=> string(9) "r_709el2r" ["to_ping"]=> string(0) "" ["pinged"]=> string(0) "" ["post_modified"]=> string(19) "2017-05-20 02:39:20" ["post_modified_gmt"]=> string(19) "2017-05-20 09:39:20" ["post_content_filtered"]=> string(0) "" ["guid"]=> string(51) "http://148.62.4.17/wp-content/uploads/R_709EL2R.pdf" ["menu_order"]=> int(0) ["post_mime_type"]=> string(15) "application/pdf" ["comment_count"]=> string(1) "0" ["filter"]=> string(3) "raw" ["status"]=> string(7) "inherit" ["attachment_authors"]=> bool(false) }