Donate
Independent, objective, nonpartisan research

R 504DNR

Authors

R 504DNR

Tagged with:

Publication PDFs

Database

This is the content currently stored in the post and postmeta tables.

View live version

object(Timber\Post)#3742 (44) { ["ImageClass"]=> string(12) "Timber\Image" ["PostClass"]=> string(11) "Timber\Post" ["TermClass"]=> string(11) "Timber\Term" ["object_type"]=> string(4) "post" ["custom"]=> array(5) { ["_wp_attached_file"]=> string(12) "R_504DNR.pdf" ["wpmf_size"]=> string(7) "4544045" ["wpmf_filetype"]=> string(3) "pdf" ["wpmf_order"]=> string(1) "0" ["searchwp_content"]=> string(221714) "The Effects of School-to-Career Programs on Postsecondary Enrollment and Employment ••• David Neumark 2004 Library of Congress Cataloging-in-Publication Data Neumark, David. The effects of school-to-career programs on postsecondary enrollment and employment / David Neumark. p. cm. Includes bibliographical references. ISBN: 1-58213-088-4 1. School-to-work transition—United States. 2. Vocational education—United States. 3. College attendance—United States. 4. High school graduates—Employment—United States. I. Public Policy Institute of California. II. Title. LC1045.N48 2004 370.11'3'0973—dc22 2004007828 Copyright © 2004 by Public Policy Institute of California All rights reserved San Francisco, CA Short sections of text, not to exceed three paragraphs, may be quoted without written permission provided that full attribution is given to the source and the above copyright notice is included. PPIC does not take or support positions on any ballot measure or on any local, state, or federal legislation, nor does it endorse, support, or oppose any political parties or candidates for public office. Research publications reflect the views of the authors and do not necessarily reflect the views of the staff, officers, or Board of Directors of the Public Policy Institute of California. Foreword One of the greatest concerns surrounding educational policymaking is whether students, especially disadvantaged students, are prepared for a successful transition from school to work. This is especially true during the high school years—a crucial time when young people need the education and motivation that will enable and encourage them to attend college or help them find higher-paying jobs immediately after they leave high school. Too often, those not going on to college simply drift from one lowpaying job to another—or worse, remain unemployed. This concern led to educational reform efforts in the 1990s that focused on school-towork programs or, as they are known in California, school-to-career programs. The School-to-Work Opportunities Act (STWOA), passed by Congress in 1994, provided more than $135 million to California over a five-year period to support increased school-to-career activities such as internships, apprenticeships, and mentoring of students on a part-time basis by local employers. STWOA was not reauthorized after its initial five years. And although certain other school-to-career programs still exist, mainly Tech Prep and Career Academies, this loss of funding appears to have left a gaping hole in efforts to prepare low-skilled youth for higher-paying jobs—the principal goal behind the original legislation. This is particularly problematical for California because, as research in other PPIC studies has shown, income inequality is higher in California than in the rest of the nation, and the difference between the incomes of the “haves” and “have nots” is largely attributable to education. Given the loss of federal STWOA funding, it is important to determine just how effective the activities supported by this program were and whether it might be in California’s best interest to restore some of the funding for these activities. In this report, David Neumark analyzes a national dataset that includes the results of broad-based iii programs similar to those established in California. He concludes that a case can be made for restoring or reallocating some funding to the types of activities supported by STWOA. The national results suggest that these activities are more effective in boosting postsecondary enrollment and employment than certain other types of existing school-to-career programs, such as Tech Prep. In a secondary and equally important analysis, Neumark also looks more specifically at existing evaluations of various programs in California, and he finds them wanting. He argues that there is a pressing need for a serious evaluation of the effectiveness of the school-to-career activities remaining in California and for any such activities that might arise in the future. He also notes that future school-to-career efforts in the state would be well served by better articulating the goals of the program and by establishing clear criteria for measuring progress toward these goals. There is compelling evidence that schooling is essential to the future well-being of California’s children. A large body of research has shown that the transition from school to a career—whether after high school graduation or postsecondary education—is a critical step in determining personal economic success. Neumark concludes that much remains to be done in California to assure that this critical step is successful. Failure will result in a setback not just for our students but for the many Californians—workers, employers, citizens, and consumers alike—who benefit from a more-skilled workforce. David W. Lyon President and CEO Public Policy Institute of California iv Summary “School-to-work,” or “school-to-career” as it is known in California, refers to programs that integrate academic and vocational skills with the goal of improving the transitions of youths from schools into their careers. This report uses national data from the 1997 National Longitudinal Survey of Youth (NLSY97) to evaluate the effectiveness of the types of school-to-career (STC) programs that were encouraged and supported in California by the grants received by the state from the federal School-to-Work Opportunities Act of 1994 (STWOA). In particular, the empirical analysis focuses on whether participation in these STC programs increases postsecondary college enrollment or employment. STWOA provided more than $1.5 billion over a five-year period to support increased school-to-work activities in the nation’s public schools. This money was made available to states to create STC systems entailing cooperation among schools, private business, and government bodies (Office of Technology Assessment, 1995). Congress passed STWOA in response to three areas of particular concern for public education identified by researchers and educators. These were (1) a lack of connection between school and work that led many youths to be unmotivated in school and to experience subsequent difficulty moving out of low-wage jobs, (2) youths completing school with insufficient skills needed for the labor market, and (3) increasing labor market demands for complex thinking, close teamwork, and the ability to learn on the job. More generally, STWOA was motivated by a concern shared by policymakers and researchers alike that school-to-work transitions of youths in the United States entail too much joblessness, job instability, and employment in dead-end jobs (for example, see U.S. General Accounting Office, 1990). To help young people develop the skills needed in the workforce and make better connections to careers, STWOA set out to increase (1) school-based initiatives such as career v links to academic curriculum and career awareness activities, (2) workbased activities such as job shadowing,1 internships, and apprenticeships, and (3) connecting activities, such as the development of partnerships with employers and postsecondary institutions. In 1996, California was awarded $130 million over five years from the National School-to-Work Office to set up an extensive STC system, and the state received a supplemental $7.2 million grant in 2000. The STC activities funded under STWOA can be viewed as one of three branches of STC activities in California. The other two branches are Career Academies (sometimes called Partnership Academies) and Tech Prep, which focus much more narrowly on specific student populations in contrast to the general and broad-based approach of STWOA. However, after its initial five years, STWOA was not reauthorized. Although funding for Career Academies and Tech Prep from the state or federal governments has continued, the state has not made up the shortfall of funding for general STC activities. Given this change in funding, it is a particularly opportune time to study the effectiveness of the types of general and broad-based STC activities in California that were encouraged and supported by the federal legislation, to ask whether the state might better achieve its goals by funding general STC activities. Despite the fact that present prospects for significant new spending on STC in California are slim, given the budget crisis, possibilities for minor new allocations or reallocation from the other two branches of STC or other areas of spending on education should not be ruled out, especially in light of the significant effect that the relatively small amounts spent in California under STWOA had on implementation of STC programs. Goals of the Research The central goal of this report is to evaluate the effectiveness of STC programs in improving labor market outcomes of individuals as they begin to develop their careers and move on to work, further education, or a combination of the two. Its particular focus is an econometric analysis of the NLSY97, estimating the effects of STC program _____________ 1In job shadowing programs, a student follows an employee for one or more days to learn about a particular occupation or industry. vi participation, nationally, on education and employment in the period immediately after leaving high school. Although enrollment and employment in the immediate postsecondary years capture only a segment of an individual’s career development, these are the two activities that are most likely to increase the skills acquired by youths making the transition from school to work. This research thus focuses attention on the transitional period that has been the concern underlying STC legislation, as well as most research on STC. The NLSY97 covers six types of STC programs that comprise many of the most common types of STC programs and activities offered in California by the local STC providers funded by STWOA. Thus, although this research does not directly address specific, local STC programs and activities offered in the state, it speaks to the effectiveness of these types of programs. The NSLY97 is invaluable for research on the effectiveness of STC programs because its richness provides a number of methods for handling a fundamental problem in the evaluation of the effectiveness of a government program. Specifically, accurate estimation of such program effects—or recovering the causal effects—requires a thorough accounting of differences between STC participants and nonparticipants that exist independently of program participation. Such estimation is a primary focus of this research, and at the same time the lack of such estimation is a major shortcoming of most of the existing research on STC programs in California and elsewhere. In addition to presenting this new evidence from the NLSY97, the report reviews past evidence on the effectiveness of STC and includes a detailed analysis of evaluations that have been conducted of specific STC programs and activities in California that were supported by STWOA. Unfortunately, these evaluations provide little if any convincing evidence of the effectiveness of STC programs, especially as regards postsecondary outcomes. Many of the evaluations fail even to use a control group, which is a prerequisite for evaluation. Most fail to use any control variables or other methods to account for differences between participants and nonparticipants, and it is probably safe to say that none does this adequately. Finally, few even look at postsecondary outcomes. The dearth of compelling evidence and the data limitations that vii characterize these evaluations of specific STC programs in California imply that—to date—the NLSY97 data are the best source of information on the effectiveness of the types of STC programs that STWOA spurred in California. Broader Policy Context In some respects, STC has been ignored in recent research and policy debates regarding educational reform. In particular, recent research on educational quality and reform has focused largely on testrelated outcomes, and this focus is strongly reflected in the No Child Left Behind Act of 2001 (NCLB), which emphasizes standardized testing in grades K–12. But another important perspective on educational quality concerns the link between education and labor market success. Before the introduction of NCLB, STC programs had become an integral part of high school education in many states, spurred in part by STWOA. Although there is a link between test scores and socioeconomic success, there is no reason to believe that test-based standards encompass all of what schools do to prepare students for the world of work. It is plausible and indeed likely that STC also has an important role to play. In one fashion or another, virtually all students move into the workplace. Therefore, for all students, the provision of information about careers, tools to make decisions about careers, and the acquisition of specific skills valued in the labor market would seem to be an important complement to the academic component of education. Moreover, many students do not attend four-year postsecondary institutions, and it is conceivable that the strong emphasis on testing may not serve these students as well as those bound for such institutions. None of this is to argue that STC is more important than a rigorous academic focus. But research that addresses the potential benefits of STC can help maintain a focus on the problems of the noncollege-bound and more generally on the link between education and labor market success. Indeed, some researchers see NCLB as likely to diminish government interest and investment in STC. Without prejudging the outcome, suffice it to say that there is good reason to keep STC “on the table” in the context of broader issues of educational reform. viii More generally, STC is one potential policy response to poverty and low-wage work. The principal goal of STC is to increase labor market skills, in part via direct acquisition of skills from participating in STC programs, and to a greater extent via longer-run effects associated with launching youths onto career paths with greater potential for growth of skills. A great deal of research and policy effort focuses on problems and policies related to poverty, including changes in the income or wage distribution, social programs to mitigate the consequences of poverty or attack its sources, and mandated higher wages for low-skill workers. Most policymakers and researchers would agree, however, that the ideal solution—albeit perhaps the most difficult to achieve—is to increase the skills and productivity of those at the bottom of the socioeconomic scale so that they can earn a market wage high enough to lift them and their families out of poverty. STC programs may be one way to help achieve that goal. Thus, research on STC has a critical role to play in assessing these efforts to raise skill levels.2 Principal Findings The evidence from the NLSY97 indicates that some STC programs (school enterprises) boost post-high school education,3 and other STC programs (cooperative education and internships/apprenticeships) boost post-high school employment.4 Tech Prep appears to reduce post-high school education, although it possibly may also increase the likelihood of full-time work. The magnitudes implied by the estimates are reasonable yet also sizable, suggesting that participation in the programs with positive effects boosts enrollment or employment by about 5 to 10 percentage points, relative to the baseline rates of about 50 percent attending college and 60 percent employed in the immediate postsecondary period. Moreover, in each case of a positive effect, there is _____________ 2Donahoe and Tienda (1999) see the problem of moving workers into higher-wage jobs as the principal school-to-work “problem.” 3“School enterprises” entail the production of goods or services by students for sale to or use by others. They typically involve students in the management of the project and may be undertaken on or off the school site. 4In “cooperative education” programs, students alternate or parallel their academic and vocational studies with a job in a related field. ix no offsetting negative effect on the other dependent variable, implying that these programs appear to induce higher enrollment or employment at the expense of being neither enrolled nor employed, rather than at the expense of an alternative “productive” activity. Finally, there is also some evidence of differences in the effects of STC programs across groups distinguished by race and ethnicity and other characteristics associated with socioeconomic status and across men and women. However, this evidence does not point to a consistent pattern indicating that STC is particularly or primarily beneficial to disadvantaged students. Instead, there is some evidence of beneficial effects for all groups, although different programs deliver different benefits. One finding that perhaps does stand out, though, is that internship/apprenticeship programs may be particularly advantageous for the less-advantaged, as these programs boost college enrollment among those with the lowest test scores and boost employment among blacks and those with less-educated mothers and living in nontraditional arrangements. Recommendations On balance, relying principally on the results from the NLSY97, the evidence provides some indication that the types of STC programs funded by STWOA in California increase postsecondary enrollment and employment. However, with the demise of STWOA, funding has continued for Career Academies and Tech Prep, whereas direct funding for general types of STC programs (such as apprenticeships, internships, co-op programs, and school enterprises) has been sharply reduced.5 Yet the evidence on Tech Prep and Career Academies does not make a strong case that these branches of STC are more effective, at best suggesting that they may boost full-time work but possibly at the expense of reducing education.6 This evidence suggests that a reconsideration of the _____________ 5As explained in Chapter 2, because there was some commingling of funds between STWOA activities and other STC or vocational educational activities, it is inaccurate to say that the demise of STWOA removed all funding for general STC activities. 6As explained below, this is despite the fact that Tech Prep entails coordination between high school and college courses. x allocation of existing funding for STC activities in California may be warranted. The evidence is not overwhelming, and this report does not present a full-blown cost-benefit analysis; but it appears that a case can be made for restoring some funding to the types of programs supported by STWOA and, barring that, possibly considering some reallocation of funds toward these types of programs. Another important observation that emerges from the research is that there is a pressing need for serious evaluation of the effectiveness of the specific STC activities remaining in California and for any new activities that might arise in the future. The main evidence provided in this report comes from a national dataset that is informative about the types of STC activities in California that were spurred by STWOA. But in a subsidiary part of this research project, a survey of local providers of these activities and of evaluations of their efforts was conducted to compile and assess existing evidence on the effectiveness of STWOArelated STC activities in California. This survey revealed that specific evaluations of STC programs in California are lacking in many respects. Thus, any future funding increases for STC should be predicated on requiring evaluation of programs. Moreover, given the evidence of beneficial effects of general STC programs nationally, and the more ambiguous evidence regarding Tech Prep and Career Academies, the optimal strategy may be to restore some funding for these general programs (or to reallocate funding) and to mandate evaluations of all three types of programs. Only this type of effort holds the promise of more reliable conclusions to be drawn regarding the effectiveness of the different types of STC efforts implemented in the state. However, simply mandating evaluation is not enough. The survey of local providers revealed not only inadequate evaluations but also wide variation in local providers’ self-reported criteria for success and data collection efforts. These findings suggest that future STC efforts in California would be well served by better articulating the goals of the program and establishing clear criteria for measuring and assessing progress toward these goals. More important, the weaknesses of the existing local evaluations of STC efforts in California strongly indicate that evaluation of STC effectiveness in the state would be well served by xi substantive expert assistance in designing data collection efforts and in conducting the evaluations. A plea for “more research and more evaluation” may come across as a not very constructive policy recommendation—one that could be applied to virtually any government program. Although better evaluation is a widespread need in policy research, this report provides a compelling case that the need is particularly strong and potentially promising with respect to STC in California. First, because of legislative developments, funding of one of the three branches of STC in California—general and broad-based STC programs and activities—has been largely eliminated, with little basis for believing that the two branches of STC that continue to be funded—Tech Prep and Career Academies—are more effective. Second, the research base from which to assess the specific STC efforts that STWOA spurred in California is very weak. And third, the research this report presents using the NLSY97, as well as some of the most recent research on STC from other researchers, suggests that more compelling methods of studying STC programs are feasible and can in some cases successfully detect programs that are effective in increasing postsecondary employment or education. xii Contents Foreword........................................................................................................................... iii Summary........................................................................................................................... v Figures ................................................................................................................................ xv Tables.................................................................................................................................. xvii Acknowledgments......................................................................................................... xix 1. INTRODUCTION........................................................................................... Goals of the Research......................................................................................... Limitations .............................................................................................................. Broader Policy Context..................................................................................... Outline of the Report........................................................................................ 1 4 5 6 7 2. SCHOOL-TO-CAREER IN CALIFORNIA...................................... 9 Career Academies (Partnership Academies) ........................................... 12 Tech Prep................................................................................................................. 13 School-to-Work Opportunities Act of 1994.......................................... 14 Legislation and Organization.................................................................... 14 Local Partnerships........................................................................................... 15 Post-STWOA State Legislation.................................................................... 18 Funding..................................................................................................................... 19 Summary.................................................................................................................. 20 3. EXISTING RESEARCH ON THE EFFECTIVENESS OF STC IN CALIFORNIA AND NATIONALLY.................................. 21 Statewide Evaluation of STC in California ............................................ 22 Local Evaluations of General STC in California ................................. 23 Tech Prep and Career Academies in California.................................... 28 STC Research for the United States........................................................... 29 Summary.................................................................................................................. 30 4. EVIDENCE ON THE EFFECTS OF SCHOOL-TOCAREER PROGRAMS ON POSTSECONDARY ENROLLMENT AND EMPLOYMENT ............................................ 33 Empirical Approach............................................................................................ 34 Dataset, Variables, and Analysis Sample.................................................. 37 xiii Overview of the NLSY97............................................................................ STC Information in the NLSY97 .......................................................... Analysis Samples.............................................................................................. Results........................................................................................................................ STC Participation........................................................................................... Basic Analysis.................................................................................................... Summary of Findings from Basic Analysis......................................... Work and Schooling Expectations......................................................... School Fixed Effects....................................................................................... Provisional Summary and Interpretation............................................ Heterogeneity in the Effects of STC..................................................... Summary.................................................................................................................. 37 37 39 40 40 43 46 47 50 52 54 61 5. CONCLUSIONS ............................................................................................... 67 Evidence from the NLSY97 ........................................................................... 69 Recommendations............................................................................................... 71 Appendix A. National Research on the Effectiveness of STC................................... B. Empirical Methods for the NLSY97 Analysis ....................................... C. Detailed Tables for NLYS97 Analysis....................................................... 77 83 93 Bibliography.................................................................................................................... 103 About the Author.......................................................................................................... 111 Related PPIC Publications....................................................................................... 113 xiv Figures 2.1. 3.1. 4.1a. 4.1b. 4.1c. 4.2a. 4.2b. 4.3a. 4.3b. 4.4a. 4.4b. 4.5a. 4.5b. 4.5c. 4.5d. STC Funding in California ................................................................... Summary of Evidence on the Effectiveness of STC on Postsecondary Enrollment and Employment ............................... STC Participation Probabilities, Differences by Demographic Group................................................................................. STC Participation Probabilities, Differences by Family Structure.......................................................................................................... STC Participation Probabilities, Differences Associated with One Standard Deviation Higher ASVAB Scores............. College Enrollment Probabilities, Effects of STC Participation .................................................................................................. Employment Probabilities, Effects of STC Participation....... College Enrollment Probabilities, Effects of STC Participation, Controlling for Prior Work and Schooling Expectations................................................................................................... Employment Probabilities, Effects of STC Participation, Controlling for Prior Work and Schooling Expectations....... College Enrollment Probabilities, Effects of STC Participation, with School Fixed Effects ......................................... Employment Probabilities, Effects of STC Participation, with School Fixed Effects........................................................................ College Enrollment Probabilities, Effects of STC Participation, Differences by Race/Ethnicity................................ Employment Probabilities, Effects of STC Participation, Differences by Race/Ethnicity.............................................................. College Enrollment Probabilities, Effects of STC Participation, Differences by ASVAB Math Knowledge Scores ................................................................................................................ Employment Probabilities, Effects of STC Participation, Differences by ASVAB Math Knowledge Scores........................ 19 31 41 42 42 43 44 48 48 50 51 56 56 58 58 xv 4.5e. 4.5f. 4.5g. 4.5h. 4.5i. 4.5j. 4.6. College Enrollment Probabilities, Effects of STC Participation, Differences by Biological Mother’s Education........................................................................................................ Employment Probabilities, Effects of STC Participation, Differences by Biological Mother’s Education............................. College Enrollment Probabilities, Effects of STC Participation, Differences by Living Arrangement..................... Employment Probabilities, Effects of STC Participation, Differences by Living Arrangement................................................... College Enrollment Probabilities, Effects of STC Participation, Differences by Sex........................................................ Employment Probabilities, Effects of STC Participation, Differences by Sex ...................................................................................... Summary of Evidence on Effectiveness of STC on Postsecondary Enrollment and Employment ............................... 59 59 60 60 62 62 64 xvi Tables 2.1. School-to-Career Programs and Funding in California .......... 11 2.2. STC Activities Reported by Local Partnerships .......................... 16 3.1. Local Partnership Evaluations .............................................................. 24 C.1. Sample Construction and Analysis Samples.................................. 93 C.2. Participation Shares and Linear Probability Estimates of Individual Characteristics Associated with STC Participation.................................................................................................. 94 C.3a. Linear Probability Estimates of the Effects of STC Participation on College Attendance and Employment.......... 96 C.3b. Linear Probability Estimates of Control Variable Coefficients for Models in Table C.3a............................................. 97 C.4. Linear Probability Estimates of the Effects of STC Participation on College Attendance and Employment, Incorporating Prior Work and Schooling Expectations.......... 99 C.5. School Fixed Effects Linear Probability Estimates of the Effects of STC Participation on College Attendance and Employment.................................................................................................. 100 C.6. School Fixed Effects Linear Probability Estimates of the Effects of STC Participation on College Attendance and Employment, Specifications Interacting STC Participation with Respondent Characteristics............................ 101 xvii Acknowledgments The material in Chapter 4 of this report is the result of collaborative research with Donna Rothstein, a research economist at the Bureau of Labor Statistics, U.S. Department of Labor. I am grateful to Anne Danenberg and Douglas Symes for research assistance and to Gary Bjork, Jon Haveman, Daiji Kawaguchi, Nan Maxwell, Margaret O’BrienStrain, Joyce Peterson, Deborah Reed, Jon Sonstelie, Michael Teitz, Maureen Waller, and seminar participants at the Public Policy Institute of California and Stanford University for helpful comments. The views expressed are those of the author and do not reflect the views or policies of the Bureau of Labor Statistics or its staff members or of the Public Policy Institute of California. xix 1. Introduction The 1994 federal School-to-Work Opportunities Act (STWOA) allocated more than $1.5 billion to support increased career preparation activities in the nation’s public schools.1 STWOA was a response to the absence in the United States of a “comprehensive and coherent system to help its youths acquire the knowledge, skills, abilities, and information about and access to the labor market necessary to make an effective transition from school to career-oriented work or to further education and training” (H. R. 2884, 103rd Congress, “School-to-Work Opportunities Act of 1994”). STWOA funds were provided to states to encourage and support the creation of school-to-work or school-to-career (STC) systems, entailing cooperation among schools, private business, and government bodies (Office of Technology Assessment, 1995).2 In particular, STWOA funds were aimed at general or broad-based STC programs and activities that served all students, helping guide them into careers that not only could be entered immediately after high school or with additional vocational or technical education but also that entailed higher education at four-year institutions. STWOA was intended to address three areas of particular concern. These included (1) a lack of connection between school and work that led many youths to be unmotivated in school and to experience subsequent difficulty moving out of low-wage jobs, (2) youths _____________ 1This amount was originally appropriated for fiscal years 1994 to 1998 for grants to states and local partnerships under STWOA (Hershey et al., 1999). 2Although “school-to-work” is the more common appellation, many states have adopted the label “school-to-career.” Given that most of the effort is devoted to high school students and that the goal is to improve the career decisionmaking of youths—which may often entail further education—the school-to-career label provides a better characterization. Indeed, in the past, school-to-work programs (especially Tech Prep) have sometimes suffered from the stigma associated with placing high school students onto vocational tracks rather than directing them toward further education (Neumark and Allen, 2003). Thus, in this report the “school-to-career” label is used. 1 completing school with insufficient skills needed for the labor market, and (3) increasing labor market demands for complex thinking, close teamwork, and the ability to learn on the job. The goal of STWOA was to help young people develop the skills needed in the workforce and establish better connections between education and subsequent careers through STC transition systems. In particular, three core goals of STWOA were to increase (1) school-based initiatives such as career awareness activities and career links to academic curriculum; (2) workbased activities such as job shadowing, internships, and apprenticeships; and (3) connecting activities, such as the development of partnerships between high schools and employers and postsecondary institutions. See Box 1.1 for a summary of the act. Under STWOA, beginning in 1996 California received $130 million over five years from the National School-to-Work Office to establish STC providers and programs, plus an additional $7.2 million supplemental grant in 2000. STWOA funds were used in California to establish an extensive statewide system of Local Partnerships (LPs) that were involved in multiple aspects of STC but which clearly played an important role in implementing the types of general or broad-based STC activities and programs encouraged by STWOA. However, the U.S. Congress did not reauthorize STWOA after its initial five-year run. As federal dollars to support STC activities wound down, the California legislature approved legislation (AB 1873) in 2000 including $5 million in state funding, which was later cut to $2 million by Governor Gray Davis. Currently, the continuation of this funding (at even lower levels) is the only direct state funding that specifically replaces funds from STWOA. The STC activities funded under STWOA can be viewed as one of three branches of STC activities in California. The other two branches are Career Academies (sometimes called Partnership Academies) and Tech Prep, which focus more narrowly on specific student populations, in contrast to the general and broad-based approach of STWOA. Although funding from other sources (both state and federal) has continued for Tech Prep and Career Academies, the loss of STWOA funds represents more than a one-third decline in funding for STC activities overall in California and will likely over time severely curtail or 2 3 even eliminate broad-based STC programs and activities. Given that funding for STC programs—if it is to continue at anywhere near the levels under STWOA—will have to come from the state, this is an ideal time to study the effectiveness of STC in achieving its goals. Goals of the Research This report focuses on the central goal of STC policy—to improve the labor market outcomes of individuals as they begin to develop their careers and move on to work, further education, or a combination of the two. In particular, the main contribution of this report is an econometric analysis of a large-scale dataset, the 1997 National Longitudinal Survey of Youth (NLSY97), estimating the effects of STC program participation, nationally, on student enrollment in postsecondary education or employment in the period immediately following high school. The NLSY97 covers six types of STC programs: job shadowing, mentoring, co-op programs, school enterprises, Tech Prep, and internships/apprenticeships. These are by no means all of the programs and activities pursued by STC providers in California using grants under STWOA, but they do represent many of the most common programs and activities pursued. This complication arises because STWOA, and its implementation in California, did not restrict STC efforts to a specific set of programs or activities but instead created a network of providers of STC programs and activities in pursuit of the specific goals of the legislation. Thus, this research does not evaluate STWOA per se but examines instead a wide set of STC programs that characterize—albeit imperfectly—the types of general and broad-based STC programs that STWOA encouraged. An important strength of the NLSY97 for this research is that it offers many ways to estimate the causal effects of participation, accounting for nonrandom selection of participants into STC programs that could lead to inaccurate estimates of the programs’ effectiveness. Estimation of the causal effects of STC programs, by accounting for differences between STC participants and nonparticipants, is a major focus of this report and at the same time a major shortcoming of most of the existing research on the topic, whether for California or elsewhere. 4 Limitations The NLSY97 dataset poses some limitations. First, it is not sufficiently large to carry out analyses specific to California. But evidence on the effects of STC participation gained from data on all states should be informative given that effects most likely generalize across states. On the other hand, policymakers in California may be most interested in assessments of the effectiveness of the specific STC programs that were established in various localities throughout the state under STWOA. Although the NLSY97 data cannot provide information specific to California, this report also presents a summary of findings from a compendium of evaluations of specific local STC programs supported by STWOA in the state. The second limitation is that the overall goal of STC policy—to improve labor market outcomes of individuals as they begin to develop their careers—is considerably more broad than simply increased postsecondary employment or enrollment and should be evaluated by labor market success defined over the longer run. However, because the NLSY97 at this point covers individuals only in their first few years after leaving high school, the analysis in this report adopts a similarly narrow focus, examining the effectiveness of STC programs in boosting immediate postsecondary enrollment and employment; this examination is quite important, because it addresses the two activities most likely to increase the skills acquired by youths making the transition from school to work. Indeed, much of the existing research on STC focuses on the years immediately after high school as the key transitional period that has been the chief concern underlying STC legislation.3 _____________ 3The early writing on school-to-work transitions in the United States that advocated the development of a school-to-work system along the lines of STWOA focused on the general problem of “churning” or “milling about” that characterized youth labor market experiences in this country, entailing initial periods of joblessness or a series of dead-end jobs (see U.S. General Accounting Office, 1990; Commission on the Skills of the American Workforce, 1990; Hamilton, 1990; Lerman and Pouncy, 1990; Glazer, 1993; and other work reviewed in Heckman, 1993.) Much of the newer research on the effects of STC programs has focused explicitly on labor market and educational outcomes in the immediate postsecondary period. 5 Broader Policy Context Research on the effectiveness of STC also fits into broader policy issues. First, research on educational quality and reform has emphasized test-related outcomes, which can perhaps be viewed as culminating in the No Child Left Behind Act of 2001 (NCLB), emphasizing standardized testing in grades K–12. Although test score achievement undoubtedly reflects some dimensions of educational quality, another important dimension is education’s contribution to labor market success. In part because of STWOA, STC programs had become well integrated into high school education in many states before the enactment of NCLB and the failure to reauthorize STWOA.4 Indeed, some researchers see NCLB as likely to diminish government interest and investment in STC.5 And although test scores likely have a positive effect on socioeconomic success, it is highly unlikely that test-based standards encompass the entirety of schools’ contributions to labor market success of students. STC may very well also have an important role to play. Nearly all students make a transition from school to the labor market, and many spend a substantial fraction of their lives working. Thus, integrating into academic curricula information about careers, tools for increasing capacity and knowledge to make better career decisions, and the provision of specific skills could be a productive complement to an academic focus. STC may be particularly important for the many students who will not attend four-year postsecondary institutions, whereas a strong emphasis on testing may be more suited to the needs of those who will attend such institutions. The argument is not that STC is more important than quality academic education but that it may offer the potential to help those less likely to attend college and, more generally, may strengthen the contribution of education to labor market success. _____________ 4Nationally, among high school students in the NLSY97 sample used in this report, participation in the different individual types of STC programs covered ranges from about 7 to 18 percent, and 42 percent of high school students participate in at least one type of program. 5See, for example, Maher (2003). 6 Second, policymakers and researchers devote a great deal of time and effort to problems and policies related to poverty, focusing on changes in inequality, social programs to try to reduce poverty or cushion its blow, and setting wage floors for low-skill workers. However, most policymakers and researchers recognize that—although difficult to achieve—the best long-term solution is to raise the skills and productivity of the most economically disadvantaged, enabling them to earn enough to raise their families out of poverty. STC programs may be one way to increase skills and productivity, especially among those who would be less likely to obtain a college education. Finally, although not directly examined in this report, it is conceivable that the benefits of STC programs are extensive, if effective programs improve the efficiency with which youths “transfer” the human capital acquired in school into productive activity in the labor market. Outline of the Report Following this introductory chapter defining the issues, explaining the goals of the research, and discussing the policy context, Chapter 2 provides an overview of STC legislation and programs in California and of federal legislation. Chapter 3 surveys prior research on the effectiveness of STC in California and elsewhere. Chapter 4 explains and discusses the econometric analysis of the 1997 National Longitudinal Survey of Youth, and Chapter 5 offers a summary of the findings and policy recommendations. 7 2. School-to-Career in California This chapter provides an overview of state and federal legislation that funds STC in California, or did so in the recent past, and of STC efforts in the state. Describing STC efforts in California is complicated because of overlapping programs and a profusion of names. One useful distinction is between vocational education and STC (or school-to-work). Although the distinction is not always sharp in practice, and not all researchers make it, the two have some conceptual and programmatic differences and the distinction is useful for delineating what is and is not studied in this report. In particular, in the NLSY97 data, STC activities do not include vocational or technical education.1 Vocational education has been supported by the federal government since the Hughes Act of 1917. At that time, and for decades afterward, it was distinguished by being isolated from more comprehensive high school curricula (Hayward and Benson, 1993), sometimes because of the perspective that vocational education programs served students “better suited to applied than academic learning” (Donahoe and Tienda, 1999). School-to-work, in contrast, refers to efforts that integrate academic and vocational skills. A major impetus in this direction was the 1990 reauthorization of the Carl D. Perkins Vocational and Applied Technology Education Act, which sought this integration in a number of ways.2 One significant vehicle for this integration was Tech Prep, which was designed to introduce some vocational education into comprehensive high school curricula while integrating these programs with postsecondary institutions. A second was the development of Career Academies, “schools within schools” that focus on an _____________ 1Instead, some schools are classified as vocational/technical, but these are excluded from the analysis. 2See Urquiola et al. (1997) for more details and an outstanding review of the history of school-to-work. 9 occupational area and integrate academic education with technical preparation. The phrase “school-to-work” came into popular use with the federal School-to-Work Opportunities Act of 1994. The STC programs envisioned under STWOA were distinguished from earlier efforts because they sought a greater integration of vocational and academic education by calling for school-to-work programs for all students. In particular, STWOA was less explicitly geared toward “underserved populations” such as minorities, women, and the economically disadvantaged, although it noted some of the particular problems of disadvantaged youths in achieving a successful school-to-work transition. It was also less explicitly restricted to pathways that lead through community or technical colleges and, instead, sought to expand STC for all students and to include paths that lead to four-year colleges and universities (Urquiola et al., 1997, pp. 23–24). In responding to STWOA, California adopted the label “school-tocareer.” However, in current use, STC is used to describe an array of programs and efforts in California, not only those funded under STWOA. Similarly, the STC label is used in a general fashion throughout this report, and the activities and programs supported by STWOA are referred to as “general” or “broad-based” STC activities or programs, in contrast to the more narrowly defined Tech Prep and Career Academy programs. Finally, as described below, there is some overlap in the current administration and funding of the different components of STC. Table 2.1 shows the legislation and resulting programs that are part of the overall STC system in California. Aside from STWOA and follow-on state efforts, as mentioned above, the other two branches of STC are Tech Prep and Career or Partnership Academies. Tech Prep is funded federally through the Perkins Act, whereas Career Academies are funded by the state’s general fund. One major component not depicted in Table 2.1 is Regional Occupational Centers and Programs (ROP or ROCP), which originated in California in 1967 and are codified through several sections of the California Education Code (primarily 10 Table 2.1 School-to-Career Programs and Funding in California Program School-to-Work Opportunities Act Program Description School-based learning, work-based learning, and connecting activities Year Established 1994 Ongoing Program No School-to-Career School-based learning, work-based learning, and connecting activities 2000 (AB 1873) Ambiguous Tech Prep Educational programs that prepare individuals for employment in occupations that do not require a baccalaureate or advanced degree 1990 (reauthorized in the 1998 Perkins Act) Yes Career (Partnership) Academies Three-year program for grades 10–12 that is structured as a school-within-a-school and incorporates rigorous integrated academics with a career focus, business partnerships, and teachers who work as a team in preparing students for careers and postsecondary education 1985 (AB 3104), codified in 1993 (California Education Code, §§ 54690–54697) Yes California’s Appropriation Amount and Source $137 million over the life of the program; federal funding from STWOA $2 million from state funding (AB 1873) $11.8 million in FY 2002–03; federal funding from Carl Perkins Act $23 million in FY 2002–03; state funding from general fund §§ 52300–52336.5).3 Although there is some intermingling of STC activities and ROCPs, especially with respect to internships, ROCP is best viewed as a vocational education program. Vocational education funding swamps the STC programs listed in Table 2.1, with state spending of $373 million on ROCP in 2002–03 (based on Average Daily Attendance) and federal spending of $120 million for vocational and technical education from the Perkins Act. Career Academies (Partnership Academies) A Career Academy is a high school program in which the curriculum organizes academic education around an industrial or occupational area such as health, finance, computers, or media. In Career Academies, a group of students typically stays together with the same set of teachers for many years, and a relatively small number of students (such as 50) are enrolled in each grade. Career Academies are intended to enable students to fulfill requirements for college entrance and to acquire workrelated knowledge and skills. Academy teachers try to establish relationships with employers, who provide support through adult mentors and internships for students, for example (Career Academy Support Network, 2003a). Maxwell and Rubin (2001) describe the model as a “school-within-a-school.” Career Academies began in Philadelphia in 1969, and the model was established in California in the 1980s (Career Academy Support Network, 2003a, 2003b). California began to fund planning and implementation grants for academies beginning with AB 3104 in 1985. Additional funding for the Career Academy model was legislated in SB 605 in 1988, and the California Education Code (§§ 54690–54697) _____________ 3ROCPs are available to high school students 16 years of age and older as well as to adult students. Three main missions of ROCPs are to give students opportunities to “(1) enter the workforce with skills and competencies to be successful; (2) pursue advanced training in higher educational institutions; or (3) upgrade existing skills and knowledge” (California Department of Education, 2003a). Over 302,000 high school students (17.1 percent) and over 155,000 adults were enrolled in courses in the state’s 73 ROCPs in 2001–02 (California Association of Regional Occupational Centers and Programs, 2003). 12 included Partnership Academies beginning in 1993 (Maxwell and Rubin, 2001; California Education Code, n.d.). In the 1,005 high schools listed in California’s 2002–03 report from the California Basic Education Data System (CBEDS), there were 290 state-funded Career Academy programs that received a total of $23 million from the state’s general fund (California Department of Education, 2002). Tech Prep Tech Prep in California is a program of technical preparation in areas such as applied sciences; mechanical, industrial, or practical arts or trades; health occupations; or business (California Department of Education, 2003b). Tech Prep programs combine two or more years of high school education with two years of postsecondary education in sequential coursework, integrating academic with vocational and technical education, to strengthen the applied academic component of vocational and technical education. Tech Prep is intended to lead to an associate degree or a certificate in a specific career field, in turn resulting in higher-wage employment and perhaps further education. Federal support for Tech Prep began with the 1990 Carl D. Perkins Vocational and Applied Technology Education Act, Title II, Part E. The initial Tech Prep funding was for articulation of programs integrating two years of high school and two years of college (“2+2” programs).4 The 1990 Perkins Act was extended through 1998 (Perkins II), although funding for some programs such as consumer and homemaking education and student guidance services was dropped. Reauthorization of the Perkins Act of 1998 (Perkins III) eliminated all programs except Tech Prep from Title II. However, the state’s grant was broadened to include business partnerships and the use of funds for work-based learning as well as the original provisions for articulation. _____________ 4“Articulation” in this context means that courses taken at one school level fulfill requirements at another level. For example, an advanced placement class taken in high school “articulates” to the same course when taken at a university, or a lower division college class when taken at community college is considered the same as if taken at a university as long as an articulation agreement exists between any two school levels. See U.S. Department of Education (2002a) for a description of current Tech Prep sequences of study. 13 Since 1990, California’s share of Tech Prep funding has been close to $12 million annually. In the 2002–03 state budget cycle, $11.8 million was allocated to Tech Prep from $138.4 million in federal funds that were targeted at vocational and technical education programs in California (California Department of Education, 2002). School-to-Work Opportunities Act of 1994 Legislation and Organization Under STWOA, states were given considerable discretion over how to structure and run their STC systems. After STWOA was signed into law in 1994, Governor Pete Wilson issued an executive order (Wilson, 1994) forming a state school-to-career taskforce charged with creating a state STC plan that would lay the groundwork to apply for federal funding. This taskforce became the 27-member STC advisory council that provided recommendations for developing a statewide STC system. It was composed of representatives from education, business, labor, and the legislature, and supported by an Interagency Partnership (IAP) consisting of the Employment Development Department (EDD), the California Department of Education (CDE), and the California Community Colleges (CCC) Chancellor’s Office. The state was divided into 12 STC regions, each containing numerous LPs. Larger counties had more than one partnership and less-populated areas had multicounty partnerships. Each LP was required to include employers, representatives from local educational agencies (elementary, middle, and secondary schools), representatives of postsecondary institutions, local educators, and representatives of organized labor, students, and parent organizations. California announced the receipt of its first federal school-to-career grant of $21.9 million on September 19, 1996 (California Employment Development Department, n.d.). According to the announcement, a total of $131.4 million in federal funds would be allocated to California over a five-year period, which would be used to provide incentives for stakeholders to form local partnerships and implement STC-based efforts in conjunction with other education and workforce initiatives. Between 14 1997 and 2000, federal funds were distributed among the partnerships and state-level administrative units.5 Local Partnerships To get a better sense of the STC activities generated by STWOA in California, a survey was conducted entailing extensive interviews of representatives of LPs in the state.6 These representatives were interviewed about the activities of their STC programs, among other things. The interviews took place from December 2002 through April 2003 and covered 38 of the 57 LPs in the state. The interviews revealed considerable complexity in the administration of STC activities. Each LP interviewed, of course, had direct responsibility for STWOA activities. In addition, many of these LPs were also involved in other STC activities. For example, some of the LPs were housed in ROCPs. Many were in Local Education Agencies (typically a district or county office of education). And some were involved in Tech Prep or moved into it as STWOA wound down. As further evidence of this commingling of functions, many LPs indicated that before STWOA they had some involvement with vocational education through Perkins money, ROCPs, and so on, as well as Tech Prep. Table 2.2 presents the simple percentages of LPs reporting different types of STC programs.7 The table shows that LPs were generally _____________ 5The first payment was $21.9 million, 70 percent of which went to LPs and 30 percent to the state. The second payment was $43.8 million, of which 80 percent went to LPs and 20 percent to the state. Subsequent payments were $32.9 million, $21.9 million, and $10.9 million, of which 90 percent went to LPs and 10 percent to the state. In 2000, a “supplemental transition” grant of $7.2 million was awarded in addition to the earlier federal grants (personal communication, Ed Armijo, California Employment Development Department, October 2002 and May 2003). 6See Neumark (2004) for full details regarding the survey and the findings. 7The upper panel lists the STC programs that were explicitly included in the interviews, and the bottom panel lists any other types of programs that were mentioned in response to general questions about STC services, or elsewhere during the interview. It is important to note that because not all LPs participated in the interviews, the programs summarized here cannot necessarily be generalized to all STWOA partnerships in the state. 15 Table 2.2 STC Activities Reported by Local Partnerships Percentage Survey responses Job shadowing Career fairs or outside speakers Career assessment or career inventory Career academy or career pathway Internships/apprenticeships Tech Prep Mentoring Certificates Other responses from narratives Teacher/staff training and professional development ROCPa Service learning Work-based/project-based learning Junior achievement Field trips Vocational education Small learning communities Industry-based training Employer interviewing service Junior chamber of commerce Career workshops 95 89 84 82 82 79 32 53 45 26 24 24 5 5 3 3 3 3 3 3 aIncludes cases where partnerships indicated internships/ apprenticeships or career pathways through ROCP, or skill certificates through ROCP. involved in a wide range of STC activities, with all or nearly all including career assessment/inventories, career fairs or outside speakers, Career Academies or Pathways, Tech Prep, job shadowing, and internships/ apprenticeships. These programs are in many ways the bread and butter of STC efforts and constitute many of the same programs that are covered in the NLSY97 data and discussed in Chapter 4. Many if not most of these programs can be viewed as serving one or more of the goals of increased school-based learning, work-based learning, and connecting activities. Apprenticeships and internships are perhaps the best examples of work-based learning. Job shadowing and mentoring are probably best thought of as connecting activities, because 16 they do not entail extensive work experience but give participants the opportunity to see the types of skills and education that jobs require. Career guidance and counseling also clearly fall under the category of connecting activities, to the extent that they help students plan their educations with regard to career goals. Co-op programs that often entail split time between work and schooling are a bit harder to characterize and probably contain all three components. Tech Prep and Career Academies, which were discussed above and do not fall directly under the rubric of STWOA, are prime examples of school-based learning, as they bring vocational and career education into the classroom.8 In addition, the survey revealed that funding from STWOA did not go solely to the creation of new STC programs serving high school students. Some appears to have been used to expand or supplement preexisting programs or to add K–8 activities to high school activities already in place. In other LPs, STC funding was used for staff development rather than to provide “hands-on” experiences to students (such as job shadowing and internships). Some partnerships focused on building and sustaining the partnership itself and using it as a resource or “clearinghouse” for local districts and businesses. Nonetheless, the interviews indicated that termination of STWOA funding did have adverse consequences for the provision of general STC programs for high school students, although in the immediate aftermath, LPs seem to have scaled back programs and tried to support them with previously existing funding sources. The array of STC activities in which LPs were involved is natural, because STWOA did not mandate specific types of programs. In part, as well, it probably reflected opportunism, as existing institutions with oversight for STC and vocational education also tapped into STWOA funds while they were available. As a result, there is really no meaningful sense in which STWOA can be viewed as a unified, well-defined program—or evaluated as one. Rather, we can only identify a set of programs supported by STWOA and study their effectiveness, which is the strategy used in this report. _____________ 8More detailed definitions of STC programs are provided in Chapter 4 in the context of those programs studied using the NLSY97. 17 Post-STWOA State Legislation California is one of many states that—after STWOA’s demise—introduced their own STC legislation with at least some supporting appropriation, although California’s appropriations for general, broad-based STC programs have been very modest. According to data compiled in Schmidt (2000), in the 2000–01 legislative year, 24 states considered legislation pertaining to STC programs, 21 passed such legislation, and in 16 cases state general funds were appropriated. In total, state general fund appropriations in that year amounted to $89 million.9 California’s first STC legislation, AB 1873, was introduced in March 2000 and appropriated $5 million in state funds to support LPs whose federal STWOA funding was ending (Schmidt, 2001). Governor Gray Davis signed the bill in September 2000 but with a reduction in funding to $2 million. Although the original legislation was a one-time appropriation (Schmidt, 2001), this amount was subsequently included in California’s annual budget; however, it did not receive its first funding until the 2001–02 budget. The 2002–03 budget appropriated another $2 million. However, in the 2003–04 final budget (AB 1765), the amount was reduced to $1.7 million, and future appropriations are subject to annual budgetary and political constraints. In addition to AB 1873 and its renewal through AB 1765, STC has surfaced on the legislative agenda in at least four other pieces of legislation (AB 1412 and SB 1934, passed in 2002; AB 1266, passed in 2003; and AB 639, under consideration in the 2003–04 session) but without any additional financial support.10 _____________ 9This figure includes California’s STC appropriation of $2 million described below, plus an appropriation of $19.7 million to Partnership Academies in the state. 10AB 1412 renamed vocational education as “Career Technical Education” in the state. SB 1934 encouraged schools to combine a rigorous academic curriculum integrating academic and career skills to prepare all students for high school graduation and career entry. But the legislation does not provide funding and restricts implementation to the use of federal funds via the Perkins legislation. AB 1266 shifted all administrative and fiscal oversight from the IAP to the CDE and made funding for the program contingent upon a budget act or other legislative appropriation. AB 639 would authorize an ROCP to establish a Career Academy. Given the relatively high level of annual funding that ROCP currently receives, the ability to use a portion of these funds 18 Funding Figure 2.1 illustrates the sources of major appropriations for STC activities in California from fiscal years 1998–99 through 2003–04. As the figure shows, after STWOA appropriations ended in 2000–01, California’s relatively low STC appropriation began in 2001–02.11 At the same time, appropriations for Partnership or Career Academies have Figure 2.1—STC Funding in California ______________________________________________________________ for Career Academies rather than for strictly vocational programs may signal a shift in the focus of ROCP away from vocational education and toward academic integration. (This information came from the Assembly Education Committee, the Senate Education Committee, and the Senate Office of Research during our attempt to ascertain whether the state planned to continue STC and whether there was any other evaluation research under way or contemplated.) 11STWOA funding was in the $20 million to $40 million range in the few years preceding those shown in the figure. 19 risen as more academies have been established, and Tech Prep funding has remained relatively constant during the six years. The significant drop in funding for general STC is very apparent in the figure. Summary STC activities in California have occurred and continue under the auspices of a number of programs, with funding from both federal and state sources. In addition to STWOA, STC efforts in California include Career Academies and Tech Prep. One important implication of this is that the failure to pick up responsibility for funding previously provided by STWOA does not mean that STC activities will cease to exist in California. As summarized in Figure 2.1, funding has continued or increased for Career Academies and Tech Prep. Furthermore, although the drop-off in STWOA-related funding represents a large share of the funding depicted in Figure 2.1, to the extent that STC activities can piggy-back onto other programs, these activities will continue. Nonetheless, it seems that, absent legislation restoring some of the funding previously provided under STWOA, general and broad-based STC activities are sure to diminish. The following chapters use a combination of original research and review of past research to provide a detailed picture of what we know about the effectiveness of the three components of STC in California in boosting postsecondary enrollment and employment, with the original research focusing on the general and broad-based STC programs supported by STWOA. The next chapter covers the existing evidence at the state level and the national level, including not only published and widely circulated studies but also information on local evaluation efforts garnered from the survey of local STC providers in California. Following that, Chapter 4 presents the main analysis in this report, using the NLSY97 data to study the effects of the types of STC programs that were supported in California (and elsewhere) by STWOA. 20 3. Existing Research on the Effectiveness of STC in California and Nationally The research presented in this report offers new evidence on the effectiveness of STC in California and the rest of the nation. But it focuses on the types of general and broad-based programs that were spurred by STWOA. Thus, it is useful to summarize briefly what other research establishes about the effectiveness of the other branches of STC in California, including studies specific to California and studies that cover the types of programs in effect in California. Combining existing research findings with the new findings reported in Chapter 4 provides the fullest possible picture of the effectiveness of California’s STC programs. Throughout, the focus is on the effects of STC programs on postsecondary enrollment and employment. In addition, both the review of existing evidence in this chapter and the new evidence in the next chapter emphasize the estimation of causal effects of STC programs. In evaluating the effects of STC programs (or most other government programs), a fundamental problem is determining whether there are preexisting differences between those who participate in the programs and those who do not—whether because of individual choices to participate or because of assignment to program participation. If account is not taken of these “pre-program differences,” then comparisons between participants and nonparticipants will typically incorrectly estimate the causal effect of the program. As an example, if high school students who are more career oriented choose to participate in STC programs, then participants may have been more likely to be employed after high school regardless of participation, and a comparison of participants to nonparticipants would overstate the extent to which STC participation 21 boosts postsecondary employment.1 In this chapter, then, we examine not only what the existing research says but how convincingly that research handles the problem of pre-program differences. Statewide Evaluation of STC in California In 2002, WestEd and MPR Associates, Inc. (“WestEd-MPR”) completed a statewide evaluation of the STC structure and activities established in California in response to STWOA (WestEd and MPR Associates, Inc., 2002). Most of this evaluation focuses on the implementation and sustainability of STC, rather than on the effects of STC on participants. However, a limited part of the WestEd-MPR report focuses on the effects of STC participation on postsecondary outcomes.2 Based on a survey of seniors and a follow-up survey on postsecondary experiences, the report concludes that there was little or no evidence of positive effects of STC participation on postsecondary enrollment or employment. Some of the analyses on which these conclusions are based incorporated a limited set of control variables for pre-program differences, including parent’s education, race, sex, and a measure of prior academic achievement.3 However, the limited set of controls seems unlikely to adequately account for pre-program differences between STC participants and nonparticipants. _____________ 1This discussion applies not only to STC but to estimating the causal effect of virtually any type of program participation. For a thorough treatment of the issues, see Heckman et al. (1999). This problem is often referred to as “endogenous selection” because it arises when individuals self-select into program participation, as in the example in the text. 2Part of the WestEd-MPR report also presents some findings on the association between STC participation and student attitudes toward work and schooling. However, from a policy perspective, this is considerably less important than the question of whether STC program participation changes observed behavior. 3The report provided no statistical results, but upon request they were provided to us by WestEd-MPR. The results summarized in the WestEd-MPR evaluation are a synthesis of findings from the individual LPs (personal communication, Robert Fitzgerald, September 2003). 22 Local Evaluations of General STC in California Another potential source of information on the effectiveness of STC programs in California is the local STC providers in the state. Because local organizations are at the forefront of STC efforts, and because STC programs are relatively new, much of the most recent or most convincing evidence may be in the hands of practitioners “on the ground.” Consequently, in the survey of LPs described in the previous chapter, respondents were also asked about formal evaluations of their local programs.4 In most cases, when an evaluation was eventually supplied, either it was in the form of a publicly available report or the LP representative gave us permission to cite an unpublished report; in others, only confidential internal evaluations were provided. In the latter cases, the promise of confidentiality given for the interviews precludes identifying the evaluated district. In the end, 17 evaluations were received. A number of the local evaluations had a common structure, because they were a by-product of the statewide STC evaluation by WestEd-MPR (2002). A subset of these evaluations used a common set of analyses; these are referred to as WestEd-MPR Core cases. The others conducted somewhat different analyses; these are referred to as WestEd-MPR Plus cases.5 The remaining evaluations were unrelated to the statewide evaluation. Table 3.1 summarizes the 17 evaluations received. Because there are multiple evaluations of the same LP, in total these evaluations cover 12 LPs. The first panel in Table 3.1 shows the WestEd-MPR Core cases.6 The last column reports any statistically significant findings related to _____________ 4The findings from the interviews are summarized here. A more detailed discussion is provided in Neumark (2004), which discusses how STC practitioners assess the success of their programs. Such an approach is useful in thinking about how assessments might be enhanced in the future. 5These evaluations used two individual-level surveys of STC participants and nonparticipants that were conducted in 2001—the Senior Survey and the Senior Followup Survey. At no time were these responses pooled and analyzed statewide, and none of the findings from these individual evaluations were covered in the WestEd-MPR report. 6Note that in the case of one LP only an executive summary was provided, and it was difficult to ascertain much from it. 23 Table 3.1 Local Partnership Evaluations 24 STC Partnership or School District Type of Program Executive summary, confidential ROP Sonoma (O’Driscoll et al., 2002a) “High-STC” Unite-LA (O’Driscoll et al., 2002b, 2003) Ventura (Christensen, 2002) “High-STC” “High-STC,” Career Pathways, internship students Verdugo (O’Driscoll et al., 2002c) “High-STC” Control Variables Control Group Used WestEd-MPR Core Case Studies Non-high-STC ? students Postsecondary Effects of STC No enrollment or employment differences Non-high-STC students No Lower enrollment, no employment difference Non-high-STC students No Higher enrollment, lower employment Converse of each group No “High-STC”: No enrollment or employment differences Career Pathways: Lower enrollment in fouryear college, higher enrollment in vocational/ technical schools, no employment difference Internships: Higher enrollment, no overall employment difference, higher full-time employment, higher receipt of benefits, and more likely to view job as “highly related” to high school education Non-high-STC students No No overall enrollment difference, higher enrollment in four-year college, lower employment Table 3.1 (continued) 25 STC Partnership or School District Orange County Coalition/ Vision 2020 (Forouzesh and Orange County, 2002) Verdugo (Butler et al., 2002a) Type of Program “High-STC” Magnet students Control Group Non-high-STC students Control Variables Used No Postsecondary Effects of STC No enrollment or employment differences WestEd-MPR Plus Case Studies Nonmagnet students Yes—demographic characteristics and prior academic achievement Not studied Unite-LA (Butler et al., 2002b) Career Academy students Partnerships Plus! (Kern County) Partnerships in youth (Fattahi, 2002) leadership workshop Nonacademy students Yes—demographic characteristics and prior academic achievement Independent Evaluations No No Not studied N/A Confidential Various programs No No N/A Confidential ? No No N/A Orange County Department of Education (Forouzesh, 2002) Participants in Career Academy, Pathway, Tech Prep, or other career-related activities Nonparticipants in career-related activities No Not studied Table 3.1 (continued) STC Partnership or School District Confidential Type of Program Career Academy students Control Group Comprehensive high school students Control Variables Used ? Postsecondary Effects of STC Possibly higher enrollment in two-year colleges, no differences in four-year enrollments (unclear, and no statistical tests) Confidential, summary STC participants No No N/A San Mateo STC Partnership (Lindler and Hill, 2000) Various programs No No N/A Partnerships Plus! (Kern County) Career Academy (Bechtold and King, 2001) students Nonacademy students No Not studied Horizons (San Bernardino) (Hemsley, 2002) Career Academy and technical education students Students with no career/technical education courses Indirect—focused only on “highachieving” students No enrollment or employment differences NOTES: Under type of program, “high-STC” refers to a combination of STC activities. A question mark indicates that the evaluation was not sufficiently detailed to answer the question. 26 postsecondary enrollment or employment effects.7 Looking across all of the postsecondary employment results reported, there is no evidence of positive employment effects and occasional evidence of negative effects. For enrollment, there is some mixed evidence of positive and negative effects, but most common, again, is a failure to find evidence of any effect. These studies are limited by the failure to account for selection into STC participation, even at the level of observable (and measured) control variables. Although data were collected on items that could have been introduced as limited controls in estimating the effects of STC (such as sex, age, ethnicity, socioeconomic status, and so on), the analyses did not account for any of these. The second panel of Table 3.1 covers the two WestEd-MPR Plus cases, first, magnet students, and second, Career Academy students. Both studies are better than those just discussed in that they take account of control variables in estimating the effects of STC. However, neither study looks at postsecondary outcomes; instead, both focus on performance while in high school.8 Of the remaining nine evaluations in the last panel, five can be identified. Unfortunately, for five of these nine evaluations no data were collected on a control group of nonparticipants, which would rule them out as providing an evaluation of the effects of STC. Instead, these evaluations focus on such aspects as implementation, surveys of program participants, and so on. Of the four studies that include control groups, only two focus on postsecondary outcomes. The first (a confidential study) finds some indication of higher two-year college enrollments among Career Academy students but no difference in four-year enrollments, although the statistical significance of the results is not _____________ 7Effects are reported that are statistically significant at the 5 percent level. It might have been desirable to also report findings that were significant at the 10 percent level. But the evaluations did not provide p-values or t-statistics; instead, they simply reported which results were significant at the 5 percent level. 8Both studies provide some evidence that STC participants do better in school (in terms of, for example, test scores, attendance, and pass rates), even after controlling for some measures of prior achievement in school. These findings echo the Manpower Demonstration Research Corporation studies of Career Academies discussed below (Kemple and Snipes, 2000; Kemple, 2001). 27 reported. The second, for the Horizons LP, finds no evidence of enrollment or employment differences. In summary, these evaluations provide little if any convincing evidence of the effectiveness of STC programs, especially as regards postsecondary outcomes. Many of the evaluations fail to even use a control group, which is a prerequisite for evaluation. Most fail to use any control variables to account for differences between participants and nonparticipants. And few look at postsecondary outcomes. Tech Prep and Career Academies in California The other two branches of the STC system in California are Tech Prep and Career Academies. There is no existing statewide or other program evaluation of Tech Prep. There is a statewide evaluation of Tech Prep program implementation for the 2001–02 academic year, but it neither evaluates student outcomes nor reviews research on student outcomes related to participation in Tech Prep (O’Driscoll et al., 2002d).9 There are a couple of studies of Career Academies in California. Maxwell and Rubin (2001) study Career Academies in an undisclosed large, inner-city California school district in a period before SWTOA. Their research is based primarily on a survey about secondary and postsecondary experiences sent to academy students and a comparison group of nonacademy students. The survey data point to increases in postsecondary enrollment for some groups (Asians and males) but no improvements in employment or wages, which the authors describe as “inconsistent with the state’s goals for the program” (p. 14). The authors also report positive effects of academy participation on employment in a job related to the high school program for some groups, although noting that very few participants or nonparticipants entered fields related to their high school course of study. However, these conclusions are of _____________ 9The report does point to some of the key problems associated with identifying and tracking Tech Prep students. In particular, there are no consistent guidelines to define Tech Prep students, and electronic data systems linking secondary and postsecondary partners are inadequate. The report concludes that the state’s Tech Prep system does not currently have the capacity to perform an adequate student outcome-based evaluation of the effectiveness of Tech Prep. 28 questionable reliability. The authors use some limited controls for preprogram differences between participants and nonparticipants; but a potentially more serious concern is the very low response rate to the survey (around 12 percent). An earlier study by Stern et al. (1988) also looked at Career Academies in California. With very limited control variables, the evidence suggests some positive effects of academies on high school performance but does not address postsecondary outcomes. STC Research for the United States Not surprisingly, research on the effects of STC outside California is more extensive and its conclusions may be informative about the types of STC programs used in the state. However, this wider research base also generally provides little basis for concluding that STC programs improve postsecondary outcomes. This section briefly reviews this research; Appendix A provides a more detailed discussion. Two reports from the National Center for Research in Vocational Education (NCRVE) provide a relatively thorough compendium of research on STC programs in a wide variety of locations through about 1997 (Stern et al., 1994; Urquiola et al., 1997). Many of the studies covered in these reviews do not even include a comparison or control group of nonparticipants, making it impossible to say anything about the causal effects of STC programs. Some of the more recent studies use a comparison group and pay more careful attention to pre-program differences, but even the best of these (Orr, 1996; Hollenbeck, 1996) are plagued by possibly important differences between participants and nonparticipants. Finally, a recent report on STWOA by Mathematica, Inc.—the national evaluation of STWOA for the U.S. Congress that was mandated by the act—does not focus on evaluating the effects of STC on participants, and the limited evidence it does present does not indicate beneficial postsecondary effects. In the national research on the effectiveness of STC, one important exception to the generally inadequate efforts to account for pre-program differences is the ongoing evaluation of Career Academies by the Manpower Demonstration Research Corporation (Kemple and Snipes, 2000; Kemple, 2001, 2003), based on random assignment of students to 29 Career Academies. The second report from this evaluation (Kemple, 2001) looked at students one year after the scheduled completion of high school and found no effect on high school graduation rates, postsecondary education, or employment. But the most recent report (Kemple, 2003) presents stronger evidence of such postsecondary effects. In particular, male participants have higher full-time employment, hours, and earnings, although there is no evidence of such positive effects for women. On the other hand, the estimates indicate that male participants are less likely to be enrolled in school or to have obtained further education; although many of the education-related differentials are large, most are not statistically significant. The author interprets the results as indicating that Career Academies boost labor market outcomes without any offsetting reductions in education. However, the results could perhaps more plausibly be read as suggesting offsetting positive employment and negative schooling effects, as the magnitudes of the negative schooling effects roughly offset the positive employment effects. Summary The findings from this review of existing research are summarized in Figure 3.1, focusing on the three components of STC in California.10 There is as yet no existing statewide evidence establishing the effectiveness of STC in California in increasing postsecondary employment or enrollment, or more generally improving youths’ labor market outcomes. Although California commissioned a statewide STC evaluation, that study contributes little empirical evidence on the effectiveness of STC in helping students make transitions to postsecondary education or employment. The state’s 2002 Tech Prep evaluation did not address student outcomes at all, and evaluations of Career Academies in California perhaps point to increased college enrollments, although the evidence is of limited quality. There was a flurry of evaluations of the effectiveness of local STC efforts in California spurred by STWOA. However, these evaluations also provided little if any convincing evidence of the effectiveness of STC _____________ 10Some of the material summarized in this figure is based on sources discussed only briefly in the text, but in more detail in Appendix A. 30 Figure 3.1—Summary of Evidence on the Effectiveness of STC on Postsecondary Enrollment and Employment programs, especially on postsecondary outcomes. Many of these evaluations do not use a control group, which makes it impossible to infer the effects of the programs, most fail to account for differences between participants and nonparticipants, and few look at postsecondary outcomes. At the national level, most of the existing evidence is plagued by the problem of pre-program differences between participants and nonparticipants. However, recent research that uses an experimental design to evaluate the effectiveness of one particular kind of STC program— Career Academies—to this point finds some beneficial effects on postsecondary outcomes for men in the form of higher wages and earnings and greater likelihood of full-time work, although the evidence suggests that there may be some tradeoffs in the form of lower participation in postsecondary education.11 The next chapter presents new evidence on the effects of the types of STC programs that were supported by STWOA in California, based on the NLSY97 data. _____________ 11Note, however, that these conclusions are the opposite of those reached by Maxwell and Rubin (2001) for Career Academies in California. 31 4. Evidence on the Effects of School-to-Career Programs on Postsecondary Enrollment and Employment1 This chapter describes econometric analyses of the effectiveness of STC programs that overlap considerably with the types of general, broad-based STC programs spurred by STWOA in California. The research takes advantage of the information on STC program participation in the NLSY97, which is the first nationally representative dataset—and indeed the first large-scale dataset—to include information on participation of high school students in STC programs. STC program participation and postsecondary outcomes may be related simply because participants differ from nonparticipants independent of any effects of STC programs; such noncausal relationships, however, do not imply that STC programs are effective. In contrast, this chapter emphasizes taking advantage of numerous features of the NLSY97 to estimate the actual (that is, causal) effects of high school STC programs on postsecondary employment and higher education. Of course, even establishing causal effects of STC on higher education and employment does not imply that the benefits of STC funding outweigh the costs. Given the difficulty of characterizing better career decisionmaking, there may never be a complete analysis of this nature. Nonetheless, information on the effects of STC programs on transitions from high school to employment or higher education can inform policymakers as to whether STC is having some of the intended effects. After all, core goals of STC are to move people into higher_____________ 1This chapter is based on Neumark and Rothstein (2003). 33 paying jobs and to encourage skill formation among new labor market entrants, both of which are likely to be furthered by increased enrollment or employment in the immediate post-high school period. Empirical Approach2 The goal of the empirical analysis is to estimate, at the individual level, the relationship between employment and enrollment in the posthigh school period and participation in STC during high school. STC participation is categorized in terms of participation in specific types of programs discussed below. Given that employment could be very shortterm, whereas college enrollment of any type is nearly always longerterm, the focus is on employment at the time respondents to the NLSY97 are surveyed and on whether the individual was enrolled in college at any time since leaving high school. The simplest form of such an analysis looks at the statistical relationship between STC participation, on the one hand, and postsecondary enrollment or employment, on the other. The results from this type of estimation will be summarized in terms of the difference in the probability of postsecondary enrollment or employment associated with participation in different types of STC programs. As an example, the statistical analysis would conclude that participation in a co-op STC program boosts the probability of postsecondary employment by 0.07.3 However, as mentioned above, there may be pre-program differences between STC participants and nonparticipants. That is, participants and nonparticipants may have had different likelihoods of postsecondary enrollment or employment even before they participated in STC programs, and it would be a mistake to interpret such differences as causal effects of STC participation. The empirical analysis takes three _____________ 2A more detailed discussion of the empirical approach and methods is provided in Appendix B. 3An estimate like this can be equivalently interpreted as a 7 percentage point increase in employment. But percentage point changes have to be distinguished from percentage changes. For example, an increase in the probability of employment from 0.50 to 0.57, and similarly an increase in the percentage employed from 50 to 57 percentage points, both represent 14 percent increases in employment. 34 different approaches to trying to account for these pre-program differences between participants and nonparticipants in an attempt to develop more reliable estimates of the effects that STC participation has on postsecondary outcomes. Each approach takes advantage of unique features of the NLSY97 that make this dataset particularly powerful for studying the effects of STC. First, the NLSY97 has information on a rich set of variables describing individual respondents to the survey, and their families. In addition to fairly typical demographic variables (sex, race, ethnicity, and age), the dataset includes three additional sets of variables that are potentially important, including information on living arrangements and the respondent’s family;4 test scores from the Armed Services Vocational Aptitude Battery (ASVAB); and self-reported measures of school behavior.5 Controlling statistically for these variables should go a long way to eliminating any influence on the estimates of the effects of STC of pre-program differences between STC participants and nonparticipants. Second, for part of the sample the NLSY97 also includes measures of individual respondents’ self-reported expectations (in the form of probabilities) for future education and employment, including receipt of a high school diploma by age 20, obtaining a four-year college degree by age 30, and working over 20 hours per week at age 30.6 Controlling for these expectations—which are reported before the STC participation measures used in the analysis—makes it even more likely that the estimates account for pre-program differences. The intuition behind the contribution of the work and schooling expectations variables is as follows. Before participating in STC, students are asked about their _____________ 4The family and living arrangement variables include urban residence; whether one lives with both biological parents, only the biological mother or father, a biological parent and a stepparent, or in some other arrangement; household size; household income; and the biological mother’s schooling. Household income is from a Round 1 parent questionnaire except in the rare event that the youth is defined as independent in 1997. 5These include whether the respondent was threatened at school or had gotten into a physical fight at school and whether the respondent had been late with no excuse two or more times or had been absent two or more weeks. All are measured in 1997. 6This information is available for the 1980 and 1981 birth cohorts, which are the oldest cohorts in the sample. 35 post-high school work and schooling expectations. Then some participate in STC and some do not, and their post-high school work and schooling behavior are subsequently observed. If, for example, after controlling for educational expectations STC participants are still more likely to be enrolled in college after leaving high school, then it is quite sensible to conclude that there is a true (that is, causal) effect of STC on postsecondary enrollment, because the educational expectations variables should have controlled for many remaining pre-program differences that are associated with postsecondary educational outcomes. Third, there may be other unmeasured pre-program differences between STC participants and nonparticipants that are not captured in any of the control variables discussed thus far. Some of these differences, however, may vary systematically across students in different schools, because of similarities either in the student body or in the educational environment. Because the NLSY97 has data on multiple students in the same school, it is possible to control, in addition, for any unmeasured pre-program differences common to all individuals within a school, identifying the effects of STC participation from the within-school differences between those who do and do not participate in STC and within-school differences in outcomes associated with this participation. Intuitively, these estimates will point to an effect of STC on postsecondary outcomes only if, for students in the same school, STC participants have different probabilities of postsecondary enrollment than nonparticipants. In contrast, average differences in STC participation rates and postsecondary outcomes across schools will not feed into these estimates of the effects of STC. These estimates are referred to as “school fixed effects” estimates, because they hold constant school-level differences common to students in the same school. Additional results discussed in Appendix B suggest that school-level differences are potentially quite important; as a result, the school fixed effects estimates are emphasized in the discussion of the results that follows. 36 Dataset, Variables, and Analysis Sample Overview of the NLSY97 The first round of the NLSY97 was administered in 1997 to a nationally representative sample of 8,984 men and women who were ages 12 to 16 as of December 31, 1996. The survey involved personal interviews with the youth and one of his or her parents. It gathered extensive information on the youth’s labor market behavior, education and training, family and community background, as well as important life events such as marriage or the birth a child. More important, in each year, respondents were asked a number of questions about participation in school programs designed to help them prepare for the world of work. This report uses data from the first four rounds of the NLSY97, from 1997 through 2000. All STC information is taken from Rounds 2–4, after the data on schooling and work expectations and school behaviors had been collected. STC Information in the NLSY97 In the NLSY97, students were surveyed about “programs schools offer to help students prepare for the world of work.” The STC programs covered include internships or apprenticeships, cooperative education, job shadowing, mentoring, work in a school-sponsored enterprise, and Tech Prep. The following definitions summarize the features of these programs.7 • Internship: For a specified period of time, students work for an employer to learn about a particular industry or occupation. Students’ workplace activities may include special projects, a sample of tasks from different jobs, or tasks from a single occupation. The internship may or may not include paid work experiences. _____________ 7These are taken from the NLSY97 School Administrator Survey, which was given to administrators of schools that were subsequently linked to the NLSY97. In the NLSY97, respondents were shown definitions that corresponded closely to these but were slightly shorter. The data from the School Administrator Survey are not used in this chapter, although they are used in some of the results discussed in Appendix B. 37 • Apprenticeship: Typically, apprenticeships are multiyear programs that combine school- and work-based learning in specific occupational areas or occupational clusters designed to lead directly into either a related postsecondary program, entry-level job, or registered apprenticeship program. They may or may not include paid work experiences. • Cooperative education: This is a method of instruction whereby students alternate or parallel their academic and vocational studies with a job in a related field. It may or may not include paid work experiences. • Job shadowing: Typically, as part of career exploration activities early in high school, a student follows an employee for one or more days to learn about a particular occupation or industry. Job shadowing is intended to help students hone their career objectives and select a career major for the latter part of high school. • Mentoring: A student is paired with an employee over an extended period of time during which the employee helps the student master certain skills and knowledge the employee possesses, models workplace behavior, challenges the student to perform well, and assesses the student’s performance. Mentoring may be combined with other work-based learning activities, such as internships or on-the-job training. • School-sponsored enterprise: This entails the production of goods or services by students for sale to or use by others. School-sponsored enterprises typically involve students in the management of the project. Enterprises may be undertaken on or off the school site.8 _____________ 8School enterprises are likely the least well understood of these programs. There is tremendous variety in school-sponsored enterprises, from businesses run by students in schools (banks for students and faculty, coffee shops, desktop publishing and multimedia presentations, snack shops, and so on) to enterprises run off premises (such as airport stores and retail outlets). Students working in these enterprises often combine their work with either formal or informal instruction in business-related fields such as accounting and management. 38 • Tech Prep: This is a planned program of study with a defined career focus that links secondary and postsecondary education.9 In Round 1, administered in 1997, the STC questions cover the types of programs in which individuals participated at any time in the past, whereas in subsequent rounds (2–4) the questions shifted to participation in the year preceding each round of the survey. The questions were asked of all 9th to 12th graders in Round 1. In Rounds 2-4, they were asked of any respondents enrolled in school (including college). However, this research focuses on participation while in high school by using only STC information for years in which respondents were enrolled in high school.10 Analysis Samples11 Our analysis is based on data from the first four rounds of the NLSY97. Respondents were ages 12–17 as of the first round in 1997, and the next three rounds were conducted in the three subsequent years. The analysis focuses on educational and employment outcomes measured as of the third or fourth rounds, in 1999 and 2000, to isolate those individuals for whom it is possible to observe the early years of their employment or higher education after leaving high school. _____________ 9Note that Tech Prep was separated from the general, broad-based STC programs supported by the STWOA that were discussed in the previous chapter. The NLSY97 was not designed, of course, specifically to evaluate the STWOA. The empirical results for Tech Prep relative to the other types of STC programs included in this list therefore help address the effectiveness of these alternative types of programs. 10The manner in which the STC data are collected raises two issues. First, because the larger share of STC participation occurs in the later years of high school, high school graduation is endogenously related to “exposure” to STC; dropouts cannot participate in high school STC programs. A study of STC and high school graduation would therefore require a very different research design that is not afforded by these data, for example, by randomizing students into a program in 9th grade and then measuring graduation rates. This type of study design is used in the Career Academy study discussed above (Kemple, 2003). Second, a moderate share of individuals report some college enrollment in the last year in which they are enrolled in high school, so in principle these individuals could be reporting STC during their short initial spell of college.As a check on the robustness of the results, the specifications reported below were re-estimated excluding the STCinformation for interview years with mixed high school and college enrollmentfrom the calculation of the STC measures.The qualitative conclusions were unaffected. 11A more detailed discussion of the analysis samples is provided in Appendix B. 39 Combined with a number of sample restrictions such as availability of data on STC participation, there are 2,933 observations in the baseline analysis sample; the samples analyzed incorporating the work and schooling expectations, and including school fixed effects, are smaller. In the former case, this is because the work and schooling expectations questions were asked only of the two oldest birth cohorts in the dataset. In the latter case, the sample size reduction stems from cases in which there were no multiple observations with available data within the same school. Results STC Participation Before reporting estimates of the models for postsecondary enrollment and employment, estimates of models for participation in STC are reported, in part to provide descriptive information on STC participation, and in part to provide a preliminary assessment of the severity of pre-program differences between STC participants and nonparticipants. Some of the key results are summarized in Figures 4.1a–4.1c.12 In Figure 4.1a, the first bar for each type of STC (and for any STC) shows the sample proportion participating. Participation rates in each type of STC range from about 0.07 to 0.18, with the rate for participation in any STC equal to 0.42. The three subsequent bars for each STC entry show how, holding other factors at their means, participation rates differ for blacks, Hispanics, and females. The figure reveals that for three types of STC programs (co-op, school enterprise, and Tech Prep), black students are significantly more likely to participate, with the estimates indicating participation probabilities that are higher by 32 to 79 percent relative to the overall sample. This race difference appears to stem from higher participation of blacks relative to whites and others in the same schools, rather than from schools with more black students being more likely to offer STC programs. _____________ 12Appendix Table C.2 reports the regression results underlying these figures. 40 Figure 4.1a—STC Participation Probabilities, Differences by Demographic Group Figures 4.1b and 4.1c report the differences in participation associated with family structure and with ASVAB test scores. Perhaps the most striking finding from these figures (as well as the full regression results reported in Appendix Table C.2) is that almost no other variables are significantly related to STC participation.13 In Figure 4.1b, the only significant differences are lower participation in job shadowing and mentoring for those who live only with a mother. And in Figure 4.1c, there are very few significant differences in participation (and slight differences overall) associated with one standard deviation higher ASVAB scores. The finding that very few variables predict STC participation suggests that, in this sample, problems from pre-program differences between STC participants and nonparticipants may not be too severe, although it is important to confront this question directly in the course _____________ 13This is also reflected in the very low R2 values for these models of individual participation (see Appendix Table C.2), which are generally one-fifth or less of the R2 values for the models for whether schools offered STC. 41 Figure 4.1b—STC Participation Probabilities, Differences by Family Structure Figure 4.1c—STC Participation Probabilities, Differences Associated with One Standard Deviation Higher ASVAB Scores 42 of estimating the effects of STC on postsecondary outcomes, given that STC participation was not randomly assigned in these data.14 Basic Analysis Figures 4.2a and 4.2b begin the analysis of the effects of STC on postsecondary enrollment and employment. For each type of STC program, the upper bar in Figure 4.2a reports the estimated effects of Figure 4.2a—College Enrollment Probabilities, Effects of STC Participation _____________ 14If STC participation were randomly assigned, none of the variables in Appendix Table C.2 would be expected to predict participation, and the evidence does not deviate much from this scenario. Excepting the estimates of the black-white difference, 7.5 percent of the remaining coefficient estimates are significant at the 5 percent level, versus the 5 percent that would be expected to be significant at this significance level solely as a result of randomness even if none of the controls were in fact related to STC participation. One potential issue is that the ASVAB test scores may appear insignificant because of a high degree of multicollinearity among them. Tests for the joint significance of the test score coefficients—which are not affected by the multicollinearity—indicated that multicollinearity is an issue, but not an important one, as only in one case (for job shadowing) does the joint test indicate significance while none of the individual tests do. And the low R2 values mentioned in the footnote above are of course not influenced by any multicollinearity problem. 43 Figure 4.2b—Employment Probabilities, Effects of STC Participation STC on enrollment, conditional on the demographic controls only, and the lower bar reports the estimated effects when the detailed controls—including those for living arrangement and family structure, ASVAB scores, and school behaviors—are included.15 The estimates in Figure 4.2a reveal considerable heterogeneity in the effects of different types of STC programs. In the estimates reflected in the upper bars, with minimal controls, mentoring and school enterprise STC programs are associated with a significantly higher probability of some college education, with the differential ranging from about a 0.07 to a 0.11 higher probability. Relative to the sample proportion of 0.50 with some college, these estimates imply increases of 14 to 22 percent in the probability of college attendance. On the other hand, Tech Prep STC programs are associated with a significantly lower likelihood of college education. Although these estimates reveal the possible magnitudes of the effects of participation in STC programs on college enrollment, they also _____________ 15The regression results and all of the regression controls on which these tables are based are reported in Appendix Tables C.3a and C.3b. 44 indicate that it may be difficult to detect as statistically significant effects on college enrollment that might be regarded as substantively important. For example, consider one of the STC programs the estimated effect of which is not statistically significant in Figure 4.2a, such as job shadowing. The standard error of this estimate is 0.026, which implies that an estimate as large as 0.042 (implying an increase in the probability of college enrollment of 8.4 percent) would not be statistically significant. This is an important caveat to keep in mind throughout the discussion of the estimates in this chapter. In particular, it is critical to remember that failure to find that an estimated effect is significantly different from zero is not the same thing as concluding that the effect is zero. The point estimate that is obtained from the regression model is still the best estimate. A lack of statistical significance of an otherwise positive estimate (as in the above example) simply means that one cannot be very confident that the estimated effect is different from zero. As reflected in the lower bars in Figure 4.2a, when the more detailed set of control variables is added, these estimated effects of participation in STC weaken somewhat. With all of these controls added, the findings indicate that only school enterprise STC programs are significantly associated with college attendance, with a positive effect estimated at a 0.088 higher probability of enrollment. The results for current employment—in Figure 4.2b—are less sensitive to the addition of the more detailed control variables than were the results for college enrollment. In all cases, the estimates point to strongly statistically significant positive effects of co-op programs, with effects on the probability of employment near 0.08, and also evidence of positive effects (near 0.06) of internship/apprenticeship programs, although the estimates of this latter effect are statistically significant only at the 10 percent level. Relative to the proportion of the sample that is currently employed, which is 0.63, the estimates imply that co-op STC programs are associated with approximately 13 percent increases in the likelihood of post-high school employment, and internship/ apprenticeship programs with increases of about 9 percent. In addition to the estimated effects of participation in STC programs, the estimates of the coefficients of the control variables are of some interest; these are reported in full in Appendix Table C.3b and 45 discussed briefly here. Women and blacks are significantly more likely to have attended some college, conditional on the observables,16 whereas blacks are less likely to be employed. In general, household structures without two biological parents are associated with a lower likelihood of college attendance and a higher likelihood of employment, and respondents from smaller households and from households with higher income, and whose mothers are more educated, are more likely to have attended some college. Presumably all of these latter effects reflect the effect of economic resources on college attendance. Also not surprisingly, the ASVAB scores—in particular for word knowledge and math knowledge—are significantly positively associated with college attendance. Finally, all four indicators of troublesome behaviors in high school are significantly associated with a lower probability of attending college, and one of these (fighting in school) is also negatively associated with employment. Thus, these control variables have rather strong associations, in the expected directions, with enrollment and employment outcomes, suggesting that they do relatively little to mediate the effects of STC participation because they are weakly associated with this participation—as documented in part in Figures 4.1a–4.1c—rather than because they are uninformative about enrollment and employment outcomes. Summary of Findings from Basic Analysis Overall, the basic multivariate analysis suggests some positive effects of STC participation on college education and on current employment. More specifically, school enterprise programs are associated with a higher likelihood of obtaining some college education, and co-op and internship/apprenticeship programs are associated with increased employment. As already noted, the extensive set of control variables does relatively little to alter the estimated relationship between STC and employment, although there could still be important unmeasured differences between participants and nonparticipants that underlie the _____________ 16It is a common finding in many datasets that, overall, blacks attend college at lower rates than whites, but that, conditional on factors such as household income, parents’ education, and test scores, blacks are more likely to attend college (see, for example, Cameron and Heckman, 2001). 46 estimated effects on either enrollment or employment. In particular, although the links between co-op or internship/apprenticeship programs and later employment may seem reasonably straightforward, the link between school enterprise and college attainment is less obvious, suggesting that the estimated effect of school enterprise programs may reflect pre-program differences rather than the effects of STC participation. On the other hand, school enterprises may provide important learning opportunities by placing business-related instruction in the context of running a student business. This could result in better academic performance (and hence greater college attendance) or may directly demonstrate to students the value of further education. The following subsections therefore focus more sharply on the question of whether these effects can be interpreted as causal effects of STC participation. Work and Schooling Expectations The next analysis introduces variables measuring respondents’ work and schooling expectations during high school. Recall that these are measured before STC participation. Thus, if there are pre-program differences between participants and nonparticipants, then including these expectations variables is likely to mute any estimated effects of STC participation on enrollment and employment outcomes. For example, those students who early on expect to work right after high school are more likely to participate in programs that will enhance the returns to that work, generating a spurious positive association between, for example, participation in co-op STC programs and postsecondary employment, which would weaken upon including controls for the earlier expectations. The results are reported in Figures 4.3a and 4.3b.17 For each dependent variable, the estimates are first reported—in the upper bars—excluding the expectations variables but including all of the other _____________ 17The regression results on which these figures are based are reported in Appendix Table C.4. 47 Figure 4.3a—College Enrollment Probabilities, Effects of STC Participation, Controlling for Prior Work and Schooling Expectations Figure 4.3b—Employment Probabilities, Effects of STC Participation, Controlling for Prior Work and Schooling Expectations 48 controls included in the second set of estimates in Figures 4.2a and 4.2b, for the subsample of observations—the 1980 and 1981 birth cohorts—for which these expectations data are available. Holding the sample constant isolates the effects of adding the expectations variables. The estimates for this subsample are generally similar to the corresponding estimates for the full sample. Only for school enterprise STC programs is there a significant effect of STC. It is positive, as for the full sample, although a bit larger (0.11 compared with 0.09). For current employment, the estimated effect of internship/apprenticeship programs falls a little and is no longer statistically significant. The estimated effect of co-op programs also falls somewhat (from 0.08 to 0.06) but remains statistically significant at the 10 percent level. The lower bars in each figure report the results when the expectations variables are added. The key result is that there is little change in the estimated coefficients or their statistical significance. The estimated effect of school enterprise on college attendance falls by only about 0.01 and remains statistically significant, and in the employment model the estimated effect of internship/apprenticeship programs is unchanged and the estimated effect of co-op programs changes only modestly. Although these latter two estimates are not statistically significant, the interest in Figures 4.3a and 4.3b lies in asking whether the addition of the expectations variables results in any appreciable changes in the estimated effects of STC. Clearly the answer is no; in other words, once the detailed set of control variables considered in Figures 4.2a and 4.2b are included, there appears to be little remaining bias in the estimated effects of STC stemming from pre-program differences that are reflected in work and schooling expectations. Although the analysis with the expectations data could be done only on a subsample, the finding of no indication of remaining bias implies that the full sample estimates can be used reliably. The evidence, therefore, still points to positive causal effects of school enterprise STC programs on college attendance, and of co-op and internship/apprenticeship programs on employment, in the immediate post-high school period; at 49 the same time, the estimates in Figures 4.3a and 4.3b perhaps suggest some caution in concluding that these effects are very robust.18 School Fixed Effects Finally, Figures 4.4a and 4.4b report on the analysis incorporating the fixed school effects.19 Recall that the merit of this analysis is that it uses only differences in STC participation among individuals in the same school to estimate the effects of STC, eliminating any role for differences in STC participation across schools that may be related to unmeasured school-level differences. For this analysis, attention is restricted to the Figure 4.4a—College Enrollment Probabilities, Effects of STC Participation, with School Fixed Effects _____________ 18Finally, the estimated coefficients of the other included variables (see Appendix Table C.4) show that expectations for a four-year degree are strongly positively associated with actual college attendance. At the same time, these expectations are strongly negatively associated with employment, whereas work expectations for age 30 are strongly positively associated with employment. This latter evidence indicates that the expectations variables are quite informative about subsequent behavior, bolstering the validity of this approach. 19The regression results on which these figures are based are reported in Appendix Table C.5. 50 Figure 4.4b—Employment Probabilities, Effects of STC Participation, with School Fixed Effects subsample of individuals with multiple observations in the same school, as it is only these individuals who reveal the effects of STC programs. The estimates are first reported without the school fixed effects for this subsample and then with the fixed effects; the comparison between these two sets of estimates isolates the effects of holding constant all unmeasured characteristics of students that are common to students in the same school.20 The upper-bar estimates in each figure—for the “fixed effects sample” but excluding the fixed effects—are quite similar to the full sample results (the lower bars in Figures 4.2a and 4.2b). There is a positive and significant estimated effect of school enterprise on college attendance that is virtually the same size (about 0.09), and positive and significant estimates of the effects of co-op and internship/apprenticeship programs on current employment. However, although the estimated _____________ 20In these specifications, the full set of controls are included, but the expectations variables are excluded because they did not matter in Figures 4.3a and 4.3b, and because including them would impose a further sizable sample restriction given that the expectations variables are asked only of the two oldest birth cohorts. 51 effect of co-op programs is of similar magnitude, the estimated effect of internship/apprenticeship programs is larger (0.12 versus 0.06). In addition, the estimated negative effect of Tech Prep programs on college attendance is larger (–0.07 versus –0.04) and statistically significant. The most important questions, though, are how the estimates change when the school fixed effects are added and what the estimates are with the fixed effects included. These latter estimates are reported by the lower bars in each figure. In the estimates for college enrollment, the effects of STC programs strengthen, with the already significant effects of school enterprise and Tech Prep programs growing and a positive and significant effect of job shadowing programs emerging as well. In the estimates for current employment, the estimated positive effect of co-op programs grows from 0.08 to 0.10, remaining statistically significant, whereas the estimated effect of internship/apprenticeship programs falls from 0.12 to 0.07 and becomes statistically significant at only the 10 percent level. The changes in the estimated effects of STC program participation are not large, but they are not trivial either, and in some cases the conclusions are substantively different (for example, with respect to job shadowing). For a number of reasons, the estimates with school fixed effects are the most reliable.21 They therefore serve as the basis for summarizing the results thus far and as the jumping off point for the remaining analyses in this chapter. Provisional Summary and Interpretation To summarize, the analysis thus far points to a number of beneficial effects from certain types of STC programs. In particular, job shadowing and school enterprise programs increase post-high school college attendance. Participating in job shadowing increases the probability of _____________ 21Statistical tests for whether the school fixed effects are needed yield p-values of 0.19 for the college enrollment specification, and 0.23 for the current employment specification. (These are Hausman tests and are computed for fixed effects versus random effects estimates.) Strictly speaking, these results suggest that the school fixed effects can be excluded. But these p-values are nonetheless relatively low, and the standard errors increase very little in the fixed effects estimation—in contrast to what occurs in many applications. Thus, because (1) there is little cost to retaining the school fixed effects, (2) the estimates are a bit different, and (3) including the school fixed effects is unlikely to introduce any bias, the school fixed effects estimates are preferable. 52 postsecondary enrollment by about 0.06, whereas the increase associated with school enterprise STC programs is twice as large. Co-op and internship/apprenticeship STC programs increase employment, with the probability rising by about 0.10 in the case of co-op programs and 0.07 in the case of internship/apprenticeship programs, although the statistical evidence for internship/apprenticeship programs is weaker. In each case, there is no offsetting negative effect on the other dependent variable; for example, co-op programs increase postsecondary employment without decreasing postsecondary enrollment. The implication is that these programs appear to induce higher enrollment or employment at the expense of being neither enrolled nor employed, which casts the results in a more favorable light. Finally, working in the other direction, Tech Prep programs appear to reduce the probability of college enrollment by about 0.10, without any offsetting increases in employment.22 A natural question that arises in light of these estimates is whether the effects that are found are “good” or “bad” from the perspective of evaluating STC programs. It has already been noted that rather than focusing on the effects of STC on careers, broadly speaking, this report instead focuses more narrowly on postsecondary enrollment and employment. Even with this more limited focus, though, one can probe a bit further in trying to assess the evidence. In particular, with respect to the finding that Tech Prep reduces college enrollment, it is of interest to ask whether there is any tradeoff in terms of work. The estimates in Figure 4.4b show no employment effect (although employment and enrollment are not necessarily measured contemporaneously). But in results not reported in the figures (or in the corresponding appendix table), a model for the probability of working full-time, conditional on _____________ 22Multinomial logit models could also be estimated for the four categories: enrolled/employed, enrolled/non-employed, non-enrolled/employed, and nonenrolled/non-employed. If the evidence indicated, for example, that a particular program raised the probability of both employment and enrollment, then there would be a potentially important refinement from the multinomial estimates (although there is no fixed effects multinomial logit estimator), telling us whether these effects came from individuals who were more likely to be both enrolled and employed or more likely to be doing one or the other relative to neither. But given that for each program the effects appear (if at all) for only employment or only enrollment, the multinomial estimates do not provide any additional information. 53 working, was estimated, and the resulting coefficient for Tech Prep, although not statistically significant, was positive and roughly of the same size (but opposite sign) as the estimated effect on enrollment. This suggests that the negative effects of Tech Prep on schooling are roughly offset by a higher incidence of full-time work.23 Nonetheless, as the returns to schooling in the form of higher wages typically outweigh the returns to experience—and even more so with regard to the return to full-time versus part-time experience—it is difficult to view the adverse effect of Tech Prep on schooling in a positive light.24 Heterogeneity in the Effects of STC To this point, the effects of STC participation have been estimated for the full sample. However, these effects may differ across racial or ethnic groups or across other characteristics of individuals that are associated with socioeconomic status or the likelihood of attending college (aside from STC). In fact, STWOA makes some reference to the problems faced by disadvantaged and minority youths. And STC practitioners commonly argue that STC programs are particularly helpful for less-advantaged youths or the broader group of those who in the absence of any intervention are unlikely to go on to college—often termed the “forgotten half” (see, for example, Donahoe and Tienda, 1999). To explore such questions, the key specifications were re-estimated allowing the effects of STC to differ across groups on the basis of race or ethnicity, the ASVAB math knowledge score, mother’s education, the family’s living arrangements, and sex. In each case, a single specification for the pooled sample was maintained but interactions of each of the STC participation variables with indicators for the groups considered (e.g., white, black, and Hispanic) were introduced. School fixed effects were included, which for reasons just discussed provide the most reliable _____________ 23Although the effect on full-time work is not significantly different from zero, the data are even more consistent with offsetting effects on full-time work and schooling. 24A possible counterargument is that the Tech Prep students who forgo postsecondary education face different relative returns to schooling and labor market experience—either lower returns to schooling or higher returns to work. There is, however, no direct evidence on this question. 54 estimates. In the case of the math knowledge test, mother’s education, and living arrangement variables, the estimates in Appendix Table C.3b indicated that these were strongly related to college attendance. And all of the variables aside from the sex of the respondent are related to socioeconomic advantage. Selected results are reported in Figures 4.5a–4.5j. A separate figure is provided for each specification; for example, Figure 4.5a reports results for postsecondary enrollment, with the effects broken out for whites, blacks, and Hispanics.25 To make the results easier to digest, for each specification, only estimated coefficients of the STC-group interactions that are significantly different from zero for at least one group are reported. It is important to keep in mind, though, that in many cases the differences across groups were not statistically significant, in which case the pooled estimates reported in Figures 4.4a and 4.4b cannot be rejected.26 The evidence in these figures should be interpreted more as simply establishing which groups display statistically significant evidence of the effects of STC programs. Furthermore, the caveat above about substantively large estimated effects falling short of statistical significance bears reiterating with respect to these figures, as the precision of the estimates falls with the disaggregation of the estimated effects; this occurs because in many cases the sample of individuals in any subgroup that participates in a particular STC program is quite small. Results from the specification allowing different effects by race and ethnicity are reported in Figures 4.5a and 4.5b. There is generally more evidence of beneficial effects of STC programs for whites, because for whites only there are (statistically significant) positive effects of internship/apprenticeship programs on college attendance and of co-op and Tech Prep programs on employment. At the same time, Tech Prep has a negative effect on schooling only for whites. Also of note is that the only evidence of positive effects of internship/apprenticeship _____________ 25The regression results on which these figures are based are reported in Appendix Table C.6. 26For this reason, in these figures, the statistical significance of the estimates (relative to zero) is not reported. But see Appendix Table C.6 for this information. 55 Figure 4.5a—College Enrollment Probabilities, Effects of STC Participation, Differences by Race/Ethnicity Figure 4.5b—Employment Probabilities, Effects of STC Participation, Differences by Race/Ethnicity 56 programs on employment arises for minorities, with the estimate large (and statistically significant) for blacks and near zero for whites. The finding that school enterprise programs have positive effects on college attendance appears to apply to all groups.27 The results disaggregating the effects by the ASVAB math knowledge score are reported next, in Figures 4.5c and 4.5d. There is no clear message here, as there are some positive and significant effects of alternative programs for each ability group. Figures 4.5e and 4.5f report results for the specification allowing different effects depending on the biological mother’s education. Here, too, there is some evidence of positive program effects (for different programs) in each of the two groups. This is also the case for the estimates in Figures 4.5g and 4.5h, which allow the effects to differ depending on living arrangement. The estimates discussed to this point concern differences among respondents that are likely related to socioeconomic advantage or disadvantage. Although it is difficult to synthesize the disparate estimates, some summary is possible. If whites, those with high ASVAB scores, those with more-educated mothers, and those living with two biological parents are regarded as relatively advantaged, then a couple of conclusions emerge. First, job shadowing and mentoring have some beneficial effects on college enrollment, but only for more-advantaged individuals. There is some evidence of positive effects of school enterprise and internship/apprenticeship STC programs on college enrollment for both more- and less-advantaged individuals and similarly some evidence of negative effects of Tech Prep programs on both moreand less-advantaged individuals. Turning to employment, the evidence of positive effects of co-op STC programs is strongest for more_____________ 27Finally, recall the higher participation rates of blacks in co-op, school enterprise, and Tech Prep programs, which were documented in Figure 4.1a. The evidence in Figures 4.5a and 4.5b suggests that this higher participation does not reflect higher returns to STC for blacks; the alternative, presumably, is that blacks for some reason are more likely assigned to such programs or choose to participate in such programs despite no greater benefit for them—an interesting issue but one that is beyond the scope of this report. 57 Figure 4.5c—College Enrollment Probabilities, Effects of STC Participation, Differences by ASVAB Math Knowledge Scores Figure 4.5d—Employment Probabilities, Effects of STC Participation, Differences by ASVAB Math Knowledge Scores 58 Figure 4.5e—College Enrollment Probabilities, Effects of STC Participation, Differences by Biological Mother’s Education Figure 4.5f—Employment Probabilities, Effects of STC Participation, Differences by Biological Mother’s Education 59 Figure 4.5g—College Enrollment Probabilities, Effects of STC Participation, Differences by Living Arrangement Figure 4.5h—Employment Probabilities, Effects of STC Participation, Differences by Living Arrangement 60 advantaged individuals, whereas the evidence of positive effects of internship/apprenticeship programs arises only for less-advantaged individuals. Job shadowing appears to have some negative effects for both types of individuals. Thus, the combined evidence does not indicate that the benefits of STC accrue primarily or even relatively more to minorities and other less-advantaged students. One exception, though, is that there is some indication that internship/apprenticeship programs may be particularly advantageous for the less-advantaged, as these programs boost college enrollment among those with the lowest test scores and boost employment among blacks and those with less-educated mothers and in nontraditional living arrangements. Finally, Figures 4.5i and 4.5j break out the results separately for men and women. There are some similarities with the combined estimates. In particular, the findings that school enterprise STC programs boost postsecondary education and that co-op programs boost employment appear to hold for both men and women. But there are also some differences. The finding that internship/apprenticeship programs boost employment is driven entirely by men, as for them the estimated effect (0.13) is positive (and significant), whereas for women it is near zero. It appears that job shadowing boosts postsecondary education for men, while internship/apprenticeship programs boost it for women. On the other hand, there is stronger evidence of a negative effect of Tech Prep on the postsecondary education of women. However, for the estimates disaggregating the effects of STC by sex, the differences between the estimated effects (evaluated one program at a time) were never statistically significant at the 10 percent level. This differed from the case for the other breakdowns, where there were always significant differences across groups for at least one of the STC programs. Thus, although the estimated sex differences are intriguing, they should be given less weight. Summary The empirical research presented in this chapter attempts to estimate the causal effects of participation in school-to-career programs on further education and employment in the years immediately after individuals 61 Figure 4.5i—College Enrollment Probabilities, Effects of STC Participation, Differences by Sex Figure 4.5j—Employment Probabilities, Effects of STC Participation, Differences by Sex 62 leave high school. Although the data analyzed are national in scope, the types of programs covered overlap substantially with those that were to some extent spurred by STWOA in California. The new NLSY97 data used in this chapter offer unparalleled opportunities to study the effects of participating in STC programs while confronting the classic problem in program evaluation—namely, pre-existing differences between STC participants and nonparticipants in the likelihoods of post-high school enrollment or employment. In particular, the NLSY97 offers a detailed battery of STC participation questions asked of respondents, data on later work and schooling expectations from questions asked before STC participation, repeated observations of individuals in the same schools, and a rich dataset generally. The analysis taking advantage of these features of the NLSY97 leads to a relatively robust set of findings. As at the end of the last chapter, Figure 4.6 summarizes the findings from this chapter. The evidence indicates that school enterprise STC programs boost post-high school education, and Tech Prep may reduce it, whereas cooperative education and internship/apprenticeship STC programs boost post-high school employment.28 The magnitudes implied by the estimates are reasonable yet also sizable, suggesting that participation in school enterprise, co-op, and internship/apprenticeship STC programs boosts the probabilities of enrollment or employment by about 0.05 to 0.10 relative to a base college attendance rate of 0.50 and a base employment rate of about 0.60. Moreover, in each case of a positive effect, there is no offsetting negative effect on the other dependent variable, implying that these programs appear to induce higher enrollment or employment more at the expense of being neither enrolled nor employed than at the expense of an alternative “productive” activity. Finally, there is also some evidence of differences in the effects of STC programs across groups distinguished by race and ethnicity and other characteristics associated with socioeconomic status, and across men and women. The most important finding regarding these differences is that STC does not appear to be particularly beneficial for _____________ 28There is also evidence that job shadowing boosts college enrollment, but this finding is not as robust across the alternative statistical analyses. 63 Figure 4.6—Summary of Evidence on Effectiveness of STC on Postsecondary Enrollment and Employment disadvantaged students. Instead, there is some evidence of beneficial effects for all groups, although different programs deliver different benefits. One finding that perhaps does stand out, though, is that internship/apprenticeship programs may be particularly advantageous for the less advantaged. This latter finding suggests that further effort 64 should go into trying to establish which populations of students gain more from different types of STC programs. There is, of course, an inherent limitation in using evidence from the NLSY97 to provide information on the effectiveness of STC in California. In particular, evidence from a study like this one can only provide a sense of the likely effects of the specific STC programs used in California, as the NLSY97 analysis is based on individual reports of participation in STC programs that may correspond only approximately to the types of programs used in California. And given the diversity across local providers documented in Chapter 2, evidence from the NLSY97 restricted to California—even if it yielded large enough samples (which it does not)—would still not suffice. Especially given the diversity of programs that flourished under STWOA, direct evaluation of programs in the state will ultimately provide the best evidence. At present, though, the NLSY97 data provide the best evidence available. 65 5. Conclusions Under the federal School-to-Work Opportunities Act of 1994, California received about $140 million to fund the development of a school-to-career system. In particular, STWOA funds were directed toward the provision of general or broad-based STC programs and activities that served all students, helping guide students not only into careers that could be entered immediately after high school or with additional vocational or technical education but also into careers that entail higher education at four-year institutions. STWOA funds were used in California to establish an extensive statewide system of Local Partnerships that were involved in multiple aspects of STC but which clearly played an important role in implementing the types of general STC activities and programs encouraged by STWOA. When STWOA was not reauthorized, this funding dried up, and state funding to cover the gap has not been forthcoming. Although federal funding has continued for Tech Prep and state funding has continued for Career or Partnership Academies—which are much more narrowly focused—the loss of STWOA funds represents more than a one-third decline in funding for STC activities overall and will likely, over time, severely curtail broad-based STC programs and activities.1 Although other states have been more active in making up for the lost federal funds, California’s current budget crisis makes it unlikely that serious state funding of broad-based STC activities and programs will be contemplated in the immediate future. In addition, school reform efforts targeting educational quality have turned increasingly toward testbased standards. However, because a successful school-to-work transition is a critical determinant of socioeconomic success, the ability _____________ 1It is important to note that funding for vocational education writ large far exceeds funding for these programs and activities. Chapter 2 explains why STC is typically construed more narrowly. 67 of the educational system to put students on the path to successful school-to-work transitions should remain an important criterion in assessing educational success. Thus, it is important to focus attention on the ability of the types of broad-based activities and programs supported by STWOA to contribute to this success, especially because, as STWOA and the response of California (and other states) to it suggest, significant efforts may result from rather small investments. The main contribution of this report is a statistical analysis of data from the 1997 National Longitudinal Survey of Youth. This data source includes information on participation by high school students in the United States in a variety of STC programs. The STC programs covered in the NLSY97 overlap substantially with the types of broad-based STC programs encouraged by STWOA and supported in California with STWOA funds, and which would presumably receive further support were the state to restore the funds that disappeared when STWOA was not reauthorized. The NLSY97 data can be used to study the consequences of participation in different types of STC programs for the early career development of the sample members. In particular, given the relatively young ages covered by the sample thus far, the analysis focuses on employment and postsecondary education in the period immediately after leaving high school. Although these outcomes fall short of characterizing the entire school-to-work transition, employment and further education are the two central methods by which individuals acquire new skills and build their careers, and hence positive effects of STC program participation on either or both of these is likely to signal the beginning of a more successful school-to-work transition. An important strength of the NLSY97 dataset is that it provides a number of means of accounting for pre-program differences between STC participants and nonparticipants. Such pre-program differences could generate either overly optimistic or overly pessimistic assessments of the likely effects of STC participation on employment or enrollment, depending on whether those relatively more or relatively less likely to be employed or enrolled after leaving high school—independently of the effects of STC—participate in STC programs. This is a fundamental and well-recognized problem in estimating the effects of a wide variety of 68 government programs, yet one that plagues much of the previous research on the effectiveness of STC. The empirical analysis that this report presents covers the entire United States, because the NLSY97 dataset is not sufficiently large to carry out analyses specific to California. But evidence on the effects of STC participation gained from the data from all states should be informative about the effects of these programs in California, given that there is no reason to believe that these effects do not generalize across states. Of course, policymakers in California may be more interested in assessments of the effectiveness of the specific STC programs that have been established in various localities throughout the state. Although the NLSY97 data cannot speak directly to such questions, this report also presents a summary of findings from a compendium of evaluations of specific local STC programs supported by STWOA in California. Unfortunately, these evaluations fail to provide much decisive evidence of any kind on the effects of these STC programs on postsecondary employment and enrollment or, indeed, on any student outcomes. Thus, research using the NLSY97 provides the best evidence currently available for assessing the effectiveness of the types of broad-based STC programs supported by STWOA in California. Evidence from the NLSY97 The statistical analysis of the NLSY97 covers six types of STC programs: job shadowing, mentoring, co-op programs, school enterprises, Tech Prep, and internships/apprenticeships. Although these six types of programs by no means cover all of the programs and activities pursued by the LPs in California that were set up using grants under STWOA, many of these are among the most common programs and activities the LPs did pursue. Nationally, participation in each of these types of STC programs among high school students ranges from about 7 percent to 18 percent, and over 42 percent of high school students covered in the NLSY97 participated in at least one of the six types of STC programs. Considering first the effects of STC participation on postsecondary education, the evidence indicates that school enterprise STC programs boost college enrollment, with the best estimates suggesting that this type 69 of program boosts the probability of enrollment in higher education in the immediate post-high school period by about 0.13, or 27 percent (given a postsecondary enrollment rate of 0.50), which is a sizable effect. On the other hand, participation in Tech Prep appears to lower the probability of postsecondary enrollment by about 0.095, or 19 percent. In neither case is the effect on enrollment offset by an effect on employment in the opposite direction. If it were, then the conclusions would be more ambiguous. For example, if school enterprise boosts college enrollment but reduces postsecondary employment, then the overall effects on skill formation would be unclear, although the returns to education are likely higher. With respect to Tech Prep, however, there is some evidence that the decline in enrollment is accompanied by an increase in full-time work as opposed to part-time work, which may to some extent mitigate the apparent adverse effects of Tech Prep on further education. The evidence on the effects of STC participation on postsecondary employment point to positive effects from the two types of STC programs that most closely combine high school education with work experience: co-op programs and internships/apprenticeships. Participation in co-op programs boosts the probability of employment in the immediate post-high school period by about 0.09, or 14 percent, and participation in internship/apprenticeship programs increases the probability of employment by about 0.07, or 11 percent, although the statistical evidence for the effects of internship/apprenticeship programs is a bit weaker. As with the enrollment results described above, these are sizable effects. Furthermore, the positive effects of co-op and internship/apprenticeship programs on postsecondary employment are not offset by reduced enrollment, so these results—as with the beneficial effects of school enterprise programs on postsecondary enrollment—indicate unambiguously positive effects on skill formation in the immediate post-high school period. That is, these programs appear to induce higher enrollment or employment mainly at the expense of being neither enrolled nor employed, rather than at the expense of an alternative “productive” activity. There is also some evidence of differences in the effects of STC programs across groups distinguished by race and ethnicity and other 70 characteristics associated with socioeconomic status, and across men and women, although the ability of the data to sharply distinguish among the effects on these different groups is somewhat limited. One finding that does appear to stand out is that internship/apprenticeship STC programs do have particularly beneficial effects for less-advantaged groups, because participation in these programs appears to boost college enrollment among those with the lowest test scores and to boost employment among blacks and those with less-educated mothers and in nontraditional living arrangements. Overall, though, the evidence does not provide a clear message that STC programs are more effective for less-advantaged than for moreadvantaged groups; instead, there is some evidence of beneficial effects for all groups, although different programs deliver diffferent benefits. The potential significance of these results stems from the argument often made by STC practitioners that STC programs are particularly helpful for less-advantaged youths or the broader group of those who in the absence of any intervention are unlikely to go on to college—often termed the “forgotten half.” The data analyzed in this report are not particularly supportive of this position, although they do suggest that some differences may exist—an issue that merits further attention in the implementation and evaluation of STC programs. On the other hand, STWOA was not motivated solely by problems experienced by the less advantaged in the school-to-work transition but was intended to serve students generally, including those who typically attend four-year colleges but would still potentially benefit from greater integration of learning and careers. Recommendations On balance, the evidence from the NLSY97 provides some indication that broad-based STC programs such as those supported by STWOA can increase postsecondary enrollment and employment. However, the prior research on Career Academies and the evidence on Tech Prep presented in this report do not make a compelling case for the effectiveness of these latter two programs, suggesting that at best they may boost the likelihood of full-time employment after leaving high school but with a tradeoff of lower postsecondary education. In light of 71 this evidence, the allocation of existing funding for STC activities merits reconsideration. In particular, with the demise of STWOA, funding has continued for Career Academies and Tech Prep, whereas direct funding for the broad-based types of STC programs (apprenticeships, internships, co-op programs, school enterprises, and so on) has all but disappeared. The evidence from the NLSY97 and the other available research is not overwhelming, but it appears that a case can be made for restoring some funding to the latter types of programs and, barring that, possibly reallocating funds from Career Academies, which are state funded, or from Tech Prep, if possible within the constraints of the federal funding. More generally, the state spends heavily on school reform efforts broadly defined. If STC delivers gains in terms of increasing the socioeconomic success of students, then there is no reason that the larger pool of state spending on educational reform efforts should not be considered as a potential source of funding for STC. At the same time, it is important not to interpret the evidence from the NLSY97 too strongly in the context of STC programs in California. Although this evidence suggests that the type of broad-based programs supported by STWOA in California may have been effective, this evidence is not specific to the efforts of the LPs established in California under STWOA. There is likely enough heterogeneity in STC programs across local jurisdictions that evaluation of local efforts is clearly warranted before drawing firm conclusions. Although there have been many evaluations of the STC efforts of these LPs in California, these evaluations are inadequate for establishing the effectiveness of these efforts and for determining which types of efforts have been most effective. The lack of evidence on the effectiveness of local STC efforts generated in the “round” of STC activities spurred by STWOA, and the limitations inherent in directly applying the findings from the NLSY97 to California’s STC programs, suggest that any future funding increases for broad-based STC activities should be predicated on requiring evaluation of programs. At the same time, given the dearth of evidence on the effectiveness of Career Academies and Tech Prep in California, these two other components of the STC system should be subject to the same evaluation requirements so that—with time—policymakers will be 72 able to make more informed judgments about the best way to allocate funds to the various components of the state’s STC system. Furthermore, the evidence from the NLSY97 on differential effects across subgroups of the population, although not strong, suggests that it would be valuable if future evaluation efforts were designed to look for such differential effects, as this may lead to more effective use of STC resources. Past experience has also demonstrated that simply mandating evaluations is insufficient. Tech Prep provides a good example of this. Perkins III requires accountability of the states in terms of a number of indicators. These cover postsecondary enrollment and employment—in particular, “Placement in, retention in, and completion of, postsecondary education or advanced training, placement in military service, or placement or retention in employment” (U.S. Congress, 1998, § 113). Yet the limitations of the state’s Tech Prep evaluation—in particular its lack of focus on student outcomes—was already noted (Chapter 3). Furthermore, a U.S. Department of Education report (1995) concluded that throughout the nation, few Tech Prep consortia were able to obtain student outcome data such as employment or further education, echoing the conclusions of California’s Tech Prep evaluation seven years later (O’Driscoll et al., 2002d). Similarly, there are evaluation requirements for state-funded Career Academies in California, but such evaluations do not appear to have been performed. Part of the reason may be language in the Education Code that provides a loophole for districts that do not wish to be evaluated. For example, § 54697 (a) of the California Education Code states, “The Superintendent of Public Instruction shall select an entity (the evaluating entity) to conduct a long-term evaluation of the Partnership Academies conducted pursuant to this article using a random assignment of pupils into program and control groups. The participation of any school district in this long-term evaluation is voluntary.” In addition, no funds have been appropriated for such an evaluation; instead, subsection (c) of the same section of the code states that the evaluating entity is responsible for securing the funding for the evaluation. 73 Finally, some of the latest state legislation on STC—AB 1873 (followed up by AB 1765)—specified accountability criteria based on past performance to be eligible for new grants. Among these criteria are “increased academic performance, postsecondary enrollment, decreased dropout rates, transition to appropriate employment, apprenticeship, or any other job training school when applicable, and measurements of pupil, parent, and employer satisfaction” (paragraph 7). But our interviews with LPs indicated that because they were under no mandate to collect data on these measures previously, most had difficulty establishing this accountability, and it is probably safe to say that few if any could do so reliably. This past experience with mandated evaluations, coupled with the poor quality of local efforts to evaluate the effectiveness of LPs established in California under STWOA, suggests that rigorous evaluation should be an important component of future legislation in support of STC activities. Clearly, the mandated evaluation will have to go beyond the types of mandates used in the past. In particular, mandating evaluation but allowing participation on a voluntary basis, or requiring funding from an external source, is unlikely to produce the type of evaluation—if any—needed to reliably assess the effectiveness of STC. And the quality and variability of the local evaluations of activities supported by STWOA indicate that assessments of STC activities would greatly benefit from outside technical assistance and uniformity in designing data collection efforts and ensuring the quality of the evaluations. Finally, the “gold standard” in evaluating most programs is in many ways experimental evidence based on random assignment, because such methods provide a simple solution to the problem posed by unobserved pre-program differences between participants and nonparticipants. Although not a panacea, experimental evidence is likely to be viewed as the most compelling. At the same time, such data as the NLSY97 do offer compelling nonexperimental approaches to this problem. Nonetheless, it is obviously worth considering whether evaluations of 74 STC programs in California could be based on experimental designs.2 And, finally, whether experimental or nonexperimental, evaluation efforts need to be extended beyond estimating the effect of one program or a set of programs. In particular, the evaluation efforts have to be extended to cost-benefit analyses and to comparisons of alternative methods of improving school-to-work transitions of California’s youths, including comparisons between STC efforts broadly defined to include all three components of California’s STC system and school reform efforts focused on other outcomes such as test-based standards. _____________ 2A useful illustration of how evaluation can lead to better policy is suggested by welfare reform. In the early 1990s, states requesting waivers from federal welfare regulations to try implementing their own welfare reforms were required to engage in “serious” evaluations of their waiver programs (typically using random assignment experimental designs). Blank (2002, p. 1122) describes the results of these evaluations—in particular the positive results of welfare-to-work evaluations—as having contributed importantly to the work-oriented federal reforms adopted in 1996. A context closer to STC programs where random assignment has been successfully applied is in the analysis of the effectiveness of the Job Corps program (see, for example, Burghardt et al., 2001). 75 Appendix A National Research on the Effectiveness of STC This appendix provides a more detailed discussion of research on the effectiveness of STC based on evidence from outside California, expanding on the brief discussion in Chapter 3. The 1994 NCRVE report provided a thorough compendium of research on STC programs through that date (Stern et al., 1994). The research summarized in this compendium provides little persuasive evidence of positive effects of these programs on adult labor market outcomes. First, many of these studies do not construct a reasonable comparison group, let alone consider the problem of selection into the program on the basis of unobserved pre-program differences. Second, even those studies that attempt to construct a good comparison group find no beneficial shortterm labor market effects, with the possible exception of those students who remained with the employer with whom they “apprenticed” during the program. Finally, some of the evidence suggests that STC programs may discourage postsecondary education. A subsequent NCRVE report (Urquiola et al., 1997) provides an update on STC. Reflecting the still scant progress toward successful evaluations of STC programs, most of this report focuses on implementation issues. The authors of the report echo a number of concerns about evaluations of STC programs that others have expressed, in particular emphasizing both the specific issues involved in drawing a causal inference and broader issues of how to define success and how to evaluate what some view as a systemic change. However, the report does discuss a couple of new studies of the effects of STC programs on postsecondary outcomes with some attention to pre-program differences. First, an evaluation of Wisconsin’s Youth Apprenticeship Program in printing, by Orr (1996), reported some evidence of higher employment, higher incidence of full-time work, and 77 higher wages for participants, relative to those who took more conventional vocational printing programs. Yet there are some differences between the treatment and control groups, as the evaluation’s author acknowledges. In particular, the apprenticeship program does not serve poorly performing students or those perceived as less likely to be hired by employers, which raises serious doubts about drawing any causal conclusions from the evidence. Second, an evaluation of the Manufacturing Technology Partnership Program in Michigan, by Hollenbeck (1996), reports mixed evidence of increased college enrollment and some evidence of higher employment. In this case, though, for one of the two cohorts studied the control group includes those not selected for the program and those who dropped out—and it is for this cohort that the positive effect on college education appears—making a causal inference suspect. And for the other cohort, the control group includes nonparticipants and nonapplicants, again raising issues of pre-program differences. A more recent major study does little to improve matters. In particular, a report on STWOA by Mathematica, Inc.—the national evaluation of STWOA for the U.S. Congress mandated by the act—does not even attempt to provide a program evaluation, arguing that STC implementation “generally involves broad and diverse initiatives that in varied ways touch most or all students, so it is impossible to distinguish between participants and an unaffected comparison group” (Hershey et al., 1999, p. xviii).1 Nonetheless, this report does present some evidence that is intended to speak to the effects of STC programs. For example, the report notes that students in paid positions arranged as part of STC programs are employed in a wider array of industries and receive more training than other students in paid positions, and it concludes that “Schools develop positions in a wide range of industries, increasing the chances that students can work in a setting relevant to their career interests” (p. 89). However, students who found these jobs as part of _____________ 1Given that the Mathematica report documents variation across school districts and states in the incidence of school-to-work partnerships supported by grants under STWOA, there seems to be a natural way to construct treatment and control groups. Instead, the report focuses on eight states all of which have a high percentage of school districts covered by such partnerships. 78 STC programs may have found the same types of jobs absent such programs; students most likely to do so may simply have sorted into STC programs. Furthermore, there is no evidence of beneficial effects on postsecondary enrollment or employment.2 To some extent paralleling this view of the existing evidence, a recent survey of published academic research on STC across the United States generally supports the claim that although STC is attracting increasing attention from researchers, the existing work has tended to shy away from trying to draw causal inferences (Hughes et al., 2001). Nonetheless, Hughes et al. draw rather rosy conclusions regarding STC, suggesting that “It is perhaps ironic that just as the major federal role in School-to-Work is winding down, the flow of evaluation research with positive findings is increasing” (p. 39). They reach this conclusion despite being able to cite only a handful of studies that use comparison or control groups, most of which do not appear to seriously wrestle with the problem of pre-program differences between participants and nonparticipants. And many of the findings cited in their survey do not appear to be based on comparisons between STC participants and nonparticipants. However, one important exception—which is also cited by Hughes et al.—is the recent (and ongoing) evaluation of Career Academies by the Manpower Demonstration Research Corporation (Kemple and Snipes, 2000; Kemple, 2001, 2003).3 The strength of this study is that it is based on random assignment of students to Career Academies, as participants were chosen randomly from applicants to the Career Academies in the study, with both participants and nonparticipants followed. The results from this study have been reported at different stages as the study participants have aged. The first paper based on this study (Kemple and Snipes, 2000) focused on the effects on participants while in high school, which is not the focus of this report. But it found some beneficial effects, such as increased exposure of students to career or _____________ 2Even if the evidence of a higher likelihood of working in a setting relevant to their career interests is taken at face value, this does not necessarily imply that such workers are doing better. 3The study covers nine schools across the nation, all located in or near urban areas. 79 technical courses and better performance in high school (improved attendance, more credits earned). The second paper (Kemple, 2001) considered students one year after the scheduled completion of high school and found no effects on high school graduation rates, postsecondary education, or employment. However, the most recent paper (Kemple, 2003) presents stronger evidence of such effects. In particular, looking at participants one, two, three, and four years after their scheduled graduation from high school, the evidence for men points to higher full-time employment, earnings, wages, and hours among participants, with these differentials statistically significant and of rather similar magnitude in each of the four years after scheduled graduation. (For women, there is no evidence of such positive effects.) With respect to schooling, Kemple examines numerous measures, including whether the individual was ever enrolled in postsecondary education, his or her highest enrollment, degrees earned, and so on. The evidence for men generates many negative differentials for participants relative to nonparticipants, although only the overall “ever enrolled” differential is statistically significant. Kemple interprets the combined evidence in an entirely positive light, concluding that “The Career Academy-induced improvements in labor market prospects did not come at the expense of opportunities for these young people to enroll in, progress through and complete postsecondary education programs” (p. 1). However, these results could plausibly be read differently. In particular, the negative schooling effects do not appear to be significantly different from what would just offset, say, the greater full-time employment of participants (on the presumption that full-time employment precludes schooling). Thus, the evidence may be most consistent with Career Academy participants substituting full-time work for schooling. One piece of evidence consistent with this is that the wage and earnings differentials that Kemple reports are essentially constant over the four years since scheduled high school graduation, suggesting that participants are not experiencing greater “success” at building careers. Instead, a constant wage or earnings differential may largely reflect a difference owing to higher incidence of full-time work among Career Academy participants—especially because the earnings differential on a monthly 80 basis is about twice as large in relative terms as the wage differential. Thus, a more nuanced view of the evidence from the study is that participation in Career Academies, for men, does seem to increase the orientation toward work but possibly at the cost of less schooling. 81 Appendix B Empirical Methods for the NLSY97 Analysis The core empirical framework is to estimate, at the individual level, the relationship between employment and enrollment in the post-high school period and participation in STC during high school.1 STC participation is categorized by participation in specific types of programs discussed below. The analysis is based on separate models for postsecondary employment and enrollment. Given that employment could be very short-term, whereas college enrollment of any type is nearly always longer-term, the focus is on employment at the time respondents to the NLSY97 are surveyed and on whether the individual was enrolled in college at any time since leaving high school.2 The basic statistical framework used is straightforward. Because the NLSY97 has repeated observations on individuals in the same school, observations on individuals are indexed by both the individual (i) and the school (j). Generically, let Yij be an indicator equal to one when individual i in school j is employed or has enrolled (depending on the analysis), and let STCij be a dummy variable for whether the individual reports participating in an STC program (a set of dummy variables for participation in a variety of STC programs is actually used in the empirical implementation). Linear regressions for Yij as a function of these variables are estimated, of the form: _____________ 1Other outcomes were also considered, including criminal activity, single parenthood, type of postsecondary education, and so on. But the combination of lower incidence of these different or more detailed outcomes and participation rates of about 0.10 to 0.20 in most types of STC programs precluded obtaining sufficiently precise estimates in these other analyses. 2Those who report their employer as the military are coded as employed. 83 Yij = a + bSTCij + e ij . (B.1)3 The estimates of b are denoted b. When a regression is estimated for a variable such as interpreted as the Yefi†fj etchtaot†fcSanTCtakpea†rotnicoipnaltyiotnheovnaltuheesporfozbearboiloitryoonfet,hbe is outcome; for e†xample, a coefficient estimate of 0.1 would imply that STC participation boosts the probability of enrollment, for e†xample, by 0.1, or 10 percentage points. Hence the regression model in this case is referred to as a “linear probability” model.4 Unbiased estimation of b requires that there be no unobserved pre- program differences—captured in eij, which implies that they affect Yij—between STC participants and nonparticipants. As in most research on program evaluatio†n, this cannot necessarily be assumed to hold, and if it does not, then b is a bi†ased estimate of the causal effect of STC programs. Neither longitudinal analysis nor data based on random assignment are available to study the effects of STC programs generally.5 As a result, in this research other methods have to be considered. Each of these methods is subject to some criticisms—but indeed the same can be said of social experiments and longitudinal estimation (in other contexts). The hope is that the “collage” of evidence from the alternative _____________ 3Reported standard errors are robust to heteroscedasticity. Also, because there are often multiple observations on students within the same school, the reported standard errors allow for heteroscedasticity of the error term across schools and nonindependence of an arbitrary nature within schools, which typically results in somewhat larger standard errors. 4Versions of all models discussed in this report were also estimated using logit specifications, and the results were very similar. Using the linear probability model simplifies the presentation and avoids distributional assumptions that underlie the logit (or probit) model. 5In general, longitudinal data on outcomes and participants before and after participation can be used to estimate causal effects of programs. In this approach, the measure of the outcome before the program participation captures the unobserved individual characteristics that might be associated with participation, and the change in the outcome then provides a causal estimate, assuming that the change in outcome did not occur for any other reason. However, in the context of STC, because the object of study is the effects of a program on individuals’ first labor market experiences, or on further school enrollment of those already enrolled, there are no meaningful observations on the outcomes of interest prior to the program, so longitudinal estimation is inapplicable. As noted in Chapter 3, only one study of STC has managed to use random assignment. 84 approaches, along with some analysis of whether each approach is likely to lessen any biases rather than exacerbate them, will help pin down the range of causal estimates of the effects of STC. Controls and Proxies The first approach to unobserved differences across individuals is to introduce into Equation (B.1) an extensive set of controls for the factors that might be correlated with STC participation and might also affect post-high school enrollment and employment. These controls can be thought of in two ways. First, they may be direct measures of some of the factors that are thought to affect enrollment and employment outcomes. Second, they may be proxies for the unobservable factors. For example, parents’ income may have a direct effect on postsecondary enrollment because it influences the affordability of college education. Alternatively, parents’ income may serve as a proxy for the differences in labor market networks faced by children from more affluent families—such as the likelihood that their networks are more rooted in jobs requiring a college degree. The distinction between control variables and proxies is important because when a particular variable is viewed as a proxy rather than a control for something that directly affects the outcome, specific questions arise as to whether it is an acceptable proxy for the unobserved differences across individuals, in the sense that its inclusion eliminates the bias from unobservable or unmeasured pre-program differences. To eliminate the bias, proxy variables have to satisfy three conditions. First, they must be related to the factors underlying the unobserved differences. Second, they must be redundant in the equation, meaning that in the hypothetical case of controlling for STC and the unobservable that actually affects behavior, the proxy has no direct effect on the dependent variables. Third, they have to capture enough of the variation in the unobservable so that once the proxies are included there is no remaining bias from unobserved pre-program differences in the estimates relating postsecondary outcomes to STC participation. As an example, if “career orientation” is viewed as a key unobserved difference across individuals, then the inclusion of variables capturing 85 race and sex in the regression would appear unlikely to satisfy the second or third conditions for a proxy. With regard to the second condition, race and sex may directly affect postsecondary outcomes even after accounting (again, hypothetically) for career orientation; with regard to the third, it seems likely that even after controlling for race and sex, there would still be remaining variation in career orientation that would be correlated with STC participation and the dependent variables. On the other hand, variables capturing work and schooling plans and expectations might more reasonably be viewed as adequate proxies; this case is discussed more below, because the NLSY97 contains information on such expectations. Of course, one can never know with certainty that a given set of proxy variables adequately captures the unobservables related to preprogram differences. But comparing the estimates of the coefficients of STC participation using a narrow set of control variables and a detailed set of proxy variables (that are assumed to satisfy the first condition) can help to gauge whether biases from unobservables remain. Specifically, if the inclusion of the detailed proxy variables has little or no effect on the estimates, then because their inclusion reduces the bias from pre-program differences, it is arguably less plausible (more so the more complete the set of proxy variables) that remaining unobservables generate a correlation between STC participation and enrollment or employment (Wooldridge, 2002). The NLSY97 offers a detailed set of control or proxy variables for characteristics of both the individual respondents and their families. In addition to fairly typical demographic variables (sex, race, ethnicity, and age), the dataset includes three additional sets of variables that are potentially important. These include data on living arrangements and the respondent’s family, test scores from the ASVAB, and self-reported measures of school behavior. This rich array of variables seems likely to capture some and perhaps a good deal of the variation in underlying propensities for post-high school enrollment or employment, including the quality and quantity of human capital investments families have made in their children, resources available in the household, educational norms in the family, labor market networks, the individual respondent’s academic intelligence, and the extent to which the respondent is learning 86 traits such as timeliness and reliability that are valued by employers. But these variables may also directly affect enrollment and employment outcomes, so they perhaps should not be viewed as proxy variables. Work and Schooling Expectations The NLSY97 data also include a particularly compelling set of proxy variables for trying to uncover the causal effects of STC. Specifically, the NLSY97 includes the respondent’s self-reported subjective probabilities for future education and employment, including receipt of a high school diploma by age 20, obtaining a four-year college degree by age 30, and working over 20 hours per week at age 30. These variables were measured in 1997, before the STC participation used in the estimation. The intuition behind using the work and schooling expectations is as follows. Before participating in STC, students are asked about their post-high school work and schooling expectations. Then some participate in STC and some do not, and their post-high school work and schooling behavior is subsequently observed. If, for example, conditional on educational expectations, STC participants are more likely to be enrolled in college after leaving high school, then it is quite sensible to infer a causal effect of STC, because the expectations questions should have controlled for remaining unobservables associated with post-high school educational decisions or outcomes. More formally, these expectations variables may come close to serving as perfect proxy variables. They should easily satisfy the redundancy condition because they should play no independent role net of the unobserved propensities for post-high school enrollment and employment for which they are proxies. And they should satisfy the third condition for a proxy—that once they are included, there should be no remaining unobserved pre-program differences between STC participants and nonparticipants that are related to postsecondary enrollment or employment. 87 Across-School vs. Within-School Variation in STC Participation: School Fixed Effects Finally, there may be other unobserved pre-program differences between STC participants and nonparticipants that are not captured in any of the control or proxy variables, yet which vary systematically across students in different schools. If so, and if there are differences in STC offerings by school, then unless all school-level characteristics that affect postsecondary outcomes are held constant, estimates of Equation (B.1) may still be biased because of unobserved pre-program differences. This in fact appears to be a potentially serious problem. In the NLSY97, in addition to the survey questions administered to individuals, a 2000 survey of schools elicits information on STC programs offered by the schools attended by survey respondents. Estimated models of the relationships between school characteristics and school offerings of STC reveal that many school-level characteristics that are likely to be associated with post-high school outcomes for students are also strongly associated with STC offerings.6 For example, the proportions of students involved in four types of problematic behavior—truancy, pregnancies, alcohol possession, and drugs—are in many cases significantly related to school STC offerings. And teacher characteristics such as pay are also strongly associated with STC offerings. Because variation in offerings of STC programs at the school level is strongly associated with characteristics of schools that are likely to be related to post-high school outcomes for their students, using variation in STC participation that is partly driven by school-level variation in offerings of STC is likely to yield biased estimates.7 _____________ 6See Neumark and Rothstein (2003) for more details and the estimation results. 7The school-level information on offerings of STC programs raises the possibility of using school-level variation in STC programs to estimate the effects of STC programs. In particular, a simple approach to try to assess whether STC programs increase post-high school education or employment would be to look directly at school-level estimates of these post-high school outcomes as functions of the STC programs that schools offer. Such regressions could be easily interpreted from a policy perspective, as they attempt to ask whether getting more STC programs to be offered by schools increases post-high school enrollment or employment. However, for these regressions to correctly answer this question, the observed variation in school STC offerings has to be unrelated to school-level differences in factors 88 The NLSY97 has one additional important feature that can be used to address this problem. Specifically, there are data on multiple students in the same school. Because of this feature, dummy variables for each school in the sample, or school fixed effects, can be added to Equation (B.1) to control for all unobserved factors that are common to students within a school. Estimation of the linear probability model including the school fixed effects (the “school fixed-effects estimator”) holds constant all other unobserved factors common to students in the same school. It therefore identifies the effects of STC participation from the withinschool differences between those who do and do not participate in STC and the differences in outcomes associated with this participation and hence does not rely at all on variation in STC participation that may be driven by the problematic across-school variation in offerings of STC. Of course, since individuals within schools differ from one another, in the within-school estimation attention must still be paid to individuallevel differences. That is, although students in the same school may have much in common, there is no reason to believe, for example, that their career orientations or other factors associated with STC participation and with postsecondary enrollment or employment do not differ. The hope, though, is that the combination of school fixed effects plus other individual-level controls and proxies will fully account for any preprogram differences between STC participants and nonparticipants. Analysis Samples The analysis is based on data from the first four rounds of the NLSY97. When the first round was administered, in 1997, respondents were ages 12–17. With the second round, then, more observations on respondents who have left high school become available. But the numbers go up considerably with the third and fourth rounds, and therefore the research focuses on educational and employment outcomes ______________________________________________________________ influencing postsecondary outcomes for these students. In contrast, for example, if STC programs tend to proliferate in underperforming schools, the beneficial effects of STC would be obscured in the across-school “experiment.” Unfortunately, as described in the text, variation in STC offerings does appear to be related to school-level characteristics. (This approach is closely related to using school offerings of STC as instrumental variables for individual participation in STC in Equation (B.1). The instrumental variables strategy is problematic for exactly the same reason.) 89 measured as of the third or fourth rounds, in 1999 and 2000.8 The samples used for the empirical analysis are obtained from the full NLSY97 sample in a number of steps. First, with respect to the basic sample, of the total 8,984 original respondents to the NLSY97, 8,510 were interviewed in 1999 or 2000 (Rounds 3 and 4). Of these, we restricted our attention to those ages 18 or older, which eliminated about 40 percent of the sample (about 3,500 observations), and to those for whom there is at least one observation (in 1999 or 2000) at which they are not enrolled in high school, yielding 4,234 observations. Because the latest observation comes from Round 4, the latest age from which the postsecondary work and schooling observations is taken is just over 21, and the mean is 19. The sample is restricted in this way to isolate those individuals for whom it is possible to observe the early years of their employment or higher education after leaving high school. In addition to meeting these criteria, information on STC participation is also required. To get an accurate reading on STC participation, and to be able to measure some behaviors and expectations as of a well-defined date before measured STC participation (given that these are measured in Round 1), the focus is on the information provided in the surveys after the first round in 1997, which for each subsequent round covers participation in the past year.9 Requiring STC information after Round 1 drops sample observations for those who did not answer the STC part of the survey after this round, either because they had not spent time in high school in 1997 or a subsequent year or in subsequent years were not enrolled as of the interview date and hence were not asked the STC questions. Coupled with some final sample restrictions on availability of the other data used in the study, this takes the sample down to 3,279 observations. For the baseline analysis sample, private schools and vocational/technical schools are excluded, leaving a sample of 2,933 observations. Finally, for two of the analyses the sample is restricted further. First, although linear models with school fixed effects can be estimated for the _____________ 8Rounds past the fourth were not available when this research was completed. 9However, the robustness of the results to using information on any STC in which the respondent had ever participated was verified. 90 full sample, observations on individuals who are the sole observations from their school contribute no identifying information. Thus, the effective sample size for this analysis includes only individuals with multiple respondents for their school; there are 2,230 such observations. Second, the work and schooling expectations data from 1997 are asked only of those born in 1980 or 1981—the two oldest birth cohorts in the dataset—leaving 2,057 observations from the baseline analysis sample.10 _____________ 10Far fewer than a proportionate number of observations are lost because the other sample restrictions leave a sample of mainly older NLSY97 respondents. Appendix Table C.1 summarizes these sample restrictions and their consequences for the samples analyzed. 91 Appendix C Detailed Tables for NLSY97 Analysis Table C.1 Sample Construction and Analysis Samples Sample Inclusion Criteria Total sample in 1997 Interviewed in 1999 or 2000 Age ≥ 18 at 1999 or 2000 interview Not enrolled in high school by 1999 or 2000 Answered STC questions covering high school after Round 1 † Complete data on baseline controls Baseline analysis sample: exclude private and vocational/technical school students Subset of baseline analysis sample with multiple observations per school Subset of baseline analysis sample with data on work and schooling expectations (asked only of those born in 1980 and 1981) No. of Observations 8,984 8,510 5,047 4,234 3,347 3,279 2,933 2,230 2,057 NOTES: The 1999 or 2000 interview is used as the “post-high school interview,” choosing the earliest one at which the respondent is age 18 or older and no longer enrolled in high school. More restrictive sample inclusion criteria are imposed in each successive row, except for the last two rows. Baseline controls include race/ethnicity, education, and family structure (whether the respondent lives with one, two, or no biological parents, and which ones, and household size). When other control variables are introduced in the regression models, dummy variables indicating missing data are included. 93 Table C.2 Participation Shares and Linear Probability Estimates of Individual Characteristics Associated with STC Participation 94 Demographic Female Black Hispanic Age Living arrangement/family Urban Biological parent and stepparent Biological mother only Biological father only Other arrangement Household size Log household income Biological mother’s schooling ASVAB Arithmetic reasoning Word knowledge Paragraph comprehension Math knowledge Job Any STC Shadowing Mentoring Co–op Share Participating in Each Type of STC Program .415 .175 .084 .135 Estimates of Participation Models School Enterprise .067 .008 .063** .022 .011 –.048* –.003 –.026 .010 –.004 –.014** .006 –.001 .023 –.007 –.005 .012 –.033 –.016 –.035* –.025 –.033 –.005 .006 .000 .014 .022 .022 .005 .004 –.016 –.034*** –.015 .032 –.004 .002 .001 –.010 .043** .004 .007 .004 –.013 .008 .026 .002 –.000 .001 –.001 .004 .053*** .004 –.006 .002 .015 –.010 .090 –.015 –.005 .002* –.001 –.006 –.011 –.016 –.000 –.001 –.014 –.007 –.015 –.013 .002 –.006 .013 –.002 .006 –.012 –.011 –.006 .017 .015 –.012 Internship/ Tech Prep Apprenticeship .110 .097 –.027** .057*** .028 –.000 –.039* .003 –.015 .024 –.019 –.001 –.001 –.001 .010 .022* –.020 –.019 .004 .008 .024 –.001 –.002 –.000 –.016 –.048 –.002 –.011*** .002 .005** –.007 –.032*** –.001 .019* Table C.2 (continued) 95 Job School Internship/ Any STC Shadowing Mentoring Co–op Enterprise Tech Prep Apprenticeship School behaviors Threatened at school .060** .022 .008 .010 .009 .023 .018 Got into physical fight at school Late with no excuse 2+ times Absent 2+ weeks R2 –.010 –.009 –.072** .016 –.021 .018 –.059* .014 –.000 –.018 –.024 .009 .014 –.009 –.046** .011 .017 .020* –.022 .016 –.022 –.002 –.002 .016 –.003 .010 –.040** .013 NOTES: Figures 4.1a–4.1c are based on the results in this table. There are 2,933 observations. ***, **, and * indicate that the estimated coefficients are significantly different from zero at the 1, 5, and 10 percent level, respectively. In the data, anyone of Hispanic origin is coded as Hispanic, and blacks must be non–Hispanic. Students in private schools and vocational/technical schools are excluded. Dummy variables are included for missing data on some individual variables. STC participation is measured based on responses from the 1998, 1999, and 2000 interviews regarding STC participation while in high school; the 2000 data are used only if the 2000 interview is the post–high school interview defined in the notes to Table C.1. All independent variables are defined as of 1997. The ASVAB test scores are standardized to have mean 0 and standard deviation of 1 for the calibration sample that took the test. The subset of ASVAB scores used are those that make up the Armed Forces Qualification Test. Statistical inferences are based on standard errors that are robust to heteroscedasticity and were adjusted to account for the clustering of observations within schools, allowing for nonindependence within schools and heteroscedasticity across schools. Table C.3a Linear Probability Estimates of the Effects of STC Participation on College Attendance and Employment 96 Some College Employment Job shadowing Mentoring Co–op School enterprise Tech Prep Internship/apprenticeship Control variables included Demographic Living arrangement/family ASVAB School behaviors R2 (1) .015 (.026) .066* (.036) –.019 (.028) .112*** (.037) –.059** (.030) .045 (.032) X .053 (2) .016 (.025) .043 (.034) –.004 (.027) .100*** (.036) –.051* (.030) .021 (.031) X X .148 (3) .042* (.024) .029 (.031) .010 (.026) .083** (.033) –.042 (.029) .022 (.030) X X X .255 (4) .037 (.023) .026 (.031) .007 (.026) .088*** (.033) –.042 (.030) .021 (.030) X X X X .274 (5) .006 (.025) –.035 (.033) .079*** (.028) .025 (.037) –.000 (.028) .053* (.030) X .019 (6) .003 (.025) –.030 (.033) .078*** (.028) .018 (.037) –.005 (.028) .058* (.030) X X .030 (7) .004 (.025) –.030 (.033) .078*** (.028) .015 (.037) –.006 (.028) .060** (.031) X X X .030 (8) –.000 (.025) –.029 (.033) .078*** (.028) .016 (.037) –.007 (.028) .059* (.030) X X X X .037 NOTES: The results in Figures 4.2a and 4.2b are based on the results in this table. There are 2,933 observations in all of the specifications. The estimated effect on probability of college attendance (columns (1)–(4)) and employment (columns (5)–(8)) are reported. School and work outcomes are measured as of the post–high school interview (1999 or 2000). Asterisks denote statistical significance of the coefficient estimates, with ***, **, and * indicating that the estimate is significantly different from zero at the 1, 5, and 10 percent level, respectively. STC participation is defined as described in Table C.2. The standard errors allow for general heteroscedasticity and were adjusted to account for the clustering of observations within schools, allowing for nonindependence within schools and heteroscedasticity across schools. The sets of control variables are listed in detail in Table C.3b. There are dummy variables for a small number of cases with missing data for the following variables: log household income, biological mother’s schooling, ASVAB scores, and urban residence. Table C.3b Linear Probability Estimates of Control Variable Coefficients for Models in Table C.3a Specification Demographic Female Black Hispanic Age Living arrangement/family Urban Biological parent and stepparent Biological mother only Biological father only Other arrangement Household size excluding youth Log household income Biological mother’s schooling ASVAB Arithmetic reasoning Word knowledge Paragraph comprehension Math knowledge Some College Table C.3a, Col. (4) (1) Employment Table C.3a, Col. (8) (2) .090*** (.017) .060*** (.023) .008 (.024) .034** (.014) –.023 (.019) –.115*** (.027) –.020 (.027) .051*** (.015) .027 (.020) –.049* (.026) –.108*** (.021) –.088* (.050) –.068 (.044) –.015*** (.006) .010** (.004) .023*** (.003) –.022 (.023) .087*** (.028) .027 (.024) .190*** (.045) –.016 (.050) –.006 (.006) .001 (.005) –.010*** (.003) –.011 (.017) .047*** (.017) .014 (.017) .156*** (.016) –.012 (.020) .023 (.019) .000 (.019) –.009 (.020) 97 Specification School behaviors Threatened at school Got into physical fight at school Late with no excuse 2+ times Absent 2+ weeks Some College Employment Table C.3a, Col. (4) Table C.3a, Col. (8) –.078*** (.023) –.096*** (.025) –.039** (.020) –.121*** (.028) –.029 (.023) –.080*** (.030) .006 (.022) –.048 (.032) NOTES: All specifications correspond to Table C.3a, columns (4) and (8). See the notes to Table C.3a for details. 98 Table C.4 Linear Probability Estimates of the Effects of STC Participation on College Attendance and Employment, Incorporating Prior Work and Schooling Expectations Some College Employment Job shadowing Mentoring Co-op School enterprise Tech Prep Internship/apprenticeship Expectations High school diploma by age 20 Four-year degree by age 30 Work over 20 hours/week at age 30 R2 (1) .024 (.028) .019 (.039) .021 (.031) .113*** (.040) –.046 (.038) .012 (.036) (2) .014 (.027) –.008 (.038) .030 (.030) .104*** (.039) –.016 (.035) .016 (.035) (3) .017 (.030) –.007 (.041) .055* (.033) –.025 (.048) .031 (.033) .052 (.037) — .010 — (.072) — .428*** — (.035) — .054 — (.064) .271 .320 .051 (4) .018 (.030) .008 (.041) .052 (.033) –.019 (.049) .030 (.033) .052 (.037) .025 (.090) –.101** (.043) .226*** (.079) .057 NOTES: The results in Figures 4.3a and 4.3b are based on the results in this table. The work and schooling expectations data come from questions asked in Round 1 (in 1997) and are asked only of those born in 1980 or 1981—the two oldest birth cohorts in the dataset. Because of this restriction, there are 2,057 observations. See the notes to Tables C.2 and C.3a for details. All of the specifications include the demographic, living arrangement/family, ASVAB, and school behavior variables that are the same as those included in columns (4) and (8) of Table C.3a. The standard errors allow for general heteroscedasticity and were adjusted to account for the clustering of observations within schools, allowing for nonindependence within schools and heteroscedasticity across schools. 99 Table C.5 School Fixed Effects Linear Probability Estimates of the Effects of STC Participation on College Attendance and Employment Job shadowing Mentoring Co-op School enterprise Tech prep Internship/apprenticeship Hausman test for excluding school fixed effects, p-value School fixed effects Some College (1) .035 (.027) .018 (.034) .004 (.031) .091** (.038) –.070** (.036) .038 (.036) — (2) .063** (.030) .048 (.039) –.013 (.035) .133*** (.048) –.095** (.040) .055 (.041) .19 Employment (3) (4) –.019 (.028) –.031 (.037) .075** (.033) –.002 (.047) .011 (.032) .116*** (.035) — –.026 (.035) –.057 (.047) .102*** (.037) –.018 (.056) .036 (.041) .073* (.043) .23 XX NOTES: The results in Figures 4.4a and 4.4b are based on the results in this table. There are 2,230 observations, because only observations on respondents in schools with multiple observations are included. All of the specifications include the demographic, living arrangement/family, ASVAB, and school behavior variables that are the same as those included in columns (4) and (8) of Table C.3a. See the notes to Table C.3a for details. The standard errors are robust to heteroscedasticity. The Hausman test is computed for the vector of STC coefficients. The test is based on random versus fixed effects and comes from the regression form of the test with standard errors robust to general heteroscedasticity and nonindependence of observations within schools (Wooldridge, 2002, Chapter 10). 100 Table C.6 School Fixed Effects Linear Probability Estimates of the Effects of STC Participation on College Attendance and Employment, Specifications Interacting STC Participation with Respondent Characteristics 101 Race/Ethnicity White Black Hispanic (1) (1’) (1’’) Job shadowing Mentoring Co–op School enterprise Tech Prep Internship/ apprenticeship .119** –.166** .084* .113 –.082 –.042 .253** .067 .086 Job shadowing Mentoring Co–op School enterprise Tech Prep .159*** –.015 .084 .109** –.023 –.085 ASVAB Math Knowledge Biological Mother’s Score Education High Some School High Medium Low College or Less (2) (2’) (2’’) (3) (3’) .122* Some College .122** .050 .009 .065 Living Arrangement Lives with Two Biological Other Living Parents Arrangement (4) (4’) .086** .037 .145** .193* .005 .166** .113* –.222*** –.033 .175* –.037 .083 –.162*** .054 –.173** .052 –.048 .178** Employment .067 .069 .112 .132* .168*** .068 .090* .047 –.125** .137*** .002 –.132** .048 .053 Sex Men Women (5) (5’) .097** .001 .032 .087* .113 –.072 .149** –.125** .005 .100* .113** .088 Table C.6 (continued) 102 Race/Ethnicity Internship/ apprenticeship Range Observations White (1) .012 — 1,295 Black Hispanic (1’) (1’’) .220** — 503 .081 — 432 ASVAB Math Knowledge Score High (2) Medium (2’) Low (2’’) > .75 687 0 – .75 < 0 547 625 Biological Mother’s Education High Some School College or Less (3) (3’) Living Arrangement Lives with Two Biological Other Living Parents Arrangement (4) (4’) .017 .145** .033 —— — 889 1,191 1,230 .117* — 1,000 Sex Men Women (5) (5’) .133** — 1,084 .015 — 1,146 NOTES: The results in Figures 4.5a–4.5j are based on the results in this table. See the notes to Tables C.3a and C.5. Specifications include STC program participation interacted with dummy variables for each group; each numbered set of columns (e.g., (1) through (1’’)) corresponds to a separate specification. They include school fixed effects and are in other respects identical to Table C.5. The “high,” “medium,” and “low” cutoffs for the estimates in columns (2)–(2’’) were chosen to assign roughly one-third of each group to each subsample. Bibliography Bechtold, Deborah A., and Dixie L. King, Partnerships Plus! 2000–2001 Evaluation Report: Partnership Academy Outcomes for Kern County, Transforming Local Communities, Bakersfield, California, 2001. Blank, Rebecca M., “Evaluating Welfare Reform in the United States,” Journal of Economic Literature, Vol. 60, No. 4, December 2002, pp. 1105–1166. Burghardt, John, Peter Z. Schochet, Sheena McConnell, Terry Johnson, R. Mark Gritz, Steven Glazerman, John Homrighausen, and Russell Jackson, Does Job Corps Work? Summary of the National Job Corps Study, Mathematica Policy Research, Inc., Princeton, New Jersey, 2001. Butler, Michael, Jia Wang, Mikala Rahn, Patricia O’Driscoll, and Van Villanueva, Evaluation of the Verdugo School-to-Career Partnership: PLUS Evaluation of Clark Magnet High School, PublicWorks, Inc., Pasadena, California, 2002a. Butler, Michael, Jia Wang, Mikala Rahn, Patricia O’Driscoll, and Van Villanueva, Evaluation of the UNITE-LA School-to-Career Partnership: PLUS Evaluation of LAUSD Career Academies, PublicWorks, Inc., Pasadena, California, 2002b. California Association of Regional Occupational Centers and Programs, ROCP Facts at a Glance 2002–2003, San Juan Capistrano, California, 2003. California Department of Education, Fiscal and Administrative Services Division, Report on the Budget Act of 2002, Sacramento, California, 2002, available at http://www.cde.ca.gov/fiscal/budgetact/ report2002.pdf (retrieved May 29, 2003). California Department of Education, Regional Occupational Centers and Programs, Sacramento, California, 2003a, available at http:// www.cde.ca.gov/rocp/ (retrieved April 20, 2003). 103 California Department of Education, What is Tech Prep? Sacramento, California, 2003b, available at http://www.cde.ca.gov/shsd/aci/tp/ (retrieved April 11, 2003). California Department of Education, “2003–04 Final Budget Summary,” Sacramento, California, 2004a, available at http://www. cde.ca.gov/fiscal/budgetact/budsum0304.pdf (retrieved 2/4/2004). California Department of Education, “Education Budgets and Related Legislation,” Sacramento, California, 2004b, available at http://www. cde.ca.gov/fiscal/budgetact (retrieved 3/22/2004). California Department of Finance, “Budget Documents,” Sacramento, CA, 2003, 2004, available at http://www.dof.ca.gov/HTML/ BUD_DOCS/Bud_link.htm (retrieved 3/22/2004). California Employment Development Department, Sacramento, California (personal communication, Edward Armijo), October 2002. California Employment Development Department, “Implementation Grant Award Announcement,” Sacramento, California, n.d., available at http://wwwstc.cahwnet.gov/page.asp?page=gi_impgrantaward. htm&cat=content (retrieved May 29, 2003). California Office of Legislative Counsel, “Bill Number: AB 1765,” Sacramento, California, 2004, available at http://www.leginfo.ca.gov/ pub/bill/asm/ab_1751-1800/ab_1765_bill_20030729_enrolled.html (retrieved 3/22/2004). Cameron, Stephen V., and James J. Heckman, “The Dynamics of Educational Attainment for Black, Hispanic, and White Males,” Journal of Political Economy, Vol. 109, No. 3, June 2001, pp. 455–499. Career Academy Support Network, What Is a Career Academy? Graduate School of Education, University of California, Berkeley, California, 2003a, available at http://casn.berkeley.edu/ ProjectDescription.html (retrieved June 19, 2003). Career Academy Support Network, Career Academy Support Network Homepage, Graduate School of Education, University of California, Berkeley, California, 2003b, available at http://casn. berkeley.edu/ (retrieved June 19, 2003). 104 Christensen, Heidi, Ventura County School-to-Career Case Studies Evaluation, Ventura County Superintendent of Schools Office, Camarillo, California, 2002. Commission on the Skills of the American Workforce, “America’s Choice: High Skills or Low Wages,” National Center on Education and the Economy, Rochester, New York, 1990. Donahoe, Debra, and Marta Tienda, “Human Asset Development and the Transition from School to Work: Policy Lessons for the 21st Century,” 1999 (unpublished manuscript). Fattahi, Behrooz, Youth Leadership Bakersfield, 2001–2002: Program Evaluation, Greater Bakersfield Chamber of Commerce, Bakersfield, California, 2002. Forouzesh, Mohammed R., Evaluation Report: Impact of AB-1873 Funding in Improving STC Delivery, Costa Mesa, California, 2002. Forouzesh, Mohammed R., and Orange County Department of Education Instructional Services Division, Orange County Coalition/Vision 2020 School-to-Career Case Studies Final Evaluation Report 2001–2002, Costa Mesa, California, 2002. Glazer, Nathan, “A Human Capital Policy for the Cities,” The Public Interest, No. 112, Summer 1993, pp. 27–49. Hamilton, Stephen, Apprenticeship for Adulthood, The Free Press, New York, 1990. Hayward, G. C., and C. S. Benson, “The Changing Role of VocationalTechnical Education in the United States,” CenterWork, Vol. 4, No. 2, National Center for Research on Vocational Education, 1993. Heckman, James, “Assessing Clinton's Program on Job Training, Workfare, and Education in the Workplace,” NBER Working Paper No. 4428, Cambridge, Massachusetts, August 1993. Heckman, James J., Robert J. Lalonde, and Jeffrey A. Smith, “The Economics and Econometrics of Active Labor Market Programs,” in Orley C. Ashenfelter and David Card, eds., Handbook of Labor Economics, Vol. 3A, Elsevier, Amsterdam, 1999, pp. 1865–2097. 105 Hemsley, Rita, San Bernardino County Horizons Partnership: Year 4 Report, California Educational Research Cooperative, University of California, Riverside, California, 2002. Hershey, Alan M., Marsha K. Silverberg, and Joshua Haimson, Expanding Options for Students: Report to Congress on the National Evaluation of School-to-Work Implementation, Mathematica Policy Research, Inc., Princeton, New Jersey, 1999. Hollenbeck, Kevin, An Evaluation of the Manufacturing Technology Partnership (MTP) Program, Technical Report No. 96-007, Upjohn Institute, Kalamazoo, Michigan, 1996. Hughes, Katherine L., Stephen F. Hamilton, and Robert J. Ivry, Schoolto-Work: Making a Difference in Education, Institute on Education and the Economy, Teachers College, Columbia University, New York, New York, 2001. Kemple, James J., Career Academies: Impacts on Students’ Initial Transitions to Post-Secondary Education and Employment, Manpower Demonstration Research Corporation, New York, New York, 2001. Kemple, James J., “Career Academy Impacts on Labor Market Outcomes and Educational Attainment,” working draft, Manpower Demonstration Research Corporation, New York, New York, 2003. Kemple, James J., and Jason C. Snipes, Career Academies: Impacts on Students’ Engagement and Performance in High School, Manpower Demonstration Research Corporation, New York, New York, 2000. Lerman, Robert I., and Hilliard Pouncy, “The Compelling Case for Youth Apprenticeship,” The Public Interest, No. 101, Fall 1990, pp. 62–77. Lindler, Vanessa, and Laura Hill, San Mateo County School-to-Career Partnership Second Year Evaluation, The Sphere Institute, Burlingame, California, 2000. Maher, Fred, “Vo-tech’s Role in Public Education Debated at Forum,” Temple Times Online Edition, Vol. 33, No. 18, February 13, 2003, available at http://www.temple.edu/temple_times/2-13-03/votech. html (retrieved August 11, 2003). 106 Maxwell, Nan L., and Victor Rubin, Career Academy Programs in California: Outcomes and Implementation, California Policy Research Center, University of California, Berkeley, California, 2001. Neumark, David, “Evaluating the Effectiveness of the School-to-Work Opportunities Act in California,” Working Paper No. 2004.02, Public Policy Institute of California, San Francisco, California, 2004. Neumark, David, and Ann Allen, “What Do We Know About the Effects of School-to-Work? A Case Study of Michigan,” Journal of Vocational Education Research, Vol. 28, No. 1, 2003, pp. 59–84. Neumark, David, and Donna Rothstein, “School-to-Career Programs and Transitions to Employment and Higher Education,” Working Paper No. 2003.22, Public Policy Institute of California, San Francisco, California, August 2003. O’Driscoll, Patricia, Mikala L. Rahn, and Gina Jesse, UNITE-LA Schoolto-Career Evaluation: Final Report 2001–2002, PublicWorks, Inc., Pasadena, California, 2003. O’Driscoll, Patricia, Mikala L. Rahn, Pam Marx, and Tom South, Core Case Study Evaluation: Sonoma County School-to-Career Partnership, PublicWorks, Inc., Pasadena, California, 2002a. O’Driscoll, Patricia, Mikala L. Rahn, Pam Marx, and Shannon Faris, Core Case Study Evaluation: UNITE-LA School-to-Career Partnership, PublicWorks, Inc., Pasadena, California, 2002b. O’Driscoll, Patricia, Mikala L. Rahn, Pam Marx, and Van Villanueva, Core Case Study Evaluation: Verdugo School-to-Career Partnership, PublicWorks, Inc., Pasadena, California, 2002c. O’Driscoll, Patricia, Pam Marx, Tom South, Van Villanueva, Mikala L. Rahn, and Gina Jesse, Tech Prep Connections in California: The Right Direction for High Schools and Community Colleges, PublicWorks, Inc., Pasadena, California, 2002d. Office of Technology Assessment, United States Congress, Learning to Work: Making the Transition from School to Work, Publication No. OTA-HER 637, U.S. Government Printing Office, Washington, D.C., 1995. 107 Orr, Margaret Terry, Wisconsin Youth Apprenticeship Program in Printing: Evaluation 1993–1995, Jobs for the Future, Boston, Massachusetts, 1996. Schmidt, Tracy, A Survey of School-to-Work Actions: The 2000 State Legislative Sessions, National Conference of State Legislators, Washington, D.C., 2000. Schmidt, Tracy J., Inside the State House: The People and Ideas that Shaped School-to-Career Legislation, National Conference of State Legislatures, Washington, D.C., 2001. Stern, David, Charles Dayton, Il-Woo Paik, Alan Weisberg, and John Evans, “Combining Academic and Vocational Courses in an Integrated Program to Reduce High School Dropout Rates: SecondYear Results from Replications of the California Peninsula Academies,” Educational Evaluation and Policy Analysis, Vol. 10, No. 2, Summer 1988, pp. 161–170. Stern, David, Neal Finkelstein, James R. Stone, John Latting, and Carolyn Dornsife, Research on School-to-Work Transition Programs in the United States, National Center for Research in Vocational Education, University of California, Berkeley, California, 1994. Urquiola, Miguel, David Stern, Ilana Horn, Carolyn Dornsife, Bernadette Chi, Lea Williams, Donna Meritt, Katherine Hughes, and Thomas Bailey, School to Work, College and Career: A Review of Policy, Practice, and Results, 1993–1997, National Center for Research in Vocational Education, University of California, Berkeley, California, 1997. U.S. Congress, Carl D. Perkins Vocational and Applied Technology Education Amendments of 1998, Public Law 105-332, 105th Congress, October 31, 1998. U.S. Department of Education, The Emergence of Tech-Prep at the State and Local Levels—1995, Washington, D.C., 1995. U.S. Department of Education, Office of Vocational and Adult Education, “FY1998 Allocations for Carl D. Perkins Vocational and Technical Education Act,” Washington, D.C., 1998. 108 U.S. Department of Education, Office of Vocational and Adult Education, “FY1999 Allocations for Carl D. Perkins Vocational and Technical Education Act,” Washington, D.C., 1999. U.S. Department of Education, Office of Vocational and Adult Education, “FY2000 Allocations for Carl D. Perkins Vocational and Technical Education Act,” Washington, D.C., 2000. U.S. Department of Education, Office of Vocational and Adult Education, “FY2001 Allocations for Carl D. Perkins Vocational and Technical Education Act,” Washington, D.C., 2001. U.S. Department of Education, Facts about the Tech-Prep Program, Washington, D.C., 2002a. U.S. Department of Education, Office of Vocational and Adult Education, “FY2002 Allocations for Carl D. Perkins Vocational and Technical Education Act,” Washington, D.C., 2002b. U.S. Department of Education, Office of Vocational and Adult Education, “FY2003 Allocations for Carl D. Perkins Vocational and Technical Education Act,” Washington, D.C., 2003. U.S. General Accounting Office, Preparing Noncollege Youth for Employment in the U.S. and Foreign Countries, Washington, D.C., May 1990. WestEd and MPR Associates, Inc., California School-to-Career: Helping Students Make Better Choices for Their Future, Final Evaluation Report, WestEd, San Francisco, California, 2002. Wilson, Pete, “Executive Order W-94-94,” Executive Department, State of California, Sacramento, California, June 28, 1994. Wooldridge, Jeffrey M., Econometric Analysis of Cross Section and Panel Data, The MIT Press, Cambridge, Massachussetts, 2002. 109 About the Author DAVID NEUMARK David Neumark is a senior fellow in economics at the Public Policy Institute of California and a research associate of the National Bureau of Economic Research. He has published numerous studies on school-towork, workplace segregation, sex discrimination, the economics of gender and the family, affirmative action, aging, minimum wages, and living wages. He is on the editorial boards of Contemporary Economic Policy and Economics of Education Review. He has also held positions as professor of economics at Michigan State University, assistant professor of economics at the University of Pennsylvania, and economist at the Federal Reserve Board. He holds a Ph.D. in economics from Harvard University. 111 Related PPIC Publications The Changing Role of Education in the California Labor Market Julian R. Betts Math Matters: The Links Between High School Curriculum, College Graduation, and Earnings Heather Rose and Julian R. Betts “At Home and in School: Racial and Ethnic Gaps in Educational Preparedness” California Counts: Population Trends and Profiles Volume 3, Number 2, November 2001 Jennifer Y. Cheng PPIC publications may be ordered by phone or from our website (800) 232-5343 [mainland U.S.] (415) 291-4400 [Canada, Hawaii, overseas] www.ppic.org 113" } ["___content":protected]=> string(102) "

R 504DNR

" ["_permalink":protected]=> string(126) "https://www.ppic.org/publication/the-effects-of-school-to-career-programs-on-postsecondary-enrollment-and-employment/r_504dnr/" ["_next":protected]=> array(0) { } ["_prev":protected]=> array(0) { } ["_css_class":protected]=> NULL ["id"]=> int(8308) ["ID"]=> int(8308) ["post_author"]=> string(1) "1" ["post_content"]=> string(0) "" ["post_date"]=> string(19) "2017-05-20 02:36:28" ["post_excerpt"]=> string(0) "" ["post_parent"]=> int(3484) ["post_status"]=> string(7) "inherit" ["post_title"]=> string(8) "R 504DNR" ["post_type"]=> string(10) "attachment" ["slug"]=> string(8) "r_504dnr" ["__type":protected]=> NULL ["_wp_attached_file"]=> string(12) "R_504DNR.pdf" ["wpmf_size"]=> string(7) "4544045" ["wpmf_filetype"]=> string(3) "pdf" ["wpmf_order"]=> string(1) "0" ["searchwp_content"]=> string(221714) "The Effects of School-to-Career Programs on Postsecondary Enrollment and Employment ••• David Neumark 2004 Library of Congress Cataloging-in-Publication Data Neumark, David. The effects of school-to-career programs on postsecondary enrollment and employment / David Neumark. p. cm. Includes bibliographical references. ISBN: 1-58213-088-4 1. School-to-work transition—United States. 2. Vocational education—United States. 3. College attendance—United States. 4. High school graduates—Employment—United States. I. Public Policy Institute of California. II. Title. LC1045.N48 2004 370.11'3'0973—dc22 2004007828 Copyright © 2004 by Public Policy Institute of California All rights reserved San Francisco, CA Short sections of text, not to exceed three paragraphs, may be quoted without written permission provided that full attribution is given to the source and the above copyright notice is included. PPIC does not take or support positions on any ballot measure or on any local, state, or federal legislation, nor does it endorse, support, or oppose any political parties or candidates for public office. Research publications reflect the views of the authors and do not necessarily reflect the views of the staff, officers, or Board of Directors of the Public Policy Institute of California. Foreword One of the greatest concerns surrounding educational policymaking is whether students, especially disadvantaged students, are prepared for a successful transition from school to work. This is especially true during the high school years—a crucial time when young people need the education and motivation that will enable and encourage them to attend college or help them find higher-paying jobs immediately after they leave high school. Too often, those not going on to college simply drift from one lowpaying job to another—or worse, remain unemployed. This concern led to educational reform efforts in the 1990s that focused on school-towork programs or, as they are known in California, school-to-career programs. The School-to-Work Opportunities Act (STWOA), passed by Congress in 1994, provided more than $135 million to California over a five-year period to support increased school-to-career activities such as internships, apprenticeships, and mentoring of students on a part-time basis by local employers. STWOA was not reauthorized after its initial five years. And although certain other school-to-career programs still exist, mainly Tech Prep and Career Academies, this loss of funding appears to have left a gaping hole in efforts to prepare low-skilled youth for higher-paying jobs—the principal goal behind the original legislation. This is particularly problematical for California because, as research in other PPIC studies has shown, income inequality is higher in California than in the rest of the nation, and the difference between the incomes of the “haves” and “have nots” is largely attributable to education. Given the loss of federal STWOA funding, it is important to determine just how effective the activities supported by this program were and whether it might be in California’s best interest to restore some of the funding for these activities. In this report, David Neumark analyzes a national dataset that includes the results of broad-based iii programs similar to those established in California. He concludes that a case can be made for restoring or reallocating some funding to the types of activities supported by STWOA. The national results suggest that these activities are more effective in boosting postsecondary enrollment and employment than certain other types of existing school-to-career programs, such as Tech Prep. In a secondary and equally important analysis, Neumark also looks more specifically at existing evaluations of various programs in California, and he finds them wanting. He argues that there is a pressing need for a serious evaluation of the effectiveness of the school-to-career activities remaining in California and for any such activities that might arise in the future. He also notes that future school-to-career efforts in the state would be well served by better articulating the goals of the program and by establishing clear criteria for measuring progress toward these goals. There is compelling evidence that schooling is essential to the future well-being of California’s children. A large body of research has shown that the transition from school to a career—whether after high school graduation or postsecondary education—is a critical step in determining personal economic success. Neumark concludes that much remains to be done in California to assure that this critical step is successful. Failure will result in a setback not just for our students but for the many Californians—workers, employers, citizens, and consumers alike—who benefit from a more-skilled workforce. David W. Lyon President and CEO Public Policy Institute of California iv Summary “School-to-work,” or “school-to-career” as it is known in California, refers to programs that integrate academic and vocational skills with the goal of improving the transitions of youths from schools into their careers. This report uses national data from the 1997 National Longitudinal Survey of Youth (NLSY97) to evaluate the effectiveness of the types of school-to-career (STC) programs that were encouraged and supported in California by the grants received by the state from the federal School-to-Work Opportunities Act of 1994 (STWOA). In particular, the empirical analysis focuses on whether participation in these STC programs increases postsecondary college enrollment or employment. STWOA provided more than $1.5 billion over a five-year period to support increased school-to-work activities in the nation’s public schools. This money was made available to states to create STC systems entailing cooperation among schools, private business, and government bodies (Office of Technology Assessment, 1995). Congress passed STWOA in response to three areas of particular concern for public education identified by researchers and educators. These were (1) a lack of connection between school and work that led many youths to be unmotivated in school and to experience subsequent difficulty moving out of low-wage jobs, (2) youths completing school with insufficient skills needed for the labor market, and (3) increasing labor market demands for complex thinking, close teamwork, and the ability to learn on the job. More generally, STWOA was motivated by a concern shared by policymakers and researchers alike that school-to-work transitions of youths in the United States entail too much joblessness, job instability, and employment in dead-end jobs (for example, see U.S. General Accounting Office, 1990). To help young people develop the skills needed in the workforce and make better connections to careers, STWOA set out to increase (1) school-based initiatives such as career v links to academic curriculum and career awareness activities, (2) workbased activities such as job shadowing,1 internships, and apprenticeships, and (3) connecting activities, such as the development of partnerships with employers and postsecondary institutions. In 1996, California was awarded $130 million over five years from the National School-to-Work Office to set up an extensive STC system, and the state received a supplemental $7.2 million grant in 2000. The STC activities funded under STWOA can be viewed as one of three branches of STC activities in California. The other two branches are Career Academies (sometimes called Partnership Academies) and Tech Prep, which focus much more narrowly on specific student populations in contrast to the general and broad-based approach of STWOA. However, after its initial five years, STWOA was not reauthorized. Although funding for Career Academies and Tech Prep from the state or federal governments has continued, the state has not made up the shortfall of funding for general STC activities. Given this change in funding, it is a particularly opportune time to study the effectiveness of the types of general and broad-based STC activities in California that were encouraged and supported by the federal legislation, to ask whether the state might better achieve its goals by funding general STC activities. Despite the fact that present prospects for significant new spending on STC in California are slim, given the budget crisis, possibilities for minor new allocations or reallocation from the other two branches of STC or other areas of spending on education should not be ruled out, especially in light of the significant effect that the relatively small amounts spent in California under STWOA had on implementation of STC programs. Goals of the Research The central goal of this report is to evaluate the effectiveness of STC programs in improving labor market outcomes of individuals as they begin to develop their careers and move on to work, further education, or a combination of the two. Its particular focus is an econometric analysis of the NLSY97, estimating the effects of STC program _____________ 1In job shadowing programs, a student follows an employee for one or more days to learn about a particular occupation or industry. vi participation, nationally, on education and employment in the period immediately after leaving high school. Although enrollment and employment in the immediate postsecondary years capture only a segment of an individual’s career development, these are the two activities that are most likely to increase the skills acquired by youths making the transition from school to work. This research thus focuses attention on the transitional period that has been the concern underlying STC legislation, as well as most research on STC. The NLSY97 covers six types of STC programs that comprise many of the most common types of STC programs and activities offered in California by the local STC providers funded by STWOA. Thus, although this research does not directly address specific, local STC programs and activities offered in the state, it speaks to the effectiveness of these types of programs. The NSLY97 is invaluable for research on the effectiveness of STC programs because its richness provides a number of methods for handling a fundamental problem in the evaluation of the effectiveness of a government program. Specifically, accurate estimation of such program effects—or recovering the causal effects—requires a thorough accounting of differences between STC participants and nonparticipants that exist independently of program participation. Such estimation is a primary focus of this research, and at the same time the lack of such estimation is a major shortcoming of most of the existing research on STC programs in California and elsewhere. In addition to presenting this new evidence from the NLSY97, the report reviews past evidence on the effectiveness of STC and includes a detailed analysis of evaluations that have been conducted of specific STC programs and activities in California that were supported by STWOA. Unfortunately, these evaluations provide little if any convincing evidence of the effectiveness of STC programs, especially as regards postsecondary outcomes. Many of the evaluations fail even to use a control group, which is a prerequisite for evaluation. Most fail to use any control variables or other methods to account for differences between participants and nonparticipants, and it is probably safe to say that none does this adequately. Finally, few even look at postsecondary outcomes. The dearth of compelling evidence and the data limitations that vii characterize these evaluations of specific STC programs in California imply that—to date—the NLSY97 data are the best source of information on the effectiveness of the types of STC programs that STWOA spurred in California. Broader Policy Context In some respects, STC has been ignored in recent research and policy debates regarding educational reform. In particular, recent research on educational quality and reform has focused largely on testrelated outcomes, and this focus is strongly reflected in the No Child Left Behind Act of 2001 (NCLB), which emphasizes standardized testing in grades K–12. But another important perspective on educational quality concerns the link between education and labor market success. Before the introduction of NCLB, STC programs had become an integral part of high school education in many states, spurred in part by STWOA. Although there is a link between test scores and socioeconomic success, there is no reason to believe that test-based standards encompass all of what schools do to prepare students for the world of work. It is plausible and indeed likely that STC also has an important role to play. In one fashion or another, virtually all students move into the workplace. Therefore, for all students, the provision of information about careers, tools to make decisions about careers, and the acquisition of specific skills valued in the labor market would seem to be an important complement to the academic component of education. Moreover, many students do not attend four-year postsecondary institutions, and it is conceivable that the strong emphasis on testing may not serve these students as well as those bound for such institutions. None of this is to argue that STC is more important than a rigorous academic focus. But research that addresses the potential benefits of STC can help maintain a focus on the problems of the noncollege-bound and more generally on the link between education and labor market success. Indeed, some researchers see NCLB as likely to diminish government interest and investment in STC. Without prejudging the outcome, suffice it to say that there is good reason to keep STC “on the table” in the context of broader issues of educational reform. viii More generally, STC is one potential policy response to poverty and low-wage work. The principal goal of STC is to increase labor market skills, in part via direct acquisition of skills from participating in STC programs, and to a greater extent via longer-run effects associated with launching youths onto career paths with greater potential for growth of skills. A great deal of research and policy effort focuses on problems and policies related to poverty, including changes in the income or wage distribution, social programs to mitigate the consequences of poverty or attack its sources, and mandated higher wages for low-skill workers. Most policymakers and researchers would agree, however, that the ideal solution—albeit perhaps the most difficult to achieve—is to increase the skills and productivity of those at the bottom of the socioeconomic scale so that they can earn a market wage high enough to lift them and their families out of poverty. STC programs may be one way to help achieve that goal. Thus, research on STC has a critical role to play in assessing these efforts to raise skill levels.2 Principal Findings The evidence from the NLSY97 indicates that some STC programs (school enterprises) boost post-high school education,3 and other STC programs (cooperative education and internships/apprenticeships) boost post-high school employment.4 Tech Prep appears to reduce post-high school education, although it possibly may also increase the likelihood of full-time work. The magnitudes implied by the estimates are reasonable yet also sizable, suggesting that participation in the programs with positive effects boosts enrollment or employment by about 5 to 10 percentage points, relative to the baseline rates of about 50 percent attending college and 60 percent employed in the immediate postsecondary period. Moreover, in each case of a positive effect, there is _____________ 2Donahoe and Tienda (1999) see the problem of moving workers into higher-wage jobs as the principal school-to-work “problem.” 3“School enterprises” entail the production of goods or services by students for sale to or use by others. They typically involve students in the management of the project and may be undertaken on or off the school site. 4In “cooperative education” programs, students alternate or parallel their academic and vocational studies with a job in a related field. ix no offsetting negative effect on the other dependent variable, implying that these programs appear to induce higher enrollment or employment at the expense of being neither enrolled nor employed, rather than at the expense of an alternative “productive” activity. Finally, there is also some evidence of differences in the effects of STC programs across groups distinguished by race and ethnicity and other characteristics associated with socioeconomic status and across men and women. However, this evidence does not point to a consistent pattern indicating that STC is particularly or primarily beneficial to disadvantaged students. Instead, there is some evidence of beneficial effects for all groups, although different programs deliver different benefits. One finding that perhaps does stand out, though, is that internship/apprenticeship programs may be particularly advantageous for the less-advantaged, as these programs boost college enrollment among those with the lowest test scores and boost employment among blacks and those with less-educated mothers and living in nontraditional arrangements. Recommendations On balance, relying principally on the results from the NLSY97, the evidence provides some indication that the types of STC programs funded by STWOA in California increase postsecondary enrollment and employment. However, with the demise of STWOA, funding has continued for Career Academies and Tech Prep, whereas direct funding for general types of STC programs (such as apprenticeships, internships, co-op programs, and school enterprises) has been sharply reduced.5 Yet the evidence on Tech Prep and Career Academies does not make a strong case that these branches of STC are more effective, at best suggesting that they may boost full-time work but possibly at the expense of reducing education.6 This evidence suggests that a reconsideration of the _____________ 5As explained in Chapter 2, because there was some commingling of funds between STWOA activities and other STC or vocational educational activities, it is inaccurate to say that the demise of STWOA removed all funding for general STC activities. 6As explained below, this is despite the fact that Tech Prep entails coordination between high school and college courses. x allocation of existing funding for STC activities in California may be warranted. The evidence is not overwhelming, and this report does not present a full-blown cost-benefit analysis; but it appears that a case can be made for restoring some funding to the types of programs supported by STWOA and, barring that, possibly considering some reallocation of funds toward these types of programs. Another important observation that emerges from the research is that there is a pressing need for serious evaluation of the effectiveness of the specific STC activities remaining in California and for any new activities that might arise in the future. The main evidence provided in this report comes from a national dataset that is informative about the types of STC activities in California that were spurred by STWOA. But in a subsidiary part of this research project, a survey of local providers of these activities and of evaluations of their efforts was conducted to compile and assess existing evidence on the effectiveness of STWOArelated STC activities in California. This survey revealed that specific evaluations of STC programs in California are lacking in many respects. Thus, any future funding increases for STC should be predicated on requiring evaluation of programs. Moreover, given the evidence of beneficial effects of general STC programs nationally, and the more ambiguous evidence regarding Tech Prep and Career Academies, the optimal strategy may be to restore some funding for these general programs (or to reallocate funding) and to mandate evaluations of all three types of programs. Only this type of effort holds the promise of more reliable conclusions to be drawn regarding the effectiveness of the different types of STC efforts implemented in the state. However, simply mandating evaluation is not enough. The survey of local providers revealed not only inadequate evaluations but also wide variation in local providers’ self-reported criteria for success and data collection efforts. These findings suggest that future STC efforts in California would be well served by better articulating the goals of the program and establishing clear criteria for measuring and assessing progress toward these goals. More important, the weaknesses of the existing local evaluations of STC efforts in California strongly indicate that evaluation of STC effectiveness in the state would be well served by xi substantive expert assistance in designing data collection efforts and in conducting the evaluations. A plea for “more research and more evaluation” may come across as a not very constructive policy recommendation—one that could be applied to virtually any government program. Although better evaluation is a widespread need in policy research, this report provides a compelling case that the need is particularly strong and potentially promising with respect to STC in California. First, because of legislative developments, funding of one of the three branches of STC in California—general and broad-based STC programs and activities—has been largely eliminated, with little basis for believing that the two branches of STC that continue to be funded—Tech Prep and Career Academies—are more effective. Second, the research base from which to assess the specific STC efforts that STWOA spurred in California is very weak. And third, the research this report presents using the NLSY97, as well as some of the most recent research on STC from other researchers, suggests that more compelling methods of studying STC programs are feasible and can in some cases successfully detect programs that are effective in increasing postsecondary employment or education. xii Contents Foreword........................................................................................................................... iii Summary........................................................................................................................... v Figures ................................................................................................................................ xv Tables.................................................................................................................................. xvii Acknowledgments......................................................................................................... xix 1. INTRODUCTION........................................................................................... Goals of the Research......................................................................................... Limitations .............................................................................................................. Broader Policy Context..................................................................................... Outline of the Report........................................................................................ 1 4 5 6 7 2. SCHOOL-TO-CAREER IN CALIFORNIA...................................... 9 Career Academies (Partnership Academies) ........................................... 12 Tech Prep................................................................................................................. 13 School-to-Work Opportunities Act of 1994.......................................... 14 Legislation and Organization.................................................................... 14 Local Partnerships........................................................................................... 15 Post-STWOA State Legislation.................................................................... 18 Funding..................................................................................................................... 19 Summary.................................................................................................................. 20 3. EXISTING RESEARCH ON THE EFFECTIVENESS OF STC IN CALIFORNIA AND NATIONALLY.................................. 21 Statewide Evaluation of STC in California ............................................ 22 Local Evaluations of General STC in California ................................. 23 Tech Prep and Career Academies in California.................................... 28 STC Research for the United States........................................................... 29 Summary.................................................................................................................. 30 4. EVIDENCE ON THE EFFECTS OF SCHOOL-TOCAREER PROGRAMS ON POSTSECONDARY ENROLLMENT AND EMPLOYMENT ............................................ 33 Empirical Approach............................................................................................ 34 Dataset, Variables, and Analysis Sample.................................................. 37 xiii Overview of the NLSY97............................................................................ STC Information in the NLSY97 .......................................................... Analysis Samples.............................................................................................. Results........................................................................................................................ STC Participation........................................................................................... Basic Analysis.................................................................................................... Summary of Findings from Basic Analysis......................................... Work and Schooling Expectations......................................................... School Fixed Effects....................................................................................... Provisional Summary and Interpretation............................................ Heterogeneity in the Effects of STC..................................................... Summary.................................................................................................................. 37 37 39 40 40 43 46 47 50 52 54 61 5. CONCLUSIONS ............................................................................................... 67 Evidence from the NLSY97 ........................................................................... 69 Recommendations............................................................................................... 71 Appendix A. National Research on the Effectiveness of STC................................... B. Empirical Methods for the NLSY97 Analysis ....................................... C. Detailed Tables for NLYS97 Analysis....................................................... 77 83 93 Bibliography.................................................................................................................... 103 About the Author.......................................................................................................... 111 Related PPIC Publications....................................................................................... 113 xiv Figures 2.1. 3.1. 4.1a. 4.1b. 4.1c. 4.2a. 4.2b. 4.3a. 4.3b. 4.4a. 4.4b. 4.5a. 4.5b. 4.5c. 4.5d. STC Funding in California ................................................................... Summary of Evidence on the Effectiveness of STC on Postsecondary Enrollment and Employment ............................... STC Participation Probabilities, Differences by Demographic Group................................................................................. STC Participation Probabilities, Differences by Family Structure.......................................................................................................... STC Participation Probabilities, Differences Associated with One Standard Deviation Higher ASVAB Scores............. College Enrollment Probabilities, Effects of STC Participation .................................................................................................. Employment Probabilities, Effects of STC Participation....... College Enrollment Probabilities, Effects of STC Participation, Controlling for Prior Work and Schooling Expectations................................................................................................... Employment Probabilities, Effects of STC Participation, Controlling for Prior Work and Schooling Expectations....... College Enrollment Probabilities, Effects of STC Participation, with School Fixed Effects ......................................... Employment Probabilities, Effects of STC Participation, with School Fixed Effects........................................................................ College Enrollment Probabilities, Effects of STC Participation, Differences by Race/Ethnicity................................ Employment Probabilities, Effects of STC Participation, Differences by Race/Ethnicity.............................................................. College Enrollment Probabilities, Effects of STC Participation, Differences by ASVAB Math Knowledge Scores ................................................................................................................ Employment Probabilities, Effects of STC Participation, Differences by ASVAB Math Knowledge Scores........................ 19 31 41 42 42 43 44 48 48 50 51 56 56 58 58 xv 4.5e. 4.5f. 4.5g. 4.5h. 4.5i. 4.5j. 4.6. College Enrollment Probabilities, Effects of STC Participation, Differences by Biological Mother’s Education........................................................................................................ Employment Probabilities, Effects of STC Participation, Differences by Biological Mother’s Education............................. College Enrollment Probabilities, Effects of STC Participation, Differences by Living Arrangement..................... Employment Probabilities, Effects of STC Participation, Differences by Living Arrangement................................................... College Enrollment Probabilities, Effects of STC Participation, Differences by Sex........................................................ Employment Probabilities, Effects of STC Participation, Differences by Sex ...................................................................................... Summary of Evidence on Effectiveness of STC on Postsecondary Enrollment and Employment ............................... 59 59 60 60 62 62 64 xvi Tables 2.1. School-to-Career Programs and Funding in California .......... 11 2.2. STC Activities Reported by Local Partnerships .......................... 16 3.1. Local Partnership Evaluations .............................................................. 24 C.1. Sample Construction and Analysis Samples.................................. 93 C.2. Participation Shares and Linear Probability Estimates of Individual Characteristics Associated with STC Participation.................................................................................................. 94 C.3a. Linear Probability Estimates of the Effects of STC Participation on College Attendance and Employment.......... 96 C.3b. Linear Probability Estimates of Control Variable Coefficients for Models in Table C.3a............................................. 97 C.4. Linear Probability Estimates of the Effects of STC Participation on College Attendance and Employment, Incorporating Prior Work and Schooling Expectations.......... 99 C.5. School Fixed Effects Linear Probability Estimates of the Effects of STC Participation on College Attendance and Employment.................................................................................................. 100 C.6. School Fixed Effects Linear Probability Estimates of the Effects of STC Participation on College Attendance and Employment, Specifications Interacting STC Participation with Respondent Characteristics............................ 101 xvii Acknowledgments The material in Chapter 4 of this report is the result of collaborative research with Donna Rothstein, a research economist at the Bureau of Labor Statistics, U.S. Department of Labor. I am grateful to Anne Danenberg and Douglas Symes for research assistance and to Gary Bjork, Jon Haveman, Daiji Kawaguchi, Nan Maxwell, Margaret O’BrienStrain, Joyce Peterson, Deborah Reed, Jon Sonstelie, Michael Teitz, Maureen Waller, and seminar participants at the Public Policy Institute of California and Stanford University for helpful comments. The views expressed are those of the author and do not reflect the views or policies of the Bureau of Labor Statistics or its staff members or of the Public Policy Institute of California. xix 1. Introduction The 1994 federal School-to-Work Opportunities Act (STWOA) allocated more than $1.5 billion to support increased career preparation activities in the nation’s public schools.1 STWOA was a response to the absence in the United States of a “comprehensive and coherent system to help its youths acquire the knowledge, skills, abilities, and information about and access to the labor market necessary to make an effective transition from school to career-oriented work or to further education and training” (H. R. 2884, 103rd Congress, “School-to-Work Opportunities Act of 1994”). STWOA funds were provided to states to encourage and support the creation of school-to-work or school-to-career (STC) systems, entailing cooperation among schools, private business, and government bodies (Office of Technology Assessment, 1995).2 In particular, STWOA funds were aimed at general or broad-based STC programs and activities that served all students, helping guide them into careers that not only could be entered immediately after high school or with additional vocational or technical education but also that entailed higher education at four-year institutions. STWOA was intended to address three areas of particular concern. These included (1) a lack of connection between school and work that led many youths to be unmotivated in school and to experience subsequent difficulty moving out of low-wage jobs, (2) youths _____________ 1This amount was originally appropriated for fiscal years 1994 to 1998 for grants to states and local partnerships under STWOA (Hershey et al., 1999). 2Although “school-to-work” is the more common appellation, many states have adopted the label “school-to-career.” Given that most of the effort is devoted to high school students and that the goal is to improve the career decisionmaking of youths—which may often entail further education—the school-to-career label provides a better characterization. Indeed, in the past, school-to-work programs (especially Tech Prep) have sometimes suffered from the stigma associated with placing high school students onto vocational tracks rather than directing them toward further education (Neumark and Allen, 2003). Thus, in this report the “school-to-career” label is used. 1 completing school with insufficient skills needed for the labor market, and (3) increasing labor market demands for complex thinking, close teamwork, and the ability to learn on the job. The goal of STWOA was to help young people develop the skills needed in the workforce and establish better connections between education and subsequent careers through STC transition systems. In particular, three core goals of STWOA were to increase (1) school-based initiatives such as career awareness activities and career links to academic curriculum; (2) workbased activities such as job shadowing, internships, and apprenticeships; and (3) connecting activities, such as the development of partnerships between high schools and employers and postsecondary institutions. See Box 1.1 for a summary of the act. Under STWOA, beginning in 1996 California received $130 million over five years from the National School-to-Work Office to establish STC providers and programs, plus an additional $7.2 million supplemental grant in 2000. STWOA funds were used in California to establish an extensive statewide system of Local Partnerships (LPs) that were involved in multiple aspects of STC but which clearly played an important role in implementing the types of general or broad-based STC activities and programs encouraged by STWOA. However, the U.S. Congress did not reauthorize STWOA after its initial five-year run. As federal dollars to support STC activities wound down, the California legislature approved legislation (AB 1873) in 2000 including $5 million in state funding, which was later cut to $2 million by Governor Gray Davis. Currently, the continuation of this funding (at even lower levels) is the only direct state funding that specifically replaces funds from STWOA. The STC activities funded under STWOA can be viewed as one of three branches of STC activities in California. The other two branches are Career Academies (sometimes called Partnership Academies) and Tech Prep, which focus more narrowly on specific student populations, in contrast to the general and broad-based approach of STWOA. Although funding from other sources (both state and federal) has continued for Tech Prep and Career Academies, the loss of STWOA funds represents more than a one-third decline in funding for STC activities overall in California and will likely over time severely curtail or 2 3 even eliminate broad-based STC programs and activities. Given that funding for STC programs—if it is to continue at anywhere near the levels under STWOA—will have to come from the state, this is an ideal time to study the effectiveness of STC in achieving its goals. Goals of the Research This report focuses on the central goal of STC policy—to improve the labor market outcomes of individuals as they begin to develop their careers and move on to work, further education, or a combination of the two. In particular, the main contribution of this report is an econometric analysis of a large-scale dataset, the 1997 National Longitudinal Survey of Youth (NLSY97), estimating the effects of STC program participation, nationally, on student enrollment in postsecondary education or employment in the period immediately following high school. The NLSY97 covers six types of STC programs: job shadowing, mentoring, co-op programs, school enterprises, Tech Prep, and internships/apprenticeships. These are by no means all of the programs and activities pursued by STC providers in California using grants under STWOA, but they do represent many of the most common programs and activities pursued. This complication arises because STWOA, and its implementation in California, did not restrict STC efforts to a specific set of programs or activities but instead created a network of providers of STC programs and activities in pursuit of the specific goals of the legislation. Thus, this research does not evaluate STWOA per se but examines instead a wide set of STC programs that characterize—albeit imperfectly—the types of general and broad-based STC programs that STWOA encouraged. An important strength of the NLSY97 for this research is that it offers many ways to estimate the causal effects of participation, accounting for nonrandom selection of participants into STC programs that could lead to inaccurate estimates of the programs’ effectiveness. Estimation of the causal effects of STC programs, by accounting for differences between STC participants and nonparticipants, is a major focus of this report and at the same time a major shortcoming of most of the existing research on the topic, whether for California or elsewhere. 4 Limitations The NLSY97 dataset poses some limitations. First, it is not sufficiently large to carry out analyses specific to California. But evidence on the effects of STC participation gained from data on all states should be informative given that effects most likely generalize across states. On the other hand, policymakers in California may be most interested in assessments of the effectiveness of the specific STC programs that were established in various localities throughout the state under STWOA. Although the NLSY97 data cannot provide information specific to California, this report also presents a summary of findings from a compendium of evaluations of specific local STC programs supported by STWOA in the state. The second limitation is that the overall goal of STC policy—to improve labor market outcomes of individuals as they begin to develop their careers—is considerably more broad than simply increased postsecondary employment or enrollment and should be evaluated by labor market success defined over the longer run. However, because the NLSY97 at this point covers individuals only in their first few years after leaving high school, the analysis in this report adopts a similarly narrow focus, examining the effectiveness of STC programs in boosting immediate postsecondary enrollment and employment; this examination is quite important, because it addresses the two activities most likely to increase the skills acquired by youths making the transition from school to work. Indeed, much of the existing research on STC focuses on the years immediately after high school as the key transitional period that has been the chief concern underlying STC legislation.3 _____________ 3The early writing on school-to-work transitions in the United States that advocated the development of a school-to-work system along the lines of STWOA focused on the general problem of “churning” or “milling about” that characterized youth labor market experiences in this country, entailing initial periods of joblessness or a series of dead-end jobs (see U.S. General Accounting Office, 1990; Commission on the Skills of the American Workforce, 1990; Hamilton, 1990; Lerman and Pouncy, 1990; Glazer, 1993; and other work reviewed in Heckman, 1993.) Much of the newer research on the effects of STC programs has focused explicitly on labor market and educational outcomes in the immediate postsecondary period. 5 Broader Policy Context Research on the effectiveness of STC also fits into broader policy issues. First, research on educational quality and reform has emphasized test-related outcomes, which can perhaps be viewed as culminating in the No Child Left Behind Act of 2001 (NCLB), emphasizing standardized testing in grades K–12. Although test score achievement undoubtedly reflects some dimensions of educational quality, another important dimension is education’s contribution to labor market success. In part because of STWOA, STC programs had become well integrated into high school education in many states before the enactment of NCLB and the failure to reauthorize STWOA.4 Indeed, some researchers see NCLB as likely to diminish government interest and investment in STC.5 And although test scores likely have a positive effect on socioeconomic success, it is highly unlikely that test-based standards encompass the entirety of schools’ contributions to labor market success of students. STC may very well also have an important role to play. Nearly all students make a transition from school to the labor market, and many spend a substantial fraction of their lives working. Thus, integrating into academic curricula information about careers, tools for increasing capacity and knowledge to make better career decisions, and the provision of specific skills could be a productive complement to an academic focus. STC may be particularly important for the many students who will not attend four-year postsecondary institutions, whereas a strong emphasis on testing may be more suited to the needs of those who will attend such institutions. The argument is not that STC is more important than quality academic education but that it may offer the potential to help those less likely to attend college and, more generally, may strengthen the contribution of education to labor market success. _____________ 4Nationally, among high school students in the NLSY97 sample used in this report, participation in the different individual types of STC programs covered ranges from about 7 to 18 percent, and 42 percent of high school students participate in at least one type of program. 5See, for example, Maher (2003). 6 Second, policymakers and researchers devote a great deal of time and effort to problems and policies related to poverty, focusing on changes in inequality, social programs to try to reduce poverty or cushion its blow, and setting wage floors for low-skill workers. However, most policymakers and researchers recognize that—although difficult to achieve—the best long-term solution is to raise the skills and productivity of the most economically disadvantaged, enabling them to earn enough to raise their families out of poverty. STC programs may be one way to increase skills and productivity, especially among those who would be less likely to obtain a college education. Finally, although not directly examined in this report, it is conceivable that the benefits of STC programs are extensive, if effective programs improve the efficiency with which youths “transfer” the human capital acquired in school into productive activity in the labor market. Outline of the Report Following this introductory chapter defining the issues, explaining the goals of the research, and discussing the policy context, Chapter 2 provides an overview of STC legislation and programs in California and of federal legislation. Chapter 3 surveys prior research on the effectiveness of STC in California and elsewhere. Chapter 4 explains and discusses the econometric analysis of the 1997 National Longitudinal Survey of Youth, and Chapter 5 offers a summary of the findings and policy recommendations. 7 2. School-to-Career in California This chapter provides an overview of state and federal legislation that funds STC in California, or did so in the recent past, and of STC efforts in the state. Describing STC efforts in California is complicated because of overlapping programs and a profusion of names. One useful distinction is between vocational education and STC (or school-to-work). Although the distinction is not always sharp in practice, and not all researchers make it, the two have some conceptual and programmatic differences and the distinction is useful for delineating what is and is not studied in this report. In particular, in the NLSY97 data, STC activities do not include vocational or technical education.1 Vocational education has been supported by the federal government since the Hughes Act of 1917. At that time, and for decades afterward, it was distinguished by being isolated from more comprehensive high school curricula (Hayward and Benson, 1993), sometimes because of the perspective that vocational education programs served students “better suited to applied than academic learning” (Donahoe and Tienda, 1999). School-to-work, in contrast, refers to efforts that integrate academic and vocational skills. A major impetus in this direction was the 1990 reauthorization of the Carl D. Perkins Vocational and Applied Technology Education Act, which sought this integration in a number of ways.2 One significant vehicle for this integration was Tech Prep, which was designed to introduce some vocational education into comprehensive high school curricula while integrating these programs with postsecondary institutions. A second was the development of Career Academies, “schools within schools” that focus on an _____________ 1Instead, some schools are classified as vocational/technical, but these are excluded from the analysis. 2See Urquiola et al. (1997) for more details and an outstanding review of the history of school-to-work. 9 occupational area and integrate academic education with technical preparation. The phrase “school-to-work” came into popular use with the federal School-to-Work Opportunities Act of 1994. The STC programs envisioned under STWOA were distinguished from earlier efforts because they sought a greater integration of vocational and academic education by calling for school-to-work programs for all students. In particular, STWOA was less explicitly geared toward “underserved populations” such as minorities, women, and the economically disadvantaged, although it noted some of the particular problems of disadvantaged youths in achieving a successful school-to-work transition. It was also less explicitly restricted to pathways that lead through community or technical colleges and, instead, sought to expand STC for all students and to include paths that lead to four-year colleges and universities (Urquiola et al., 1997, pp. 23–24). In responding to STWOA, California adopted the label “school-tocareer.” However, in current use, STC is used to describe an array of programs and efforts in California, not only those funded under STWOA. Similarly, the STC label is used in a general fashion throughout this report, and the activities and programs supported by STWOA are referred to as “general” or “broad-based” STC activities or programs, in contrast to the more narrowly defined Tech Prep and Career Academy programs. Finally, as described below, there is some overlap in the current administration and funding of the different components of STC. Table 2.1 shows the legislation and resulting programs that are part of the overall STC system in California. Aside from STWOA and follow-on state efforts, as mentioned above, the other two branches of STC are Tech Prep and Career or Partnership Academies. Tech Prep is funded federally through the Perkins Act, whereas Career Academies are funded by the state’s general fund. One major component not depicted in Table 2.1 is Regional Occupational Centers and Programs (ROP or ROCP), which originated in California in 1967 and are codified through several sections of the California Education Code (primarily 10 Table 2.1 School-to-Career Programs and Funding in California Program School-to-Work Opportunities Act Program Description School-based learning, work-based learning, and connecting activities Year Established 1994 Ongoing Program No School-to-Career School-based learning, work-based learning, and connecting activities 2000 (AB 1873) Ambiguous Tech Prep Educational programs that prepare individuals for employment in occupations that do not require a baccalaureate or advanced degree 1990 (reauthorized in the 1998 Perkins Act) Yes Career (Partnership) Academies Three-year program for grades 10–12 that is structured as a school-within-a-school and incorporates rigorous integrated academics with a career focus, business partnerships, and teachers who work as a team in preparing students for careers and postsecondary education 1985 (AB 3104), codified in 1993 (California Education Code, §§ 54690–54697) Yes California’s Appropriation Amount and Source $137 million over the life of the program; federal funding from STWOA $2 million from state funding (AB 1873) $11.8 million in FY 2002–03; federal funding from Carl Perkins Act $23 million in FY 2002–03; state funding from general fund §§ 52300–52336.5).3 Although there is some intermingling of STC activities and ROCPs, especially with respect to internships, ROCP is best viewed as a vocational education program. Vocational education funding swamps the STC programs listed in Table 2.1, with state spending of $373 million on ROCP in 2002–03 (based on Average Daily Attendance) and federal spending of $120 million for vocational and technical education from the Perkins Act. Career Academies (Partnership Academies) A Career Academy is a high school program in which the curriculum organizes academic education around an industrial or occupational area such as health, finance, computers, or media. In Career Academies, a group of students typically stays together with the same set of teachers for many years, and a relatively small number of students (such as 50) are enrolled in each grade. Career Academies are intended to enable students to fulfill requirements for college entrance and to acquire workrelated knowledge and skills. Academy teachers try to establish relationships with employers, who provide support through adult mentors and internships for students, for example (Career Academy Support Network, 2003a). Maxwell and Rubin (2001) describe the model as a “school-within-a-school.” Career Academies began in Philadelphia in 1969, and the model was established in California in the 1980s (Career Academy Support Network, 2003a, 2003b). California began to fund planning and implementation grants for academies beginning with AB 3104 in 1985. Additional funding for the Career Academy model was legislated in SB 605 in 1988, and the California Education Code (§§ 54690–54697) _____________ 3ROCPs are available to high school students 16 years of age and older as well as to adult students. Three main missions of ROCPs are to give students opportunities to “(1) enter the workforce with skills and competencies to be successful; (2) pursue advanced training in higher educational institutions; or (3) upgrade existing skills and knowledge” (California Department of Education, 2003a). Over 302,000 high school students (17.1 percent) and over 155,000 adults were enrolled in courses in the state’s 73 ROCPs in 2001–02 (California Association of Regional Occupational Centers and Programs, 2003). 12 included Partnership Academies beginning in 1993 (Maxwell and Rubin, 2001; California Education Code, n.d.). In the 1,005 high schools listed in California’s 2002–03 report from the California Basic Education Data System (CBEDS), there were 290 state-funded Career Academy programs that received a total of $23 million from the state’s general fund (California Department of Education, 2002). Tech Prep Tech Prep in California is a program of technical preparation in areas such as applied sciences; mechanical, industrial, or practical arts or trades; health occupations; or business (California Department of Education, 2003b). Tech Prep programs combine two or more years of high school education with two years of postsecondary education in sequential coursework, integrating academic with vocational and technical education, to strengthen the applied academic component of vocational and technical education. Tech Prep is intended to lead to an associate degree or a certificate in a specific career field, in turn resulting in higher-wage employment and perhaps further education. Federal support for Tech Prep began with the 1990 Carl D. Perkins Vocational and Applied Technology Education Act, Title II, Part E. The initial Tech Prep funding was for articulation of programs integrating two years of high school and two years of college (“2+2” programs).4 The 1990 Perkins Act was extended through 1998 (Perkins II), although funding for some programs such as consumer and homemaking education and student guidance services was dropped. Reauthorization of the Perkins Act of 1998 (Perkins III) eliminated all programs except Tech Prep from Title II. However, the state’s grant was broadened to include business partnerships and the use of funds for work-based learning as well as the original provisions for articulation. _____________ 4“Articulation” in this context means that courses taken at one school level fulfill requirements at another level. For example, an advanced placement class taken in high school “articulates” to the same course when taken at a university, or a lower division college class when taken at community college is considered the same as if taken at a university as long as an articulation agreement exists between any two school levels. See U.S. Department of Education (2002a) for a description of current Tech Prep sequences of study. 13 Since 1990, California’s share of Tech Prep funding has been close to $12 million annually. In the 2002–03 state budget cycle, $11.8 million was allocated to Tech Prep from $138.4 million in federal funds that were targeted at vocational and technical education programs in California (California Department of Education, 2002). School-to-Work Opportunities Act of 1994 Legislation and Organization Under STWOA, states were given considerable discretion over how to structure and run their STC systems. After STWOA was signed into law in 1994, Governor Pete Wilson issued an executive order (Wilson, 1994) forming a state school-to-career taskforce charged with creating a state STC plan that would lay the groundwork to apply for federal funding. This taskforce became the 27-member STC advisory council that provided recommendations for developing a statewide STC system. It was composed of representatives from education, business, labor, and the legislature, and supported by an Interagency Partnership (IAP) consisting of the Employment Development Department (EDD), the California Department of Education (CDE), and the California Community Colleges (CCC) Chancellor’s Office. The state was divided into 12 STC regions, each containing numerous LPs. Larger counties had more than one partnership and less-populated areas had multicounty partnerships. Each LP was required to include employers, representatives from local educational agencies (elementary, middle, and secondary schools), representatives of postsecondary institutions, local educators, and representatives of organized labor, students, and parent organizations. California announced the receipt of its first federal school-to-career grant of $21.9 million on September 19, 1996 (California Employment Development Department, n.d.). According to the announcement, a total of $131.4 million in federal funds would be allocated to California over a five-year period, which would be used to provide incentives for stakeholders to form local partnerships and implement STC-based efforts in conjunction with other education and workforce initiatives. Between 14 1997 and 2000, federal funds were distributed among the partnerships and state-level administrative units.5 Local Partnerships To get a better sense of the STC activities generated by STWOA in California, a survey was conducted entailing extensive interviews of representatives of LPs in the state.6 These representatives were interviewed about the activities of their STC programs, among other things. The interviews took place from December 2002 through April 2003 and covered 38 of the 57 LPs in the state. The interviews revealed considerable complexity in the administration of STC activities. Each LP interviewed, of course, had direct responsibility for STWOA activities. In addition, many of these LPs were also involved in other STC activities. For example, some of the LPs were housed in ROCPs. Many were in Local Education Agencies (typically a district or county office of education). And some were involved in Tech Prep or moved into it as STWOA wound down. As further evidence of this commingling of functions, many LPs indicated that before STWOA they had some involvement with vocational education through Perkins money, ROCPs, and so on, as well as Tech Prep. Table 2.2 presents the simple percentages of LPs reporting different types of STC programs.7 The table shows that LPs were generally _____________ 5The first payment was $21.9 million, 70 percent of which went to LPs and 30 percent to the state. The second payment was $43.8 million, of which 80 percent went to LPs and 20 percent to the state. Subsequent payments were $32.9 million, $21.9 million, and $10.9 million, of which 90 percent went to LPs and 10 percent to the state. In 2000, a “supplemental transition” grant of $7.2 million was awarded in addition to the earlier federal grants (personal communication, Ed Armijo, California Employment Development Department, October 2002 and May 2003). 6See Neumark (2004) for full details regarding the survey and the findings. 7The upper panel lists the STC programs that were explicitly included in the interviews, and the bottom panel lists any other types of programs that were mentioned in response to general questions about STC services, or elsewhere during the interview. It is important to note that because not all LPs participated in the interviews, the programs summarized here cannot necessarily be generalized to all STWOA partnerships in the state. 15 Table 2.2 STC Activities Reported by Local Partnerships Percentage Survey responses Job shadowing Career fairs or outside speakers Career assessment or career inventory Career academy or career pathway Internships/apprenticeships Tech Prep Mentoring Certificates Other responses from narratives Teacher/staff training and professional development ROCPa Service learning Work-based/project-based learning Junior achievement Field trips Vocational education Small learning communities Industry-based training Employer interviewing service Junior chamber of commerce Career workshops 95 89 84 82 82 79 32 53 45 26 24 24 5 5 3 3 3 3 3 3 aIncludes cases where partnerships indicated internships/ apprenticeships or career pathways through ROCP, or skill certificates through ROCP. involved in a wide range of STC activities, with all or nearly all including career assessment/inventories, career fairs or outside speakers, Career Academies or Pathways, Tech Prep, job shadowing, and internships/ apprenticeships. These programs are in many ways the bread and butter of STC efforts and constitute many of the same programs that are covered in the NLSY97 data and discussed in Chapter 4. Many if not most of these programs can be viewed as serving one or more of the goals of increased school-based learning, work-based learning, and connecting activities. Apprenticeships and internships are perhaps the best examples of work-based learning. Job shadowing and mentoring are probably best thought of as connecting activities, because 16 they do not entail extensive work experience but give participants the opportunity to see the types of skills and education that jobs require. Career guidance and counseling also clearly fall under the category of connecting activities, to the extent that they help students plan their educations with regard to career goals. Co-op programs that often entail split time between work and schooling are a bit harder to characterize and probably contain all three components. Tech Prep and Career Academies, which were discussed above and do not fall directly under the rubric of STWOA, are prime examples of school-based learning, as they bring vocational and career education into the classroom.8 In addition, the survey revealed that funding from STWOA did not go solely to the creation of new STC programs serving high school students. Some appears to have been used to expand or supplement preexisting programs or to add K–8 activities to high school activities already in place. In other LPs, STC funding was used for staff development rather than to provide “hands-on” experiences to students (such as job shadowing and internships). Some partnerships focused on building and sustaining the partnership itself and using it as a resource or “clearinghouse” for local districts and businesses. Nonetheless, the interviews indicated that termination of STWOA funding did have adverse consequences for the provision of general STC programs for high school students, although in the immediate aftermath, LPs seem to have scaled back programs and tried to support them with previously existing funding sources. The array of STC activities in which LPs were involved is natural, because STWOA did not mandate specific types of programs. In part, as well, it probably reflected opportunism, as existing institutions with oversight for STC and vocational education also tapped into STWOA funds while they were available. As a result, there is really no meaningful sense in which STWOA can be viewed as a unified, well-defined program—or evaluated as one. Rather, we can only identify a set of programs supported by STWOA and study their effectiveness, which is the strategy used in this report. _____________ 8More detailed definitions of STC programs are provided in Chapter 4 in the context of those programs studied using the NLSY97. 17 Post-STWOA State Legislation California is one of many states that—after STWOA’s demise—introduced their own STC legislation with at least some supporting appropriation, although California’s appropriations for general, broad-based STC programs have been very modest. According to data compiled in Schmidt (2000), in the 2000–01 legislative year, 24 states considered legislation pertaining to STC programs, 21 passed such legislation, and in 16 cases state general funds were appropriated. In total, state general fund appropriations in that year amounted to $89 million.9 California’s first STC legislation, AB 1873, was introduced in March 2000 and appropriated $5 million in state funds to support LPs whose federal STWOA funding was ending (Schmidt, 2001). Governor Gray Davis signed the bill in September 2000 but with a reduction in funding to $2 million. Although the original legislation was a one-time appropriation (Schmidt, 2001), this amount was subsequently included in California’s annual budget; however, it did not receive its first funding until the 2001–02 budget. The 2002–03 budget appropriated another $2 million. However, in the 2003–04 final budget (AB 1765), the amount was reduced to $1.7 million, and future appropriations are subject to annual budgetary and political constraints. In addition to AB 1873 and its renewal through AB 1765, STC has surfaced on the legislative agenda in at least four other pieces of legislation (AB 1412 and SB 1934, passed in 2002; AB 1266, passed in 2003; and AB 639, under consideration in the 2003–04 session) but without any additional financial support.10 _____________ 9This figure includes California’s STC appropriation of $2 million described below, plus an appropriation of $19.7 million to Partnership Academies in the state. 10AB 1412 renamed vocational education as “Career Technical Education” in the state. SB 1934 encouraged schools to combine a rigorous academic curriculum integrating academic and career skills to prepare all students for high school graduation and career entry. But the legislation does not provide funding and restricts implementation to the use of federal funds via the Perkins legislation. AB 1266 shifted all administrative and fiscal oversight from the IAP to the CDE and made funding for the program contingent upon a budget act or other legislative appropriation. AB 639 would authorize an ROCP to establish a Career Academy. Given the relatively high level of annual funding that ROCP currently receives, the ability to use a portion of these funds 18 Funding Figure 2.1 illustrates the sources of major appropriations for STC activities in California from fiscal years 1998–99 through 2003–04. As the figure shows, after STWOA appropriations ended in 2000–01, California’s relatively low STC appropriation began in 2001–02.11 At the same time, appropriations for Partnership or Career Academies have Figure 2.1—STC Funding in California ______________________________________________________________ for Career Academies rather than for strictly vocational programs may signal a shift in the focus of ROCP away from vocational education and toward academic integration. (This information came from the Assembly Education Committee, the Senate Education Committee, and the Senate Office of Research during our attempt to ascertain whether the state planned to continue STC and whether there was any other evaluation research under way or contemplated.) 11STWOA funding was in the $20 million to $40 million range in the few years preceding those shown in the figure. 19 risen as more academies have been established, and Tech Prep funding has remained relatively constant during the six years. The significant drop in funding for general STC is very apparent in the figure. Summary STC activities in California have occurred and continue under the auspices of a number of programs, with funding from both federal and state sources. In addition to STWOA, STC efforts in California include Career Academies and Tech Prep. One important implication of this is that the failure to pick up responsibility for funding previously provided by STWOA does not mean that STC activities will cease to exist in California. As summarized in Figure 2.1, funding has continued or increased for Career Academies and Tech Prep. Furthermore, although the drop-off in STWOA-related funding represents a large share of the funding depicted in Figure 2.1, to the extent that STC activities can piggy-back onto other programs, these activities will continue. Nonetheless, it seems that, absent legislation restoring some of the funding previously provided under STWOA, general and broad-based STC activities are sure to diminish. The following chapters use a combination of original research and review of past research to provide a detailed picture of what we know about the effectiveness of the three components of STC in California in boosting postsecondary enrollment and employment, with the original research focusing on the general and broad-based STC programs supported by STWOA. The next chapter covers the existing evidence at the state level and the national level, including not only published and widely circulated studies but also information on local evaluation efforts garnered from the survey of local STC providers in California. Following that, Chapter 4 presents the main analysis in this report, using the NLSY97 data to study the effects of the types of STC programs that were supported in California (and elsewhere) by STWOA. 20 3. Existing Research on the Effectiveness of STC in California and Nationally The research presented in this report offers new evidence on the effectiveness of STC in California and the rest of the nation. But it focuses on the types of general and broad-based programs that were spurred by STWOA. Thus, it is useful to summarize briefly what other research establishes about the effectiveness of the other branches of STC in California, including studies specific to California and studies that cover the types of programs in effect in California. Combining existing research findings with the new findings reported in Chapter 4 provides the fullest possible picture of the effectiveness of California’s STC programs. Throughout, the focus is on the effects of STC programs on postsecondary enrollment and employment. In addition, both the review of existing evidence in this chapter and the new evidence in the next chapter emphasize the estimation of causal effects of STC programs. In evaluating the effects of STC programs (or most other government programs), a fundamental problem is determining whether there are preexisting differences between those who participate in the programs and those who do not—whether because of individual choices to participate or because of assignment to program participation. If account is not taken of these “pre-program differences,” then comparisons between participants and nonparticipants will typically incorrectly estimate the causal effect of the program. As an example, if high school students who are more career oriented choose to participate in STC programs, then participants may have been more likely to be employed after high school regardless of participation, and a comparison of participants to nonparticipants would overstate the extent to which STC participation 21 boosts postsecondary employment.1 In this chapter, then, we examine not only what the existing research says but how convincingly that research handles the problem of pre-program differences. Statewide Evaluation of STC in California In 2002, WestEd and MPR Associates, Inc. (“WestEd-MPR”) completed a statewide evaluation of the STC structure and activities established in California in response to STWOA (WestEd and MPR Associates, Inc., 2002). Most of this evaluation focuses on the implementation and sustainability of STC, rather than on the effects of STC on participants. However, a limited part of the WestEd-MPR report focuses on the effects of STC participation on postsecondary outcomes.2 Based on a survey of seniors and a follow-up survey on postsecondary experiences, the report concludes that there was little or no evidence of positive effects of STC participation on postsecondary enrollment or employment. Some of the analyses on which these conclusions are based incorporated a limited set of control variables for pre-program differences, including parent’s education, race, sex, and a measure of prior academic achievement.3 However, the limited set of controls seems unlikely to adequately account for pre-program differences between STC participants and nonparticipants. _____________ 1This discussion applies not only to STC but to estimating the causal effect of virtually any type of program participation. For a thorough treatment of the issues, see Heckman et al. (1999). This problem is often referred to as “endogenous selection” because it arises when individuals self-select into program participation, as in the example in the text. 2Part of the WestEd-MPR report also presents some findings on the association between STC participation and student attitudes toward work and schooling. However, from a policy perspective, this is considerably less important than the question of whether STC program participation changes observed behavior. 3The report provided no statistical results, but upon request they were provided to us by WestEd-MPR. The results summarized in the WestEd-MPR evaluation are a synthesis of findings from the individual LPs (personal communication, Robert Fitzgerald, September 2003). 22 Local Evaluations of General STC in California Another potential source of information on the effectiveness of STC programs in California is the local STC providers in the state. Because local organizations are at the forefront of STC efforts, and because STC programs are relatively new, much of the most recent or most convincing evidence may be in the hands of practitioners “on the ground.” Consequently, in the survey of LPs described in the previous chapter, respondents were also asked about formal evaluations of their local programs.4 In most cases, when an evaluation was eventually supplied, either it was in the form of a publicly available report or the LP representative gave us permission to cite an unpublished report; in others, only confidential internal evaluations were provided. In the latter cases, the promise of confidentiality given for the interviews precludes identifying the evaluated district. In the end, 17 evaluations were received. A number of the local evaluations had a common structure, because they were a by-product of the statewide STC evaluation by WestEd-MPR (2002). A subset of these evaluations used a common set of analyses; these are referred to as WestEd-MPR Core cases. The others conducted somewhat different analyses; these are referred to as WestEd-MPR Plus cases.5 The remaining evaluations were unrelated to the statewide evaluation. Table 3.1 summarizes the 17 evaluations received. Because there are multiple evaluations of the same LP, in total these evaluations cover 12 LPs. The first panel in Table 3.1 shows the WestEd-MPR Core cases.6 The last column reports any statistically significant findings related to _____________ 4The findings from the interviews are summarized here. A more detailed discussion is provided in Neumark (2004), which discusses how STC practitioners assess the success of their programs. Such an approach is useful in thinking about how assessments might be enhanced in the future. 5These evaluations used two individual-level surveys of STC participants and nonparticipants that were conducted in 2001—the Senior Survey and the Senior Followup Survey. At no time were these responses pooled and analyzed statewide, and none of the findings from these individual evaluations were covered in the WestEd-MPR report. 6Note that in the case of one LP only an executive summary was provided, and it was difficult to ascertain much from it. 23 Table 3.1 Local Partnership Evaluations 24 STC Partnership or School District Type of Program Executive summary, confidential ROP Sonoma (O’Driscoll et al., 2002a) “High-STC” Unite-LA (O’Driscoll et al., 2002b, 2003) Ventura (Christensen, 2002) “High-STC” “High-STC,” Career Pathways, internship students Verdugo (O’Driscoll et al., 2002c) “High-STC” Control Variables Control Group Used WestEd-MPR Core Case Studies Non-high-STC ? students Postsecondary Effects of STC No enrollment or employment differences Non-high-STC students No Lower enrollment, no employment difference Non-high-STC students No Higher enrollment, lower employment Converse of each group No “High-STC”: No enrollment or employment differences Career Pathways: Lower enrollment in fouryear college, higher enrollment in vocational/ technical schools, no employment difference Internships: Higher enrollment, no overall employment difference, higher full-time employment, higher receipt of benefits, and more likely to view job as “highly related” to high school education Non-high-STC students No No overall enrollment difference, higher enrollment in four-year college, lower employment Table 3.1 (continued) 25 STC Partnership or School District Orange County Coalition/ Vision 2020 (Forouzesh and Orange County, 2002) Verdugo (Butler et al., 2002a) Type of Program “High-STC” Magnet students Control Group Non-high-STC students Control Variables Used No Postsecondary Effects of STC No enrollment or employment differences WestEd-MPR Plus Case Studies Nonmagnet students Yes—demographic characteristics and prior academic achievement Not studied Unite-LA (Butler et al., 2002b) Career Academy students Partnerships Plus! (Kern County) Partnerships in youth (Fattahi, 2002) leadership workshop Nonacademy students Yes—demographic characteristics and prior academic achievement Independent Evaluations No No Not studied N/A Confidential Various programs No No N/A Confidential ? No No N/A Orange County Department of Education (Forouzesh, 2002) Participants in Career Academy, Pathway, Tech Prep, or other career-related activities Nonparticipants in career-related activities No Not studied Table 3.1 (continued) STC Partnership or School District Confidential Type of Program Career Academy students Control Group Comprehensive high school students Control Variables Used ? Postsecondary Effects of STC Possibly higher enrollment in two-year colleges, no differences in four-year enrollments (unclear, and no statistical tests) Confidential, summary STC participants No No N/A San Mateo STC Partnership (Lindler and Hill, 2000) Various programs No No N/A Partnerships Plus! (Kern County) Career Academy (Bechtold and King, 2001) students Nonacademy students No Not studied Horizons (San Bernardino) (Hemsley, 2002) Career Academy and technical education students Students with no career/technical education courses Indirect—focused only on “highachieving” students No enrollment or employment differences NOTES: Under type of program, “high-STC” refers to a combination of STC activities. A question mark indicates that the evaluation was not sufficiently detailed to answer the question. 26 postsecondary enrollment or employment effects.7 Looking across all of the postsecondary employment results reported, there is no evidence of positive employment effects and occasional evidence of negative effects. For enrollment, there is some mixed evidence of positive and negative effects, but most common, again, is a failure to find evidence of any effect. These studies are limited by the failure to account for selection into STC participation, even at the level of observable (and measured) control variables. Although data were collected on items that could have been introduced as limited controls in estimating the effects of STC (such as sex, age, ethnicity, socioeconomic status, and so on), the analyses did not account for any of these. The second panel of Table 3.1 covers the two WestEd-MPR Plus cases, first, magnet students, and second, Career Academy students. Both studies are better than those just discussed in that they take account of control variables in estimating the effects of STC. However, neither study looks at postsecondary outcomes; instead, both focus on performance while in high school.8 Of the remaining nine evaluations in the last panel, five can be identified. Unfortunately, for five of these nine evaluations no data were collected on a control group of nonparticipants, which would rule them out as providing an evaluation of the effects of STC. Instead, these evaluations focus on such aspects as implementation, surveys of program participants, and so on. Of the four studies that include control groups, only two focus on postsecondary outcomes. The first (a confidential study) finds some indication of higher two-year college enrollments among Career Academy students but no difference in four-year enrollments, although the statistical significance of the results is not _____________ 7Effects are reported that are statistically significant at the 5 percent level. It might have been desirable to also report findings that were significant at the 10 percent level. But the evaluations did not provide p-values or t-statistics; instead, they simply reported which results were significant at the 5 percent level. 8Both studies provide some evidence that STC participants do better in school (in terms of, for example, test scores, attendance, and pass rates), even after controlling for some measures of prior achievement in school. These findings echo the Manpower Demonstration Research Corporation studies of Career Academies discussed below (Kemple and Snipes, 2000; Kemple, 2001). 27 reported. The second, for the Horizons LP, finds no evidence of enrollment or employment differences. In summary, these evaluations provide little if any convincing evidence of the effectiveness of STC programs, especially as regards postsecondary outcomes. Many of the evaluations fail to even use a control group, which is a prerequisite for evaluation. Most fail to use any control variables to account for differences between participants and nonparticipants. And few look at postsecondary outcomes. Tech Prep and Career Academies in California The other two branches of the STC system in California are Tech Prep and Career Academies. There is no existing statewide or other program evaluation of Tech Prep. There is a statewide evaluation of Tech Prep program implementation for the 2001–02 academic year, but it neither evaluates student outcomes nor reviews research on student outcomes related to participation in Tech Prep (O’Driscoll et al., 2002d).9 There are a couple of studies of Career Academies in California. Maxwell and Rubin (2001) study Career Academies in an undisclosed large, inner-city California school district in a period before SWTOA. Their research is based primarily on a survey about secondary and postsecondary experiences sent to academy students and a comparison group of nonacademy students. The survey data point to increases in postsecondary enrollment for some groups (Asians and males) but no improvements in employment or wages, which the authors describe as “inconsistent with the state’s goals for the program” (p. 14). The authors also report positive effects of academy participation on employment in a job related to the high school program for some groups, although noting that very few participants or nonparticipants entered fields related to their high school course of study. However, these conclusions are of _____________ 9The report does point to some of the key problems associated with identifying and tracking Tech Prep students. In particular, there are no consistent guidelines to define Tech Prep students, and electronic data systems linking secondary and postsecondary partners are inadequate. The report concludes that the state’s Tech Prep system does not currently have the capacity to perform an adequate student outcome-based evaluation of the effectiveness of Tech Prep. 28 questionable reliability. The authors use some limited controls for preprogram differences between participants and nonparticipants; but a potentially more serious concern is the very low response rate to the survey (around 12 percent). An earlier study by Stern et al. (1988) also looked at Career Academies in California. With very limited control variables, the evidence suggests some positive effects of academies on high school performance but does not address postsecondary outcomes. STC Research for the United States Not surprisingly, research on the effects of STC outside California is more extensive and its conclusions may be informative about the types of STC programs used in the state. However, this wider research base also generally provides little basis for concluding that STC programs improve postsecondary outcomes. This section briefly reviews this research; Appendix A provides a more detailed discussion. Two reports from the National Center for Research in Vocational Education (NCRVE) provide a relatively thorough compendium of research on STC programs in a wide variety of locations through about 1997 (Stern et al., 1994; Urquiola et al., 1997). Many of the studies covered in these reviews do not even include a comparison or control group of nonparticipants, making it impossible to say anything about the causal effects of STC programs. Some of the more recent studies use a comparison group and pay more careful attention to pre-program differences, but even the best of these (Orr, 1996; Hollenbeck, 1996) are plagued by possibly important differences between participants and nonparticipants. Finally, a recent report on STWOA by Mathematica, Inc.—the national evaluation of STWOA for the U.S. Congress that was mandated by the act—does not focus on evaluating the effects of STC on participants, and the limited evidence it does present does not indicate beneficial postsecondary effects. In the national research on the effectiveness of STC, one important exception to the generally inadequate efforts to account for pre-program differences is the ongoing evaluation of Career Academies by the Manpower Demonstration Research Corporation (Kemple and Snipes, 2000; Kemple, 2001, 2003), based on random assignment of students to 29 Career Academies. The second report from this evaluation (Kemple, 2001) looked at students one year after the scheduled completion of high school and found no effect on high school graduation rates, postsecondary education, or employment. But the most recent report (Kemple, 2003) presents stronger evidence of such postsecondary effects. In particular, male participants have higher full-time employment, hours, and earnings, although there is no evidence of such positive effects for women. On the other hand, the estimates indicate that male participants are less likely to be enrolled in school or to have obtained further education; although many of the education-related differentials are large, most are not statistically significant. The author interprets the results as indicating that Career Academies boost labor market outcomes without any offsetting reductions in education. However, the results could perhaps more plausibly be read as suggesting offsetting positive employment and negative schooling effects, as the magnitudes of the negative schooling effects roughly offset the positive employment effects. Summary The findings from this review of existing research are summarized in Figure 3.1, focusing on the three components of STC in California.10 There is as yet no existing statewide evidence establishing the effectiveness of STC in California in increasing postsecondary employment or enrollment, or more generally improving youths’ labor market outcomes. Although California commissioned a statewide STC evaluation, that study contributes little empirical evidence on the effectiveness of STC in helping students make transitions to postsecondary education or employment. The state’s 2002 Tech Prep evaluation did not address student outcomes at all, and evaluations of Career Academies in California perhaps point to increased college enrollments, although the evidence is of limited quality. There was a flurry of evaluations of the effectiveness of local STC efforts in California spurred by STWOA. However, these evaluations also provided little if any convincing evidence of the effectiveness of STC _____________ 10Some of the material summarized in this figure is based on sources discussed only briefly in the text, but in more detail in Appendix A. 30 Figure 3.1—Summary of Evidence on the Effectiveness of STC on Postsecondary Enrollment and Employment programs, especially on postsecondary outcomes. Many of these evaluations do not use a control group, which makes it impossible to infer the effects of the programs, most fail to account for differences between participants and nonparticipants, and few look at postsecondary outcomes. At the national level, most of the existing evidence is plagued by the problem of pre-program differences between participants and nonparticipants. However, recent research that uses an experimental design to evaluate the effectiveness of one particular kind of STC program— Career Academies—to this point finds some beneficial effects on postsecondary outcomes for men in the form of higher wages and earnings and greater likelihood of full-time work, although the evidence suggests that there may be some tradeoffs in the form of lower participation in postsecondary education.11 The next chapter presents new evidence on the effects of the types of STC programs that were supported by STWOA in California, based on the NLSY97 data. _____________ 11Note, however, that these conclusions are the opposite of those reached by Maxwell and Rubin (2001) for Career Academies in California. 31 4. Evidence on the Effects of School-to-Career Programs on Postsecondary Enrollment and Employment1 This chapter describes econometric analyses of the effectiveness of STC programs that overlap considerably with the types of general, broad-based STC programs spurred by STWOA in California. The research takes advantage of the information on STC program participation in the NLSY97, which is the first nationally representative dataset—and indeed the first large-scale dataset—to include information on participation of high school students in STC programs. STC program participation and postsecondary outcomes may be related simply because participants differ from nonparticipants independent of any effects of STC programs; such noncausal relationships, however, do not imply that STC programs are effective. In contrast, this chapter emphasizes taking advantage of numerous features of the NLSY97 to estimate the actual (that is, causal) effects of high school STC programs on postsecondary employment and higher education. Of course, even establishing causal effects of STC on higher education and employment does not imply that the benefits of STC funding outweigh the costs. Given the difficulty of characterizing better career decisionmaking, there may never be a complete analysis of this nature. Nonetheless, information on the effects of STC programs on transitions from high school to employment or higher education can inform policymakers as to whether STC is having some of the intended effects. After all, core goals of STC are to move people into higher_____________ 1This chapter is based on Neumark and Rothstein (2003). 33 paying jobs and to encourage skill formation among new labor market entrants, both of which are likely to be furthered by increased enrollment or employment in the immediate post-high school period. Empirical Approach2 The goal of the empirical analysis is to estimate, at the individual level, the relationship between employment and enrollment in the posthigh school period and participation in STC during high school. STC participation is categorized in terms of participation in specific types of programs discussed below. Given that employment could be very shortterm, whereas college enrollment of any type is nearly always longerterm, the focus is on employment at the time respondents to the NLSY97 are surveyed and on whether the individual was enrolled in college at any time since leaving high school. The simplest form of such an analysis looks at the statistical relationship between STC participation, on the one hand, and postsecondary enrollment or employment, on the other. The results from this type of estimation will be summarized in terms of the difference in the probability of postsecondary enrollment or employment associated with participation in different types of STC programs. As an example, the statistical analysis would conclude that participation in a co-op STC program boosts the probability of postsecondary employment by 0.07.3 However, as mentioned above, there may be pre-program differences between STC participants and nonparticipants. That is, participants and nonparticipants may have had different likelihoods of postsecondary enrollment or employment even before they participated in STC programs, and it would be a mistake to interpret such differences as causal effects of STC participation. The empirical analysis takes three _____________ 2A more detailed discussion of the empirical approach and methods is provided in Appendix B. 3An estimate like this can be equivalently interpreted as a 7 percentage point increase in employment. But percentage point changes have to be distinguished from percentage changes. For example, an increase in the probability of employment from 0.50 to 0.57, and similarly an increase in the percentage employed from 50 to 57 percentage points, both represent 14 percent increases in employment. 34 different approaches to trying to account for these pre-program differences between participants and nonparticipants in an attempt to develop more reliable estimates of the effects that STC participation has on postsecondary outcomes. Each approach takes advantage of unique features of the NLSY97 that make this dataset particularly powerful for studying the effects of STC. First, the NLSY97 has information on a rich set of variables describing individual respondents to the survey, and their families. In addition to fairly typical demographic variables (sex, race, ethnicity, and age), the dataset includes three additional sets of variables that are potentially important, including information on living arrangements and the respondent’s family;4 test scores from the Armed Services Vocational Aptitude Battery (ASVAB); and self-reported measures of school behavior.5 Controlling statistically for these variables should go a long way to eliminating any influence on the estimates of the effects of STC of pre-program differences between STC participants and nonparticipants. Second, for part of the sample the NLSY97 also includes measures of individual respondents’ self-reported expectations (in the form of probabilities) for future education and employment, including receipt of a high school diploma by age 20, obtaining a four-year college degree by age 30, and working over 20 hours per week at age 30.6 Controlling for these expectations—which are reported before the STC participation measures used in the analysis—makes it even more likely that the estimates account for pre-program differences. The intuition behind the contribution of the work and schooling expectations variables is as follows. Before participating in STC, students are asked about their _____________ 4The family and living arrangement variables include urban residence; whether one lives with both biological parents, only the biological mother or father, a biological parent and a stepparent, or in some other arrangement; household size; household income; and the biological mother’s schooling. Household income is from a Round 1 parent questionnaire except in the rare event that the youth is defined as independent in 1997. 5These include whether the respondent was threatened at school or had gotten into a physical fight at school and whether the respondent had been late with no excuse two or more times or had been absent two or more weeks. All are measured in 1997. 6This information is available for the 1980 and 1981 birth cohorts, which are the oldest cohorts in the sample. 35 post-high school work and schooling expectations. Then some participate in STC and some do not, and their post-high school work and schooling behavior are subsequently observed. If, for example, after controlling for educational expectations STC participants are still more likely to be enrolled in college after leaving high school, then it is quite sensible to conclude that there is a true (that is, causal) effect of STC on postsecondary enrollment, because the educational expectations variables should have controlled for many remaining pre-program differences that are associated with postsecondary educational outcomes. Third, there may be other unmeasured pre-program differences between STC participants and nonparticipants that are not captured in any of the control variables discussed thus far. Some of these differences, however, may vary systematically across students in different schools, because of similarities either in the student body or in the educational environment. Because the NLSY97 has data on multiple students in the same school, it is possible to control, in addition, for any unmeasured pre-program differences common to all individuals within a school, identifying the effects of STC participation from the within-school differences between those who do and do not participate in STC and within-school differences in outcomes associated with this participation. Intuitively, these estimates will point to an effect of STC on postsecondary outcomes only if, for students in the same school, STC participants have different probabilities of postsecondary enrollment than nonparticipants. In contrast, average differences in STC participation rates and postsecondary outcomes across schools will not feed into these estimates of the effects of STC. These estimates are referred to as “school fixed effects” estimates, because they hold constant school-level differences common to students in the same school. Additional results discussed in Appendix B suggest that school-level differences are potentially quite important; as a result, the school fixed effects estimates are emphasized in the discussion of the results that follows. 36 Dataset, Variables, and Analysis Sample Overview of the NLSY97 The first round of the NLSY97 was administered in 1997 to a nationally representative sample of 8,984 men and women who were ages 12 to 16 as of December 31, 1996. The survey involved personal interviews with the youth and one of his or her parents. It gathered extensive information on the youth’s labor market behavior, education and training, family and community background, as well as important life events such as marriage or the birth a child. More important, in each year, respondents were asked a number of questions about participation in school programs designed to help them prepare for the world of work. This report uses data from the first four rounds of the NLSY97, from 1997 through 2000. All STC information is taken from Rounds 2–4, after the data on schooling and work expectations and school behaviors had been collected. STC Information in the NLSY97 In the NLSY97, students were surveyed about “programs schools offer to help students prepare for the world of work.” The STC programs covered include internships or apprenticeships, cooperative education, job shadowing, mentoring, work in a school-sponsored enterprise, and Tech Prep. The following definitions summarize the features of these programs.7 • Internship: For a specified period of time, students work for an employer to learn about a particular industry or occupation. Students’ workplace activities may include special projects, a sample of tasks from different jobs, or tasks from a single occupation. The internship may or may not include paid work experiences. _____________ 7These are taken from the NLSY97 School Administrator Survey, which was given to administrators of schools that were subsequently linked to the NLSY97. In the NLSY97, respondents were shown definitions that corresponded closely to these but were slightly shorter. The data from the School Administrator Survey are not used in this chapter, although they are used in some of the results discussed in Appendix B. 37 • Apprenticeship: Typically, apprenticeships are multiyear programs that combine school- and work-based learning in specific occupational areas or occupational clusters designed to lead directly into either a related postsecondary program, entry-level job, or registered apprenticeship program. They may or may not include paid work experiences. • Cooperative education: This is a method of instruction whereby students alternate or parallel their academic and vocational studies with a job in a related field. It may or may not include paid work experiences. • Job shadowing: Typically, as part of career exploration activities early in high school, a student follows an employee for one or more days to learn about a particular occupation or industry. Job shadowing is intended to help students hone their career objectives and select a career major for the latter part of high school. • Mentoring: A student is paired with an employee over an extended period of time during which the employee helps the student master certain skills and knowledge the employee possesses, models workplace behavior, challenges the student to perform well, and assesses the student’s performance. Mentoring may be combined with other work-based learning activities, such as internships or on-the-job training. • School-sponsored enterprise: This entails the production of goods or services by students for sale to or use by others. School-sponsored enterprises typically involve students in the management of the project. Enterprises may be undertaken on or off the school site.8 _____________ 8School enterprises are likely the least well understood of these programs. There is tremendous variety in school-sponsored enterprises, from businesses run by students in schools (banks for students and faculty, coffee shops, desktop publishing and multimedia presentations, snack shops, and so on) to enterprises run off premises (such as airport stores and retail outlets). Students working in these enterprises often combine their work with either formal or informal instruction in business-related fields such as accounting and management. 38 • Tech Prep: This is a planned program of study with a defined career focus that links secondary and postsecondary education.9 In Round 1, administered in 1997, the STC questions cover the types of programs in which individuals participated at any time in the past, whereas in subsequent rounds (2–4) the questions shifted to participation in the year preceding each round of the survey. The questions were asked of all 9th to 12th graders in Round 1. In Rounds 2-4, they were asked of any respondents enrolled in school (including college). However, this research focuses on participation while in high school by using only STC information for years in which respondents were enrolled in high school.10 Analysis Samples11 Our analysis is based on data from the first four rounds of the NLSY97. Respondents were ages 12–17 as of the first round in 1997, and the next three rounds were conducted in the three subsequent years. The analysis focuses on educational and employment outcomes measured as of the third or fourth rounds, in 1999 and 2000, to isolate those individuals for whom it is possible to observe the early years of their employment or higher education after leaving high school. _____________ 9Note that Tech Prep was separated from the general, broad-based STC programs supported by the STWOA that were discussed in the previous chapter. The NLSY97 was not designed, of course, specifically to evaluate the STWOA. The empirical results for Tech Prep relative to the other types of STC programs included in this list therefore help address the effectiveness of these alternative types of programs. 10The manner in which the STC data are collected raises two issues. First, because the larger share of STC participation occurs in the later years of high school, high school graduation is endogenously related to “exposure” to STC; dropouts cannot participate in high school STC programs. A study of STC and high school graduation would therefore require a very different research design that is not afforded by these data, for example, by randomizing students into a program in 9th grade and then measuring graduation rates. This type of study design is used in the Career Academy study discussed above (Kemple, 2003). Second, a moderate share of individuals report some college enrollment in the last year in which they are enrolled in high school, so in principle these individuals could be reporting STC during their short initial spell of college.As a check on the robustness of the results, the specifications reported below were re-estimated excluding the STCinformation for interview years with mixed high school and college enrollmentfrom the calculation of the STC measures.The qualitative conclusions were unaffected. 11A more detailed discussion of the analysis samples is provided in Appendix B. 39 Combined with a number of sample restrictions such as availability of data on STC participation, there are 2,933 observations in the baseline analysis sample; the samples analyzed incorporating the work and schooling expectations, and including school fixed effects, are smaller. In the former case, this is because the work and schooling expectations questions were asked only of the two oldest birth cohorts in the dataset. In the latter case, the sample size reduction stems from cases in which there were no multiple observations with available data within the same school. Results STC Participation Before reporting estimates of the models for postsecondary enrollment and employment, estimates of models for participation in STC are reported, in part to provide descriptive information on STC participation, and in part to provide a preliminary assessment of the severity of pre-program differences between STC participants and nonparticipants. Some of the key results are summarized in Figures 4.1a–4.1c.12 In Figure 4.1a, the first bar for each type of STC (and for any STC) shows the sample proportion participating. Participation rates in each type of STC range from about 0.07 to 0.18, with the rate for participation in any STC equal to 0.42. The three subsequent bars for each STC entry show how, holding other factors at their means, participation rates differ for blacks, Hispanics, and females. The figure reveals that for three types of STC programs (co-op, school enterprise, and Tech Prep), black students are significantly more likely to participate, with the estimates indicating participation probabilities that are higher by 32 to 79 percent relative to the overall sample. This race difference appears to stem from higher participation of blacks relative to whites and others in the same schools, rather than from schools with more black students being more likely to offer STC programs. _____________ 12Appendix Table C.2 reports the regression results underlying these figures. 40 Figure 4.1a—STC Participation Probabilities, Differences by Demographic Group Figures 4.1b and 4.1c report the differences in participation associated with family structure and with ASVAB test scores. Perhaps the most striking finding from these figures (as well as the full regression results reported in Appendix Table C.2) is that almost no other variables are significantly related to STC participation.13 In Figure 4.1b, the only significant differences are lower participation in job shadowing and mentoring for those who live only with a mother. And in Figure 4.1c, there are very few significant differences in participation (and slight differences overall) associated with one standard deviation higher ASVAB scores. The finding that very few variables predict STC participation suggests that, in this sample, problems from pre-program differences between STC participants and nonparticipants may not be too severe, although it is important to confront this question directly in the course _____________ 13This is also reflected in the very low R2 values for these models of individual participation (see Appendix Table C.2), which are generally one-fifth or less of the R2 values for the models for whether schools offered STC. 41 Figure 4.1b—STC Participation Probabilities, Differences by Family Structure Figure 4.1c—STC Participation Probabilities, Differences Associated with One Standard Deviation Higher ASVAB Scores 42 of estimating the effects of STC on postsecondary outcomes, given that STC participation was not randomly assigned in these data.14 Basic Analysis Figures 4.2a and 4.2b begin the analysis of the effects of STC on postsecondary enrollment and employment. For each type of STC program, the upper bar in Figure 4.2a reports the estimated effects of Figure 4.2a—College Enrollment Probabilities, Effects of STC Participation _____________ 14If STC participation were randomly assigned, none of the variables in Appendix Table C.2 would be expected to predict participation, and the evidence does not deviate much from this scenario. Excepting the estimates of the black-white difference, 7.5 percent of the remaining coefficient estimates are significant at the 5 percent level, versus the 5 percent that would be expected to be significant at this significance level solely as a result of randomness even if none of the controls were in fact related to STC participation. One potential issue is that the ASVAB test scores may appear insignificant because of a high degree of multicollinearity among them. Tests for the joint significance of the test score coefficients—which are not affected by the multicollinearity—indicated that multicollinearity is an issue, but not an important one, as only in one case (for job shadowing) does the joint test indicate significance while none of the individual tests do. And the low R2 values mentioned in the footnote above are of course not influenced by any multicollinearity problem. 43 Figure 4.2b—Employment Probabilities, Effects of STC Participation STC on enrollment, conditional on the demographic controls only, and the lower bar reports the estimated effects when the detailed controls—including those for living arrangement and family structure, ASVAB scores, and school behaviors—are included.15 The estimates in Figure 4.2a reveal considerable heterogeneity in the effects of different types of STC programs. In the estimates reflected in the upper bars, with minimal controls, mentoring and school enterprise STC programs are associated with a significantly higher probability of some college education, with the differential ranging from about a 0.07 to a 0.11 higher probability. Relative to the sample proportion of 0.50 with some college, these estimates imply increases of 14 to 22 percent in the probability of college attendance. On the other hand, Tech Prep STC programs are associated with a significantly lower likelihood of college education. Although these estimates reveal the possible magnitudes of the effects of participation in STC programs on college enrollment, they also _____________ 15The regression results and all of the regression controls on which these tables are based are reported in Appendix Tables C.3a and C.3b. 44 indicate that it may be difficult to detect as statistically significant effects on college enrollment that might be regarded as substantively important. For example, consider one of the STC programs the estimated effect of which is not statistically significant in Figure 4.2a, such as job shadowing. The standard error of this estimate is 0.026, which implies that an estimate as large as 0.042 (implying an increase in the probability of college enrollment of 8.4 percent) would not be statistically significant. This is an important caveat to keep in mind throughout the discussion of the estimates in this chapter. In particular, it is critical to remember that failure to find that an estimated effect is significantly different from zero is not the same thing as concluding that the effect is zero. The point estimate that is obtained from the regression model is still the best estimate. A lack of statistical significance of an otherwise positive estimate (as in the above example) simply means that one cannot be very confident that the estimated effect is different from zero. As reflected in the lower bars in Figure 4.2a, when the more detailed set of control variables is added, these estimated effects of participation in STC weaken somewhat. With all of these controls added, the findings indicate that only school enterprise STC programs are significantly associated with college attendance, with a positive effect estimated at a 0.088 higher probability of enrollment. The results for current employment—in Figure 4.2b—are less sensitive to the addition of the more detailed control variables than were the results for college enrollment. In all cases, the estimates point to strongly statistically significant positive effects of co-op programs, with effects on the probability of employment near 0.08, and also evidence of positive effects (near 0.06) of internship/apprenticeship programs, although the estimates of this latter effect are statistically significant only at the 10 percent level. Relative to the proportion of the sample that is currently employed, which is 0.63, the estimates imply that co-op STC programs are associated with approximately 13 percent increases in the likelihood of post-high school employment, and internship/ apprenticeship programs with increases of about 9 percent. In addition to the estimated effects of participation in STC programs, the estimates of the coefficients of the control variables are of some interest; these are reported in full in Appendix Table C.3b and 45 discussed briefly here. Women and blacks are significantly more likely to have attended some college, conditional on the observables,16 whereas blacks are less likely to be employed. In general, household structures without two biological parents are associated with a lower likelihood of college attendance and a higher likelihood of employment, and respondents from smaller households and from households with higher income, and whose mothers are more educated, are more likely to have attended some college. Presumably all of these latter effects reflect the effect of economic resources on college attendance. Also not surprisingly, the ASVAB scores—in particular for word knowledge and math knowledge—are significantly positively associated with college attendance. Finally, all four indicators of troublesome behaviors in high school are significantly associated with a lower probability of attending college, and one of these (fighting in school) is also negatively associated with employment. Thus, these control variables have rather strong associations, in the expected directions, with enrollment and employment outcomes, suggesting that they do relatively little to mediate the effects of STC participation because they are weakly associated with this participation—as documented in part in Figures 4.1a–4.1c—rather than because they are uninformative about enrollment and employment outcomes. Summary of Findings from Basic Analysis Overall, the basic multivariate analysis suggests some positive effects of STC participation on college education and on current employment. More specifically, school enterprise programs are associated with a higher likelihood of obtaining some college education, and co-op and internship/apprenticeship programs are associated with increased employment. As already noted, the extensive set of control variables does relatively little to alter the estimated relationship between STC and employment, although there could still be important unmeasured differences between participants and nonparticipants that underlie the _____________ 16It is a common finding in many datasets that, overall, blacks attend college at lower rates than whites, but that, conditional on factors such as household income, parents’ education, and test scores, blacks are more likely to attend college (see, for example, Cameron and Heckman, 2001). 46 estimated effects on either enrollment or employment. In particular, although the links between co-op or internship/apprenticeship programs and later employment may seem reasonably straightforward, the link between school enterprise and college attainment is less obvious, suggesting that the estimated effect of school enterprise programs may reflect pre-program differences rather than the effects of STC participation. On the other hand, school enterprises may provide important learning opportunities by placing business-related instruction in the context of running a student business. This could result in better academic performance (and hence greater college attendance) or may directly demonstrate to students the value of further education. The following subsections therefore focus more sharply on the question of whether these effects can be interpreted as causal effects of STC participation. Work and Schooling Expectations The next analysis introduces variables measuring respondents’ work and schooling expectations during high school. Recall that these are measured before STC participation. Thus, if there are pre-program differences between participants and nonparticipants, then including these expectations variables is likely to mute any estimated effects of STC participation on enrollment and employment outcomes. For example, those students who early on expect to work right after high school are more likely to participate in programs that will enhance the returns to that work, generating a spurious positive association between, for example, participation in co-op STC programs and postsecondary employment, which would weaken upon including controls for the earlier expectations. The results are reported in Figures 4.3a and 4.3b.17 For each dependent variable, the estimates are first reported—in the upper bars—excluding the expectations variables but including all of the other _____________ 17The regression results on which these figures are based are reported in Appendix Table C.4. 47 Figure 4.3a—College Enrollment Probabilities, Effects of STC Participation, Controlling for Prior Work and Schooling Expectations Figure 4.3b—Employment Probabilities, Effects of STC Participation, Controlling for Prior Work and Schooling Expectations 48 controls included in the second set of estimates in Figures 4.2a and 4.2b, for the subsample of observations—the 1980 and 1981 birth cohorts—for which these expectations data are available. Holding the sample constant isolates the effects of adding the expectations variables. The estimates for this subsample are generally similar to the corresponding estimates for the full sample. Only for school enterprise STC programs is there a significant effect of STC. It is positive, as for the full sample, although a bit larger (0.11 compared with 0.09). For current employment, the estimated effect of internship/apprenticeship programs falls a little and is no longer statistically significant. The estimated effect of co-op programs also falls somewhat (from 0.08 to 0.06) but remains statistically significant at the 10 percent level. The lower bars in each figure report the results when the expectations variables are added. The key result is that there is little change in the estimated coefficients or their statistical significance. The estimated effect of school enterprise on college attendance falls by only about 0.01 and remains statistically significant, and in the employment model the estimated effect of internship/apprenticeship programs is unchanged and the estimated effect of co-op programs changes only modestly. Although these latter two estimates are not statistically significant, the interest in Figures 4.3a and 4.3b lies in asking whether the addition of the expectations variables results in any appreciable changes in the estimated effects of STC. Clearly the answer is no; in other words, once the detailed set of control variables considered in Figures 4.2a and 4.2b are included, there appears to be little remaining bias in the estimated effects of STC stemming from pre-program differences that are reflected in work and schooling expectations. Although the analysis with the expectations data could be done only on a subsample, the finding of no indication of remaining bias implies that the full sample estimates can be used reliably. The evidence, therefore, still points to positive causal effects of school enterprise STC programs on college attendance, and of co-op and internship/apprenticeship programs on employment, in the immediate post-high school period; at 49 the same time, the estimates in Figures 4.3a and 4.3b perhaps suggest some caution in concluding that these effects are very robust.18 School Fixed Effects Finally, Figures 4.4a and 4.4b report on the analysis incorporating the fixed school effects.19 Recall that the merit of this analysis is that it uses only differences in STC participation among individuals in the same school to estimate the effects of STC, eliminating any role for differences in STC participation across schools that may be related to unmeasured school-level differences. For this analysis, attention is restricted to the Figure 4.4a—College Enrollment Probabilities, Effects of STC Participation, with School Fixed Effects _____________ 18Finally, the estimated coefficients of the other included variables (see Appendix Table C.4) show that expectations for a four-year degree are strongly positively associated with actual college attendance. At the same time, these expectations are strongly negatively associated with employment, whereas work expectations for age 30 are strongly positively associated with employment. This latter evidence indicates that the expectations variables are quite informative about subsequent behavior, bolstering the validity of this approach. 19The regression results on which these figures are based are reported in Appendix Table C.5. 50 Figure 4.4b—Employment Probabilities, Effects of STC Participation, with School Fixed Effects subsample of individuals with multiple observations in the same school, as it is only these individuals who reveal the effects of STC programs. The estimates are first reported without the school fixed effects for this subsample and then with the fixed effects; the comparison between these two sets of estimates isolates the effects of holding constant all unmeasured characteristics of students that are common to students in the same school.20 The upper-bar estimates in each figure—for the “fixed effects sample” but excluding the fixed effects—are quite similar to the full sample results (the lower bars in Figures 4.2a and 4.2b). There is a positive and significant estimated effect of school enterprise on college attendance that is virtually the same size (about 0.09), and positive and significant estimates of the effects of co-op and internship/apprenticeship programs on current employment. However, although the estimated _____________ 20In these specifications, the full set of controls are included, but the expectations variables are excluded because they did not matter in Figures 4.3a and 4.3b, and because including them would impose a further sizable sample restriction given that the expectations variables are asked only of the two oldest birth cohorts. 51 effect of co-op programs is of similar magnitude, the estimated effect of internship/apprenticeship programs is larger (0.12 versus 0.06). In addition, the estimated negative effect of Tech Prep programs on college attendance is larger (–0.07 versus –0.04) and statistically significant. The most important questions, though, are how the estimates change when the school fixed effects are added and what the estimates are with the fixed effects included. These latter estimates are reported by the lower bars in each figure. In the estimates for college enrollment, the effects of STC programs strengthen, with the already significant effects of school enterprise and Tech Prep programs growing and a positive and significant effect of job shadowing programs emerging as well. In the estimates for current employment, the estimated positive effect of co-op programs grows from 0.08 to 0.10, remaining statistically significant, whereas the estimated effect of internship/apprenticeship programs falls from 0.12 to 0.07 and becomes statistically significant at only the 10 percent level. The changes in the estimated effects of STC program participation are not large, but they are not trivial either, and in some cases the conclusions are substantively different (for example, with respect to job shadowing). For a number of reasons, the estimates with school fixed effects are the most reliable.21 They therefore serve as the basis for summarizing the results thus far and as the jumping off point for the remaining analyses in this chapter. Provisional Summary and Interpretation To summarize, the analysis thus far points to a number of beneficial effects from certain types of STC programs. In particular, job shadowing and school enterprise programs increase post-high school college attendance. Participating in job shadowing increases the probability of _____________ 21Statistical tests for whether the school fixed effects are needed yield p-values of 0.19 for the college enrollment specification, and 0.23 for the current employment specification. (These are Hausman tests and are computed for fixed effects versus random effects estimates.) Strictly speaking, these results suggest that the school fixed effects can be excluded. But these p-values are nonetheless relatively low, and the standard errors increase very little in the fixed effects estimation—in contrast to what occurs in many applications. Thus, because (1) there is little cost to retaining the school fixed effects, (2) the estimates are a bit different, and (3) including the school fixed effects is unlikely to introduce any bias, the school fixed effects estimates are preferable. 52 postsecondary enrollment by about 0.06, whereas the increase associated with school enterprise STC programs is twice as large. Co-op and internship/apprenticeship STC programs increase employment, with the probability rising by about 0.10 in the case of co-op programs and 0.07 in the case of internship/apprenticeship programs, although the statistical evidence for internship/apprenticeship programs is weaker. In each case, there is no offsetting negative effect on the other dependent variable; for example, co-op programs increase postsecondary employment without decreasing postsecondary enrollment. The implication is that these programs appear to induce higher enrollment or employment at the expense of being neither enrolled nor employed, which casts the results in a more favorable light. Finally, working in the other direction, Tech Prep programs appear to reduce the probability of college enrollment by about 0.10, without any offsetting increases in employment.22 A natural question that arises in light of these estimates is whether the effects that are found are “good” or “bad” from the perspective of evaluating STC programs. It has already been noted that rather than focusing on the effects of STC on careers, broadly speaking, this report instead focuses more narrowly on postsecondary enrollment and employment. Even with this more limited focus, though, one can probe a bit further in trying to assess the evidence. In particular, with respect to the finding that Tech Prep reduces college enrollment, it is of interest to ask whether there is any tradeoff in terms of work. The estimates in Figure 4.4b show no employment effect (although employment and enrollment are not necessarily measured contemporaneously). But in results not reported in the figures (or in the corresponding appendix table), a model for the probability of working full-time, conditional on _____________ 22Multinomial logit models could also be estimated for the four categories: enrolled/employed, enrolled/non-employed, non-enrolled/employed, and nonenrolled/non-employed. If the evidence indicated, for example, that a particular program raised the probability of both employment and enrollment, then there would be a potentially important refinement from the multinomial estimates (although there is no fixed effects multinomial logit estimator), telling us whether these effects came from individuals who were more likely to be both enrolled and employed or more likely to be doing one or the other relative to neither. But given that for each program the effects appear (if at all) for only employment or only enrollment, the multinomial estimates do not provide any additional information. 53 working, was estimated, and the resulting coefficient for Tech Prep, although not statistically significant, was positive and roughly of the same size (but opposite sign) as the estimated effect on enrollment. This suggests that the negative effects of Tech Prep on schooling are roughly offset by a higher incidence of full-time work.23 Nonetheless, as the returns to schooling in the form of higher wages typically outweigh the returns to experience—and even more so with regard to the return to full-time versus part-time experience—it is difficult to view the adverse effect of Tech Prep on schooling in a positive light.24 Heterogeneity in the Effects of STC To this point, the effects of STC participation have been estimated for the full sample. However, these effects may differ across racial or ethnic groups or across other characteristics of individuals that are associated with socioeconomic status or the likelihood of attending college (aside from STC). In fact, STWOA makes some reference to the problems faced by disadvantaged and minority youths. And STC practitioners commonly argue that STC programs are particularly helpful for less-advantaged youths or the broader group of those who in the absence of any intervention are unlikely to go on to college—often termed the “forgotten half” (see, for example, Donahoe and Tienda, 1999). To explore such questions, the key specifications were re-estimated allowing the effects of STC to differ across groups on the basis of race or ethnicity, the ASVAB math knowledge score, mother’s education, the family’s living arrangements, and sex. In each case, a single specification for the pooled sample was maintained but interactions of each of the STC participation variables with indicators for the groups considered (e.g., white, black, and Hispanic) were introduced. School fixed effects were included, which for reasons just discussed provide the most reliable _____________ 23Although the effect on full-time work is not significantly different from zero, the data are even more consistent with offsetting effects on full-time work and schooling. 24A possible counterargument is that the Tech Prep students who forgo postsecondary education face different relative returns to schooling and labor market experience—either lower returns to schooling or higher returns to work. There is, however, no direct evidence on this question. 54 estimates. In the case of the math knowledge test, mother’s education, and living arrangement variables, the estimates in Appendix Table C.3b indicated that these were strongly related to college attendance. And all of the variables aside from the sex of the respondent are related to socioeconomic advantage. Selected results are reported in Figures 4.5a–4.5j. A separate figure is provided for each specification; for example, Figure 4.5a reports results for postsecondary enrollment, with the effects broken out for whites, blacks, and Hispanics.25 To make the results easier to digest, for each specification, only estimated coefficients of the STC-group interactions that are significantly different from zero for at least one group are reported. It is important to keep in mind, though, that in many cases the differences across groups were not statistically significant, in which case the pooled estimates reported in Figures 4.4a and 4.4b cannot be rejected.26 The evidence in these figures should be interpreted more as simply establishing which groups display statistically significant evidence of the effects of STC programs. Furthermore, the caveat above about substantively large estimated effects falling short of statistical significance bears reiterating with respect to these figures, as the precision of the estimates falls with the disaggregation of the estimated effects; this occurs because in many cases the sample of individuals in any subgroup that participates in a particular STC program is quite small. Results from the specification allowing different effects by race and ethnicity are reported in Figures 4.5a and 4.5b. There is generally more evidence of beneficial effects of STC programs for whites, because for whites only there are (statistically significant) positive effects of internship/apprenticeship programs on college attendance and of co-op and Tech Prep programs on employment. At the same time, Tech Prep has a negative effect on schooling only for whites. Also of note is that the only evidence of positive effects of internship/apprenticeship _____________ 25The regression results on which these figures are based are reported in Appendix Table C.6. 26For this reason, in these figures, the statistical significance of the estimates (relative to zero) is not reported. But see Appendix Table C.6 for this information. 55 Figure 4.5a—College Enrollment Probabilities, Effects of STC Participation, Differences by Race/Ethnicity Figure 4.5b—Employment Probabilities, Effects of STC Participation, Differences by Race/Ethnicity 56 programs on employment arises for minorities, with the estimate large (and statistically significant) for blacks and near zero for whites. The finding that school enterprise programs have positive effects on college attendance appears to apply to all groups.27 The results disaggregating the effects by the ASVAB math knowledge score are reported next, in Figures 4.5c and 4.5d. There is no clear message here, as there are some positive and significant effects of alternative programs for each ability group. Figures 4.5e and 4.5f report results for the specification allowing different effects depending on the biological mother’s education. Here, too, there is some evidence of positive program effects (for different programs) in each of the two groups. This is also the case for the estimates in Figures 4.5g and 4.5h, which allow the effects to differ depending on living arrangement. The estimates discussed to this point concern differences among respondents that are likely related to socioeconomic advantage or disadvantage. Although it is difficult to synthesize the disparate estimates, some summary is possible. If whites, those with high ASVAB scores, those with more-educated mothers, and those living with two biological parents are regarded as relatively advantaged, then a couple of conclusions emerge. First, job shadowing and mentoring have some beneficial effects on college enrollment, but only for more-advantaged individuals. There is some evidence of positive effects of school enterprise and internship/apprenticeship STC programs on college enrollment for both more- and less-advantaged individuals and similarly some evidence of negative effects of Tech Prep programs on both moreand less-advantaged individuals. Turning to employment, the evidence of positive effects of co-op STC programs is strongest for more_____________ 27Finally, recall the higher participation rates of blacks in co-op, school enterprise, and Tech Prep programs, which were documented in Figure 4.1a. The evidence in Figures 4.5a and 4.5b suggests that this higher participation does not reflect higher returns to STC for blacks; the alternative, presumably, is that blacks for some reason are more likely assigned to such programs or choose to participate in such programs despite no greater benefit for them—an interesting issue but one that is beyond the scope of this report. 57 Figure 4.5c—College Enrollment Probabilities, Effects of STC Participation, Differences by ASVAB Math Knowledge Scores Figure 4.5d—Employment Probabilities, Effects of STC Participation, Differences by ASVAB Math Knowledge Scores 58 Figure 4.5e—College Enrollment Probabilities, Effects of STC Participation, Differences by Biological Mother’s Education Figure 4.5f—Employment Probabilities, Effects of STC Participation, Differences by Biological Mother’s Education 59 Figure 4.5g—College Enrollment Probabilities, Effects of STC Participation, Differences by Living Arrangement Figure 4.5h—Employment Probabilities, Effects of STC Participation, Differences by Living Arrangement 60 advantaged individuals, whereas the evidence of positive effects of internship/apprenticeship programs arises only for less-advantaged individuals. Job shadowing appears to have some negative effects for both types of individuals. Thus, the combined evidence does not indicate that the benefits of STC accrue primarily or even relatively more to minorities and other less-advantaged students. One exception, though, is that there is some indication that internship/apprenticeship programs may be particularly advantageous for the less-advantaged, as these programs boost college enrollment among those with the lowest test scores and boost employment among blacks and those with less-educated mothers and in nontraditional living arrangements. Finally, Figures 4.5i and 4.5j break out the results separately for men and women. There are some similarities with the combined estimates. In particular, the findings that school enterprise STC programs boost postsecondary education and that co-op programs boost employment appear to hold for both men and women. But there are also some differences. The finding that internship/apprenticeship programs boost employment is driven entirely by men, as for them the estimated effect (0.13) is positive (and significant), whereas for women it is near zero. It appears that job shadowing boosts postsecondary education for men, while internship/apprenticeship programs boost it for women. On the other hand, there is stronger evidence of a negative effect of Tech Prep on the postsecondary education of women. However, for the estimates disaggregating the effects of STC by sex, the differences between the estimated effects (evaluated one program at a time) were never statistically significant at the 10 percent level. This differed from the case for the other breakdowns, where there were always significant differences across groups for at least one of the STC programs. Thus, although the estimated sex differences are intriguing, they should be given less weight. Summary The empirical research presented in this chapter attempts to estimate the causal effects of participation in school-to-career programs on further education and employment in the years immediately after individuals 61 Figure 4.5i—College Enrollment Probabilities, Effects of STC Participation, Differences by Sex Figure 4.5j—Employment Probabilities, Effects of STC Participation, Differences by Sex 62 leave high school. Although the data analyzed are national in scope, the types of programs covered overlap substantially with those that were to some extent spurred by STWOA in California. The new NLSY97 data used in this chapter offer unparalleled opportunities to study the effects of participating in STC programs while confronting the classic problem in program evaluation—namely, pre-existing differences between STC participants and nonparticipants in the likelihoods of post-high school enrollment or employment. In particular, the NLSY97 offers a detailed battery of STC participation questions asked of respondents, data on later work and schooling expectations from questions asked before STC participation, repeated observations of individuals in the same schools, and a rich dataset generally. The analysis taking advantage of these features of the NLSY97 leads to a relatively robust set of findings. As at the end of the last chapter, Figure 4.6 summarizes the findings from this chapter. The evidence indicates that school enterprise STC programs boost post-high school education, and Tech Prep may reduce it, whereas cooperative education and internship/apprenticeship STC programs boost post-high school employment.28 The magnitudes implied by the estimates are reasonable yet also sizable, suggesting that participation in school enterprise, co-op, and internship/apprenticeship STC programs boosts the probabilities of enrollment or employment by about 0.05 to 0.10 relative to a base college attendance rate of 0.50 and a base employment rate of about 0.60. Moreover, in each case of a positive effect, there is no offsetting negative effect on the other dependent variable, implying that these programs appear to induce higher enrollment or employment more at the expense of being neither enrolled nor employed than at the expense of an alternative “productive” activity. Finally, there is also some evidence of differences in the effects of STC programs across groups distinguished by race and ethnicity and other characteristics associated with socioeconomic status, and across men and women. The most important finding regarding these differences is that STC does not appear to be particularly beneficial for _____________ 28There is also evidence that job shadowing boosts college enrollment, but this finding is not as robust across the alternative statistical analyses. 63 Figure 4.6—Summary of Evidence on Effectiveness of STC on Postsecondary Enrollment and Employment disadvantaged students. Instead, there is some evidence of beneficial effects for all groups, although different programs deliver different benefits. One finding that perhaps does stand out, though, is that internship/apprenticeship programs may be particularly advantageous for the less advantaged. This latter finding suggests that further effort 64 should go into trying to establish which populations of students gain more from different types of STC programs. There is, of course, an inherent limitation in using evidence from the NLSY97 to provide information on the effectiveness of STC in California. In particular, evidence from a study like this one can only provide a sense of the likely effects of the specific STC programs used in California, as the NLSY97 analysis is based on individual reports of participation in STC programs that may correspond only approximately to the types of programs used in California. And given the diversity across local providers documented in Chapter 2, evidence from the NLSY97 restricted to California—even if it yielded large enough samples (which it does not)—would still not suffice. Especially given the diversity of programs that flourished under STWOA, direct evaluation of programs in the state will ultimately provide the best evidence. At present, though, the NLSY97 data provide the best evidence available. 65 5. Conclusions Under the federal School-to-Work Opportunities Act of 1994, California received about $140 million to fund the development of a school-to-career system. In particular, STWOA funds were directed toward the provision of general or broad-based STC programs and activities that served all students, helping guide students not only into careers that could be entered immediately after high school or with additional vocational or technical education but also into careers that entail higher education at four-year institutions. STWOA funds were used in California to establish an extensive statewide system of Local Partnerships that were involved in multiple aspects of STC but which clearly played an important role in implementing the types of general STC activities and programs encouraged by STWOA. When STWOA was not reauthorized, this funding dried up, and state funding to cover the gap has not been forthcoming. Although federal funding has continued for Tech Prep and state funding has continued for Career or Partnership Academies—which are much more narrowly focused—the loss of STWOA funds represents more than a one-third decline in funding for STC activities overall and will likely, over time, severely curtail broad-based STC programs and activities.1 Although other states have been more active in making up for the lost federal funds, California’s current budget crisis makes it unlikely that serious state funding of broad-based STC activities and programs will be contemplated in the immediate future. In addition, school reform efforts targeting educational quality have turned increasingly toward testbased standards. However, because a successful school-to-work transition is a critical determinant of socioeconomic success, the ability _____________ 1It is important to note that funding for vocational education writ large far exceeds funding for these programs and activities. Chapter 2 explains why STC is typically construed more narrowly. 67 of the educational system to put students on the path to successful school-to-work transitions should remain an important criterion in assessing educational success. Thus, it is important to focus attention on the ability of the types of broad-based activities and programs supported by STWOA to contribute to this success, especially because, as STWOA and the response of California (and other states) to it suggest, significant efforts may result from rather small investments. The main contribution of this report is a statistical analysis of data from the 1997 National Longitudinal Survey of Youth. This data source includes information on participation by high school students in the United States in a variety of STC programs. The STC programs covered in the NLSY97 overlap substantially with the types of broad-based STC programs encouraged by STWOA and supported in California with STWOA funds, and which would presumably receive further support were the state to restore the funds that disappeared when STWOA was not reauthorized. The NLSY97 data can be used to study the consequences of participation in different types of STC programs for the early career development of the sample members. In particular, given the relatively young ages covered by the sample thus far, the analysis focuses on employment and postsecondary education in the period immediately after leaving high school. Although these outcomes fall short of characterizing the entire school-to-work transition, employment and further education are the two central methods by which individuals acquire new skills and build their careers, and hence positive effects of STC program participation on either or both of these is likely to signal the beginning of a more successful school-to-work transition. An important strength of the NLSY97 dataset is that it provides a number of means of accounting for pre-program differences between STC participants and nonparticipants. Such pre-program differences could generate either overly optimistic or overly pessimistic assessments of the likely effects of STC participation on employment or enrollment, depending on whether those relatively more or relatively less likely to be employed or enrolled after leaving high school—independently of the effects of STC—participate in STC programs. This is a fundamental and well-recognized problem in estimating the effects of a wide variety of 68 government programs, yet one that plagues much of the previous research on the effectiveness of STC. The empirical analysis that this report presents covers the entire United States, because the NLSY97 dataset is not sufficiently large to carry out analyses specific to California. But evidence on the effects of STC participation gained from the data from all states should be informative about the effects of these programs in California, given that there is no reason to believe that these effects do not generalize across states. Of course, policymakers in California may be more interested in assessments of the effectiveness of the specific STC programs that have been established in various localities throughout the state. Although the NLSY97 data cannot speak directly to such questions, this report also presents a summary of findings from a compendium of evaluations of specific local STC programs supported by STWOA in California. Unfortunately, these evaluations fail to provide much decisive evidence of any kind on the effects of these STC programs on postsecondary employment and enrollment or, indeed, on any student outcomes. Thus, research using the NLSY97 provides the best evidence currently available for assessing the effectiveness of the types of broad-based STC programs supported by STWOA in California. Evidence from the NLSY97 The statistical analysis of the NLSY97 covers six types of STC programs: job shadowing, mentoring, co-op programs, school enterprises, Tech Prep, and internships/apprenticeships. Although these six types of programs by no means cover all of the programs and activities pursued by the LPs in California that were set up using grants under STWOA, many of these are among the most common programs and activities the LPs did pursue. Nationally, participation in each of these types of STC programs among high school students ranges from about 7 percent to 18 percent, and over 42 percent of high school students covered in the NLSY97 participated in at least one of the six types of STC programs. Considering first the effects of STC participation on postsecondary education, the evidence indicates that school enterprise STC programs boost college enrollment, with the best estimates suggesting that this type 69 of program boosts the probability of enrollment in higher education in the immediate post-high school period by about 0.13, or 27 percent (given a postsecondary enrollment rate of 0.50), which is a sizable effect. On the other hand, participation in Tech Prep appears to lower the probability of postsecondary enrollment by about 0.095, or 19 percent. In neither case is the effect on enrollment offset by an effect on employment in the opposite direction. If it were, then the conclusions would be more ambiguous. For example, if school enterprise boosts college enrollment but reduces postsecondary employment, then the overall effects on skill formation would be unclear, although the returns to education are likely higher. With respect to Tech Prep, however, there is some evidence that the decline in enrollment is accompanied by an increase in full-time work as opposed to part-time work, which may to some extent mitigate the apparent adverse effects of Tech Prep on further education. The evidence on the effects of STC participation on postsecondary employment point to positive effects from the two types of STC programs that most closely combine high school education with work experience: co-op programs and internships/apprenticeships. Participation in co-op programs boosts the probability of employment in the immediate post-high school period by about 0.09, or 14 percent, and participation in internship/apprenticeship programs increases the probability of employment by about 0.07, or 11 percent, although the statistical evidence for the effects of internship/apprenticeship programs is a bit weaker. As with the enrollment results described above, these are sizable effects. Furthermore, the positive effects of co-op and internship/apprenticeship programs on postsecondary employment are not offset by reduced enrollment, so these results—as with the beneficial effects of school enterprise programs on postsecondary enrollment—indicate unambiguously positive effects on skill formation in the immediate post-high school period. That is, these programs appear to induce higher enrollment or employment mainly at the expense of being neither enrolled nor employed, rather than at the expense of an alternative “productive” activity. There is also some evidence of differences in the effects of STC programs across groups distinguished by race and ethnicity and other 70 characteristics associated with socioeconomic status, and across men and women, although the ability of the data to sharply distinguish among the effects on these different groups is somewhat limited. One finding that does appear to stand out is that internship/apprenticeship STC programs do have particularly beneficial effects for less-advantaged groups, because participation in these programs appears to boost college enrollment among those with the lowest test scores and to boost employment among blacks and those with less-educated mothers and in nontraditional living arrangements. Overall, though, the evidence does not provide a clear message that STC programs are more effective for less-advantaged than for moreadvantaged groups; instead, there is some evidence of beneficial effects for all groups, although different programs deliver diffferent benefits. The potential significance of these results stems from the argument often made by STC practitioners that STC programs are particularly helpful for less-advantaged youths or the broader group of those who in the absence of any intervention are unlikely to go on to college—often termed the “forgotten half.” The data analyzed in this report are not particularly supportive of this position, although they do suggest that some differences may exist—an issue that merits further attention in the implementation and evaluation of STC programs. On the other hand, STWOA was not motivated solely by problems experienced by the less advantaged in the school-to-work transition but was intended to serve students generally, including those who typically attend four-year colleges but would still potentially benefit from greater integration of learning and careers. Recommendations On balance, the evidence from the NLSY97 provides some indication that broad-based STC programs such as those supported by STWOA can increase postsecondary enrollment and employment. However, the prior research on Career Academies and the evidence on Tech Prep presented in this report do not make a compelling case for the effectiveness of these latter two programs, suggesting that at best they may boost the likelihood of full-time employment after leaving high school but with a tradeoff of lower postsecondary education. In light of 71 this evidence, the allocation of existing funding for STC activities merits reconsideration. In particular, with the demise of STWOA, funding has continued for Career Academies and Tech Prep, whereas direct funding for the broad-based types of STC programs (apprenticeships, internships, co-op programs, school enterprises, and so on) has all but disappeared. The evidence from the NLSY97 and the other available research is not overwhelming, but it appears that a case can be made for restoring some funding to the latter types of programs and, barring that, possibly reallocating funds from Career Academies, which are state funded, or from Tech Prep, if possible within the constraints of the federal funding. More generally, the state spends heavily on school reform efforts broadly defined. If STC delivers gains in terms of increasing the socioeconomic success of students, then there is no reason that the larger pool of state spending on educational reform efforts should not be considered as a potential source of funding for STC. At the same time, it is important not to interpret the evidence from the NLSY97 too strongly in the context of STC programs in California. Although this evidence suggests that the type of broad-based programs supported by STWOA in California may have been effective, this evidence is not specific to the efforts of the LPs established in California under STWOA. There is likely enough heterogeneity in STC programs across local jurisdictions that evaluation of local efforts is clearly warranted before drawing firm conclusions. Although there have been many evaluations of the STC efforts of these LPs in California, these evaluations are inadequate for establishing the effectiveness of these efforts and for determining which types of efforts have been most effective. The lack of evidence on the effectiveness of local STC efforts generated in the “round” of STC activities spurred by STWOA, and the limitations inherent in directly applying the findings from the NLSY97 to California’s STC programs, suggest that any future funding increases for broad-based STC activities should be predicated on requiring evaluation of programs. At the same time, given the dearth of evidence on the effectiveness of Career Academies and Tech Prep in California, these two other components of the STC system should be subject to the same evaluation requirements so that—with time—policymakers will be 72 able to make more informed judgments about the best way to allocate funds to the various components of the state’s STC system. Furthermore, the evidence from the NLSY97 on differential effects across subgroups of the population, although not strong, suggests that it would be valuable if future evaluation efforts were designed to look for such differential effects, as this may lead to more effective use of STC resources. Past experience has also demonstrated that simply mandating evaluations is insufficient. Tech Prep provides a good example of this. Perkins III requires accountability of the states in terms of a number of indicators. These cover postsecondary enrollment and employment—in particular, “Placement in, retention in, and completion of, postsecondary education or advanced training, placement in military service, or placement or retention in employment” (U.S. Congress, 1998, § 113). Yet the limitations of the state’s Tech Prep evaluation—in particular its lack of focus on student outcomes—was already noted (Chapter 3). Furthermore, a U.S. Department of Education report (1995) concluded that throughout the nation, few Tech Prep consortia were able to obtain student outcome data such as employment or further education, echoing the conclusions of California’s Tech Prep evaluation seven years later (O’Driscoll et al., 2002d). Similarly, there are evaluation requirements for state-funded Career Academies in California, but such evaluations do not appear to have been performed. Part of the reason may be language in the Education Code that provides a loophole for districts that do not wish to be evaluated. For example, § 54697 (a) of the California Education Code states, “The Superintendent of Public Instruction shall select an entity (the evaluating entity) to conduct a long-term evaluation of the Partnership Academies conducted pursuant to this article using a random assignment of pupils into program and control groups. The participation of any school district in this long-term evaluation is voluntary.” In addition, no funds have been appropriated for such an evaluation; instead, subsection (c) of the same section of the code states that the evaluating entity is responsible for securing the funding for the evaluation. 73 Finally, some of the latest state legislation on STC—AB 1873 (followed up by AB 1765)—specified accountability criteria based on past performance to be eligible for new grants. Among these criteria are “increased academic performance, postsecondary enrollment, decreased dropout rates, transition to appropriate employment, apprenticeship, or any other job training school when applicable, and measurements of pupil, parent, and employer satisfaction” (paragraph 7). But our interviews with LPs indicated that because they were under no mandate to collect data on these measures previously, most had difficulty establishing this accountability, and it is probably safe to say that few if any could do so reliably. This past experience with mandated evaluations, coupled with the poor quality of local efforts to evaluate the effectiveness of LPs established in California under STWOA, suggests that rigorous evaluation should be an important component of future legislation in support of STC activities. Clearly, the mandated evaluation will have to go beyond the types of mandates used in the past. In particular, mandating evaluation but allowing participation on a voluntary basis, or requiring funding from an external source, is unlikely to produce the type of evaluation—if any—needed to reliably assess the effectiveness of STC. And the quality and variability of the local evaluations of activities supported by STWOA indicate that assessments of STC activities would greatly benefit from outside technical assistance and uniformity in designing data collection efforts and ensuring the quality of the evaluations. Finally, the “gold standard” in evaluating most programs is in many ways experimental evidence based on random assignment, because such methods provide a simple solution to the problem posed by unobserved pre-program differences between participants and nonparticipants. Although not a panacea, experimental evidence is likely to be viewed as the most compelling. At the same time, such data as the NLSY97 do offer compelling nonexperimental approaches to this problem. Nonetheless, it is obviously worth considering whether evaluations of 74 STC programs in California could be based on experimental designs.2 And, finally, whether experimental or nonexperimental, evaluation efforts need to be extended beyond estimating the effect of one program or a set of programs. In particular, the evaluation efforts have to be extended to cost-benefit analyses and to comparisons of alternative methods of improving school-to-work transitions of California’s youths, including comparisons between STC efforts broadly defined to include all three components of California’s STC system and school reform efforts focused on other outcomes such as test-based standards. _____________ 2A useful illustration of how evaluation can lead to better policy is suggested by welfare reform. In the early 1990s, states requesting waivers from federal welfare regulations to try implementing their own welfare reforms were required to engage in “serious” evaluations of their waiver programs (typically using random assignment experimental designs). Blank (2002, p. 1122) describes the results of these evaluations—in particular the positive results of welfare-to-work evaluations—as having contributed importantly to the work-oriented federal reforms adopted in 1996. A context closer to STC programs where random assignment has been successfully applied is in the analysis of the effectiveness of the Job Corps program (see, for example, Burghardt et al., 2001). 75 Appendix A National Research on the Effectiveness of STC This appendix provides a more detailed discussion of research on the effectiveness of STC based on evidence from outside California, expanding on the brief discussion in Chapter 3. The 1994 NCRVE report provided a thorough compendium of research on STC programs through that date (Stern et al., 1994). The research summarized in this compendium provides little persuasive evidence of positive effects of these programs on adult labor market outcomes. First, many of these studies do not construct a reasonable comparison group, let alone consider the problem of selection into the program on the basis of unobserved pre-program differences. Second, even those studies that attempt to construct a good comparison group find no beneficial shortterm labor market effects, with the possible exception of those students who remained with the employer with whom they “apprenticed” during the program. Finally, some of the evidence suggests that STC programs may discourage postsecondary education. A subsequent NCRVE report (Urquiola et al., 1997) provides an update on STC. Reflecting the still scant progress toward successful evaluations of STC programs, most of this report focuses on implementation issues. The authors of the report echo a number of concerns about evaluations of STC programs that others have expressed, in particular emphasizing both the specific issues involved in drawing a causal inference and broader issues of how to define success and how to evaluate what some view as a systemic change. However, the report does discuss a couple of new studies of the effects of STC programs on postsecondary outcomes with some attention to pre-program differences. First, an evaluation of Wisconsin’s Youth Apprenticeship Program in printing, by Orr (1996), reported some evidence of higher employment, higher incidence of full-time work, and 77 higher wages for participants, relative to those who took more conventional vocational printing programs. Yet there are some differences between the treatment and control groups, as the evaluation’s author acknowledges. In particular, the apprenticeship program does not serve poorly performing students or those perceived as less likely to be hired by employers, which raises serious doubts about drawing any causal conclusions from the evidence. Second, an evaluation of the Manufacturing Technology Partnership Program in Michigan, by Hollenbeck (1996), reports mixed evidence of increased college enrollment and some evidence of higher employment. In this case, though, for one of the two cohorts studied the control group includes those not selected for the program and those who dropped out—and it is for this cohort that the positive effect on college education appears—making a causal inference suspect. And for the other cohort, the control group includes nonparticipants and nonapplicants, again raising issues of pre-program differences. A more recent major study does little to improve matters. In particular, a report on STWOA by Mathematica, Inc.—the national evaluation of STWOA for the U.S. Congress mandated by the act—does not even attempt to provide a program evaluation, arguing that STC implementation “generally involves broad and diverse initiatives that in varied ways touch most or all students, so it is impossible to distinguish between participants and an unaffected comparison group” (Hershey et al., 1999, p. xviii).1 Nonetheless, this report does present some evidence that is intended to speak to the effects of STC programs. For example, the report notes that students in paid positions arranged as part of STC programs are employed in a wider array of industries and receive more training than other students in paid positions, and it concludes that “Schools develop positions in a wide range of industries, increasing the chances that students can work in a setting relevant to their career interests” (p. 89). However, students who found these jobs as part of _____________ 1Given that the Mathematica report documents variation across school districts and states in the incidence of school-to-work partnerships supported by grants under STWOA, there seems to be a natural way to construct treatment and control groups. Instead, the report focuses on eight states all of which have a high percentage of school districts covered by such partnerships. 78 STC programs may have found the same types of jobs absent such programs; students most likely to do so may simply have sorted into STC programs. Furthermore, there is no evidence of beneficial effects on postsecondary enrollment or employment.2 To some extent paralleling this view of the existing evidence, a recent survey of published academic research on STC across the United States generally supports the claim that although STC is attracting increasing attention from researchers, the existing work has tended to shy away from trying to draw causal inferences (Hughes et al., 2001). Nonetheless, Hughes et al. draw rather rosy conclusions regarding STC, suggesting that “It is perhaps ironic that just as the major federal role in School-to-Work is winding down, the flow of evaluation research with positive findings is increasing” (p. 39). They reach this conclusion despite being able to cite only a handful of studies that use comparison or control groups, most of which do not appear to seriously wrestle with the problem of pre-program differences between participants and nonparticipants. And many of the findings cited in their survey do not appear to be based on comparisons between STC participants and nonparticipants. However, one important exception—which is also cited by Hughes et al.—is the recent (and ongoing) evaluation of Career Academies by the Manpower Demonstration Research Corporation (Kemple and Snipes, 2000; Kemple, 2001, 2003).3 The strength of this study is that it is based on random assignment of students to Career Academies, as participants were chosen randomly from applicants to the Career Academies in the study, with both participants and nonparticipants followed. The results from this study have been reported at different stages as the study participants have aged. The first paper based on this study (Kemple and Snipes, 2000) focused on the effects on participants while in high school, which is not the focus of this report. But it found some beneficial effects, such as increased exposure of students to career or _____________ 2Even if the evidence of a higher likelihood of working in a setting relevant to their career interests is taken at face value, this does not necessarily imply that such workers are doing better. 3The study covers nine schools across the nation, all located in or near urban areas. 79 technical courses and better performance in high school (improved attendance, more credits earned). The second paper (Kemple, 2001) considered students one year after the scheduled completion of high school and found no effects on high school graduation rates, postsecondary education, or employment. However, the most recent paper (Kemple, 2003) presents stronger evidence of such effects. In particular, looking at participants one, two, three, and four years after their scheduled graduation from high school, the evidence for men points to higher full-time employment, earnings, wages, and hours among participants, with these differentials statistically significant and of rather similar magnitude in each of the four years after scheduled graduation. (For women, there is no evidence of such positive effects.) With respect to schooling, Kemple examines numerous measures, including whether the individual was ever enrolled in postsecondary education, his or her highest enrollment, degrees earned, and so on. The evidence for men generates many negative differentials for participants relative to nonparticipants, although only the overall “ever enrolled” differential is statistically significant. Kemple interprets the combined evidence in an entirely positive light, concluding that “The Career Academy-induced improvements in labor market prospects did not come at the expense of opportunities for these young people to enroll in, progress through and complete postsecondary education programs” (p. 1). However, these results could plausibly be read differently. In particular, the negative schooling effects do not appear to be significantly different from what would just offset, say, the greater full-time employment of participants (on the presumption that full-time employment precludes schooling). Thus, the evidence may be most consistent with Career Academy participants substituting full-time work for schooling. One piece of evidence consistent with this is that the wage and earnings differentials that Kemple reports are essentially constant over the four years since scheduled high school graduation, suggesting that participants are not experiencing greater “success” at building careers. Instead, a constant wage or earnings differential may largely reflect a difference owing to higher incidence of full-time work among Career Academy participants—especially because the earnings differential on a monthly 80 basis is about twice as large in relative terms as the wage differential. Thus, a more nuanced view of the evidence from the study is that participation in Career Academies, for men, does seem to increase the orientation toward work but possibly at the cost of less schooling. 81 Appendix B Empirical Methods for the NLSY97 Analysis The core empirical framework is to estimate, at the individual level, the relationship between employment and enrollment in the post-high school period and participation in STC during high school.1 STC participation is categorized by participation in specific types of programs discussed below. The analysis is based on separate models for postsecondary employment and enrollment. Given that employment could be very short-term, whereas college enrollment of any type is nearly always longer-term, the focus is on employment at the time respondents to the NLSY97 are surveyed and on whether the individual was enrolled in college at any time since leaving high school.2 The basic statistical framework used is straightforward. Because the NLSY97 has repeated observations on individuals in the same school, observations on individuals are indexed by both the individual (i) and the school (j). Generically, let Yij be an indicator equal to one when individual i in school j is employed or has enrolled (depending on the analysis), and let STCij be a dummy variable for whether the individual reports participating in an STC program (a set of dummy variables for participation in a variety of STC programs is actually used in the empirical implementation). Linear regressions for Yij as a function of these variables are estimated, of the form: _____________ 1Other outcomes were also considered, including criminal activity, single parenthood, type of postsecondary education, and so on. But the combination of lower incidence of these different or more detailed outcomes and participation rates of about 0.10 to 0.20 in most types of STC programs precluded obtaining sufficiently precise estimates in these other analyses. 2Those who report their employer as the military are coded as employed. 83 Yij = a + bSTCij + e ij . (B.1)3 The estimates of b are denoted b. When a regression is estimated for a variable such as interpreted as the Yefi†fj etchtaot†fcSanTCtakpea†rotnicoipnaltyiotnheovnaltuheesporfozbearboiloitryoonfet,hbe is outcome; for e†xample, a coefficient estimate of 0.1 would imply that STC participation boosts the probability of enrollment, for e†xample, by 0.1, or 10 percentage points. Hence the regression model in this case is referred to as a “linear probability” model.4 Unbiased estimation of b requires that there be no unobserved pre- program differences—captured in eij, which implies that they affect Yij—between STC participants and nonparticipants. As in most research on program evaluatio†n, this cannot necessarily be assumed to hold, and if it does not, then b is a bi†ased estimate of the causal effect of STC programs. Neither longitudinal analysis nor data based on random assignment are available to study the effects of STC programs generally.5 As a result, in this research other methods have to be considered. Each of these methods is subject to some criticisms—but indeed the same can be said of social experiments and longitudinal estimation (in other contexts). The hope is that the “collage” of evidence from the alternative _____________ 3Reported standard errors are robust to heteroscedasticity. Also, because there are often multiple observations on students within the same school, the reported standard errors allow for heteroscedasticity of the error term across schools and nonindependence of an arbitrary nature within schools, which typically results in somewhat larger standard errors. 4Versions of all models discussed in this report were also estimated using logit specifications, and the results were very similar. Using the linear probability model simplifies the presentation and avoids distributional assumptions that underlie the logit (or probit) model. 5In general, longitudinal data on outcomes and participants before and after participation can be used to estimate causal effects of programs. In this approach, the measure of the outcome before the program participation captures the unobserved individual characteristics that might be associated with participation, and the change in the outcome then provides a causal estimate, assuming that the change in outcome did not occur for any other reason. However, in the context of STC, because the object of study is the effects of a program on individuals’ first labor market experiences, or on further school enrollment of those already enrolled, there are no meaningful observations on the outcomes of interest prior to the program, so longitudinal estimation is inapplicable. As noted in Chapter 3, only one study of STC has managed to use random assignment. 84 approaches, along with some analysis of whether each approach is likely to lessen any biases rather than exacerbate them, will help pin down the range of causal estimates of the effects of STC. Controls and Proxies The first approach to unobserved differences across individuals is to introduce into Equation (B.1) an extensive set of controls for the factors that might be correlated with STC participation and might also affect post-high school enrollment and employment. These controls can be thought of in two ways. First, they may be direct measures of some of the factors that are thought to affect enrollment and employment outcomes. Second, they may be proxies for the unobservable factors. For example, parents’ income may have a direct effect on postsecondary enrollment because it influences the affordability of college education. Alternatively, parents’ income may serve as a proxy for the differences in labor market networks faced by children from more affluent families—such as the likelihood that their networks are more rooted in jobs requiring a college degree. The distinction between control variables and proxies is important because when a particular variable is viewed as a proxy rather than a control for something that directly affects the outcome, specific questions arise as to whether it is an acceptable proxy for the unobserved differences across individuals, in the sense that its inclusion eliminates the bias from unobservable or unmeasured pre-program differences. To eliminate the bias, proxy variables have to satisfy three conditions. First, they must be related to the factors underlying the unobserved differences. Second, they must be redundant in the equation, meaning that in the hypothetical case of controlling for STC and the unobservable that actually affects behavior, the proxy has no direct effect on the dependent variables. Third, they have to capture enough of the variation in the unobservable so that once the proxies are included there is no remaining bias from unobserved pre-program differences in the estimates relating postsecondary outcomes to STC participation. As an example, if “career orientation” is viewed as a key unobserved difference across individuals, then the inclusion of variables capturing 85 race and sex in the regression would appear unlikely to satisfy the second or third conditions for a proxy. With regard to the second condition, race and sex may directly affect postsecondary outcomes even after accounting (again, hypothetically) for career orientation; with regard to the third, it seems likely that even after controlling for race and sex, there would still be remaining variation in career orientation that would be correlated with STC participation and the dependent variables. On the other hand, variables capturing work and schooling plans and expectations might more reasonably be viewed as adequate proxies; this case is discussed more below, because the NLSY97 contains information on such expectations. Of course, one can never know with certainty that a given set of proxy variables adequately captures the unobservables related to preprogram differences. But comparing the estimates of the coefficients of STC participation using a narrow set of control variables and a detailed set of proxy variables (that are assumed to satisfy the first condition) can help to gauge whether biases from unobservables remain. Specifically, if the inclusion of the detailed proxy variables has little or no effect on the estimates, then because their inclusion reduces the bias from pre-program differences, it is arguably less plausible (more so the more complete the set of proxy variables) that remaining unobservables generate a correlation between STC participation and enrollment or employment (Wooldridge, 2002). The NLSY97 offers a detailed set of control or proxy variables for characteristics of both the individual respondents and their families. In addition to fairly typical demographic variables (sex, race, ethnicity, and age), the dataset includes three additional sets of variables that are potentially important. These include data on living arrangements and the respondent’s family, test scores from the ASVAB, and self-reported measures of school behavior. This rich array of variables seems likely to capture some and perhaps a good deal of the variation in underlying propensities for post-high school enrollment or employment, including the quality and quantity of human capital investments families have made in their children, resources available in the household, educational norms in the family, labor market networks, the individual respondent’s academic intelligence, and the extent to which the respondent is learning 86 traits such as timeliness and reliability that are valued by employers. But these variables may also directly affect enrollment and employment outcomes, so they perhaps should not be viewed as proxy variables. Work and Schooling Expectations The NLSY97 data also include a particularly compelling set of proxy variables for trying to uncover the causal effects of STC. Specifically, the NLSY97 includes the respondent’s self-reported subjective probabilities for future education and employment, including receipt of a high school diploma by age 20, obtaining a four-year college degree by age 30, and working over 20 hours per week at age 30. These variables were measured in 1997, before the STC participation used in the estimation. The intuition behind using the work and schooling expectations is as follows. Before participating in STC, students are asked about their post-high school work and schooling expectations. Then some participate in STC and some do not, and their post-high school work and schooling behavior is subsequently observed. If, for example, conditional on educational expectations, STC participants are more likely to be enrolled in college after leaving high school, then it is quite sensible to infer a causal effect of STC, because the expectations questions should have controlled for remaining unobservables associated with post-high school educational decisions or outcomes. More formally, these expectations variables may come close to serving as perfect proxy variables. They should easily satisfy the redundancy condition because they should play no independent role net of the unobserved propensities for post-high school enrollment and employment for which they are proxies. And they should satisfy the third condition for a proxy—that once they are included, there should be no remaining unobserved pre-program differences between STC participants and nonparticipants that are related to postsecondary enrollment or employment. 87 Across-School vs. Within-School Variation in STC Participation: School Fixed Effects Finally, there may be other unobserved pre-program differences between STC participants and nonparticipants that are not captured in any of the control or proxy variables, yet which vary systematically across students in different schools. If so, and if there are differences in STC offerings by school, then unless all school-level characteristics that affect postsecondary outcomes are held constant, estimates of Equation (B.1) may still be biased because of unobserved pre-program differences. This in fact appears to be a potentially serious problem. In the NLSY97, in addition to the survey questions administered to individuals, a 2000 survey of schools elicits information on STC programs offered by the schools attended by survey respondents. Estimated models of the relationships between school characteristics and school offerings of STC reveal that many school-level characteristics that are likely to be associated with post-high school outcomes for students are also strongly associated with STC offerings.6 For example, the proportions of students involved in four types of problematic behavior—truancy, pregnancies, alcohol possession, and drugs—are in many cases significantly related to school STC offerings. And teacher characteristics such as pay are also strongly associated with STC offerings. Because variation in offerings of STC programs at the school level is strongly associated with characteristics of schools that are likely to be related to post-high school outcomes for their students, using variation in STC participation that is partly driven by school-level variation in offerings of STC is likely to yield biased estimates.7 _____________ 6See Neumark and Rothstein (2003) for more details and the estimation results. 7The school-level information on offerings of STC programs raises the possibility of using school-level variation in STC programs to estimate the effects of STC programs. In particular, a simple approach to try to assess whether STC programs increase post-high school education or employment would be to look directly at school-level estimates of these post-high school outcomes as functions of the STC programs that schools offer. Such regressions could be easily interpreted from a policy perspective, as they attempt to ask whether getting more STC programs to be offered by schools increases post-high school enrollment or employment. However, for these regressions to correctly answer this question, the observed variation in school STC offerings has to be unrelated to school-level differences in factors 88 The NLSY97 has one additional important feature that can be used to address this problem. Specifically, there are data on multiple students in the same school. Because of this feature, dummy variables for each school in the sample, or school fixed effects, can be added to Equation (B.1) to control for all unobserved factors that are common to students within a school. Estimation of the linear probability model including the school fixed effects (the “school fixed-effects estimator”) holds constant all other unobserved factors common to students in the same school. It therefore identifies the effects of STC participation from the withinschool differences between those who do and do not participate in STC and the differences in outcomes associated with this participation and hence does not rely at all on variation in STC participation that may be driven by the problematic across-school variation in offerings of STC. Of course, since individuals within schools differ from one another, in the within-school estimation attention must still be paid to individuallevel differences. That is, although students in the same school may have much in common, there is no reason to believe, for example, that their career orientations or other factors associated with STC participation and with postsecondary enrollment or employment do not differ. The hope, though, is that the combination of school fixed effects plus other individual-level controls and proxies will fully account for any preprogram differences between STC participants and nonparticipants. Analysis Samples The analysis is based on data from the first four rounds of the NLSY97. When the first round was administered, in 1997, respondents were ages 12–17. With the second round, then, more observations on respondents who have left high school become available. But the numbers go up considerably with the third and fourth rounds, and therefore the research focuses on educational and employment outcomes ______________________________________________________________ influencing postsecondary outcomes for these students. In contrast, for example, if STC programs tend to proliferate in underperforming schools, the beneficial effects of STC would be obscured in the across-school “experiment.” Unfortunately, as described in the text, variation in STC offerings does appear to be related to school-level characteristics. (This approach is closely related to using school offerings of STC as instrumental variables for individual participation in STC in Equation (B.1). The instrumental variables strategy is problematic for exactly the same reason.) 89 measured as of the third or fourth rounds, in 1999 and 2000.8 The samples used for the empirical analysis are obtained from the full NLSY97 sample in a number of steps. First, with respect to the basic sample, of the total 8,984 original respondents to the NLSY97, 8,510 were interviewed in 1999 or 2000 (Rounds 3 and 4). Of these, we restricted our attention to those ages 18 or older, which eliminated about 40 percent of the sample (about 3,500 observations), and to those for whom there is at least one observation (in 1999 or 2000) at which they are not enrolled in high school, yielding 4,234 observations. Because the latest observation comes from Round 4, the latest age from which the postsecondary work and schooling observations is taken is just over 21, and the mean is 19. The sample is restricted in this way to isolate those individuals for whom it is possible to observe the early years of their employment or higher education after leaving high school. In addition to meeting these criteria, information on STC participation is also required. To get an accurate reading on STC participation, and to be able to measure some behaviors and expectations as of a well-defined date before measured STC participation (given that these are measured in Round 1), the focus is on the information provided in the surveys after the first round in 1997, which for each subsequent round covers participation in the past year.9 Requiring STC information after Round 1 drops sample observations for those who did not answer the STC part of the survey after this round, either because they had not spent time in high school in 1997 or a subsequent year or in subsequent years were not enrolled as of the interview date and hence were not asked the STC questions. Coupled with some final sample restrictions on availability of the other data used in the study, this takes the sample down to 3,279 observations. For the baseline analysis sample, private schools and vocational/technical schools are excluded, leaving a sample of 2,933 observations. Finally, for two of the analyses the sample is restricted further. First, although linear models with school fixed effects can be estimated for the _____________ 8Rounds past the fourth were not available when this research was completed. 9However, the robustness of the results to using information on any STC in which the respondent had ever participated was verified. 90 full sample, observations on individuals who are the sole observations from their school contribute no identifying information. Thus, the effective sample size for this analysis includes only individuals with multiple respondents for their school; there are 2,230 such observations. Second, the work and schooling expectations data from 1997 are asked only of those born in 1980 or 1981—the two oldest birth cohorts in the dataset—leaving 2,057 observations from the baseline analysis sample.10 _____________ 10Far fewer than a proportionate number of observations are lost because the other sample restrictions leave a sample of mainly older NLSY97 respondents. Appendix Table C.1 summarizes these sample restrictions and their consequences for the samples analyzed. 91 Appendix C Detailed Tables for NLSY97 Analysis Table C.1 Sample Construction and Analysis Samples Sample Inclusion Criteria Total sample in 1997 Interviewed in 1999 or 2000 Age ≥ 18 at 1999 or 2000 interview Not enrolled in high school by 1999 or 2000 Answered STC questions covering high school after Round 1 † Complete data on baseline controls Baseline analysis sample: exclude private and vocational/technical school students Subset of baseline analysis sample with multiple observations per school Subset of baseline analysis sample with data on work and schooling expectations (asked only of those born in 1980 and 1981) No. of Observations 8,984 8,510 5,047 4,234 3,347 3,279 2,933 2,230 2,057 NOTES: The 1999 or 2000 interview is used as the “post-high school interview,” choosing the earliest one at which the respondent is age 18 or older and no longer enrolled in high school. More restrictive sample inclusion criteria are imposed in each successive row, except for the last two rows. Baseline controls include race/ethnicity, education, and family structure (whether the respondent lives with one, two, or no biological parents, and which ones, and household size). When other control variables are introduced in the regression models, dummy variables indicating missing data are included. 93 Table C.2 Participation Shares and Linear Probability Estimates of Individual Characteristics Associated with STC Participation 94 Demographic Female Black Hispanic Age Living arrangement/family Urban Biological parent and stepparent Biological mother only Biological father only Other arrangement Household size Log household income Biological mother’s schooling ASVAB Arithmetic reasoning Word knowledge Paragraph comprehension Math knowledge Job Any STC Shadowing Mentoring Co–op Share Participating in Each Type of STC Program .415 .175 .084 .135 Estimates of Participation Models School Enterprise .067 .008 .063** .022 .011 –.048* –.003 –.026 .010 –.004 –.014** .006 –.001 .023 –.007 –.005 .012 –.033 –.016 –.035* –.025 –.033 –.005 .006 .000 .014 .022 .022 .005 .004 –.016 –.034*** –.015 .032 –.004 .002 .001 –.010 .043** .004 .007 .004 –.013 .008 .026 .002 –.000 .001 –.001 .004 .053*** .004 –.006 .002 .015 –.010 .090 –.015 –.005 .002* –.001 –.006 –.011 –.016 –.000 –.001 –.014 –.007 –.015 –.013 .002 –.006 .013 –.002 .006 –.012 –.011 –.006 .017 .015 –.012 Internship/ Tech Prep Apprenticeship .110 .097 –.027** .057*** .028 –.000 –.039* .003 –.015 .024 –.019 –.001 –.001 –.001 .010 .022* –.020 –.019 .004 .008 .024 –.001 –.002 –.000 –.016 –.048 –.002 –.011*** .002 .005** –.007 –.032*** –.001 .019* Table C.2 (continued) 95 Job School Internship/ Any STC Shadowing Mentoring Co–op Enterprise Tech Prep Apprenticeship School behaviors Threatened at school .060** .022 .008 .010 .009 .023 .018 Got into physical fight at school Late with no excuse 2+ times Absent 2+ weeks R2 –.010 –.009 –.072** .016 –.021 .018 –.059* .014 –.000 –.018 –.024 .009 .014 –.009 –.046** .011 .017 .020* –.022 .016 –.022 –.002 –.002 .016 –.003 .010 –.040** .013 NOTES: Figures 4.1a–4.1c are based on the results in this table. There are 2,933 observations. ***, **, and * indicate that the estimated coefficients are significantly different from zero at the 1, 5, and 10 percent level, respectively. In the data, anyone of Hispanic origin is coded as Hispanic, and blacks must be non–Hispanic. Students in private schools and vocational/technical schools are excluded. Dummy variables are included for missing data on some individual variables. STC participation is measured based on responses from the 1998, 1999, and 2000 interviews regarding STC participation while in high school; the 2000 data are used only if the 2000 interview is the post–high school interview defined in the notes to Table C.1. All independent variables are defined as of 1997. The ASVAB test scores are standardized to have mean 0 and standard deviation of 1 for the calibration sample that took the test. The subset of ASVAB scores used are those that make up the Armed Forces Qualification Test. Statistical inferences are based on standard errors that are robust to heteroscedasticity and were adjusted to account for the clustering of observations within schools, allowing for nonindependence within schools and heteroscedasticity across schools. Table C.3a Linear Probability Estimates of the Effects of STC Participation on College Attendance and Employment 96 Some College Employment Job shadowing Mentoring Co–op School enterprise Tech Prep Internship/apprenticeship Control variables included Demographic Living arrangement/family ASVAB School behaviors R2 (1) .015 (.026) .066* (.036) –.019 (.028) .112*** (.037) –.059** (.030) .045 (.032) X .053 (2) .016 (.025) .043 (.034) –.004 (.027) .100*** (.036) –.051* (.030) .021 (.031) X X .148 (3) .042* (.024) .029 (.031) .010 (.026) .083** (.033) –.042 (.029) .022 (.030) X X X .255 (4) .037 (.023) .026 (.031) .007 (.026) .088*** (.033) –.042 (.030) .021 (.030) X X X X .274 (5) .006 (.025) –.035 (.033) .079*** (.028) .025 (.037) –.000 (.028) .053* (.030) X .019 (6) .003 (.025) –.030 (.033) .078*** (.028) .018 (.037) –.005 (.028) .058* (.030) X X .030 (7) .004 (.025) –.030 (.033) .078*** (.028) .015 (.037) –.006 (.028) .060** (.031) X X X .030 (8) –.000 (.025) –.029 (.033) .078*** (.028) .016 (.037) –.007 (.028) .059* (.030) X X X X .037 NOTES: The results in Figures 4.2a and 4.2b are based on the results in this table. There are 2,933 observations in all of the specifications. The estimated effect on probability of college attendance (columns (1)–(4)) and employment (columns (5)–(8)) are reported. School and work outcomes are measured as of the post–high school interview (1999 or 2000). Asterisks denote statistical significance of the coefficient estimates, with ***, **, and * indicating that the estimate is significantly different from zero at the 1, 5, and 10 percent level, respectively. STC participation is defined as described in Table C.2. The standard errors allow for general heteroscedasticity and were adjusted to account for the clustering of observations within schools, allowing for nonindependence within schools and heteroscedasticity across schools. The sets of control variables are listed in detail in Table C.3b. There are dummy variables for a small number of cases with missing data for the following variables: log household income, biological mother’s schooling, ASVAB scores, and urban residence. Table C.3b Linear Probability Estimates of Control Variable Coefficients for Models in Table C.3a Specification Demographic Female Black Hispanic Age Living arrangement/family Urban Biological parent and stepparent Biological mother only Biological father only Other arrangement Household size excluding youth Log household income Biological mother’s schooling ASVAB Arithmetic reasoning Word knowledge Paragraph comprehension Math knowledge Some College Table C.3a, Col. (4) (1) Employment Table C.3a, Col. (8) (2) .090*** (.017) .060*** (.023) .008 (.024) .034** (.014) –.023 (.019) –.115*** (.027) –.020 (.027) .051*** (.015) .027 (.020) –.049* (.026) –.108*** (.021) –.088* (.050) –.068 (.044) –.015*** (.006) .010** (.004) .023*** (.003) –.022 (.023) .087*** (.028) .027 (.024) .190*** (.045) –.016 (.050) –.006 (.006) .001 (.005) –.010*** (.003) –.011 (.017) .047*** (.017) .014 (.017) .156*** (.016) –.012 (.020) .023 (.019) .000 (.019) –.009 (.020) 97 Specification School behaviors Threatened at school Got into physical fight at school Late with no excuse 2+ times Absent 2+ weeks Some College Employment Table C.3a, Col. (4) Table C.3a, Col. (8) –.078*** (.023) –.096*** (.025) –.039** (.020) –.121*** (.028) –.029 (.023) –.080*** (.030) .006 (.022) –.048 (.032) NOTES: All specifications correspond to Table C.3a, columns (4) and (8). See the notes to Table C.3a for details. 98 Table C.4 Linear Probability Estimates of the Effects of STC Participation on College Attendance and Employment, Incorporating Prior Work and Schooling Expectations Some College Employment Job shadowing Mentoring Co-op School enterprise Tech Prep Internship/apprenticeship Expectations High school diploma by age 20 Four-year degree by age 30 Work over 20 hours/week at age 30 R2 (1) .024 (.028) .019 (.039) .021 (.031) .113*** (.040) –.046 (.038) .012 (.036) (2) .014 (.027) –.008 (.038) .030 (.030) .104*** (.039) –.016 (.035) .016 (.035) (3) .017 (.030) –.007 (.041) .055* (.033) –.025 (.048) .031 (.033) .052 (.037) — .010 — (.072) — .428*** — (.035) — .054 — (.064) .271 .320 .051 (4) .018 (.030) .008 (.041) .052 (.033) –.019 (.049) .030 (.033) .052 (.037) .025 (.090) –.101** (.043) .226*** (.079) .057 NOTES: The results in Figures 4.3a and 4.3b are based on the results in this table. The work and schooling expectations data come from questions asked in Round 1 (in 1997) and are asked only of those born in 1980 or 1981—the two oldest birth cohorts in the dataset. Because of this restriction, there are 2,057 observations. See the notes to Tables C.2 and C.3a for details. All of the specifications include the demographic, living arrangement/family, ASVAB, and school behavior variables that are the same as those included in columns (4) and (8) of Table C.3a. The standard errors allow for general heteroscedasticity and were adjusted to account for the clustering of observations within schools, allowing for nonindependence within schools and heteroscedasticity across schools. 99 Table C.5 School Fixed Effects Linear Probability Estimates of the Effects of STC Participation on College Attendance and Employment Job shadowing Mentoring Co-op School enterprise Tech prep Internship/apprenticeship Hausman test for excluding school fixed effects, p-value School fixed effects Some College (1) .035 (.027) .018 (.034) .004 (.031) .091** (.038) –.070** (.036) .038 (.036) — (2) .063** (.030) .048 (.039) –.013 (.035) .133*** (.048) –.095** (.040) .055 (.041) .19 Employment (3) (4) –.019 (.028) –.031 (.037) .075** (.033) –.002 (.047) .011 (.032) .116*** (.035) — –.026 (.035) –.057 (.047) .102*** (.037) –.018 (.056) .036 (.041) .073* (.043) .23 XX NOTES: The results in Figures 4.4a and 4.4b are based on the results in this table. There are 2,230 observations, because only observations on respondents in schools with multiple observations are included. All of the specifications include the demographic, living arrangement/family, ASVAB, and school behavior variables that are the same as those included in columns (4) and (8) of Table C.3a. See the notes to Table C.3a for details. The standard errors are robust to heteroscedasticity. The Hausman test is computed for the vector of STC coefficients. The test is based on random versus fixed effects and comes from the regression form of the test with standard errors robust to general heteroscedasticity and nonindependence of observations within schools (Wooldridge, 2002, Chapter 10). 100 Table C.6 School Fixed Effects Linear Probability Estimates of the Effects of STC Participation on College Attendance and Employment, Specifications Interacting STC Participation with Respondent Characteristics 101 Race/Ethnicity White Black Hispanic (1) (1’) (1’’) Job shadowing Mentoring Co–op School enterprise Tech Prep Internship/ apprenticeship .119** –.166** .084* .113 –.082 –.042 .253** .067 .086 Job shadowing Mentoring Co–op School enterprise Tech Prep .159*** –.015 .084 .109** –.023 –.085 ASVAB Math Knowledge Biological Mother’s Score Education High Some School High Medium Low College or Less (2) (2’) (2’’) (3) (3’) .122* Some College .122** .050 .009 .065 Living Arrangement Lives with Two Biological Other Living Parents Arrangement (4) (4’) .086** .037 .145** .193* .005 .166** .113* –.222*** –.033 .175* –.037 .083 –.162*** .054 –.173** .052 –.048 .178** Employment .067 .069 .112 .132* .168*** .068 .090* .047 –.125** .137*** .002 –.132** .048 .053 Sex Men Women (5) (5’) .097** .001 .032 .087* .113 –.072 .149** –.125** .005 .100* .113** .088 Table C.6 (continued) 102 Race/Ethnicity Internship/ apprenticeship Range Observations White (1) .012 — 1,295 Black Hispanic (1’) (1’’) .220** — 503 .081 — 432 ASVAB Math Knowledge Score High (2) Medium (2’) Low (2’’) > .75 687 0 – .75 < 0 547 625 Biological Mother’s Education High Some School College or Less (3) (3’) Living Arrangement Lives with Two Biological Other Living Parents Arrangement (4) (4’) .017 .145** .033 —— — 889 1,191 1,230 .117* — 1,000 Sex Men Women (5) (5’) .133** — 1,084 .015 — 1,146 NOTES: The results in Figures 4.5a–4.5j are based on the results in this table. See the notes to Tables C.3a and C.5. Specifications include STC program participation interacted with dummy variables for each group; each numbered set of columns (e.g., (1) through (1’’)) corresponds to a separate specification. They include school fixed effects and are in other respects identical to Table C.5. The “high,” “medium,” and “low” cutoffs for the estimates in columns (2)–(2’’) were chosen to assign roughly one-third of each group to each subsample. Bibliography Bechtold, Deborah A., and Dixie L. King, Partnerships Plus! 2000–2001 Evaluation Report: Partnership Academy Outcomes for Kern County, Transforming Local Communities, Bakersfield, California, 2001. Blank, Rebecca M., “Evaluating Welfare Reform in the United States,” Journal of Economic Literature, Vol. 60, No. 4, December 2002, pp. 1105–1166. Burghardt, John, Peter Z. Schochet, Sheena McConnell, Terry Johnson, R. Mark Gritz, Steven Glazerman, John Homrighausen, and Russell Jackson, Does Job Corps Work? Summary of the National Job Corps Study, Mathematica Policy Research, Inc., Princeton, New Jersey, 2001. Butler, Michael, Jia Wang, Mikala Rahn, Patricia O’Driscoll, and Van Villanueva, Evaluation of the Verdugo School-to-Career Partnership: PLUS Evaluation of Clark Magnet High School, PublicWorks, Inc., Pasadena, California, 2002a. Butler, Michael, Jia Wang, Mikala Rahn, Patricia O’Driscoll, and Van Villanueva, Evaluation of the UNITE-LA School-to-Career Partnership: PLUS Evaluation of LAUSD Career Academies, PublicWorks, Inc., Pasadena, California, 2002b. California Association of Regional Occupational Centers and Programs, ROCP Facts at a Glance 2002–2003, San Juan Capistrano, California, 2003. California Department of Education, Fiscal and Administrative Services Division, Report on the Budget Act of 2002, Sacramento, California, 2002, available at http://www.cde.ca.gov/fiscal/budgetact/ report2002.pdf (retrieved May 29, 2003). California Department of Education, Regional Occupational Centers and Programs, Sacramento, California, 2003a, available at http:// www.cde.ca.gov/rocp/ (retrieved April 20, 2003). 103 California Department of Education, What is Tech Prep? Sacramento, California, 2003b, available at http://www.cde.ca.gov/shsd/aci/tp/ (retrieved April 11, 2003). California Department of Education, “2003–04 Final Budget Summary,” Sacramento, California, 2004a, available at http://www. cde.ca.gov/fiscal/budgetact/budsum0304.pdf (retrieved 2/4/2004). California Department of Education, “Education Budgets and Related Legislation,” Sacramento, California, 2004b, available at http://www. cde.ca.gov/fiscal/budgetact (retrieved 3/22/2004). California Department of Finance, “Budget Documents,” Sacramento, CA, 2003, 2004, available at http://www.dof.ca.gov/HTML/ BUD_DOCS/Bud_link.htm (retrieved 3/22/2004). California Employment Development Department, Sacramento, California (personal communication, Edward Armijo), October 2002. California Employment Development Department, “Implementation Grant Award Announcement,” Sacramento, California, n.d., available at http://wwwstc.cahwnet.gov/page.asp?page=gi_impgrantaward. htm&cat=content (retrieved May 29, 2003). California Office of Legislative Counsel, “Bill Number: AB 1765,” Sacramento, California, 2004, available at http://www.leginfo.ca.gov/ pub/bill/asm/ab_1751-1800/ab_1765_bill_20030729_enrolled.html (retrieved 3/22/2004). Cameron, Stephen V., and James J. Heckman, “The Dynamics of Educational Attainment for Black, Hispanic, and White Males,” Journal of Political Economy, Vol. 109, No. 3, June 2001, pp. 455–499. Career Academy Support Network, What Is a Career Academy? Graduate School of Education, University of California, Berkeley, California, 2003a, available at http://casn.berkeley.edu/ ProjectDescription.html (retrieved June 19, 2003). Career Academy Support Network, Career Academy Support Network Homepage, Graduate School of Education, University of California, Berkeley, California, 2003b, available at http://casn. berkeley.edu/ (retrieved June 19, 2003). 104 Christensen, Heidi, Ventura County School-to-Career Case Studies Evaluation, Ventura County Superintendent of Schools Office, Camarillo, California, 2002. Commission on the Skills of the American Workforce, “America’s Choice: High Skills or Low Wages,” National Center on Education and the Economy, Rochester, New York, 1990. Donahoe, Debra, and Marta Tienda, “Human Asset Development and the Transition from School to Work: Policy Lessons for the 21st Century,” 1999 (unpublished manuscript). Fattahi, Behrooz, Youth Leadership Bakersfield, 2001–2002: Program Evaluation, Greater Bakersfield Chamber of Commerce, Bakersfield, California, 2002. Forouzesh, Mohammed R., Evaluation Report: Impact of AB-1873 Funding in Improving STC Delivery, Costa Mesa, California, 2002. Forouzesh, Mohammed R., and Orange County Department of Education Instructional Services Division, Orange County Coalition/Vision 2020 School-to-Career Case Studies Final Evaluation Report 2001–2002, Costa Mesa, California, 2002. Glazer, Nathan, “A Human Capital Policy for the Cities,” The Public Interest, No. 112, Summer 1993, pp. 27–49. Hamilton, Stephen, Apprenticeship for Adulthood, The Free Press, New York, 1990. Hayward, G. C., and C. S. Benson, “The Changing Role of VocationalTechnical Education in the United States,” CenterWork, Vol. 4, No. 2, National Center for Research on Vocational Education, 1993. Heckman, James, “Assessing Clinton's Program on Job Training, Workfare, and Education in the Workplace,” NBER Working Paper No. 4428, Cambridge, Massachusetts, August 1993. Heckman, James J., Robert J. Lalonde, and Jeffrey A. Smith, “The Economics and Econometrics of Active Labor Market Programs,” in Orley C. Ashenfelter and David Card, eds., Handbook of Labor Economics, Vol. 3A, Elsevier, Amsterdam, 1999, pp. 1865–2097. 105 Hemsley, Rita, San Bernardino County Horizons Partnership: Year 4 Report, California Educational Research Cooperative, University of California, Riverside, California, 2002. Hershey, Alan M., Marsha K. Silverberg, and Joshua Haimson, Expanding Options for Students: Report to Congress on the National Evaluation of School-to-Work Implementation, Mathematica Policy Research, Inc., Princeton, New Jersey, 1999. Hollenbeck, Kevin, An Evaluation of the Manufacturing Technology Partnership (MTP) Program, Technical Report No. 96-007, Upjohn Institute, Kalamazoo, Michigan, 1996. Hughes, Katherine L., Stephen F. Hamilton, and Robert J. Ivry, Schoolto-Work: Making a Difference in Education, Institute on Education and the Economy, Teachers College, Columbia University, New York, New York, 2001. Kemple, James J., Career Academies: Impacts on Students’ Initial Transitions to Post-Secondary Education and Employment, Manpower Demonstration Research Corporation, New York, New York, 2001. Kemple, James J., “Career Academy Impacts on Labor Market Outcomes and Educational Attainment,” working draft, Manpower Demonstration Research Corporation, New York, New York, 2003. Kemple, James J., and Jason C. Snipes, Career Academies: Impacts on Students’ Engagement and Performance in High School, Manpower Demonstration Research Corporation, New York, New York, 2000. Lerman, Robert I., and Hilliard Pouncy, “The Compelling Case for Youth Apprenticeship,” The Public Interest, No. 101, Fall 1990, pp. 62–77. Lindler, Vanessa, and Laura Hill, San Mateo County School-to-Career Partnership Second Year Evaluation, The Sphere Institute, Burlingame, California, 2000. Maher, Fred, “Vo-tech’s Role in Public Education Debated at Forum,” Temple Times Online Edition, Vol. 33, No. 18, February 13, 2003, available at http://www.temple.edu/temple_times/2-13-03/votech. html (retrieved August 11, 2003). 106 Maxwell, Nan L., and Victor Rubin, Career Academy Programs in California: Outcomes and Implementation, California Policy Research Center, University of California, Berkeley, California, 2001. Neumark, David, “Evaluating the Effectiveness of the School-to-Work Opportunities Act in California,” Working Paper No. 2004.02, Public Policy Institute of California, San Francisco, California, 2004. Neumark, David, and Ann Allen, “What Do We Know About the Effects of School-to-Work? A Case Study of Michigan,” Journal of Vocational Education Research, Vol. 28, No. 1, 2003, pp. 59–84. Neumark, David, and Donna Rothstein, “School-to-Career Programs and Transitions to Employment and Higher Education,” Working Paper No. 2003.22, Public Policy Institute of California, San Francisco, California, August 2003. O’Driscoll, Patricia, Mikala L. Rahn, and Gina Jesse, UNITE-LA Schoolto-Career Evaluation: Final Report 2001–2002, PublicWorks, Inc., Pasadena, California, 2003. O’Driscoll, Patricia, Mikala L. Rahn, Pam Marx, and Tom South, Core Case Study Evaluation: Sonoma County School-to-Career Partnership, PublicWorks, Inc., Pasadena, California, 2002a. O’Driscoll, Patricia, Mikala L. Rahn, Pam Marx, and Shannon Faris, Core Case Study Evaluation: UNITE-LA School-to-Career Partnership, PublicWorks, Inc., Pasadena, California, 2002b. O’Driscoll, Patricia, Mikala L. Rahn, Pam Marx, and Van Villanueva, Core Case Study Evaluation: Verdugo School-to-Career Partnership, PublicWorks, Inc., Pasadena, California, 2002c. O’Driscoll, Patricia, Pam Marx, Tom South, Van Villanueva, Mikala L. Rahn, and Gina Jesse, Tech Prep Connections in California: The Right Direction for High Schools and Community Colleges, PublicWorks, Inc., Pasadena, California, 2002d. Office of Technology Assessment, United States Congress, Learning to Work: Making the Transition from School to Work, Publication No. OTA-HER 637, U.S. Government Printing Office, Washington, D.C., 1995. 107 Orr, Margaret Terry, Wisconsin Youth Apprenticeship Program in Printing: Evaluation 1993–1995, Jobs for the Future, Boston, Massachusetts, 1996. Schmidt, Tracy, A Survey of School-to-Work Actions: The 2000 State Legislative Sessions, National Conference of State Legislators, Washington, D.C., 2000. Schmidt, Tracy J., Inside the State House: The People and Ideas that Shaped School-to-Career Legislation, National Conference of State Legislatures, Washington, D.C., 2001. Stern, David, Charles Dayton, Il-Woo Paik, Alan Weisberg, and John Evans, “Combining Academic and Vocational Courses in an Integrated Program to Reduce High School Dropout Rates: SecondYear Results from Replications of the California Peninsula Academies,” Educational Evaluation and Policy Analysis, Vol. 10, No. 2, Summer 1988, pp. 161–170. Stern, David, Neal Finkelstein, James R. Stone, John Latting, and Carolyn Dornsife, Research on School-to-Work Transition Programs in the United States, National Center for Research in Vocational Education, University of California, Berkeley, California, 1994. Urquiola, Miguel, David Stern, Ilana Horn, Carolyn Dornsife, Bernadette Chi, Lea Williams, Donna Meritt, Katherine Hughes, and Thomas Bailey, School to Work, College and Career: A Review of Policy, Practice, and Results, 1993–1997, National Center for Research in Vocational Education, University of California, Berkeley, California, 1997. U.S. Congress, Carl D. Perkins Vocational and Applied Technology Education Amendments of 1998, Public Law 105-332, 105th Congress, October 31, 1998. U.S. Department of Education, The Emergence of Tech-Prep at the State and Local Levels—1995, Washington, D.C., 1995. U.S. Department of Education, Office of Vocational and Adult Education, “FY1998 Allocations for Carl D. Perkins Vocational and Technical Education Act,” Washington, D.C., 1998. 108 U.S. Department of Education, Office of Vocational and Adult Education, “FY1999 Allocations for Carl D. Perkins Vocational and Technical Education Act,” Washington, D.C., 1999. U.S. Department of Education, Office of Vocational and Adult Education, “FY2000 Allocations for Carl D. Perkins Vocational and Technical Education Act,” Washington, D.C., 2000. U.S. Department of Education, Office of Vocational and Adult Education, “FY2001 Allocations for Carl D. Perkins Vocational and Technical Education Act,” Washington, D.C., 2001. U.S. Department of Education, Facts about the Tech-Prep Program, Washington, D.C., 2002a. U.S. Department of Education, Office of Vocational and Adult Education, “FY2002 Allocations for Carl D. Perkins Vocational and Technical Education Act,” Washington, D.C., 2002b. U.S. Department of Education, Office of Vocational and Adult Education, “FY2003 Allocations for Carl D. Perkins Vocational and Technical Education Act,” Washington, D.C., 2003. U.S. General Accounting Office, Preparing Noncollege Youth for Employment in the U.S. and Foreign Countries, Washington, D.C., May 1990. WestEd and MPR Associates, Inc., California School-to-Career: Helping Students Make Better Choices for Their Future, Final Evaluation Report, WestEd, San Francisco, California, 2002. Wilson, Pete, “Executive Order W-94-94,” Executive Department, State of California, Sacramento, California, June 28, 1994. Wooldridge, Jeffrey M., Econometric Analysis of Cross Section and Panel Data, The MIT Press, Cambridge, Massachussetts, 2002. 109 About the Author DAVID NEUMARK David Neumark is a senior fellow in economics at the Public Policy Institute of California and a research associate of the National Bureau of Economic Research. He has published numerous studies on school-towork, workplace segregation, sex discrimination, the economics of gender and the family, affirmative action, aging, minimum wages, and living wages. He is on the editorial boards of Contemporary Economic Policy and Economics of Education Review. He has also held positions as professor of economics at Michigan State University, assistant professor of economics at the University of Pennsylvania, and economist at the Federal Reserve Board. He holds a Ph.D. in economics from Harvard University. 111 Related PPIC Publications The Changing Role of Education in the California Labor Market Julian R. Betts Math Matters: The Links Between High School Curriculum, College Graduation, and Earnings Heather Rose and Julian R. Betts “At Home and in School: Racial and Ethnic Gaps in Educational Preparedness” California Counts: Population Trends and Profiles Volume 3, Number 2, November 2001 Jennifer Y. Cheng PPIC publications may be ordered by phone or from our website (800) 232-5343 [mainland U.S.] (415) 291-4400 [Canada, Hawaii, overseas] www.ppic.org 113" ["post_date_gmt"]=> string(19) "2017-05-20 09:36:28" ["comment_status"]=> string(4) "open" ["ping_status"]=> string(6) "closed" ["post_password"]=> string(0) "" ["post_name"]=> string(8) "r_504dnr" ["to_ping"]=> string(0) "" ["pinged"]=> string(0) "" ["post_modified"]=> string(19) "2017-05-20 02:36:28" ["post_modified_gmt"]=> string(19) "2017-05-20 09:36:28" ["post_content_filtered"]=> string(0) "" ["guid"]=> string(50) "http://148.62.4.17/wp-content/uploads/R_504DNR.pdf" ["menu_order"]=> int(0) ["post_mime_type"]=> string(15) "application/pdf" ["comment_count"]=> string(1) "0" ["filter"]=> string(3) "raw" ["status"]=> string(7) "inherit" ["attachment_authors"]=> bool(false) }