Donate
Independent, objective, nonpartisan research

Academic Progress for English Learners: The Role Of School Language Environment and Course Placement in Grades 6-12, Technical Appendix

Database

This is the content currently stored in the post and postmeta tables.

View live version

object(Timber\Post)#3742 (44) { ["ImageClass"]=> string(12) "Timber\Image" ["PostClass"]=> string(11) "Timber\Post" ["TermClass"]=> string(11) "Timber\Term" ["object_type"]=> string(4) "post" ["custom"]=> array(5) { ["_wp_attached_file"]=> string(20) "0119lhr-appendix.pdf" ["wpmf_size"]=> string(7) "1529304" ["wpmf_filetype"]=> string(3) "pdf" ["wpmf_order"]=> string(1) "0" ["searchwp_content"]=> string(77015) "Academic Progress for English Learners The Role of School Language Environment and Course Placement in Grades 6-12 Technical Appendices CONTENTS Appendix A. LTEL and LAEL Definitions Appendix B. Course Assignment Appendix C. Qualitative Approach Appendix D . Regressions and Results Laura Hill, Julian Betts, Megan Hopkins, Magaly Lavadenz, Karen Bachofer, Joseph Hayes, Andrew Lee, Marco A. Murillo, Tara Vahdani, and Andrew C. Zau Supported with funding from the William T. Grant Foundation PPIC.ORG Technical Appendices Academic Progress for English Learners 2 Appendix A: LTEL and LAEL Definitions The official definition of LTELs in California is ELs who have been designated as ELs for at least six years. LAUSD defines them as ELs who have been designated as ELs for at least five years. SDUSD treats ELs who have spent five years in the district as students at -risk of becoming LTELs. Given these concerns in the two districts, we defined LTELs as those who have spent 5 or more years in the given district as ELs without having been reclassified. We replicated many of our regression results using the “six -year” definition and results were highly similar. We define a LAEL as a student whose first enrollment in the given district occurs in grade 6 or higher, the student is determined to be an English Learner, and scores at the lowest (Beginning) level on first taking the CELDT. English Learners who fall into neither category are excluded from our analyses. PPIC.ORG Technical Appendices Academic Progress for English Learners 3 Appendix B: Course Assignmen t Overview of district policy in LAUSD Figure B1 below illustrates how the LAUSD ELD placement policy environment evolved over the years of our study, particularly in terms of shifts in the tests of English -language ability that were meant to guide that placement. Throughout all study years, a student’ s DPI and CELDT were district-sanctioned indicators for making initial placement decisions. The ELA -CST was similarly referenced from 2006 through its phase -out in 2013 (with scores from this end year still us ed in 2014). Although the ELA periodic assessme nt was used for placement in 2015 only, the DIBELS, SRI, and CAHSEE, whose placement importance also began in that year, continued as reference points in 2016. However, these exam scores were formally irrelevant for subsequent placement decisions, because ELD course advancement was directed by ELD course grades from 2006 through 2013. Starting in 2014, a student’s years in U.S. schools was the most important factor for both initial and subsequent placement. FIGURE B1 Overview of assessments used to place secondary EL students in ELD coursework, LAUSD The next sections step through LAUSD’s guidance on initial course placement (at matriculation), subsequent course placement, and how we calculate the whether an EL student is placed in ELD courses that are correct, too high, or too low. Initial ELD Placement As per district guidelines, any secondary school student (i.e. enrolled in 6 th grade or higher) that carries a “Limited English Proficient” language status should be enrolled in ELD coursework. What mak es an initial placement “proper”, “too high”, or “too low” depends on the school year (placement policies for LAUSD changed 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 ELD placement depends on student CELDT and CST scores, with advancement hinging on the attainment of a passing grade . (formal LTEL -oriented courses do not yet exist) Formal LTEL courses come into being. CELDT and CST scores still inform placement, but years in US schools becomes a major factor in placement and progre ssion . Course marks are irrelevant in determining advancement CSTfades in relevance for non - LTEL pl ace me nt; the CAHSEE and ELA/Li te racy Periodic Assessments asce nd Periodic assessments give way to the DIBELS and SRI exams. CELDT and CAHSEE mai ntai n relevancy CELDT (Placement Policy Shifts) (Standardized Exam Coverage) CST DIBELS SRI CAHSEE DPI ELA -PA PPIC.ORG Technical Appendices Academic Progress for English Learners 4 over time), the student’s available test scores, and (from 2014 onwards) the student’s total years enrolled in U.S. schools. In cases where a student experiences their first secondary school year as an English Learner (EL) but is not enrolled in ELD coursework, that lapse (and that during any semester that follows) is noted – but the appropriateness of their first recorded instance of secondary school ELD enrollment (if it exists) is still evaluated based on the initial placement criteria below. It must be noted that ELD course placement is not always straightforward: policy guidelines indicate score ranges that should direct a student into a given level of ELD, but a student’s scores across multiple measures may not align neatly into one ELD level. This uncertainty becomes compounded when a student’s exam scores may not exist; evaluations of the appropriateness of a student’s initial p lacement are therefore dependent on the available data. In cases of conflicting test data, school administrators must exercise some discretion in these placement decisions, with the Language Appraisal Team “assist[ing] in determining the most appropriate placement.” 1 General Placement Rules Given the inherent vagaries of assessing proper placement due to data limitations and Language Appraisal Team discretion, we established certain guiding rules. If the entirety of a student’s available placement data ( e.g. CELDT score) all jointly direct that student to a given course placement, it is understood as being “proper”. If one component of a student’s placement data direct them to a higher level of ELD than that to which they are actually enrolled, that place ment is considered as being “too low”. In contrast, for a student’s initial placement to be labeled as “too high”, it is not only necessary that all the available placement data components point to a lower - tiered ELD course; it is additionally necessary th at no relevant placement data are missing, the logic being that the missing component could presumably have buoyed that student’s placement above the level to which they are directed by their available criteria. In the absence of any cause to label an init ial placement as “too high” or “too low” it is defaulted to “proper” status – unless there is an absence of any ELD coursework to begin with in which case (with rare exceptions) that placement is labeled simply as “No ELD”. Pre- 2014 Prior to the 2013 -14 academic year there were no ELD courses that were formally designated for LTEL students. All EL students, irrespective of years spent in U.S. schools are assigned to ELD courses (titled “ESL” in these years) on the basis of CELDT, CST -ELA, and DPI scores (see Table B1). As per the discussion in the ‘General Placement Rules’ section, the lack of student DPI data means that no student’s initial ELD assignment will be labeled as “too high” in these years. A “too low” label is certainly possible, for example i f a student were placed in “Beginning ESL 1B” yet possessed a CELDT score of 2 or higher. A small minority of EL -designated students in these years may be designated as having “proper” initial placement even in the absence of any ELD coursework, with CEL DT or CST-ELA scores high enough to effectively “test out” of ELD while simultaneously falling short of actual reclassification as English -proficient. 1 English Learner Placement Guide REF -5151, 2010. PPIC.ORG Technical Appendices Academic Progress for English Learners 5 TABLE B1 Pre- 2014 English -Learner Initial Placement Guidelines SOURCE: LAUSD English Learner Placement Guide REF -5151, 2007 -2013 2013 -14 A key change in placement guidelines moving into the 2013- 14 academic year was the shift towards factoring in a student’s cumulative years in U.S. schools. For all EL students, different -tiered ELD courses began being associated with “years in U.S. schools” thresholds insofar as enrollment in a given ELD course became prohibited if the student had already experienced a given number of years in U.S. schools. For example, a student enrolled in ELD 2A who had spent 4 years in U.S. schools would be considered as having been placed “too low” since – regardless of test scores – their placement stands in violation of the maximum year threshold for that tier of course. For LTEL students (now guided into courses created exclusively for these types of ELs), years in U.S. schools functions more as a barrier to entry than as a potential route towards a placement being labeled “too low”. Specifically, EL students are only eligible for either of the two LTEL -level courses if they have spent more than 4.5 years in U.S. schools – fewer years, in the context of such a placement, would be considered “too high”. PPIC.ORG Technical Appendices Academic Progress for English Learners 6 TABLE B2A 2013 -14 English -Learner Initial Placement Guidelines (non -LTELs) SOURCE: LAUSD English Learner Placement Guide REF -6046.0, 2014 TABLE B2B 2013 -14 English -Learner Initial Placement Guidelines (LTELs) SOURCE: LAUSD English Learner Placement Guide REF -6046.0, 2014 2014-15 From the 2013- 14 academic year onwards, the basic structure of ELD placement delineating between mainstream ELD and LTEL -oriented ELD remained intact. Years in U.S. schools, as well as DPI and CELDT scores persisted as relevant decision -making factors. However, in subsequent years other exams are sometimes swapped in or out as relevant placement considerations. The ELA CST was discontinued in the 2012- 13 academic year – although older scores were used as a placement reference in 2014. The ELA Periodic Assessment #2 and the California High School Exit Examination (CAHSEE) (for which we possess score data) and the Literacy Periodic Assessment (LPA #3) (for which we do not) were phased in. These exam scores are drawn from particular grades and/or school years depending on whether a student is being placed for mainstream or LT EL-oriented ELD (see Tables B3A/B3B). PPIC.ORG Technical Appendices Academic Progress for English Learners 7 TABLE B3A 2014 -15 English -Learner Initial Placement Guidelines (non -LTELs) SOURCE: LAUSD English Learner Placement Guide REF -6046.1, 2015 TABLE B3B 2014 -15 English -Learner Initial Placement Guidelines (LTELs) SOURCE : LAUSD English Learner Placement Guide REF -6046.1, 2015 2015-16 In the 2015- 16 academic year, similar to the one preceding it, certain exams were phased in and out as points of reference for placement, but the use of years in U.S. schools and CELDT score remained unchanged. For mainstream ELD placement the Literacy and ELA Periodic Assessments (LPA/ELA PA) were replaced by the Scholastic Reading Inventory (SRI) and Dynamic Indicators of Basic Early Literacy Skills (DIBELS). For LTEL placement the tw o periodic assessments were likewise dropped in favor of the SRI and DIBELS (see Table B4A/B4B). PPIC.ORG Technical Appendices Academic Progress for English Learners 8 TABLE B4A 2015 -16 English -Learner Initial Placement Guidelines (non -LTELs) SOURCE: LAUSD English Learner Placement Guide REF -6046.3, 2016 TABLE B4B 2015 -16 English -Learner Initial Placement Guidelines (LTELs) SOURCE: LAUSD English Learner Placement Guide REF -6046.3, 2016 Subsequent ELD Placement After the semester of initial ELD placement, unless an EL student is reclassified as English -proficient, any subsequent semester of enrollment in LAUSD should feature sequential ELD coursework in line with district policy memorandums. Like the initial placement criteria, subsequent placement criteria evolved over the study years. As with initial ELD placement, we e valuate subsequent placement as being degrees of “proper”, “too high”, and “too low” according to a student’s course progression – initially modified by their course grades, then by their years in U.S. schools. In cases where there is a lapse in ELD coursework after the initial placement, that ‘no -ELD’ gap is noted – but if any subsequent ELD enrollment occurs, it is evaluated in the context of the preceding ELD enrollment. In all cases of non -reclassified ELs, however, once a student has been initially placed in ELD, their subsequent placement is not formally influenced on the basis of any newer test. Placement in ELD Courses for Students with Disabilities Whenever a student is enrolled in an ELD course directed towards students with disabilities, it is u nderstood that – although district policy recommends enrollment for certain disabled students according to their test scores PPIC.ORG Technical Appendices Academic Progress for English Learners 9 and years in U.S. schools – the student’s IEP team is allowed significant discretion in that placement. 2 In light of this unique role, any initial or subsequent placement into/out of this subset of ELD courses is considered “proper”. Pre-2014 Prior to the formal creation of ELD courses for LTELs in the 2013 -14 academic year, EL enrollment occurred along a tiered ‘ESL’ course seque nce, upon completion of which a non- reclassified student would be expected to exhibit continuous enrollment in grade -level ELA (see Figure B2). Advancement to the next course in the sequence was mandated if the student received a passing mark in their curr ent ELD class; failure would justify repetition. If a student skips ahead in the sequence or advances without attaining a passing grade in their current course, their subsequent placement is considered “too high”, whereas repetition in the absence of a failing mark or regression along the sequence under any circumstance is assessed as being placed “too low”. Post-2013 With the establishment of LTEL -specific courses in 2013 -14, ELD course grades became irrelevant to subsequent placement; a student’s prog ression to the next sequential ELD course is automatic and “proper”, provided the ‘years in U.S. schools’ thresholds are respected (see Tables A2 -A4). Repetition of any ELD course, regardless of a student’s marks, was likewise considered acceptable if alig ned with these year limits. A student is considered as having placed “too high” if they skip courses in the sequence – or are enrolled in LTEL courses before accruing the proper number of years in U.S. schools. Likewise, a student is still considered as pl acing “too low” if they regress in the sequence – but also if they exceed the year threshold for a given course. The new LTEL -oriented ELD courses exist in two tiers, “Literacy and Language” and the more rigorous “Advanced ELD”. Sequences of progression exist within each tier, but once the student completes its most advanced level they must repeat it until reclassification – or shift to the “Advanced ELD” tier in the case of students completing the “Literacy and Language” sequence. “Development ELS” is a supplemental learning course for LTELs that is considered a “proper” placement wherever it appears in the LTEL course sequence. Synthesizing Student-Year Placement Both initial and subsequent ELD placements are evaluated in terms of being “proper”, “too high”, “too low”, and exhibiting “no ELD”, but within a given semester a student’s situation may constitute a blend of these categories. For example, a student enrolled in ELD 3A is expected to progress to ELD 3B in the subsequent semester and simultaneous ly enroll in a grade-level ELA course. Supposing that transition to ELD 3B occurs – but without the requisite grade -level ELA – that subsequent placement would be considered both “proper” (on account of the correct sequential transition to ELD 3B) and “too low” (on account of this lack of requisite ELA) in equal proportion. If that same ELD 3A student were to instead jump to ELD 4B – also without the requisite grade -level ELA – that subsequent placement would be both “too high” and “too low”. In the same way that a given semester’s placement is established as being some mix of “proper”, “too high”, “too low” , or “no ELD”, a student in a given year is understood as making a ‘contribution’ to these categories on the basis of their fall and sprin g placement. If an EL were initially placed in ELD 1B in the fall semester (in full accord with district guidelines) and enrolled in no ELD course (but was not reclassified) in the spring, they would contribute 0.5 to “no ELD” and 0.5 to “proper” for that year. In a given year these contributions are summed up 2 Scheduling ELD Instruction for ELs with Disabilities REF -5994.1, 2015 PPIC.ORG Technical Appendices Academic Progress for English Learners 10 across all ELs so as to establish annual rates of “proper”, “too low”, “too high”, and “no ELD” placement within grades, schools, and the district as a whole. Classrooms With More Than One ELD Level Interviews in LAUSD suggested that some school staff struggled to create classroom environments in which a single level of ELD instruction was appropriate for enrolled EL students. For most years we are able to estimate how many ELD courses (and ELA cours es) are being taught simultaneously in the same classroom. We found that rates of multi -rostering were almost always under 5 percent for LTELs but started to climb for LAELs beginning in the 2009 -10 school year. FIGURE B2 Multi- Rostering Rates Increasing for Late Arriving ELs SOURCE: Authors ’ calculations from LAUSD data NOTE : Course period data not available for the 2013 -14 and 2014- 15 school years. PPIC.ORG Technical Appendices Academic Progress for English Learners 11 FIGURE B3 Pre- 2014 ELD Course Sequence PPIC.ORG Technical Appendices Academic Progress for English Learners 12 FIGURE B4 Post -2013 ELD Course Sequence PPIC.ORG Technical Appendices Academic Progress for English Learners 13 Overview of district policy in SDUSD Secondary School Placement Guidelines for English Learners in San Diego Unified School District San Diego’s English Learners are purposefully placed in English coursework based on their length of enrollment and overall Engli sh proficiency (OPL) as determined by the California English Language Development Test (CELDT). The CELDT is administered when students first enroll in the district and every fall thereafter until they are reclassified as fluent English proficient. CELDT reports five OPLs: beginning, early intermediate, intermediate, early advanced, and advanced. Prior to the 2008 -09 academic year, the only criterion used to determine English course placement was the OPL. ELs at the earlier stages of English fluency (i .e., OPLs of beginning, early intermediate, or the bottom one -third of intermediate) were placed into Structured English Immersion (SEI) coursework. SEI coursework clusters students by proficiency level, and emphasizes acquiring English as a second language (ESL). ELs in the later stages of English proficiency (i.e., OPLs of the top two -thirds of intermediate, early advanced, and advanced), but not yet reclassified, were placed in Mainstream English Cluster (MEC) coursework. MEC coursework provides Engli sh language development (ELD) supports for ELs enrolled in grade -level English classes alongside their native English -speaking peers. Beginning with the 2008- 09 academic year, San Diego’s EL placement guidelines changed, and length of enrollment was adde d as a placement criterion. In an effort to prevent students from spending extended time in SEI/ESL coursework and “getting stuck” at earlier stages of English fluency, the district decided to advance students through the course sequence at a faster pace than using OPL alone would allow. As a result, students with OPLs at the beginning, early intermediate, and the bottom one -third of intermediate levels were placed into the higher -level MEC/ELD coursework, regardless of their OPLs – if they had been enrolled in the district for four or more years. Under the prior guidelines, students would not be placed in MEC/ELD coursework until their OPLs were in the top two- thirds of the intermediate range or higher, no matter how long they had been enrolled. Lengt h of enrollment has also been used to guide placement of students enrolled less than four years since the 2008- 09 academic year. These students are enrolled in SEI/ESL coursework if their OPLs are at the beginning, early intermediate, or the bottom one -th ird of intermediate. However, students with higher OPLs are assigned higher level coursework (as described below), not “held back” based on their length of enrollment. The following tables provide general course placement guidelines for students at th e middle and high school levels. Please note that, in some academic years covered by our study, there were minor variations to the course placement guidelines that follow. These variations were taken into account in our analyses, but are not detailed in this document. Middle School Placement Guidelines Newcomer classes are designed for recently arrived English Learners at the Beginning level of English fluency who have not previously attended school or whose formal education has been interrupted. Three -period Newcomer classes focus on developing English fluency and basic literacy skills. They also introduce the basics of other academic disciplines and orient students to schooling in the United States. For the balance of the day, Newcomer students are enrolled in general education coursework (such as ph ysical education, visual and performing arts) with their English speaking peers. Newcomer students with OPLs of early intermediate and the lower one -third of the intermediate range – or students enrolled in schools that do not have enough students to offe r Newcomer courses – are enrolled in the SEI/ESL courses described in the next section, regardless of their PPIC.ORG Technical Appendices Academic Progress for English Learners 14 schooling experience. Students typically remain in Newcomer classes for only one year. Newcomer coursework, including criteria for placement, is provided in the following table. Newcomer Course Sequence for Middle School Grade Level Length of Enrollment OPL Course Grade 6 ≤1 Year B ESL Literacy 6 th Core Newcomer (3 periods) Grades 7-8 ≤1 Year B ESL Literacy 7 th-8th Core Newcomer (3 periods) Note: B = Beginning Middle school English Learners who do not yet have OPLs in the upper two- thirds of the Intermediate range are enrolled in the following SEI/ESL courses, according to their length of enrollment and OPL level. The higher of the two crite ria determines placement. For example, the appropriate placement for a student enrolled less than one year with an OPL of Early Intermediate would be ESL Level 2, not ESL Level 1. The SEI/ESL sequence of coursework provides opportunities for students to r ead and write across a variety of genres, express themselves for different audiences and purposes, and acquire academic language needed to access content in other subject areas. SEI/ESL coursework, including criteria for placement, is provided in the foll owing table. Note that students enrolled for four or more years are enrolled in the MEC/ELD course sequence (see following section), regardless of their OPL. SEI/ESL Course Sequence for Middle School Grade Level Length of Enrollment OPL Course Grades 6-8 ≤1 Year B ESL Level 1 5 th-8th (2 periods) 1.0 -2.5 Years EI ESL Level 2 5 th-8th (2 periods) 2.5 to 3.9 Years Lower 1/3 of I ESL Level 3 5 th-8th (2 periods) Note: B = Beginning, EI = Early Intermediate, I = Intermediate Middle school English Learners who have OPLs in the upper two -thirds of the intermediate range and higher OR who have been enrolled the district for four or more years (regardless of OPL) are enrolled in MEC/ELD courses. Courses in this sequence feature t he same content as grade-level English courses taken by English -fluent students, but utilize Specially Designed Academic Instruction in English (SDAIE) strategies to help English Learners learn grade -level material. From 2010- 2011 onward, the district’s c ourse placement guidelines specified that, whenever possible, MEC/ELD classrooms should have approximately one -third ELs and two -thirds native English -speaking students. MEC/ELD coursework, including criteria for placement, is provided in the following ta ble. One important feature of the district’s MEC/ELD courses is that they are two periods in length; grade -level English courses for English proficient students at most grade levels are one period in length. PPIC.ORG Technical Appendices Academic Progress for English Learners 15 MEC/ELD Course Sequence for Middle School Grade Level Length of Enrollment OPL Course Grade 6 4+ Years OR Top 2/3 of I ELD or Grade -Level English 6 th (2 period course) Grade 7 4+ Years OR Top 2/3 of I ELD or Grade -Level English 7 th (2 period course) Grade 8 4+ Years OR Top 2/3 of I ELD or Grade -Level English 8 th (2 period course) Note: I = Intermediate High School Placement Guidelines As at the high school level, Newcomer classes are designed for recently arrived English Learners at the Beginning level of English fluency who have not previously attended school or whose formal education has been interrupted. These three- period classes f ocus on developing English fluency and basic literacy skills. They also introduce the basics of other academic disciplines and orient students to schooling in the United States. For the balance of the day, Newcomer students are enrolled in general educat ion coursework (such as physical education, visual and performing arts) with their English speaking peers. Newcomer students with OPLs in the upper one -third of the beginning range, early intermediate level, or the lower one -third of the intermediate rang e – or students enrolled in schools that do not have enough students to offer Newcomer courses – are enrolled in the SEI/ESL courses described in the next section, regardless of their schooling experience. At the high school level, students typically rema in in Newcomer classes for only one semester. High School Newcomer coursework, including criteria for placement, is provided in the following table. Newcomer Course Sequence for High School Grade Level Length of Enrollment OPL Course Grade 9 <6 Months Low to Mid B ESL Literacy Core 9 th Newcomer (3 periods) Grades 10 -12 <6 Months Low to Mid B ESL Literacy Block 10 th-12 th Newcomer (3 periods) Grades 10 -12 <6 Months Low to Mid B ESL Literacy Core 10 th-12 th Newcomer (6 periods) Note: B = Beginning High school English Learners who do not yet have OPLs in the upper two- thirds of the intermediate range are enrolled in the following SEI/ESL courses, according to their length of enrollment and OPL level. The higher of the two criteria determines placement. For example, the appropriate placement for a student enrolled less than one year with an OPL of Early Intermediate would be ESL 3- 4 Literacy Block, not ESL 1-2 Literacy Block. The PPIC.ORG Technical Appendices Academic Progress for English Learners 16 SEI/ESL sequence of coursework provides opportunities for students to read and write across a variety of genres, express themselves for different audiences and purposes, and acquire academic language needed to access content in other subject areas. SEI/ESL coursework, including criteria for placement, is provided in the fol lowing table. Note that students enrolled for four or more years are enrolled in the MEC/ELD course sequence (see following section), regardless of their OPL. SEI/ESL Sequence for High School Grade Level Length of Enrollment OPL Course Grades 9 -12 ≤1 Year B ESL 1 -2 Literacy Block (2 periods) 1.0-2.5 Years EI ESL 3-4 Literacy Block (2 periods) 2.5 to 3.9 Years Lower 1/3 of I ESL 5 -6 Literacy Block (2 periods) Note: B = Beginning, EI = Early Intermediate, I = Intermediate High school English Learners who have OPLs in the upper two- thirds of the intermediate range (and higher) OR who have been enrolled the district for four or more years (regardless of OPL) are enrolled in MEC/ELD courses. Courses in this sequence feature t he same content as grade-level English courses taken by English -fluent students, but utilize Specially Designed Academic Instruction in English (SDAIE) strategies to help English Learners learn grade -level material. While not required, it is preferred tha t English Learners be enrolled in the two-period (Block) versions of these English courses. From 2010- 2011 onward, the district’s course placement guidelines specified that, whenever possible, MEC/ELD classrooms should have approximately one- third ELs and two-thirds native English -speaking students. MEC/ELD coursework, including criteria for placement, is provided in the following table. MEC/ELD Sequence for High School Grade Level Length of Enrollment OPL Course Grade 9 4+ Y ears OR Top 2/3 of I English 1-2 (1 period) or English 1- 2 Block (2 periods) Grade 10 4+ Years OR Top 2/3 of I English 3- 4 (1 period) or English 3- 4 Block (2 periods) Grade 11 4+ Years OR Top 2/3 of I American Literature (1 period) or American Literature Block (2 periods) Grade 12 4+ Years OR Top 2/3 of I Contemporary Voices in Literature (1 period) or World Literature (1 period) Note: I = Intermediate PPIC.ORG Technical Appendices Academic Progress for English Learners 17 Appendix C: Qualitative Appro ach Data Collection Qualitative data collection and analyses were conducted in two phases. In the first phase, semi -structured interviews with central office administrators and leaders were conducted to understand each district’s policy as it pertains to t he course placement and reclassification for Long -term English Learner (LTEL) and Late Arrival English Learner (LAEL) students. Five district-level interviews were conducted at SDUSD, while six district - level interviews were conducted at LAUSD. Interviewees were selected based on their position and the extent to which their work intersects with English Learner (EL) policies and programs. For example, in LAUSD, we interviewed the Executive Director of the Multilingual and Multicultural Education Department and the Coordinator of EL while in SDUSD, the Director of the Office of English Language Acquisition and the Program Manager in the Office of English Language Acquisition were among the interv iewees. Interviews were semi- structured to ensure data alignment among multiple participants (Miles & Huberman, 1994; Patton, 1990). The interview protocol was developed to understand the ways in which central office administrators’ work intersects with secondary EL policies and programs, and specifically to explore their involvement in developing and implementing EL course assignment and reclassification policies. We also collected district policy documents pertaining to EL identification, course assign ment, and reclassification to identify how these policies are meant to be implemented in each district. These documents allowed us to triangulate findings from policy documents, and to understand how and why there may be deviations from district policy. In phase 2 we conducted school -level interviews to explore how school settings and instructional programs support (or do not support) LTELs and LAELs, as well as the challenges and opportunities school staff encounter as they implement EL policies in the context of these settings and programs. Findings from our quantitative analysis were used to inform our qualitative data collection with respect to site selection and interview protocol development. A matrix was created that placed schools according to % EL, the Herfindahl index that measures the language homogeneity of ELs, and CELDT gains ( see Table C-1). Ten schools from each district were identified and invited to participate based on the matrix. Although we sought to include schools across all cell s in the matrix, we were not able to collect data from all the schools we invited to participate. School level interviews were conducted at seventeen schools in total -- eight LAUSD schools (46 interviews) and nine SDUSD schools (48 interviews) for a total of 94 interviews during fall 2017 and early winter 2018. We interviewed at least one school from each cell of our 2x2x2 school selection matrix in both districts. Interviews were semi -structured, with the aim of understanding how school leaders, counselo rs, and teachers support LTELs and/or LAELs in particular school settings and instructional programs. The interview protocol focused on understanding the school’s EL population, the staff member’s roles and responsibilities with respect to supporting LTELs and LAELs, and the broader scope of supports for LTELs/LAELs, and how programs and policies are implemented for these groups of students. PPIC.ORG Technical Appendices Academic Progress for English Learners 18 TABLE C1 School Selection Criteria Matrix District Low % EL High % EL Low Herf High Herf Low Herf High Herf Low CELDT gains LA: Middle School SD: 1 Middle School (no data collected; 1 High school LA: 1 Middle School SD: 1 Middle School LA: 2 High Schools SD: 1 High School LA: 1 High School SD: 1 Middle School High CELDT gains LA: 1 Middle School (no data collected); 1 High school SD: 1 Middle School LA: 1 Middle School (no data collected) SD: 1 High School LA: 1 Middle School SD: 1 Middle School; 1 High School LA: 1 Middle School SD: 1 Middle School Analysis Preliminary analysis consisted in developing district -level and school -based memos based on our initial observations and impressions from interview data. The Los Angeles and San Diego teams developed memos around core topic areas: interviewee information, school characteristics, student population, programs, course placement, reclassification, and other (e.g., professional development). We then engaged in some initial coding around the study’s research questions to assess the extent to which we needed to dive deeper into the transcripts to generate preliminary findings. The teams each read four site memos from across the two districts to generate a list of initial codes. We concurrently coded one interview from each district with the following codes: o Sch ool Information o Course Placement Processes o Course Placement Challenges o Reclassification Processes o Reclassification Challenges o LTEL Supports o LAEL Supports The team created a cross- site data matrix that identified the seven agreed upon codes that mapped onto the research questions. The Los Angeles and San Diego teams completed the cross- site matrix the process using the subsets of school sites memos and trans cripts from each district. We added three additional categories/codes that were not specifically derived from the original set of research questions and that were identified upon review of sites/interview data: o General challenges, needs and lack of suppor t o Supports beyond coursework o Other observations The team discussed the cross- site matrix and to determine a process for identifying initial patterns in the data related to our research questions. We created brief school summaries to provide some context fo r each of the sites. We then developed two data displays: one for each district related to course placement. PPIC.ORG Technical Appendices Academic Progress for English Learners 19 Course Placement Analysis (Spring/Summer) The qualitative team met in May 2018 to discuss the cross-district course placement analysis. An ini tial set of codes were developed focused on EL demographics/info, course placement, course descriptions, compliance and monitoring, EL school supports, and teacher supports. The team selected one school from each district (with pseudonyms Central HS and K ennedy HS) to code independently. The team met again to discuss the coding and process and refine the codes. All interview transcripts were loaded into the Dedoose data analysis software program, chosen because it facilitates collaborative qualitative an d mixed methods analysis. We used a hybrid approach to coding documents and interviews (Miles & Huberman, 1994), beginning with a list of codes that capture official district policies, policy implementation processes, and any challenges to implementation and/or deviations from policy. In addition to these a priori codes, we derived codes inductively for new ideas and themes, first by assignment general descriptive codes and then by comparing, contrasting, and grouping codes until we had a final list (Strau ss & Corbin, 1990). The final code list was comprised of 18 codes that reflected issues related to LTEL/LAEL student characteristics and course placement. They included student behaviors or needs, course description, course placement, assessment, afterschool EL supports, in- school EL supports, compliance and monitoring, and teacher capacity. Each team coded the remainder of the school interviews. During the coding process, we created a memo outline to capture school -level findings across each school sit e. Each memo captured the themes discussed in the outline and included participant quotes to support major findings. The memo consisted of the following themes: o School characteristics o Course placement o Other supports offered to ELs that may explain outcomes o Implementation supports and challenges related to student population o Implementation supports and challenges to leadership and capacity  Teacher Capacity  Professional Development o Other Issues Each area/theme included sub -categories for LAEL and LTEL students to distinguish differences across course placement and supports for each student group. A memo was written for each school. To integrate findings across the two districts, members from the qualitative team were assigned an area/theme to review and analyze across all the memos. For example, the LA team reviewed “Other supports offered to ELs that may explain outcomes” and “Implementation supports and challenges to leadership and capacity” . All members reviewed school characteristics. Each team developed high -level themes and findings based on the assigned memo area/theme, and the team met to discuss the findings and determine similarities and differences between schools and districts. To summarize findings, the qualitative team created a matrix to detail the course placement processes, support, and teacher capacity across each district and by LAEL and LTEL student. PPIC.ORG Technical Appendices Academic Progress for English Learners 20 Appendix D: Regressions and Results We performed regression analyses fo r the academic outcomes of LTELs and LAELs. Our sample, running from the 2006- 07 through the 2015- 16 school years, includes all secondary school (grade 6 -12) students who fall into one of three language groups: never EL, LAEL (now or in a prior year), and LTEL (now or in a prior year). We exclude a small number of EL students who are not yet LTEL because they have not been ELs in the district for five years, and arrived in elementary school, so that they are not LAELs. Consider the following model of an outcome for student i in grade g in school year t , S igt, where LTEL and LAEL are dummy variables indicating long -term and late- arriving ELs, respectively. For outcomes available annually, including CST Z -scores, CELDT scaled scores, and grade point average (GPA), a common and quite general way to measure student progress year by year is to model the annual outcome as a function of the lagged outcome from the prior year, plus other explanatory factors : (1) &#55349;&#56390;&#55349;&#56390; &#55349;&#56406;&#55349;&#56406;&#55349;&#56406;&#55349;&#56406;&#55349;&#56406;&#55349;&#56406; =&#55349;&#57084;&#55349;&#57084;+ &#55349;&#57091;&#55349;&#57091;&#55349;&#56390;&#55349;&#56390; &#55349;&#56406;&#55349;&#56406;, PPIC.ORG Technical Appendices Academic Progress for English Learners 21 indicated by the subscript b. The measure w e chose was the CST ELA score, observed in grade 5 if available, and if not available in grade 5 then in grades 4, 3. 2, 6 or higher, in that order. 3 Because the baseline performance levels were measured in different grades for different students, we also include in (3) dummy variables PERFGRADE indicating the grade in which each baseline measure is observed. A school with a higher EL percentage is likely to devote considerable effort to teaching English as a second language, which could benefit these stu dents. On the other hand, the relative lack of never -EL students in these schools could slow EL s’ academic progress because it lowers the frequency with which ELs converse with peers fluent in English. It is also conceivable that a school’s EL percentage could influence never -ELs’ academic progress by changing classroom heterogeneity, pedagogy, and standards. We test whether there is a relationship between a school’s EL perce ntage, which we refer to as PCT EL st, and student progress. It is especially interesting to know whether there is a relationship between this variable and the respective progress of never ELs , LTELs, and LAELs, and whether these associations differ. We modify model ( 1) above by adding P CTEL st which is interacted three times, with indicators for the two EL student types and never ELs: (4) &#55349;&#56390;&#55349;&#56390; &#55349;&#56406;&#55349;&#56406;&#55349;&#56406;&#55349;&#56406;&#55349;&#56406;&#55349;&#56406;&#55349;&#56406;&#55349;&#56406; = &#55349;&#57084;&#55349;&#57084;+ &#55349;&#57091;&#55349;&#57091;&#55349;&#56390;&#55349;&#56390; &#55349;&#56406;&#55349;&#56406;, 3 Because we needed a baseline achievement measure that included never EL students, we did not use the CELDT. Similarly we did not use baseline math scores as the CST math test students take in the upper secondary grades varie s depending on the level of math course taken, making interpretation difficult. PPIC.ORG Technical Appendices Academic Progress for English Learners 22 while at the other extreme a value of 0 means that no two ELs at a school speak the same home language. These two regressors are added to (4) in the same way that PCTEL was, that is, each interacted with dummy variables for the three language groups in the sample: never EL, LAEL and LTEL. The model that we report in Appendix Table D 4A further interacts the indicators for the three language groups with the share of ELs at the school who in the given year were placed into no ELD course at all, the share of ELs in an ELD course at a higher level than expected given district guidelines, and the share of ELs in an ELD course at a lower level than expected. The omitted variable here is the share of ELs placed into an EL group correctly. The full model is shown below. (6) &#55349;&#56390;&#55349;&#56390; &#55349;&#56406;&#55349;&#56406;&#55349;&#56406;&#55349;&#56406;&#55349;&#56406;&#55349;&#56406; =&#55349;&#57084;&#55349;&#57084;+ &#55349;&#57091;&#55349;&#57091;&#55349;&#56390;&#55349;&#56390; &#55349;&#56406;&#55349;&#56406;, 4 A simplification in the notation in (6) is that we do not include a student subscript i on the language environment variables LANG hst. This is appropriate for five of the s ix language variables, which are measured for an entire school in a given year. It is not accurate for Same_a, which is the p ercentage of all students at the school who speak the same home language as student i. This variable will take on different values for two students attending the same school in a given year if the two students speak different home languages. But for the other five LANG variables, such as the percentage of students who are ELs at the school, the values are constants across all students attending a given school in a given year. 5 If we took these models where we instead control for ELD placem ent for individual EL students as valid, which we do not, placement into no ELD classes would be predicted to lead to no impact on LAELs in SDUSD e xcept for a positive effect on CELDT scores, negative impacts on LAELs in LAUSD for all four outcomes, positive impacts on LTELs in SDUSD for CST ELA, CELDT and GPA, but negative effects for math CST scores, and in LAUSD negative impacts on LTELs for CST E LA and math scores , but positive (and implausibly large ) effects on CELDT scores. PPIC.ORG Technical Appendices Academic Progress for English Learners 23 As another robustness check, we replaced the overall percentage of ELs with no ELD, too high or too low an ELD course with measures of these specific to LAELs and LTELs. We do not prefer this approach because for LAELs the number of LAELs in a given school is often very small. This means that when we calculate the share of that group receiving a certain qualit y of ELD placement, such as too high a placement, the LAEL student plays a large role in calculating that mean, which creates a similar risk of endogeneity to that described earlier. The other issue is that because the number of LAELs in a given school can be small, calculating a mean share of LAELs who receive a given type of ELD will tend to be a quite imprecise measure. For instance, in San Diego when we calculated the number of LAELs for each school/year observation, the median school/year had only 4 LA ELs enrolled. When we replaced the ELD placement shares by shares calculated separately for LAELs and LTELs, in SDUSD the most important change was that the coefficients on the no ELD share for LAELs became insignificant in all the models in Tables D 4A a nd D 4B, except for the model of whether the student was ever reclassified, in which the coefficient for the share of LAELs with no ELD remained negative and significant . The change for LAUSD was quite similar. All of the coefficients on the course placemen t variables for LAELs which had been significant became insignificant, with one exception. In LAUSD, school shares of low course ELD placement for LAELs gained significance for GPA growth, and were positive. There were no changes on LAUSD’s coefficients f or course placement among LTELs. The drop in the frequency with which the no ELD variable was significant for LAELs could reflect both the greater risk of this measure being endogenous than was the case in our main model, where we calculated mean ELD cours e placement for all ELs rather than for small subgroups of ELs, as well as the lower precision of the course placement variables when they were estimated on subgroups. We also examine the impact of differentiated language classes for LT ELs and beginning LAELs in the two districts. We assess the relation between enrollment in these classes and student outcomes by estimating versions of models ( 6) that also include controls for whether the given school in the given year offered any newcomer classes or ALD c lasses. Again, we do not control for whether individual students participated in these courses as individual students’ course placement is likely to be endogenous. The resulting models are identical to (6) but add six new interaction terms between ALD and newcomer class provision on the one hand and the three language groups on the other. Results are summarized in Appendix Table D 5. The main text expresses our concern that a lack of precision, especially in LAUSD, renders the results for ALD suspect. We h ave more confidence in the newcomer results, especially in SDUSD where newcomer courses were provided in some schools for all but the earliest and latest years of our study. In addition to the results tabulated later in this appendix and discussed in the main text, we performed one robustness check regarding the variable measuring the percent of students who are ELs. There is a clear potential for nonlinear relations between this explanatory variable and outcomes. The careful reader may have noticed that in theory the correlation between academic outcomes and some of the above variables, especially the three variables that measure the language environment, could be either positive or negative. For instance, we hypothesized that in a school with few ELs, an increase in the share of ELs could lead to the creation of classes geared to the needs of the English Learners, which could benefit both ELs and their non- EL schoolmates. But at higher pre -existing levels of the share of ELs, a further increase in the share of ELs in a school could create situations where students could comfortably spend most of their school day speaking a language other than English at school, which could slow their acquisition of English fluency and academic progress, and perhaps have co nsequences for never ELs as well. One can easily imagine a positive relation between academic outcomes and the percent EL at low levels, turning to negative effect at higher levels. PPIC.ORG Technical Appendices Academic Progress for English Learners 24 To test this, we estimated models that replaced variables like percent E L with indicators for each tenth of the possible range (e.g. 0- 9.99% EL, 10-19.99% EL and so on). What we looked for in particular was whether an insignificant overall relation resulted from positive correlations in one range and negative correlations in t he other. We studied this possibility for the other language environment, including the percentage of all students speaking the same home language, and language homogeneity among ELs, as well as for the measures of incorrect ELD course placement. We found several interesting patterns, but did not discover any “hidden” relations that washed out in our simpler models where we looked for an overall relation. On the following pages we show the means and standard deviations of the four outcomes we model along with the explanatory variables. Means for each of three language groups are shown along with the means for the combined sample. The data cover 2006 -07 through 2015- 16. Readers will note that the mean CELDT scores of never EL students are reported. This ar ises because students who report speaking a home language other than English are given the CELDT to determine if they are fluent in English, and some of these students are determined not to be ELs. PPIC.ORG Technical Appendices Academic Progress for English Learners 25 TABLE D1 Means and Standard Deviations of Key Explanatory Variables and Dependent Variables for LAUSD Population: never EL everLAEL everLTEL5 All mean SD mean SD mean SD mean SD Variable name Percent EL * non-EL 18.58 11.87 n/a n/a n/a n/a 9.66 12.63 Percent EL * LAEL n/a n/a 30.72 14.50 n/a n/a 2.42 9.23 Percent EL * LTEL5 n/a n/a n/a n/a 26.91 11.70 10.79 15.13 Same Home Language among All Students * non-EL 45.93 23.63 n/a n/a n/a n/a 23.91 28.59 Same Home Language among All Students * LAEL n/a n/a 64.69 25.35 n/a n/a 5.10 18.83 Same Home Language among All Students * LTEL5 n/a n/a n/a n/a 67.85 22.22 27.19 36.10 Linguistic Homogeneity * non- EL 0.80 0.20 n/a n/a n/a n/a 0.42 0.42 Linguistic Homogeneity * LAEL n/a n/a 0.88 0.13 n/a n/a 0.07 0.24 Linguistic Homogeneity * LTEL5 n/a n/a n/a n/a 0.89 0.14 0.36 0.45 Any ALD Courses * non-EL 0.25 0.44 n/a n/a n/a n/a 0.13 0.34 Any ALD Courses* LAEL n/a n/a 0.23 0.42 n/a n/a 0.02 0.13 Any ALD Courses* LTEL5 n/a n/a n/a n/a 0.22 0.41 0.09 0.28 Any Newcomer Courses * non -EL 0.47 0.50 n/a n/a n/a n/a 0.25 0.43 Any Newcomer Courses * LAEL n/a n/a 0.52 0.50 n/a n/a 0.04 0.20 Any Newcomer Courses * LTEL5 n/a n/a n/a n/a 0.50 0.50 0.20 0.40 Share ELs Placed into Too Low ELD * non-EL 0.05 0.06 n/a n/a n/a n/a 0.03 0.05 Share ELs Placed into Too Low ELD * LAEL n/a n/a 0.04 0.05 n/a n/a n/a 0.02 Share ELs Placed into Too Low ELD * LTEL5 n/a n/a n/a n/a 0.05 0.06 0.02 0.05 Share ELs Placed into Too High ELD * non-EL 0.03 0.04 n/a n/a n/a n/a 0.02 0.03 Share ELs Placed into Too High ELD * LAEL n/a n/a 0.03 0.03 n/a n/a n/a 0.01 Share ELs Placed into Too High ELD * LTEL5 n/a n/a n/a n/a 0.03 0.03 0.01 0.03 Share ELs Placed into No ELD * non-EL 0.36 0.20 n/a n/a n/a n/a 0.19 0.23 Share ELs Placed into No ELD * LAEL n/a n/a 0.38 0.22 n/a n/a 0.03 0.12 Share ELs Placed into No ELD * LTEL5 n/a n/a n/a n/a 0.37 0.19 0.15 0.21 PPIC.ORG Technical Appendices Academic Progress for English Learners 26 Population: never EL everLAEL everLTEL5 All mean SD mean SD mean SD mean SD Share ELs Placed into Correct ELD * non-EL 0.46 0.17 n/a n/a n/a n/a 0.24 0.26 Share ELs Placed into Correct ELD * LAEL n/a n/a 0.49 0.19 n/a n/a 0.04 0.14 Share ELs Placed into Correct ELD * LTEL5 n/a n/a n/a n/a 0.48 0.16 0.19 0.26 LTEL5 status n/a n/a n/a n/a 1.00 n/a 0.40 0.49 Ever -LAEL status n/a n/a 1.00 n/a n/a n/a 0.08 0.27 Female 0.49 0.50 0.42 0.49 0.46 0.50 0.48 0.50 Special Education status 0.10 0.30 0.35 0.48 0.13 0.34 0.13 0.34 Spanish spoken at home 0.24 0.43 0.89 0.31 0.93 0.25 0.57 0.50 Parental education level: Less Than High School Diploma 0.10 0.30 0.21 0.41 0.30 0.46 0.19 0.39 High School Diploma 0.14 0.35 0.09 0.29 0.16 0.36 0.14 0.35 Some College 0.13 0.34 0.03 0.16 0.05 0.22 0.09 0.29 College Graduate 0.10 0.30 0.02 0.14 0.03 0.17 0.06 0.25 Graduate School 0.05 0.21 0.01 0.09 0.01 0.11 0.03 0.17 Parental Education Missing 0.48 0.50 0.64 0.48 0.45 0.50 0.48 0.50 Grade (mean only) 8.91 9.44 8.84 8.92 Total Enrollment (School Level) 2063 1017 2112 1232 2066 1091 2068 1066 Grade Configuration: 6-8 0.38 0.48 0.24 0.43 0.38 0.49 0.37 0.48 Grade Configuration: Other Than 6-8 or 9- 12 0.12 0.32 0.13 0.34 0.10 0.30 0.11 0.31 Dependent Variables CST-ELA z -score -0.03 1.00 -1.26 0.66 -0.74 0.74 -0.41 0.96 CST -Math z -score -0.10 1.01 -1.22 0.64 -0.70 0.71 -0.41 0.94 PPIC.ORG Technical Appendices Academic Progress for English Learners 27 Population: never EL everLAEL everLTEL5 All mean SD mean SD mean SD mean SD Grade Point Average (Annual) 2.35 0.96 2.20 0.99 1.98 0.93 2.19 0.97 CELDT Overall Score 607 100 447 109 567 47 534 88 TABLE D2 Means and Standard Deviations of Key Explanatory Variables and Dependent Variables for SDUSD Population: never EL everLAEL everLTEL5 All mean SD mean SD mean SD mean SD Variable name Percent EL * non-EL 12.70 10.70 0.11 1.70 n/a n/a 9.81 10.80 Percent EL * LAEL n/a n/a 29.01 15.82 n/a n/a 0.52 4.41 Percent EL * LTEL5 n/a n/a n/a n/a 23.56 15.05 4.95 11.82 Same Home Language among All Students * non-EL 49.74 21.27 0.13 2.70 n/a n/a 38.40 28.01 Same Home Language among All Students * LAEL n/a n/a 36.40 27.54 n/a n/a 0.66 6.10 Same Home Language among All Students * LTEL5 n/a n/a n/a n/a 40.64 23.78 8.53 19.82 Linguistic Homogeneity * non- EL 0.59 0.22 n/a 0.05 n/a n/a 0.46 0.32 Linguistic Homogeneity * LAEL n/a n/a 0.61 0.24 n/a n/a 0.01 0.09 Linguistic Homogeneity * LTEL5 n/a n/a n/a n/a 0.67 0.22 0.14 0.29 Any ALD Courses * non-EL 0.05 0.21 n/a n/a n/a n/a 0.04 0.19 Any ALD Courses* LAEL n/a n/a 0.04 0.20 n/a n/a n/a 0.03 Any ALD Courses* LTEL5 n/a n/a n/a n/a 0.05 0.22 0.01 0.10 Any Newcomer Courses * non -EL 0.36 0.19 n/a 0.02 n/a n/a 0.03 0.17 Any Newcomer Courses * LAEL n/a n/a 0.06 0.24 n/a n/a n/a 0.03 Any Newcomer Courses * LTEL5 n/a n/a n/a n/a 0.05 0.23 0.01 0.11 Share ELs Placed into Too Low ELD * non-EL 0.05 0.09 n/a 0.01 n/a n/a 0.04 0.08 Share ELs Placed into Too Low ELD * LAEL n/a n/a 0.07 0.07 n/a n/a n/a 0.01 Share ELs Placed into Too Low ELD * LTEL5 n/a n/a n/a n/a 0.06 0.10 0.01 0.05 PPIC.ORG Technical Appendices Academic Progress for English Learners 28 Population: never EL everLAEL everLTEL5 All mean SD mean SD mean SD mean SD Share ELs Placed into Too High ELD * non-EL 0.15 0.14 n/a 0.01 n/a n/a 0.12 0.14 Share ELs Placed into Too High ELD * LAEL n/a n/a 0.20 0.13 n/a n/a n/a 0.03 Share ELs Placed into Too High ELD * LTEL5 n/a n/a n/a n/a 0.14 0.13 0.03 0.08 Share ELs Placed into No ELD * non-EL 0.31 0.27 n/a 0.03 n/a n/a 0.24 0.27 Share ELs Placed into No ELD * LAEL n/a n/a 0.22 0.22 n/a n/a n/a 0.04 Share ELs Placed into No ELD * LTEL5 n/a n/a n/a n/a 0.30 0.27 0.06 0.17 Share ELs Placed into Correct ELD * non-EL 0.49 0.26 n/a 0.04 n/a n/a 0.38 0.31 Share ELs Placed into Correct ELD * LAEL n/a n/a 0.51 0.23 n/a n/a 0.01 0.08 Share ELs Placed into Correct ELD * LTEL5 n/a n/a n/a n/a 0.50 0.25 0.11 0.23 LTEL5 status n/a n/a n/a n/a 1.00 n/a 0.21 0.41 Ever -LAEL status n/a n/a 1.00 n/a n/a n/a 0.02 0.13 Female 0.49 0.50 0.48 0.50 0.43 0.50 0.48 0.50 Special Education status 0.13 0.34 0.03 0.16 0.27 0.44 0.16 0.37 Spanish spoken at home 0.10 0.30 0.64 0.48 0.84 0.37 0.26 0.44 Parental education level: Less Than High School Diploma 0.03 0.17 0.21 0.41 0.24 0.43 0.08 0.27 High School Diploma 0.11 0.31 0.12 0.32 0.18 0.39 0.13 0.33 Some College 0.19 0.39 0.04 0.19 0.07 0.25 0.16 0.37 College Graduate 0.19 0.39 0.05 0.22 0.04 0.20 0.15 0.36 Graduate School 0.13 0.34 0.03 0.17 0.02 0.14 0.11 0.31 Parental Education Missing 0.35 0.48 0.55 0.50 0.45 0.50 0.38 0.49 Grade (mean only) 9.09 9.76 8.93 9.07 Total Enrollment (School Level) 1386 737 1266 798 1150 710 1334 739 PPIC.ORG Technical Appendices Academic Progress for English Learners 29 Population: never EL everLAEL everLTEL5 All mean SD mean SD mean SD mean SD Grade Configuration: 6-8 0.31 0.46 0.19 0.39 0.33 0.47 0.31 0.46 Grade Configuration: Other Than 6-8 or 9- 12 0.14 0.34 0.03 0.16 0.10 0.30 0.13 0.33 Dependent Variables CST-ELA z -score 0.33 0.99 -1.07 0.72 -0.83 0.74 0.06 1.06 CST -Math z -score 0.19 0.97 -0.63 0.95 -0.65 0.71 -0.02 0.98 Grade Point Average (Annual) 2.84 0.92 2.46 1.01 2.09 0.97 2.68 0.99 CELDT Overall Score 643 54 455 104 555 57 548 72 PPIC.ORG Technical Appendices Academic Progress for English Learners 30 We constructed measures of correct course placement for both school districts over time. Those measures are used in regression analyses (Tables D 4A and D 4B). Here, we report the regression coefficients of our estimates of what predicts school level correct course placement. Our regressions are estimated at the student level, which implicitly weighs each school by the number of students involved in constructing each of the measures. TABLE D3 Correct School Placement Regression Results, LAUSD and SDUSD Variables LAUSD SDUSD LAUSD SDUSD Percent EL -0.000471 -0.00332 (0.000644) (0.00169) Number EL per grade 0.000354** -8.53e -05 (4.83e -05) (0.000321) Same Home Language among All Students 3.20e-05 -0.000971* -0.000292* -0.000692 (0.000154) (0.000412) (0.000133) (0.000426) Linguistic Homogeneity 0.0744 -0.0224 -0.00965 -0.0795 (0.0539) (0.113) (0.0473) (0.107) Constant 0.415** 0.602** 0.448** 0.579** (0.0434) (0.0651) (0.0401) (0.0706) Observations 2,374,793 474,201 2,374,790 474,201 R-squared 0.005 0.032 0.052 0.010 r2_a 0.00474 0.0321 0.0523 0.0101 rss 66108 30436 62948 31126 df_m 3 3 3 3 LogL 882829 -21782 940982 -27095 Robust standard errors in parentheses ** p<0.01, * p<0.05 PPIC.ORG Technical Appendices Academic Progress for English Learners 31 Table D 4A shows the main regression results for four annual outcomes. Statistically significant results are in bold. Cells in light gra y were not estimated because the relevant language group (never ELs) was not included in the CELDT models. Cells in darker gray represent coefficients that were statistically significant in the models shown below that include all the estimators shown, but in simpler models that included only the giv en set of explanatory variables, such as the three language homogeneity variables, the coefficient was not significant. This could indicate that the significant results in the gray cells below are not robust and that significance could reflect collinearity with the other regressors. TABLE D4A Annual Outcome Regression Results Regressor Variable CST ELA Z -Score CELDT Score CST Math Z -Score Annual GPA LAUSD SDUSD LAUSD SDUSD LAUSD SDUSD LAUSD SDUSD % EL * Never EL 0.168* -0.000426 0.235 -0.00201 -0.0167 -0.00240 (0.0738) (0.00161) (0.194) (0.00574) (0.0441) (0.00205) % EL * LAEL 0.227** -0.00160 13.01 -0.548** 0.354 -0.00374 0.0329 -0.000102 (0.0786) (0.00174) (10.22) (0.153) (0.220) (0.00622) (0.0399) (0.0 0245) % EL * LTEL 0.279** 0.00113 -13.21 -0.467** 0.469* 0.000661 -0.0255 -0.00157 (0.0725) (0.00165) (8.950) (0.115) (0.184) (0.00547) (0.0423) (0.00218) % Same Home Lan - 0.000423** 0.00103** 0.000814** 0.000643* 0.000335** 0.00130** guage * Never EL (0.000115) (0.000218) (0.000176) (0.000308) (7.25e -05) (0.000240) % Same Home Lan - 0.000240 0.00325** 0.0319 0.153** -0.000906 0.00676* 0.000134 -0.00120* guage * LAEL (0.000236) (0.000440) (0.0290) (0.0514) (0.000569) (0.00268) (0.000109) (0.000507) % Same Home Lan - -9.22e -05 0.000720** 0.0669* -0.0154 -2.07e -05 0.000555 0.000118 -0.000143 guage * LTEL (0.000148) (0.000255) (0.0263) (0.0337) (0.000308) (0.000484) (7.50e -05) (0.000348) Language Homo - -0.0390 -0.0937* -0.175** 0.0127 0.00794 0.0183 geneity * Never EL (0.0239) (0.0463) (0.0660) (0.120) (0.0180) (0.0587) Language Homo - 0.0926* -0.123 -3.088 -22.34* 0.107 -0.386 0.0412 -0.0105 geneity * LAEL (0.0416) (0.0787) (6.187) (10.25) (0.0836) (0.282) (0.0278) (0.0845) Language Homo - 0.104** -0.132* 4.349 -4.769 -0.000602 -0.0398 0.0689** 0.00146 geneity * LTEL (0.0264) (0.0537) (3.784) (7.074) (0.0669) (0.121) (0.0221) (0.0575) LTEL5 Currently or in -0.209** -0.163** 25.18** -20.78** -0.274** -0.252** -0.0980** -0.0528* Past (0.0246) (0.0212) (4.232) (5.017) (0.0506) (0.0837) (0.0124) (0.0254) Ever LAEL -0.277** -0.141* -0.212* -0.570 -0.0787** -0.0365 (0.0378) (0.0641) (0.0936) (0.297) (0.0187) (0.0390) PPIC.ORG Technical Appendices Academic Progress for English Learners 32 Re gressor Variable CST ELA Z -Score CELDT Score CST Math Z -Score Annual GPA LAUSD SDUSD LAUSD SDUSD LAUSD SDUSD LAUSD SDUSD Share ELD Too Low * 0.0102 -0.200* 0.0589 -0.00245 0.0681 -0.0224 Never EL (0.0644) (0.0762) (0.141) (0.170) (0.0381) (0.111) Share ELD Too Low * -0.213* -0.384 -35.90* -16.01 -0.330 4.187** 0.0242 -0.110 LAEL (0.0979) (0.248) (17.40) (24.31) (0.218) (1.102) (0.0518) (0.248) Share ELD Too Low * -0.112 -0.317** 26.31** -6.532 0.00670 0.199 0.0716 -0.113 LTEL (0.0680) (0.0948) (7.385) (7.910) (0.0889) (0.234) (0.0400) (0.164) Share ELD Too High -0.278 0.0208 -0.0682 -0.245** 0.0458 0.136* * Never EL (0.187) (0.0403) (0.236) (0.0906) (0.0491) (0.0536) Share ELD Too High 0.373 0.0214 -22.64 -38.81** 1.280* 0.696 0.179** -0.180 * LAEL (0.226) (0.104) (16.42) (13.53) (0.636) (0.527) (0.0687) (0.108) Share ELD Too High -0.184 -0.0375 23.47** 4.280 0.278 0.01 53 0.123 -0.0540 * LTEL (0.174) (0.0490) (9.015) (6.288) (0.288) (0.110) (0.0797) (0.0796) Share With No ELD -0.0268 -0.0493* 0.0199 0.0156 0.0134 0.0272 * Never EL (0.0277) (0.0191) (0.0573) (0.0936) (0.0139) (0.0318) Share With No ELD -0.0578 -0.132* 3.722 21.16** -0.218* 0.678 0.0304* 0.176** * LAEL (0.0466) (0.0556) (3.637) (6.966) (0.0983) (0.357) (0.0147) (0.0595) Share With No ELD -0.0977** -0.0617* -1.091 4.175 -0.0356 0.0312 0.0445** 0.0381 * LTEL (0.0290) (0.0245) (3.108) (2.703) (0.0573) (0.107) (0.0170 ) (0.0458) Constant -0.391** 1.468** 223.7** 197.3** -0.154 1.863** 0.140* 0.444** (0.125) (0.0873) (28.85) (15.50) (0.182) (0.267) (0.0579) (0.106) Observations 1,243,696 233,612 490,417 57,368 406,686 67,891 930,845 303,388 R-squared 0.713 0.727 0.722 0.632 0.690 0.673 0.881 0.670 Adjusted R -squared 0.713 0.726 0.721 0.630 0.690 0.672 0.881 0.669 Sum of Squared Residuals 330696 71248 8.840e+08 7.240e+07 119837 21604 87922 96424 df_m 125 91 109 73 115 54 130 87 LogL -941001 -192775 -2.534e+06 -286204 -328596 -57465 -222579 -256608 Robust standard errors in parentheses ** p<0.01, * p<0.05 NOTE S: Cells in gr ay indicate coefficients that are significant only in the model with all the language variables, but was not significant in models that included only that specific language variable . Coefficients in bold are significant at less than the 0.05 level . PPIC.ORG Technical Appendices Academic Progress for English Learners 33 TABLE D4B Long Term Outcome Regression Results Graduate on Time Ever Reclassified LAUSD SDUSD LAUSD SDUSD VARIABLES % EL * Never EL -0.206 -0.00134 (0.173) (0.000874) % EL * LAEL 0.0147 -0.00283* 0.363** -0.00453** (0.183) (0.00122) (0.0999) (0.00170) % EL * LTEL -0.275 -0.00131 0.447** -0.00581** (0.190) (0.000819) (0.107) (0.00167) % Same Home Lan - 0.000452** 0.000642** guage * Never EL (8.34e -05) (0.000203) % Same Home Lan - -0.000636** -0.000104 -0.000391 0.000817 guage * LAEL (0.000235) (0.000618) (0.000308) (0.000471) % Same Home Lan - 0.000575** -0.000256 0.000124 0.000417 guage * LTEL (0.000153) (0.000310) (0.000248) (0.000420) Language Homo - -0.112* 0.00621 geneity * Never EL (0.0492) (0.0274) Language Homo - -0.216** -0.0313 0.0485 -0.142 geneity * LAEL (0.0729) (0.0691) (0.0478) (0.0947) Language Homo - -0.151** -0.0728 0.0276 -0.167 geneity * LTEL (0.0559) (0.0384) (0.0345) (0.0849) LTEL5 Currently or in 0.105** 0.0954** 0.231** 0.0839 Past (0.0223) (0.0235) (0.0272) (0.0426) Ever LAEL -0.0575 0.0696 (0.0411) (0.0511) Share ELD Too Low * -0.113 0.102** Never EL (0.0752) (0.0322) Share ELD Too Low * -0.0914 -0.135 0.704** -0.397* LAEL (0.154) (0.245) (0.0812) (0.194) Share ELD Too Low * -0.135 -0.0632 0.657** -0.350** LTEL (0.0938) (0.0784) (0.0718) (0.104) Share ELD Too High 0.262** 0.0819** *Never EL (0.0843) (0.0235) Share ELD Too High -0.0168 -0.101 -0.167 0.180* *LAEL (0.243) (0.0927) (0.0998) (0.0881) Share ELD Too High -0.104 -0.231** -0.297** -0.0105 * L TEL (0.114) (0.0421) (0.0738) (0.0510) Share With No ELD 0.0121 0.0436** *Never EL (0.0277) (0.0128) Share With No ELD 0.124* 0.174* 0.0471 -0.248** *LAEL (0.0535) (0.0700) (0.0273) (0.0600) Share With No ELD -0.147** 0.0423* -0.0208 -0.275** * L TEL (0.0368) (0.0193) (0.0251) (0.0337) Constant 0.885** 0.613** -0.201 0.727** (0.0763) (0.0517) (0.217) (0.169) PPIC.ORG Technical Appendices Academic Progress for English Learners 34 Graduate on Time Ever Reclassified LAUSD SDUSD LAUSD SDUSD VARIABLES Observations 212,098 220,577 956,480 89,040 R-squared 0.200 0.250 0.072 0.329 Adjusted R -squared 0.199 0.248 0.0701 0.326 Sum of Squared Residuals 21514 24163 208655 14932 df_m 123 89 124 85 LogL -58278 -69092 -629029 -46849 NOTES: ** p<0.01, * p<0.05 Coefficients in bold are significant at less than the 0.05 level . Robust standard errors in parentheses . PPIC.ORG Technical Appendices Academic Progress for English Learners 35 Table D5 shows regression results that control for newcomer supports and LTEL course availability. TABLE D5 Supplementary Results that Add LTEL and Newcomer Course Availability to the Earlier Models Regressor Variable CST ELA Z -Score CELDT Score CST Math Z -Score Annual GPA LAUSD SDUSD LAUSD SDUSD LAUSD SDUSD LAUSD SDUSD Any Newcomer Program*Never EL 0.000518 -0.00351 -0.00687 0.149** 0.00626 -0.0224 (0.00522) (0.0164) (0.0119) (0.0266) (0.00334) (0.0225) Any Newcomer Program*LAEL -0.0107 -0.0171 0.712 -8.425 -0.0266 0.166 0.0129* -0.0140 (0.00913) (0.0345) (1.134) (6.260) (0.0211) (0.174) (0.00589) (0.0515) Any Newcomer Program*LTEL -0.0103 -0.00309 -0.0112 3.797* -0.0332** 0.126** 0.00509 -0.0220 (0.00603) (0.0217) (0.653) (1.486) (0.00952) (0.0240) (0.00466) (0.0440) Any LTEL Course*Never EL 0.000557 (0.0250) Any LTEL Course*LAEL 7.571 -0.0721 (4.804) (0.0685) Any LTEL Course*LTEL 1.599 0.00618 (2.248) (0.0358) Observations 1,286,105 233,612 524,182 57,368 416,710 67,891 997,786 303,388 R-squared 0.711 0.727 0.725 0.633 0.681 0.673 0.887 0.670 r2_a 0.711 0.726 0.724 0.630 0.681 0.672 0.887 0.669 rss 343338 71248 9.670e+08 7.230e+07 116133 21590 93085 96418 df_m 151 91 130 77 135 54 151 87 LogL -975651 -192775 -2.715e+06 -286182 -325079 -57443 -232408 -256599 Robust standard errors in parentheses. ** p<0.01, * p<0.05 Coefficients in bold are significant at less th an the 0.05 level NOTE : Cells in gr ay indicate coefficients that could not be estimated, or coefficients that in the case of LAUSD were estimable but only on sma ll atypical samples, and are thus suppressed. PPIC.ORG Technical Appendices Academic Progress for English Learners 36 TABLE D6 Supplementary Results for SDUSD that Add the Ratio of ELs to EL Support Teachers to the Earlier Models in Table D 4A Regressor Variable CST ELA Z- Score CELDT Score CST Math Z- Score Annual GPA EL to ELST Ratio *Never EL -0.413 2.159* 0.223 (0.375) (0.938) (0.791) EL to ELST Ratio *LAEL -6.062 706.3 20.68 14.78* (5.150) (457.9) (19.54) (6.464) EL to ELST Ratio *LTEL 1.008 107.7 2.468 5.375* (0.985) (108.2) (1.476) (2.076) Observations 232,897 47,276 67,710 238,055 R-squared 0.727 0.628 0.672 0.674 r2_a 0.726 0.625 0.671 0.674 rss 70893 5.740e+07 21501 73953 df_m 82 69 49 75 LogL -191959 -234948 -57241 -198634 Robust standard errors in parentheses. ** p<0.01, * p<0.05 Coefficients in bold are significant at less than the 0.05 level NOTE : Cells in gr ay indicate coefficie nts that could not be estimated, because never EL students were not given the CELDT test . PPIC.ORG Technical Appendices Academic Progress for English Learners 37 Statistical Precision In evaluating the coefficients on the variables related to the language environment, the precision of the estimates matters. We can obtain the 95% confidence interval for any coefficient by multiplying the standard error (s.e.) by 1.96, so that the 95% confidence interval will be [estimate -1.96*s.e., estimate + 1.96*s.e.]. This analysis seems most important for our findings of “no effect.” The most striking “no effect” results are that for never EL students, the percentage of students who are EL is not nega tively related to any of the academic outcomes we model. The standard errors for this variable, shown in the first row of Table D 4A, are small. The minimum effect of raising the percent EL by 1 on the CST ELA growth for LAUSD is 0.168 - 1.96*0.0738=0.023. The minimum possible effects within the 95% confidence interval for San Diego is -0.004. Similar conclusions result when looking at the other outcomes for percentage EL for the other two linguistic groups, and for the variable measuring the percentage of all students speaking the same home language. A s noted in the main text, the precision of our estimates for the EL language homogeneity variable is lower. This could reflect the relatively smaller variation in this measure, especially in LAUSD. This inde x can theoretically range from 0 to 1, and Table 1 shows that the mean in LAUSD, depending on the linguistic group, ranges from 0.8 for never ELs to 0.89 for LTELs. That said, even with this lack of precision, when we claim that language homogeneity has a significant negative or positive relation to an outcome, we are 95% certain that the sign is correct. But we are unsure of the true size of the relation. Perhaps of greater concern is the many cases where language homogeneity is not significant, for example for math CST for LTELs and GPA models for Late Arriving ELs. To give one example, in the model of GPA for LAUSD, the coefficient and standard error for language homogeneity for LAELs are 0.0412 and 0.0278, meaning that the upper bound of the confidence interval is 0.095, and the lower bound is a small negative predicted effect. A nother factor to consider is whether any of our outcomes displays bunching of observations near a ceiling or floor. For example, in theory our result that LAELs’ CELDT scores rise more quickly than those of LTELs could arise if LTELs are already scoring near the maximum possible CELDT score. Conversely, if we find no significant effect of a given variable on the CST scores of LAELs, could it be because almost all LAELs have CST scores near the minimum possible. To test whether there is massive clumping of outcome values near a ceiling or floor, we can look at the distribution of proficiency levels on the CST for all groups and for ELs, for the CELDT as well. B ecause it is hard to show floor or ceiling effects for the CST using Z scores, we instead show the distribution of test scores across language groups using proficiency levels. Similarly for the CELDT we start by showing the distribution of students by performance levels. For annual GPA we divide students into four groups, based on whether GPA is less than 1, at least equal to 1 and less than 2, at least equal to 2 and less than 3, and at least equal to 3 and less than or equal to 4. PPIC.ORG Technical Appendices Academic Progress for English Learners 38 TABLE D7 The Percentage of Students by CST ELA Proficiency Level District Language Group Far Below Basic Below Basic Basic Proficient Advanced LAUSD Never EL 9.6 14.2 27.8 27.1 21.3 LAEL 49.2 32.3 14.7 2.9 0.5 LTEL 18.3 28.7 37.0 13.4 2.7 SDUSD Never EL 4.7 8.4 22.1 31.0 33.74 LAEL 38.6 32.7 21.3 6.1 1.4 LTEL 18.5 30.1 37.7 11.6 2.2 TABLE D8 The P ercentage of Students by CST Math Proficiency Level , Grades 6 and 7 Only District Language Group Far Below Basic Below Basic Basic Proficient Advanced LAUSD Never EL 8.1 20.5 26.0 27.1 18.4 LAEL 44.2 40.8 10.6 3.6 0.8 LTEL 15.5 38.4 30.5 13.1 2.4 SDUSD Never EL 3.6 12.2 23.1 34.8 26.3 LAEL 21.8 34.2 23.6 12.2 8.3 LTEL 12.2 34.1 34.3 16.5 3.0 TABLE D9 The P ercentage of Students by CELDT Performance Level District Language Group Beginning Earl y Intermediate Intermediate Earl y Advanced Advanced LAUSD LAEL 49.7 23.0 19.2 7.3 0.9 LTEL 0.0 10.2 43.7 38.2 8.0 SDUSD LAEL 49.8 21.1 19.4 8.6 1.1 LTEL 4.5 12.5 42.1 34.9 6.1 TABLE D10 The Percentage of Students by Annual GPA Level District Language Group 0-0.999 1-1.999 2-2.999 3-4 LAUSD Never EL 8.6 24.3 36.5 30.6 LAEL 12.5 26.7 34.5 26.4 LTEL 14.5 34.0 34.8 16.7 SDUSD Never EL 4.5 12.7 30.2 52.6 LAEL 9.2 20.7 33.6 36.4 LTEL 6.6 16.2 31.6 45.6 Our concern is that if a large percentage of students is in the top or bottom group, some of them could be achieving so high or so low that a change in their underlying performance would not lead to a change in their recorded achievement measure. For all but one case (CELDT) for LAELs, a majority of students in each language group are in the middle ranges of these measures. Thus for CST scores and GPA we find little evidence of a major problem of bunching. PPIC.ORG Technical Appendices Academic Progress for English Learners 39 The exception is the CELDT performance levels for LAELs. For instance in San Diego, 49.8% of LAELs are in the bottom (Beginning) group. We took a closer look at the underlying scale scores for the CELDT to look for bunching of students at the bottom possible score. The minimum scaled score is 248 in grades 6-8 and 251 in grades 9 -12, and the respective maximum allowed scores are 741 and 761. Very few LTELs have scores near the minimum or maximum. Figures D1, D2 show the histograms for LAEL and LTEL students in LAUSD, while Figures D3 and D4 show the corresponding information for SDUSD. For LAELs about 6 percent in SDUSD and 4 percent in LAUSD have CELDT scores within 10 points of the minimum of 248, as shown by the bottom bar in the histogram, so it is possible for a very small number of these students that the CELDT test would not detect actual gains in English proficiency if the student’s initial proficiency was below the bottom limit. Theoretically this could be a concern because for these students small increases in English proficiency might not be detectable on the test. But the proportion of students for whom this could be an issue is very small. FIGURE D1 Histogram of CELDT Total Scaled Scores for LAEL Students in LAUSD 0 2 4 6 8 10 Per ce nt 2 00 400 60 0 8 00 to _s cl PPIC.ORG Technical Appendices Academic Progress for English Learners 40 FIGURE D2 Histogram of CELDT Total Scaled Scores for LTEL Students in LAUSD FI GURE D3 Histogram of CELDT Total Scaled Scores for LAEL Students in SDUSD 0 2 4 6 8 Per ce nt 2 00 400 60 0 8 00 to _s cl 0 2 4 6 Percent 200 300 400 500 600 700 CELDT to scaled PPIC.ORG Technical Appendices Academic Progress for English Learners 41 FIGURE D4 Histogram of CELDT Total Scaled Scores for LTEL Students in SDUSD For the other outcomes, we find even less evidence of ceiling or floor effects. 0 2 4 6 8 10 Percent 200 400 600 800 CELDT to scaled The Public Policy Institute of California is dedicated to informing and improving public policy in California through independent, objective, nonpartisan research. Public Policy Institute of California 500 Washington Street, Suite 600 San Francisco, CA 94111 T: 415.291.4400 F: 415.291.4401 PPIC.ORG P PIC Sacramento Center Senator Office Building 1121 L Street, Suite 801 Sacramento, CA 95814 T: 916.440.1120 F: 916.440.1121" } ["___content":protected]=> string(237) "

Academic Progress for English Learners: The Role Of School Language Environment and Course Placement in Grades 6-12, Technical Appendix

" ["_permalink":protected]=> string(165) "https://www.ppic.org/publication/academic-progress-for-english-learners-the-role-of-school-language-environment-and-course-placement-in-grades-6-12/0119lhr-appendix/" ["_next":protected]=> array(0) { } ["_prev":protected]=> array(0) { } ["_css_class":protected]=> NULL ["id"]=> int(18107) ["ID"]=> int(18107) ["post_author"]=> string(1) "4" ["post_content"]=> string(0) "" ["post_date"]=> string(19) "2019-01-23 10:07:10" ["post_excerpt"]=> string(0) "" ["post_parent"]=> int(18052) ["post_status"]=> string(7) "inherit" ["post_title"]=> string(135) "Academic Progress for English Learners: The Role Of School Language Environment and Course Placement in Grades 6-12, Technical Appendix" ["post_type"]=> string(10) "attachment" ["slug"]=> string(16) "0119lhr-appendix" ["__type":protected]=> NULL ["_wp_attached_file"]=> string(20) "0119lhr-appendix.pdf" ["wpmf_size"]=> string(7) "1529304" ["wpmf_filetype"]=> string(3) "pdf" ["wpmf_order"]=> string(1) "0" ["searchwp_content"]=> string(77015) "Academic Progress for English Learners The Role of School Language Environment and Course Placement in Grades 6-12 Technical Appendices CONTENTS Appendix A. LTEL and LAEL Definitions Appendix B. Course Assignment Appendix C. Qualitative Approach Appendix D . Regressions and Results Laura Hill, Julian Betts, Megan Hopkins, Magaly Lavadenz, Karen Bachofer, Joseph Hayes, Andrew Lee, Marco A. Murillo, Tara Vahdani, and Andrew C. Zau Supported with funding from the William T. Grant Foundation PPIC.ORG Technical Appendices Academic Progress for English Learners 2 Appendix A: LTEL and LAEL Definitions The official definition of LTELs in California is ELs who have been designated as ELs for at least six years. LAUSD defines them as ELs who have been designated as ELs for at least five years. SDUSD treats ELs who have spent five years in the district as students at -risk of becoming LTELs. Given these concerns in the two districts, we defined LTELs as those who have spent 5 or more years in the given district as ELs without having been reclassified. We replicated many of our regression results using the “six -year” definition and results were highly similar. We define a LAEL as a student whose first enrollment in the given district occurs in grade 6 or higher, the student is determined to be an English Learner, and scores at the lowest (Beginning) level on first taking the CELDT. English Learners who fall into neither category are excluded from our analyses. PPIC.ORG Technical Appendices Academic Progress for English Learners 3 Appendix B: Course Assignmen t Overview of district policy in LAUSD Figure B1 below illustrates how the LAUSD ELD placement policy environment evolved over the years of our study, particularly in terms of shifts in the tests of English -language ability that were meant to guide that placement. Throughout all study years, a student’ s DPI and CELDT were district-sanctioned indicators for making initial placement decisions. The ELA -CST was similarly referenced from 2006 through its phase -out in 2013 (with scores from this end year still us ed in 2014). Although the ELA periodic assessme nt was used for placement in 2015 only, the DIBELS, SRI, and CAHSEE, whose placement importance also began in that year, continued as reference points in 2016. However, these exam scores were formally irrelevant for subsequent placement decisions, because ELD course advancement was directed by ELD course grades from 2006 through 2013. Starting in 2014, a student’s years in U.S. schools was the most important factor for both initial and subsequent placement. FIGURE B1 Overview of assessments used to place secondary EL students in ELD coursework, LAUSD The next sections step through LAUSD’s guidance on initial course placement (at matriculation), subsequent course placement, and how we calculate the whether an EL student is placed in ELD courses that are correct, too high, or too low. Initial ELD Placement As per district guidelines, any secondary school student (i.e. enrolled in 6 th grade or higher) that carries a “Limited English Proficient” language status should be enrolled in ELD coursework. What mak es an initial placement “proper”, “too high”, or “too low” depends on the school year (placement policies for LAUSD changed 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 ELD placement depends on student CELDT and CST scores, with advancement hinging on the attainment of a passing grade . (formal LTEL -oriented courses do not yet exist) Formal LTEL courses come into being. CELDT and CST scores still inform placement, but years in US schools becomes a major factor in placement and progre ssion . Course marks are irrelevant in determining advancement CSTfades in relevance for non - LTEL pl ace me nt; the CAHSEE and ELA/Li te racy Periodic Assessments asce nd Periodic assessments give way to the DIBELS and SRI exams. CELDT and CAHSEE mai ntai n relevancy CELDT (Placement Policy Shifts) (Standardized Exam Coverage) CST DIBELS SRI CAHSEE DPI ELA -PA PPIC.ORG Technical Appendices Academic Progress for English Learners 4 over time), the student’s available test scores, and (from 2014 onwards) the student’s total years enrolled in U.S. schools. In cases where a student experiences their first secondary school year as an English Learner (EL) but is not enrolled in ELD coursework, that lapse (and that during any semester that follows) is noted – but the appropriateness of their first recorded instance of secondary school ELD enrollment (if it exists) is still evaluated based on the initial placement criteria below. It must be noted that ELD course placement is not always straightforward: policy guidelines indicate score ranges that should direct a student into a given level of ELD, but a student’s scores across multiple measures may not align neatly into one ELD level. This uncertainty becomes compounded when a student’s exam scores may not exist; evaluations of the appropriateness of a student’s initial p lacement are therefore dependent on the available data. In cases of conflicting test data, school administrators must exercise some discretion in these placement decisions, with the Language Appraisal Team “assist[ing] in determining the most appropriate placement.” 1 General Placement Rules Given the inherent vagaries of assessing proper placement due to data limitations and Language Appraisal Team discretion, we established certain guiding rules. If the entirety of a student’s available placement data ( e.g. CELDT score) all jointly direct that student to a given course placement, it is understood as being “proper”. If one component of a student’s placement data direct them to a higher level of ELD than that to which they are actually enrolled, that place ment is considered as being “too low”. In contrast, for a student’s initial placement to be labeled as “too high”, it is not only necessary that all the available placement data components point to a lower - tiered ELD course; it is additionally necessary th at no relevant placement data are missing, the logic being that the missing component could presumably have buoyed that student’s placement above the level to which they are directed by their available criteria. In the absence of any cause to label an init ial placement as “too high” or “too low” it is defaulted to “proper” status – unless there is an absence of any ELD coursework to begin with in which case (with rare exceptions) that placement is labeled simply as “No ELD”. Pre- 2014 Prior to the 2013 -14 academic year there were no ELD courses that were formally designated for LTEL students. All EL students, irrespective of years spent in U.S. schools are assigned to ELD courses (titled “ESL” in these years) on the basis of CELDT, CST -ELA, and DPI scores (see Table B1). As per the discussion in the ‘General Placement Rules’ section, the lack of student DPI data means that no student’s initial ELD assignment will be labeled as “too high” in these years. A “too low” label is certainly possible, for example i f a student were placed in “Beginning ESL 1B” yet possessed a CELDT score of 2 or higher. A small minority of EL -designated students in these years may be designated as having “proper” initial placement even in the absence of any ELD coursework, with CEL DT or CST-ELA scores high enough to effectively “test out” of ELD while simultaneously falling short of actual reclassification as English -proficient. 1 English Learner Placement Guide REF -5151, 2010. PPIC.ORG Technical Appendices Academic Progress for English Learners 5 TABLE B1 Pre- 2014 English -Learner Initial Placement Guidelines SOURCE: LAUSD English Learner Placement Guide REF -5151, 2007 -2013 2013 -14 A key change in placement guidelines moving into the 2013- 14 academic year was the shift towards factoring in a student’s cumulative years in U.S. schools. For all EL students, different -tiered ELD courses began being associated with “years in U.S. schools” thresholds insofar as enrollment in a given ELD course became prohibited if the student had already experienced a given number of years in U.S. schools. For example, a student enrolled in ELD 2A who had spent 4 years in U.S. schools would be considered as having been placed “too low” since – regardless of test scores – their placement stands in violation of the maximum year threshold for that tier of course. For LTEL students (now guided into courses created exclusively for these types of ELs), years in U.S. schools functions more as a barrier to entry than as a potential route towards a placement being labeled “too low”. Specifically, EL students are only eligible for either of the two LTEL -level courses if they have spent more than 4.5 years in U.S. schools – fewer years, in the context of such a placement, would be considered “too high”. PPIC.ORG Technical Appendices Academic Progress for English Learners 6 TABLE B2A 2013 -14 English -Learner Initial Placement Guidelines (non -LTELs) SOURCE: LAUSD English Learner Placement Guide REF -6046.0, 2014 TABLE B2B 2013 -14 English -Learner Initial Placement Guidelines (LTELs) SOURCE: LAUSD English Learner Placement Guide REF -6046.0, 2014 2014-15 From the 2013- 14 academic year onwards, the basic structure of ELD placement delineating between mainstream ELD and LTEL -oriented ELD remained intact. Years in U.S. schools, as well as DPI and CELDT scores persisted as relevant decision -making factors. However, in subsequent years other exams are sometimes swapped in or out as relevant placement considerations. The ELA CST was discontinued in the 2012- 13 academic year – although older scores were used as a placement reference in 2014. The ELA Periodic Assessment #2 and the California High School Exit Examination (CAHSEE) (for which we possess score data) and the Literacy Periodic Assessment (LPA #3) (for which we do not) were phased in. These exam scores are drawn from particular grades and/or school years depending on whether a student is being placed for mainstream or LT EL-oriented ELD (see Tables B3A/B3B). PPIC.ORG Technical Appendices Academic Progress for English Learners 7 TABLE B3A 2014 -15 English -Learner Initial Placement Guidelines (non -LTELs) SOURCE: LAUSD English Learner Placement Guide REF -6046.1, 2015 TABLE B3B 2014 -15 English -Learner Initial Placement Guidelines (LTELs) SOURCE : LAUSD English Learner Placement Guide REF -6046.1, 2015 2015-16 In the 2015- 16 academic year, similar to the one preceding it, certain exams were phased in and out as points of reference for placement, but the use of years in U.S. schools and CELDT score remained unchanged. For mainstream ELD placement the Literacy and ELA Periodic Assessments (LPA/ELA PA) were replaced by the Scholastic Reading Inventory (SRI) and Dynamic Indicators of Basic Early Literacy Skills (DIBELS). For LTEL placement the tw o periodic assessments were likewise dropped in favor of the SRI and DIBELS (see Table B4A/B4B). PPIC.ORG Technical Appendices Academic Progress for English Learners 8 TABLE B4A 2015 -16 English -Learner Initial Placement Guidelines (non -LTELs) SOURCE: LAUSD English Learner Placement Guide REF -6046.3, 2016 TABLE B4B 2015 -16 English -Learner Initial Placement Guidelines (LTELs) SOURCE: LAUSD English Learner Placement Guide REF -6046.3, 2016 Subsequent ELD Placement After the semester of initial ELD placement, unless an EL student is reclassified as English -proficient, any subsequent semester of enrollment in LAUSD should feature sequential ELD coursework in line with district policy memorandums. Like the initial placement criteria, subsequent placement criteria evolved over the study years. As with initial ELD placement, we e valuate subsequent placement as being degrees of “proper”, “too high”, and “too low” according to a student’s course progression – initially modified by their course grades, then by their years in U.S. schools. In cases where there is a lapse in ELD coursework after the initial placement, that ‘no -ELD’ gap is noted – but if any subsequent ELD enrollment occurs, it is evaluated in the context of the preceding ELD enrollment. In all cases of non -reclassified ELs, however, once a student has been initially placed in ELD, their subsequent placement is not formally influenced on the basis of any newer test. Placement in ELD Courses for Students with Disabilities Whenever a student is enrolled in an ELD course directed towards students with disabilities, it is u nderstood that – although district policy recommends enrollment for certain disabled students according to their test scores PPIC.ORG Technical Appendices Academic Progress for English Learners 9 and years in U.S. schools – the student’s IEP team is allowed significant discretion in that placement. 2 In light of this unique role, any initial or subsequent placement into/out of this subset of ELD courses is considered “proper”. Pre-2014 Prior to the formal creation of ELD courses for LTELs in the 2013 -14 academic year, EL enrollment occurred along a tiered ‘ESL’ course seque nce, upon completion of which a non- reclassified student would be expected to exhibit continuous enrollment in grade -level ELA (see Figure B2). Advancement to the next course in the sequence was mandated if the student received a passing mark in their curr ent ELD class; failure would justify repetition. If a student skips ahead in the sequence or advances without attaining a passing grade in their current course, their subsequent placement is considered “too high”, whereas repetition in the absence of a failing mark or regression along the sequence under any circumstance is assessed as being placed “too low”. Post-2013 With the establishment of LTEL -specific courses in 2013 -14, ELD course grades became irrelevant to subsequent placement; a student’s prog ression to the next sequential ELD course is automatic and “proper”, provided the ‘years in U.S. schools’ thresholds are respected (see Tables A2 -A4). Repetition of any ELD course, regardless of a student’s marks, was likewise considered acceptable if alig ned with these year limits. A student is considered as having placed “too high” if they skip courses in the sequence – or are enrolled in LTEL courses before accruing the proper number of years in U.S. schools. Likewise, a student is still considered as pl acing “too low” if they regress in the sequence – but also if they exceed the year threshold for a given course. The new LTEL -oriented ELD courses exist in two tiers, “Literacy and Language” and the more rigorous “Advanced ELD”. Sequences of progression exist within each tier, but once the student completes its most advanced level they must repeat it until reclassification – or shift to the “Advanced ELD” tier in the case of students completing the “Literacy and Language” sequence. “Development ELS” is a supplemental learning course for LTELs that is considered a “proper” placement wherever it appears in the LTEL course sequence. Synthesizing Student-Year Placement Both initial and subsequent ELD placements are evaluated in terms of being “proper”, “too high”, “too low”, and exhibiting “no ELD”, but within a given semester a student’s situation may constitute a blend of these categories. For example, a student enrolled in ELD 3A is expected to progress to ELD 3B in the subsequent semester and simultaneous ly enroll in a grade-level ELA course. Supposing that transition to ELD 3B occurs – but without the requisite grade -level ELA – that subsequent placement would be considered both “proper” (on account of the correct sequential transition to ELD 3B) and “too low” (on account of this lack of requisite ELA) in equal proportion. If that same ELD 3A student were to instead jump to ELD 4B – also without the requisite grade -level ELA – that subsequent placement would be both “too high” and “too low”. In the same way that a given semester’s placement is established as being some mix of “proper”, “too high”, “too low” , or “no ELD”, a student in a given year is understood as making a ‘contribution’ to these categories on the basis of their fall and sprin g placement. If an EL were initially placed in ELD 1B in the fall semester (in full accord with district guidelines) and enrolled in no ELD course (but was not reclassified) in the spring, they would contribute 0.5 to “no ELD” and 0.5 to “proper” for that year. In a given year these contributions are summed up 2 Scheduling ELD Instruction for ELs with Disabilities REF -5994.1, 2015 PPIC.ORG Technical Appendices Academic Progress for English Learners 10 across all ELs so as to establish annual rates of “proper”, “too low”, “too high”, and “no ELD” placement within grades, schools, and the district as a whole. Classrooms With More Than One ELD Level Interviews in LAUSD suggested that some school staff struggled to create classroom environments in which a single level of ELD instruction was appropriate for enrolled EL students. For most years we are able to estimate how many ELD courses (and ELA cours es) are being taught simultaneously in the same classroom. We found that rates of multi -rostering were almost always under 5 percent for LTELs but started to climb for LAELs beginning in the 2009 -10 school year. FIGURE B2 Multi- Rostering Rates Increasing for Late Arriving ELs SOURCE: Authors ’ calculations from LAUSD data NOTE : Course period data not available for the 2013 -14 and 2014- 15 school years. PPIC.ORG Technical Appendices Academic Progress for English Learners 11 FIGURE B3 Pre- 2014 ELD Course Sequence PPIC.ORG Technical Appendices Academic Progress for English Learners 12 FIGURE B4 Post -2013 ELD Course Sequence PPIC.ORG Technical Appendices Academic Progress for English Learners 13 Overview of district policy in SDUSD Secondary School Placement Guidelines for English Learners in San Diego Unified School District San Diego’s English Learners are purposefully placed in English coursework based on their length of enrollment and overall Engli sh proficiency (OPL) as determined by the California English Language Development Test (CELDT). The CELDT is administered when students first enroll in the district and every fall thereafter until they are reclassified as fluent English proficient. CELDT reports five OPLs: beginning, early intermediate, intermediate, early advanced, and advanced. Prior to the 2008 -09 academic year, the only criterion used to determine English course placement was the OPL. ELs at the earlier stages of English fluency (i .e., OPLs of beginning, early intermediate, or the bottom one -third of intermediate) were placed into Structured English Immersion (SEI) coursework. SEI coursework clusters students by proficiency level, and emphasizes acquiring English as a second language (ESL). ELs in the later stages of English proficiency (i.e., OPLs of the top two -thirds of intermediate, early advanced, and advanced), but not yet reclassified, were placed in Mainstream English Cluster (MEC) coursework. MEC coursework provides Engli sh language development (ELD) supports for ELs enrolled in grade -level English classes alongside their native English -speaking peers. Beginning with the 2008- 09 academic year, San Diego’s EL placement guidelines changed, and length of enrollment was adde d as a placement criterion. In an effort to prevent students from spending extended time in SEI/ESL coursework and “getting stuck” at earlier stages of English fluency, the district decided to advance students through the course sequence at a faster pace than using OPL alone would allow. As a result, students with OPLs at the beginning, early intermediate, and the bottom one -third of intermediate levels were placed into the higher -level MEC/ELD coursework, regardless of their OPLs – if they had been enrolled in the district for four or more years. Under the prior guidelines, students would not be placed in MEC/ELD coursework until their OPLs were in the top two- thirds of the intermediate range or higher, no matter how long they had been enrolled. Lengt h of enrollment has also been used to guide placement of students enrolled less than four years since the 2008- 09 academic year. These students are enrolled in SEI/ESL coursework if their OPLs are at the beginning, early intermediate, or the bottom one -th ird of intermediate. However, students with higher OPLs are assigned higher level coursework (as described below), not “held back” based on their length of enrollment. The following tables provide general course placement guidelines for students at th e middle and high school levels. Please note that, in some academic years covered by our study, there were minor variations to the course placement guidelines that follow. These variations were taken into account in our analyses, but are not detailed in this document. Middle School Placement Guidelines Newcomer classes are designed for recently arrived English Learners at the Beginning level of English fluency who have not previously attended school or whose formal education has been interrupted. Three -period Newcomer classes focus on developing English fluency and basic literacy skills. They also introduce the basics of other academic disciplines and orient students to schooling in the United States. For the balance of the day, Newcomer students are enrolled in general education coursework (such as ph ysical education, visual and performing arts) with their English speaking peers. Newcomer students with OPLs of early intermediate and the lower one -third of the intermediate range – or students enrolled in schools that do not have enough students to offe r Newcomer courses – are enrolled in the SEI/ESL courses described in the next section, regardless of their PPIC.ORG Technical Appendices Academic Progress for English Learners 14 schooling experience. Students typically remain in Newcomer classes for only one year. Newcomer coursework, including criteria for placement, is provided in the following table. Newcomer Course Sequence for Middle School Grade Level Length of Enrollment OPL Course Grade 6 ≤1 Year B ESL Literacy 6 th Core Newcomer (3 periods) Grades 7-8 ≤1 Year B ESL Literacy 7 th-8th Core Newcomer (3 periods) Note: B = Beginning Middle school English Learners who do not yet have OPLs in the upper two- thirds of the Intermediate range are enrolled in the following SEI/ESL courses, according to their length of enrollment and OPL level. The higher of the two crite ria determines placement. For example, the appropriate placement for a student enrolled less than one year with an OPL of Early Intermediate would be ESL Level 2, not ESL Level 1. The SEI/ESL sequence of coursework provides opportunities for students to r ead and write across a variety of genres, express themselves for different audiences and purposes, and acquire academic language needed to access content in other subject areas. SEI/ESL coursework, including criteria for placement, is provided in the foll owing table. Note that students enrolled for four or more years are enrolled in the MEC/ELD course sequence (see following section), regardless of their OPL. SEI/ESL Course Sequence for Middle School Grade Level Length of Enrollment OPL Course Grades 6-8 ≤1 Year B ESL Level 1 5 th-8th (2 periods) 1.0 -2.5 Years EI ESL Level 2 5 th-8th (2 periods) 2.5 to 3.9 Years Lower 1/3 of I ESL Level 3 5 th-8th (2 periods) Note: B = Beginning, EI = Early Intermediate, I = Intermediate Middle school English Learners who have OPLs in the upper two -thirds of the intermediate range and higher OR who have been enrolled the district for four or more years (regardless of OPL) are enrolled in MEC/ELD courses. Courses in this sequence feature t he same content as grade-level English courses taken by English -fluent students, but utilize Specially Designed Academic Instruction in English (SDAIE) strategies to help English Learners learn grade -level material. From 2010- 2011 onward, the district’s c ourse placement guidelines specified that, whenever possible, MEC/ELD classrooms should have approximately one -third ELs and two -thirds native English -speaking students. MEC/ELD coursework, including criteria for placement, is provided in the following ta ble. One important feature of the district’s MEC/ELD courses is that they are two periods in length; grade -level English courses for English proficient students at most grade levels are one period in length. PPIC.ORG Technical Appendices Academic Progress for English Learners 15 MEC/ELD Course Sequence for Middle School Grade Level Length of Enrollment OPL Course Grade 6 4+ Years OR Top 2/3 of I ELD or Grade -Level English 6 th (2 period course) Grade 7 4+ Years OR Top 2/3 of I ELD or Grade -Level English 7 th (2 period course) Grade 8 4+ Years OR Top 2/3 of I ELD or Grade -Level English 8 th (2 period course) Note: I = Intermediate High School Placement Guidelines As at the high school level, Newcomer classes are designed for recently arrived English Learners at the Beginning level of English fluency who have not previously attended school or whose formal education has been interrupted. These three- period classes f ocus on developing English fluency and basic literacy skills. They also introduce the basics of other academic disciplines and orient students to schooling in the United States. For the balance of the day, Newcomer students are enrolled in general educat ion coursework (such as physical education, visual and performing arts) with their English speaking peers. Newcomer students with OPLs in the upper one -third of the beginning range, early intermediate level, or the lower one -third of the intermediate rang e – or students enrolled in schools that do not have enough students to offer Newcomer courses – are enrolled in the SEI/ESL courses described in the next section, regardless of their schooling experience. At the high school level, students typically rema in in Newcomer classes for only one semester. High School Newcomer coursework, including criteria for placement, is provided in the following table. Newcomer Course Sequence for High School Grade Level Length of Enrollment OPL Course Grade 9 <6 Months Low to Mid B ESL Literacy Core 9 th Newcomer (3 periods) Grades 10 -12 <6 Months Low to Mid B ESL Literacy Block 10 th-12 th Newcomer (3 periods) Grades 10 -12 <6 Months Low to Mid B ESL Literacy Core 10 th-12 th Newcomer (6 periods) Note: B = Beginning High school English Learners who do not yet have OPLs in the upper two- thirds of the intermediate range are enrolled in the following SEI/ESL courses, according to their length of enrollment and OPL level. The higher of the two criteria determines placement. For example, the appropriate placement for a student enrolled less than one year with an OPL of Early Intermediate would be ESL 3- 4 Literacy Block, not ESL 1-2 Literacy Block. The PPIC.ORG Technical Appendices Academic Progress for English Learners 16 SEI/ESL sequence of coursework provides opportunities for students to read and write across a variety of genres, express themselves for different audiences and purposes, and acquire academic language needed to access content in other subject areas. SEI/ESL coursework, including criteria for placement, is provided in the fol lowing table. Note that students enrolled for four or more years are enrolled in the MEC/ELD course sequence (see following section), regardless of their OPL. SEI/ESL Sequence for High School Grade Level Length of Enrollment OPL Course Grades 9 -12 ≤1 Year B ESL 1 -2 Literacy Block (2 periods) 1.0-2.5 Years EI ESL 3-4 Literacy Block (2 periods) 2.5 to 3.9 Years Lower 1/3 of I ESL 5 -6 Literacy Block (2 periods) Note: B = Beginning, EI = Early Intermediate, I = Intermediate High school English Learners who have OPLs in the upper two- thirds of the intermediate range (and higher) OR who have been enrolled the district for four or more years (regardless of OPL) are enrolled in MEC/ELD courses. Courses in this sequence feature t he same content as grade-level English courses taken by English -fluent students, but utilize Specially Designed Academic Instruction in English (SDAIE) strategies to help English Learners learn grade -level material. While not required, it is preferred tha t English Learners be enrolled in the two-period (Block) versions of these English courses. From 2010- 2011 onward, the district’s course placement guidelines specified that, whenever possible, MEC/ELD classrooms should have approximately one- third ELs and two-thirds native English -speaking students. MEC/ELD coursework, including criteria for placement, is provided in the following table. MEC/ELD Sequence for High School Grade Level Length of Enrollment OPL Course Grade 9 4+ Y ears OR Top 2/3 of I English 1-2 (1 period) or English 1- 2 Block (2 periods) Grade 10 4+ Years OR Top 2/3 of I English 3- 4 (1 period) or English 3- 4 Block (2 periods) Grade 11 4+ Years OR Top 2/3 of I American Literature (1 period) or American Literature Block (2 periods) Grade 12 4+ Years OR Top 2/3 of I Contemporary Voices in Literature (1 period) or World Literature (1 period) Note: I = Intermediate PPIC.ORG Technical Appendices Academic Progress for English Learners 17 Appendix C: Qualitative Appro ach Data Collection Qualitative data collection and analyses were conducted in two phases. In the first phase, semi -structured interviews with central office administrators and leaders were conducted to understand each district’s policy as it pertains to t he course placement and reclassification for Long -term English Learner (LTEL) and Late Arrival English Learner (LAEL) students. Five district-level interviews were conducted at SDUSD, while six district - level interviews were conducted at LAUSD. Interviewees were selected based on their position and the extent to which their work intersects with English Learner (EL) policies and programs. For example, in LAUSD, we interviewed the Executive Director of the Multilingual and Multicultural Education Department and the Coordinator of EL while in SDUSD, the Director of the Office of English Language Acquisition and the Program Manager in the Office of English Language Acquisition were among the interv iewees. Interviews were semi- structured to ensure data alignment among multiple participants (Miles & Huberman, 1994; Patton, 1990). The interview protocol was developed to understand the ways in which central office administrators’ work intersects with secondary EL policies and programs, and specifically to explore their involvement in developing and implementing EL course assignment and reclassification policies. We also collected district policy documents pertaining to EL identification, course assign ment, and reclassification to identify how these policies are meant to be implemented in each district. These documents allowed us to triangulate findings from policy documents, and to understand how and why there may be deviations from district policy. In phase 2 we conducted school -level interviews to explore how school settings and instructional programs support (or do not support) LTELs and LAELs, as well as the challenges and opportunities school staff encounter as they implement EL policies in the context of these settings and programs. Findings from our quantitative analysis were used to inform our qualitative data collection with respect to site selection and interview protocol development. A matrix was created that placed schools according to % EL, the Herfindahl index that measures the language homogeneity of ELs, and CELDT gains ( see Table C-1). Ten schools from each district were identified and invited to participate based on the matrix. Although we sought to include schools across all cell s in the matrix, we were not able to collect data from all the schools we invited to participate. School level interviews were conducted at seventeen schools in total -- eight LAUSD schools (46 interviews) and nine SDUSD schools (48 interviews) for a total of 94 interviews during fall 2017 and early winter 2018. We interviewed at least one school from each cell of our 2x2x2 school selection matrix in both districts. Interviews were semi -structured, with the aim of understanding how school leaders, counselo rs, and teachers support LTELs and/or LAELs in particular school settings and instructional programs. The interview protocol focused on understanding the school’s EL population, the staff member’s roles and responsibilities with respect to supporting LTELs and LAELs, and the broader scope of supports for LTELs/LAELs, and how programs and policies are implemented for these groups of students. PPIC.ORG Technical Appendices Academic Progress for English Learners 18 TABLE C1 School Selection Criteria Matrix District Low % EL High % EL Low Herf High Herf Low Herf High Herf Low CELDT gains LA: Middle School SD: 1 Middle School (no data collected; 1 High school LA: 1 Middle School SD: 1 Middle School LA: 2 High Schools SD: 1 High School LA: 1 High School SD: 1 Middle School High CELDT gains LA: 1 Middle School (no data collected); 1 High school SD: 1 Middle School LA: 1 Middle School (no data collected) SD: 1 High School LA: 1 Middle School SD: 1 Middle School; 1 High School LA: 1 Middle School SD: 1 Middle School Analysis Preliminary analysis consisted in developing district -level and school -based memos based on our initial observations and impressions from interview data. The Los Angeles and San Diego teams developed memos around core topic areas: interviewee information, school characteristics, student population, programs, course placement, reclassification, and other (e.g., professional development). We then engaged in some initial coding around the study’s research questions to assess the extent to which we needed to dive deeper into the transcripts to generate preliminary findings. The teams each read four site memos from across the two districts to generate a list of initial codes. We concurrently coded one interview from each district with the following codes: o Sch ool Information o Course Placement Processes o Course Placement Challenges o Reclassification Processes o Reclassification Challenges o LTEL Supports o LAEL Supports The team created a cross- site data matrix that identified the seven agreed upon codes that mapped onto the research questions. The Los Angeles and San Diego teams completed the cross- site matrix the process using the subsets of school sites memos and trans cripts from each district. We added three additional categories/codes that were not specifically derived from the original set of research questions and that were identified upon review of sites/interview data: o General challenges, needs and lack of suppor t o Supports beyond coursework o Other observations The team discussed the cross- site matrix and to determine a process for identifying initial patterns in the data related to our research questions. We created brief school summaries to provide some context fo r each of the sites. We then developed two data displays: one for each district related to course placement. PPIC.ORG Technical Appendices Academic Progress for English Learners 19 Course Placement Analysis (Spring/Summer) The qualitative team met in May 2018 to discuss the cross-district course placement analysis. An ini tial set of codes were developed focused on EL demographics/info, course placement, course descriptions, compliance and monitoring, EL school supports, and teacher supports. The team selected one school from each district (with pseudonyms Central HS and K ennedy HS) to code independently. The team met again to discuss the coding and process and refine the codes. All interview transcripts were loaded into the Dedoose data analysis software program, chosen because it facilitates collaborative qualitative an d mixed methods analysis. We used a hybrid approach to coding documents and interviews (Miles & Huberman, 1994), beginning with a list of codes that capture official district policies, policy implementation processes, and any challenges to implementation and/or deviations from policy. In addition to these a priori codes, we derived codes inductively for new ideas and themes, first by assignment general descriptive codes and then by comparing, contrasting, and grouping codes until we had a final list (Strau ss & Corbin, 1990). The final code list was comprised of 18 codes that reflected issues related to LTEL/LAEL student characteristics and course placement. They included student behaviors or needs, course description, course placement, assessment, afterschool EL supports, in- school EL supports, compliance and monitoring, and teacher capacity. Each team coded the remainder of the school interviews. During the coding process, we created a memo outline to capture school -level findings across each school sit e. Each memo captured the themes discussed in the outline and included participant quotes to support major findings. The memo consisted of the following themes: o School characteristics o Course placement o Other supports offered to ELs that may explain outcomes o Implementation supports and challenges related to student population o Implementation supports and challenges to leadership and capacity  Teacher Capacity  Professional Development o Other Issues Each area/theme included sub -categories for LAEL and LTEL students to distinguish differences across course placement and supports for each student group. A memo was written for each school. To integrate findings across the two districts, members from the qualitative team were assigned an area/theme to review and analyze across all the memos. For example, the LA team reviewed “Other supports offered to ELs that may explain outcomes” and “Implementation supports and challenges to leadership and capacity” . All members reviewed school characteristics. Each team developed high -level themes and findings based on the assigned memo area/theme, and the team met to discuss the findings and determine similarities and differences between schools and districts. To summarize findings, the qualitative team created a matrix to detail the course placement processes, support, and teacher capacity across each district and by LAEL and LTEL student. PPIC.ORG Technical Appendices Academic Progress for English Learners 20 Appendix D: Regressions and Results We performed regression analyses fo r the academic outcomes of LTELs and LAELs. Our sample, running from the 2006- 07 through the 2015- 16 school years, includes all secondary school (grade 6 -12) students who fall into one of three language groups: never EL, LAEL (now or in a prior year), and LTEL (now or in a prior year). We exclude a small number of EL students who are not yet LTEL because they have not been ELs in the district for five years, and arrived in elementary school, so that they are not LAELs. Consider the following model of an outcome for student i in grade g in school year t , S igt, where LTEL and LAEL are dummy variables indicating long -term and late- arriving ELs, respectively. For outcomes available annually, including CST Z -scores, CELDT scaled scores, and grade point average (GPA), a common and quite general way to measure student progress year by year is to model the annual outcome as a function of the lagged outcome from the prior year, plus other explanatory factors : (1) &#55349;&#56390;&#55349;&#56390; &#55349;&#56406;&#55349;&#56406;&#55349;&#56406;&#55349;&#56406;&#55349;&#56406;&#55349;&#56406; =&#55349;&#57084;&#55349;&#57084;+ &#55349;&#57091;&#55349;&#57091;&#55349;&#56390;&#55349;&#56390; &#55349;&#56406;&#55349;&#56406;, PPIC.ORG Technical Appendices Academic Progress for English Learners 21 indicated by the subscript b. The measure w e chose was the CST ELA score, observed in grade 5 if available, and if not available in grade 5 then in grades 4, 3. 2, 6 or higher, in that order. 3 Because the baseline performance levels were measured in different grades for different students, we also include in (3) dummy variables PERFGRADE indicating the grade in which each baseline measure is observed. A school with a higher EL percentage is likely to devote considerable effort to teaching English as a second language, which could benefit these stu dents. On the other hand, the relative lack of never -EL students in these schools could slow EL s’ academic progress because it lowers the frequency with which ELs converse with peers fluent in English. It is also conceivable that a school’s EL percentage could influence never -ELs’ academic progress by changing classroom heterogeneity, pedagogy, and standards. We test whether there is a relationship between a school’s EL perce ntage, which we refer to as PCT EL st, and student progress. It is especially interesting to know whether there is a relationship between this variable and the respective progress of never ELs , LTELs, and LAELs, and whether these associations differ. We modify model ( 1) above by adding P CTEL st which is interacted three times, with indicators for the two EL student types and never ELs: (4) &#55349;&#56390;&#55349;&#56390; &#55349;&#56406;&#55349;&#56406;&#55349;&#56406;&#55349;&#56406;&#55349;&#56406;&#55349;&#56406;&#55349;&#56406;&#55349;&#56406; = &#55349;&#57084;&#55349;&#57084;+ &#55349;&#57091;&#55349;&#57091;&#55349;&#56390;&#55349;&#56390; &#55349;&#56406;&#55349;&#56406;, 3 Because we needed a baseline achievement measure that included never EL students, we did not use the CELDT. Similarly we did not use baseline math scores as the CST math test students take in the upper secondary grades varie s depending on the level of math course taken, making interpretation difficult. PPIC.ORG Technical Appendices Academic Progress for English Learners 22 while at the other extreme a value of 0 means that no two ELs at a school speak the same home language. These two regressors are added to (4) in the same way that PCTEL was, that is, each interacted with dummy variables for the three language groups in the sample: never EL, LAEL and LTEL. The model that we report in Appendix Table D 4A further interacts the indicators for the three language groups with the share of ELs at the school who in the given year were placed into no ELD course at all, the share of ELs in an ELD course at a higher level than expected given district guidelines, and the share of ELs in an ELD course at a lower level than expected. The omitted variable here is the share of ELs placed into an EL group correctly. The full model is shown below. (6) &#55349;&#56390;&#55349;&#56390; &#55349;&#56406;&#55349;&#56406;&#55349;&#56406;&#55349;&#56406;&#55349;&#56406;&#55349;&#56406; =&#55349;&#57084;&#55349;&#57084;+ &#55349;&#57091;&#55349;&#57091;&#55349;&#56390;&#55349;&#56390; &#55349;&#56406;&#55349;&#56406;, 4 A simplification in the notation in (6) is that we do not include a student subscript i on the language environment variables LANG hst. This is appropriate for five of the s ix language variables, which are measured for an entire school in a given year. It is not accurate for Same_a, which is the p ercentage of all students at the school who speak the same home language as student i. This variable will take on different values for two students attending the same school in a given year if the two students speak different home languages. But for the other five LANG variables, such as the percentage of students who are ELs at the school, the values are constants across all students attending a given school in a given year. 5 If we took these models where we instead control for ELD placem ent for individual EL students as valid, which we do not, placement into no ELD classes would be predicted to lead to no impact on LAELs in SDUSD e xcept for a positive effect on CELDT scores, negative impacts on LAELs in LAUSD for all four outcomes, positive impacts on LTELs in SDUSD for CST ELA, CELDT and GPA, but negative effects for math CST scores, and in LAUSD negative impacts on LTELs for CST E LA and math scores , but positive (and implausibly large ) effects on CELDT scores. PPIC.ORG Technical Appendices Academic Progress for English Learners 23 As another robustness check, we replaced the overall percentage of ELs with no ELD, too high or too low an ELD course with measures of these specific to LAELs and LTELs. We do not prefer this approach because for LAELs the number of LAELs in a given school is often very small. This means that when we calculate the share of that group receiving a certain qualit y of ELD placement, such as too high a placement, the LAEL student plays a large role in calculating that mean, which creates a similar risk of endogeneity to that described earlier. The other issue is that because the number of LAELs in a given school can be small, calculating a mean share of LAELs who receive a given type of ELD will tend to be a quite imprecise measure. For instance, in San Diego when we calculated the number of LAELs for each school/year observation, the median school/year had only 4 LA ELs enrolled. When we replaced the ELD placement shares by shares calculated separately for LAELs and LTELs, in SDUSD the most important change was that the coefficients on the no ELD share for LAELs became insignificant in all the models in Tables D 4A a nd D 4B, except for the model of whether the student was ever reclassified, in which the coefficient for the share of LAELs with no ELD remained negative and significant . The change for LAUSD was quite similar. All of the coefficients on the course placemen t variables for LAELs which had been significant became insignificant, with one exception. In LAUSD, school shares of low course ELD placement for LAELs gained significance for GPA growth, and were positive. There were no changes on LAUSD’s coefficients f or course placement among LTELs. The drop in the frequency with which the no ELD variable was significant for LAELs could reflect both the greater risk of this measure being endogenous than was the case in our main model, where we calculated mean ELD cours e placement for all ELs rather than for small subgroups of ELs, as well as the lower precision of the course placement variables when they were estimated on subgroups. We also examine the impact of differentiated language classes for LT ELs and beginning LAELs in the two districts. We assess the relation between enrollment in these classes and student outcomes by estimating versions of models ( 6) that also include controls for whether the given school in the given year offered any newcomer classes or ALD c lasses. Again, we do not control for whether individual students participated in these courses as individual students’ course placement is likely to be endogenous. The resulting models are identical to (6) but add six new interaction terms between ALD and newcomer class provision on the one hand and the three language groups on the other. Results are summarized in Appendix Table D 5. The main text expresses our concern that a lack of precision, especially in LAUSD, renders the results for ALD suspect. We h ave more confidence in the newcomer results, especially in SDUSD where newcomer courses were provided in some schools for all but the earliest and latest years of our study. In addition to the results tabulated later in this appendix and discussed in the main text, we performed one robustness check regarding the variable measuring the percent of students who are ELs. There is a clear potential for nonlinear relations between this explanatory variable and outcomes. The careful reader may have noticed that in theory the correlation between academic outcomes and some of the above variables, especially the three variables that measure the language environment, could be either positive or negative. For instance, we hypothesized that in a school with few ELs, an increase in the share of ELs could lead to the creation of classes geared to the needs of the English Learners, which could benefit both ELs and their non- EL schoolmates. But at higher pre -existing levels of the share of ELs, a further increase in the share of ELs in a school could create situations where students could comfortably spend most of their school day speaking a language other than English at school, which could slow their acquisition of English fluency and academic progress, and perhaps have co nsequences for never ELs as well. One can easily imagine a positive relation between academic outcomes and the percent EL at low levels, turning to negative effect at higher levels. PPIC.ORG Technical Appendices Academic Progress for English Learners 24 To test this, we estimated models that replaced variables like percent E L with indicators for each tenth of the possible range (e.g. 0- 9.99% EL, 10-19.99% EL and so on). What we looked for in particular was whether an insignificant overall relation resulted from positive correlations in one range and negative correlations in t he other. We studied this possibility for the other language environment, including the percentage of all students speaking the same home language, and language homogeneity among ELs, as well as for the measures of incorrect ELD course placement. We found several interesting patterns, but did not discover any “hidden” relations that washed out in our simpler models where we looked for an overall relation. On the following pages we show the means and standard deviations of the four outcomes we model along with the explanatory variables. Means for each of three language groups are shown along with the means for the combined sample. The data cover 2006 -07 through 2015- 16. Readers will note that the mean CELDT scores of never EL students are reported. This ar ises because students who report speaking a home language other than English are given the CELDT to determine if they are fluent in English, and some of these students are determined not to be ELs. PPIC.ORG Technical Appendices Academic Progress for English Learners 25 TABLE D1 Means and Standard Deviations of Key Explanatory Variables and Dependent Variables for LAUSD Population: never EL everLAEL everLTEL5 All mean SD mean SD mean SD mean SD Variable name Percent EL * non-EL 18.58 11.87 n/a n/a n/a n/a 9.66 12.63 Percent EL * LAEL n/a n/a 30.72 14.50 n/a n/a 2.42 9.23 Percent EL * LTEL5 n/a n/a n/a n/a 26.91 11.70 10.79 15.13 Same Home Language among All Students * non-EL 45.93 23.63 n/a n/a n/a n/a 23.91 28.59 Same Home Language among All Students * LAEL n/a n/a 64.69 25.35 n/a n/a 5.10 18.83 Same Home Language among All Students * LTEL5 n/a n/a n/a n/a 67.85 22.22 27.19 36.10 Linguistic Homogeneity * non- EL 0.80 0.20 n/a n/a n/a n/a 0.42 0.42 Linguistic Homogeneity * LAEL n/a n/a 0.88 0.13 n/a n/a 0.07 0.24 Linguistic Homogeneity * LTEL5 n/a n/a n/a n/a 0.89 0.14 0.36 0.45 Any ALD Courses * non-EL 0.25 0.44 n/a n/a n/a n/a 0.13 0.34 Any ALD Courses* LAEL n/a n/a 0.23 0.42 n/a n/a 0.02 0.13 Any ALD Courses* LTEL5 n/a n/a n/a n/a 0.22 0.41 0.09 0.28 Any Newcomer Courses * non -EL 0.47 0.50 n/a n/a n/a n/a 0.25 0.43 Any Newcomer Courses * LAEL n/a n/a 0.52 0.50 n/a n/a 0.04 0.20 Any Newcomer Courses * LTEL5 n/a n/a n/a n/a 0.50 0.50 0.20 0.40 Share ELs Placed into Too Low ELD * non-EL 0.05 0.06 n/a n/a n/a n/a 0.03 0.05 Share ELs Placed into Too Low ELD * LAEL n/a n/a 0.04 0.05 n/a n/a n/a 0.02 Share ELs Placed into Too Low ELD * LTEL5 n/a n/a n/a n/a 0.05 0.06 0.02 0.05 Share ELs Placed into Too High ELD * non-EL 0.03 0.04 n/a n/a n/a n/a 0.02 0.03 Share ELs Placed into Too High ELD * LAEL n/a n/a 0.03 0.03 n/a n/a n/a 0.01 Share ELs Placed into Too High ELD * LTEL5 n/a n/a n/a n/a 0.03 0.03 0.01 0.03 Share ELs Placed into No ELD * non-EL 0.36 0.20 n/a n/a n/a n/a 0.19 0.23 Share ELs Placed into No ELD * LAEL n/a n/a 0.38 0.22 n/a n/a 0.03 0.12 Share ELs Placed into No ELD * LTEL5 n/a n/a n/a n/a 0.37 0.19 0.15 0.21 PPIC.ORG Technical Appendices Academic Progress for English Learners 26 Population: never EL everLAEL everLTEL5 All mean SD mean SD mean SD mean SD Share ELs Placed into Correct ELD * non-EL 0.46 0.17 n/a n/a n/a n/a 0.24 0.26 Share ELs Placed into Correct ELD * LAEL n/a n/a 0.49 0.19 n/a n/a 0.04 0.14 Share ELs Placed into Correct ELD * LTEL5 n/a n/a n/a n/a 0.48 0.16 0.19 0.26 LTEL5 status n/a n/a n/a n/a 1.00 n/a 0.40 0.49 Ever -LAEL status n/a n/a 1.00 n/a n/a n/a 0.08 0.27 Female 0.49 0.50 0.42 0.49 0.46 0.50 0.48 0.50 Special Education status 0.10 0.30 0.35 0.48 0.13 0.34 0.13 0.34 Spanish spoken at home 0.24 0.43 0.89 0.31 0.93 0.25 0.57 0.50 Parental education level: Less Than High School Diploma 0.10 0.30 0.21 0.41 0.30 0.46 0.19 0.39 High School Diploma 0.14 0.35 0.09 0.29 0.16 0.36 0.14 0.35 Some College 0.13 0.34 0.03 0.16 0.05 0.22 0.09 0.29 College Graduate 0.10 0.30 0.02 0.14 0.03 0.17 0.06 0.25 Graduate School 0.05 0.21 0.01 0.09 0.01 0.11 0.03 0.17 Parental Education Missing 0.48 0.50 0.64 0.48 0.45 0.50 0.48 0.50 Grade (mean only) 8.91 9.44 8.84 8.92 Total Enrollment (School Level) 2063 1017 2112 1232 2066 1091 2068 1066 Grade Configuration: 6-8 0.38 0.48 0.24 0.43 0.38 0.49 0.37 0.48 Grade Configuration: Other Than 6-8 or 9- 12 0.12 0.32 0.13 0.34 0.10 0.30 0.11 0.31 Dependent Variables CST-ELA z -score -0.03 1.00 -1.26 0.66 -0.74 0.74 -0.41 0.96 CST -Math z -score -0.10 1.01 -1.22 0.64 -0.70 0.71 -0.41 0.94 PPIC.ORG Technical Appendices Academic Progress for English Learners 27 Population: never EL everLAEL everLTEL5 All mean SD mean SD mean SD mean SD Grade Point Average (Annual) 2.35 0.96 2.20 0.99 1.98 0.93 2.19 0.97 CELDT Overall Score 607 100 447 109 567 47 534 88 TABLE D2 Means and Standard Deviations of Key Explanatory Variables and Dependent Variables for SDUSD Population: never EL everLAEL everLTEL5 All mean SD mean SD mean SD mean SD Variable name Percent EL * non-EL 12.70 10.70 0.11 1.70 n/a n/a 9.81 10.80 Percent EL * LAEL n/a n/a 29.01 15.82 n/a n/a 0.52 4.41 Percent EL * LTEL5 n/a n/a n/a n/a 23.56 15.05 4.95 11.82 Same Home Language among All Students * non-EL 49.74 21.27 0.13 2.70 n/a n/a 38.40 28.01 Same Home Language among All Students * LAEL n/a n/a 36.40 27.54 n/a n/a 0.66 6.10 Same Home Language among All Students * LTEL5 n/a n/a n/a n/a 40.64 23.78 8.53 19.82 Linguistic Homogeneity * non- EL 0.59 0.22 n/a 0.05 n/a n/a 0.46 0.32 Linguistic Homogeneity * LAEL n/a n/a 0.61 0.24 n/a n/a 0.01 0.09 Linguistic Homogeneity * LTEL5 n/a n/a n/a n/a 0.67 0.22 0.14 0.29 Any ALD Courses * non-EL 0.05 0.21 n/a n/a n/a n/a 0.04 0.19 Any ALD Courses* LAEL n/a n/a 0.04 0.20 n/a n/a n/a 0.03 Any ALD Courses* LTEL5 n/a n/a n/a n/a 0.05 0.22 0.01 0.10 Any Newcomer Courses * non -EL 0.36 0.19 n/a 0.02 n/a n/a 0.03 0.17 Any Newcomer Courses * LAEL n/a n/a 0.06 0.24 n/a n/a n/a 0.03 Any Newcomer Courses * LTEL5 n/a n/a n/a n/a 0.05 0.23 0.01 0.11 Share ELs Placed into Too Low ELD * non-EL 0.05 0.09 n/a 0.01 n/a n/a 0.04 0.08 Share ELs Placed into Too Low ELD * LAEL n/a n/a 0.07 0.07 n/a n/a n/a 0.01 Share ELs Placed into Too Low ELD * LTEL5 n/a n/a n/a n/a 0.06 0.10 0.01 0.05 PPIC.ORG Technical Appendices Academic Progress for English Learners 28 Population: never EL everLAEL everLTEL5 All mean SD mean SD mean SD mean SD Share ELs Placed into Too High ELD * non-EL 0.15 0.14 n/a 0.01 n/a n/a 0.12 0.14 Share ELs Placed into Too High ELD * LAEL n/a n/a 0.20 0.13 n/a n/a n/a 0.03 Share ELs Placed into Too High ELD * LTEL5 n/a n/a n/a n/a 0.14 0.13 0.03 0.08 Share ELs Placed into No ELD * non-EL 0.31 0.27 n/a 0.03 n/a n/a 0.24 0.27 Share ELs Placed into No ELD * LAEL n/a n/a 0.22 0.22 n/a n/a n/a 0.04 Share ELs Placed into No ELD * LTEL5 n/a n/a n/a n/a 0.30 0.27 0.06 0.17 Share ELs Placed into Correct ELD * non-EL 0.49 0.26 n/a 0.04 n/a n/a 0.38 0.31 Share ELs Placed into Correct ELD * LAEL n/a n/a 0.51 0.23 n/a n/a 0.01 0.08 Share ELs Placed into Correct ELD * LTEL5 n/a n/a n/a n/a 0.50 0.25 0.11 0.23 LTEL5 status n/a n/a n/a n/a 1.00 n/a 0.21 0.41 Ever -LAEL status n/a n/a 1.00 n/a n/a n/a 0.02 0.13 Female 0.49 0.50 0.48 0.50 0.43 0.50 0.48 0.50 Special Education status 0.13 0.34 0.03 0.16 0.27 0.44 0.16 0.37 Spanish spoken at home 0.10 0.30 0.64 0.48 0.84 0.37 0.26 0.44 Parental education level: Less Than High School Diploma 0.03 0.17 0.21 0.41 0.24 0.43 0.08 0.27 High School Diploma 0.11 0.31 0.12 0.32 0.18 0.39 0.13 0.33 Some College 0.19 0.39 0.04 0.19 0.07 0.25 0.16 0.37 College Graduate 0.19 0.39 0.05 0.22 0.04 0.20 0.15 0.36 Graduate School 0.13 0.34 0.03 0.17 0.02 0.14 0.11 0.31 Parental Education Missing 0.35 0.48 0.55 0.50 0.45 0.50 0.38 0.49 Grade (mean only) 9.09 9.76 8.93 9.07 Total Enrollment (School Level) 1386 737 1266 798 1150 710 1334 739 PPIC.ORG Technical Appendices Academic Progress for English Learners 29 Population: never EL everLAEL everLTEL5 All mean SD mean SD mean SD mean SD Grade Configuration: 6-8 0.31 0.46 0.19 0.39 0.33 0.47 0.31 0.46 Grade Configuration: Other Than 6-8 or 9- 12 0.14 0.34 0.03 0.16 0.10 0.30 0.13 0.33 Dependent Variables CST-ELA z -score 0.33 0.99 -1.07 0.72 -0.83 0.74 0.06 1.06 CST -Math z -score 0.19 0.97 -0.63 0.95 -0.65 0.71 -0.02 0.98 Grade Point Average (Annual) 2.84 0.92 2.46 1.01 2.09 0.97 2.68 0.99 CELDT Overall Score 643 54 455 104 555 57 548 72 PPIC.ORG Technical Appendices Academic Progress for English Learners 30 We constructed measures of correct course placement for both school districts over time. Those measures are used in regression analyses (Tables D 4A and D 4B). Here, we report the regression coefficients of our estimates of what predicts school level correct course placement. Our regressions are estimated at the student level, which implicitly weighs each school by the number of students involved in constructing each of the measures. TABLE D3 Correct School Placement Regression Results, LAUSD and SDUSD Variables LAUSD SDUSD LAUSD SDUSD Percent EL -0.000471 -0.00332 (0.000644) (0.00169) Number EL per grade 0.000354** -8.53e -05 (4.83e -05) (0.000321) Same Home Language among All Students 3.20e-05 -0.000971* -0.000292* -0.000692 (0.000154) (0.000412) (0.000133) (0.000426) Linguistic Homogeneity 0.0744 -0.0224 -0.00965 -0.0795 (0.0539) (0.113) (0.0473) (0.107) Constant 0.415** 0.602** 0.448** 0.579** (0.0434) (0.0651) (0.0401) (0.0706) Observations 2,374,793 474,201 2,374,790 474,201 R-squared 0.005 0.032 0.052 0.010 r2_a 0.00474 0.0321 0.0523 0.0101 rss 66108 30436 62948 31126 df_m 3 3 3 3 LogL 882829 -21782 940982 -27095 Robust standard errors in parentheses ** p<0.01, * p<0.05 PPIC.ORG Technical Appendices Academic Progress for English Learners 31 Table D 4A shows the main regression results for four annual outcomes. Statistically significant results are in bold. Cells in light gra y were not estimated because the relevant language group (never ELs) was not included in the CELDT models. Cells in darker gray represent coefficients that were statistically significant in the models shown below that include all the estimators shown, but in simpler models that included only the giv en set of explanatory variables, such as the three language homogeneity variables, the coefficient was not significant. This could indicate that the significant results in the gray cells below are not robust and that significance could reflect collinearity with the other regressors. TABLE D4A Annual Outcome Regression Results Regressor Variable CST ELA Z -Score CELDT Score CST Math Z -Score Annual GPA LAUSD SDUSD LAUSD SDUSD LAUSD SDUSD LAUSD SDUSD % EL * Never EL 0.168* -0.000426 0.235 -0.00201 -0.0167 -0.00240 (0.0738) (0.00161) (0.194) (0.00574) (0.0441) (0.00205) % EL * LAEL 0.227** -0.00160 13.01 -0.548** 0.354 -0.00374 0.0329 -0.000102 (0.0786) (0.00174) (10.22) (0.153) (0.220) (0.00622) (0.0399) (0.0 0245) % EL * LTEL 0.279** 0.00113 -13.21 -0.467** 0.469* 0.000661 -0.0255 -0.00157 (0.0725) (0.00165) (8.950) (0.115) (0.184) (0.00547) (0.0423) (0.00218) % Same Home Lan - 0.000423** 0.00103** 0.000814** 0.000643* 0.000335** 0.00130** guage * Never EL (0.000115) (0.000218) (0.000176) (0.000308) (7.25e -05) (0.000240) % Same Home Lan - 0.000240 0.00325** 0.0319 0.153** -0.000906 0.00676* 0.000134 -0.00120* guage * LAEL (0.000236) (0.000440) (0.0290) (0.0514) (0.000569) (0.00268) (0.000109) (0.000507) % Same Home Lan - -9.22e -05 0.000720** 0.0669* -0.0154 -2.07e -05 0.000555 0.000118 -0.000143 guage * LTEL (0.000148) (0.000255) (0.0263) (0.0337) (0.000308) (0.000484) (7.50e -05) (0.000348) Language Homo - -0.0390 -0.0937* -0.175** 0.0127 0.00794 0.0183 geneity * Never EL (0.0239) (0.0463) (0.0660) (0.120) (0.0180) (0.0587) Language Homo - 0.0926* -0.123 -3.088 -22.34* 0.107 -0.386 0.0412 -0.0105 geneity * LAEL (0.0416) (0.0787) (6.187) (10.25) (0.0836) (0.282) (0.0278) (0.0845) Language Homo - 0.104** -0.132* 4.349 -4.769 -0.000602 -0.0398 0.0689** 0.00146 geneity * LTEL (0.0264) (0.0537) (3.784) (7.074) (0.0669) (0.121) (0.0221) (0.0575) LTEL5 Currently or in -0.209** -0.163** 25.18** -20.78** -0.274** -0.252** -0.0980** -0.0528* Past (0.0246) (0.0212) (4.232) (5.017) (0.0506) (0.0837) (0.0124) (0.0254) Ever LAEL -0.277** -0.141* -0.212* -0.570 -0.0787** -0.0365 (0.0378) (0.0641) (0.0936) (0.297) (0.0187) (0.0390) PPIC.ORG Technical Appendices Academic Progress for English Learners 32 Re gressor Variable CST ELA Z -Score CELDT Score CST Math Z -Score Annual GPA LAUSD SDUSD LAUSD SDUSD LAUSD SDUSD LAUSD SDUSD Share ELD Too Low * 0.0102 -0.200* 0.0589 -0.00245 0.0681 -0.0224 Never EL (0.0644) (0.0762) (0.141) (0.170) (0.0381) (0.111) Share ELD Too Low * -0.213* -0.384 -35.90* -16.01 -0.330 4.187** 0.0242 -0.110 LAEL (0.0979) (0.248) (17.40) (24.31) (0.218) (1.102) (0.0518) (0.248) Share ELD Too Low * -0.112 -0.317** 26.31** -6.532 0.00670 0.199 0.0716 -0.113 LTEL (0.0680) (0.0948) (7.385) (7.910) (0.0889) (0.234) (0.0400) (0.164) Share ELD Too High -0.278 0.0208 -0.0682 -0.245** 0.0458 0.136* * Never EL (0.187) (0.0403) (0.236) (0.0906) (0.0491) (0.0536) Share ELD Too High 0.373 0.0214 -22.64 -38.81** 1.280* 0.696 0.179** -0.180 * LAEL (0.226) (0.104) (16.42) (13.53) (0.636) (0.527) (0.0687) (0.108) Share ELD Too High -0.184 -0.0375 23.47** 4.280 0.278 0.01 53 0.123 -0.0540 * LTEL (0.174) (0.0490) (9.015) (6.288) (0.288) (0.110) (0.0797) (0.0796) Share With No ELD -0.0268 -0.0493* 0.0199 0.0156 0.0134 0.0272 * Never EL (0.0277) (0.0191) (0.0573) (0.0936) (0.0139) (0.0318) Share With No ELD -0.0578 -0.132* 3.722 21.16** -0.218* 0.678 0.0304* 0.176** * LAEL (0.0466) (0.0556) (3.637) (6.966) (0.0983) (0.357) (0.0147) (0.0595) Share With No ELD -0.0977** -0.0617* -1.091 4.175 -0.0356 0.0312 0.0445** 0.0381 * LTEL (0.0290) (0.0245) (3.108) (2.703) (0.0573) (0.107) (0.0170 ) (0.0458) Constant -0.391** 1.468** 223.7** 197.3** -0.154 1.863** 0.140* 0.444** (0.125) (0.0873) (28.85) (15.50) (0.182) (0.267) (0.0579) (0.106) Observations 1,243,696 233,612 490,417 57,368 406,686 67,891 930,845 303,388 R-squared 0.713 0.727 0.722 0.632 0.690 0.673 0.881 0.670 Adjusted R -squared 0.713 0.726 0.721 0.630 0.690 0.672 0.881 0.669 Sum of Squared Residuals 330696 71248 8.840e+08 7.240e+07 119837 21604 87922 96424 df_m 125 91 109 73 115 54 130 87 LogL -941001 -192775 -2.534e+06 -286204 -328596 -57465 -222579 -256608 Robust standard errors in parentheses ** p<0.01, * p<0.05 NOTE S: Cells in gr ay indicate coefficients that are significant only in the model with all the language variables, but was not significant in models that included only that specific language variable . Coefficients in bold are significant at less than the 0.05 level . PPIC.ORG Technical Appendices Academic Progress for English Learners 33 TABLE D4B Long Term Outcome Regression Results Graduate on Time Ever Reclassified LAUSD SDUSD LAUSD SDUSD VARIABLES % EL * Never EL -0.206 -0.00134 (0.173) (0.000874) % EL * LAEL 0.0147 -0.00283* 0.363** -0.00453** (0.183) (0.00122) (0.0999) (0.00170) % EL * LTEL -0.275 -0.00131 0.447** -0.00581** (0.190) (0.000819) (0.107) (0.00167) % Same Home Lan - 0.000452** 0.000642** guage * Never EL (8.34e -05) (0.000203) % Same Home Lan - -0.000636** -0.000104 -0.000391 0.000817 guage * LAEL (0.000235) (0.000618) (0.000308) (0.000471) % Same Home Lan - 0.000575** -0.000256 0.000124 0.000417 guage * LTEL (0.000153) (0.000310) (0.000248) (0.000420) Language Homo - -0.112* 0.00621 geneity * Never EL (0.0492) (0.0274) Language Homo - -0.216** -0.0313 0.0485 -0.142 geneity * LAEL (0.0729) (0.0691) (0.0478) (0.0947) Language Homo - -0.151** -0.0728 0.0276 -0.167 geneity * LTEL (0.0559) (0.0384) (0.0345) (0.0849) LTEL5 Currently or in 0.105** 0.0954** 0.231** 0.0839 Past (0.0223) (0.0235) (0.0272) (0.0426) Ever LAEL -0.0575 0.0696 (0.0411) (0.0511) Share ELD Too Low * -0.113 0.102** Never EL (0.0752) (0.0322) Share ELD Too Low * -0.0914 -0.135 0.704** -0.397* LAEL (0.154) (0.245) (0.0812) (0.194) Share ELD Too Low * -0.135 -0.0632 0.657** -0.350** LTEL (0.0938) (0.0784) (0.0718) (0.104) Share ELD Too High 0.262** 0.0819** *Never EL (0.0843) (0.0235) Share ELD Too High -0.0168 -0.101 -0.167 0.180* *LAEL (0.243) (0.0927) (0.0998) (0.0881) Share ELD Too High -0.104 -0.231** -0.297** -0.0105 * L TEL (0.114) (0.0421) (0.0738) (0.0510) Share With No ELD 0.0121 0.0436** *Never EL (0.0277) (0.0128) Share With No ELD 0.124* 0.174* 0.0471 -0.248** *LAEL (0.0535) (0.0700) (0.0273) (0.0600) Share With No ELD -0.147** 0.0423* -0.0208 -0.275** * L TEL (0.0368) (0.0193) (0.0251) (0.0337) Constant 0.885** 0.613** -0.201 0.727** (0.0763) (0.0517) (0.217) (0.169) PPIC.ORG Technical Appendices Academic Progress for English Learners 34 Graduate on Time Ever Reclassified LAUSD SDUSD LAUSD SDUSD VARIABLES Observations 212,098 220,577 956,480 89,040 R-squared 0.200 0.250 0.072 0.329 Adjusted R -squared 0.199 0.248 0.0701 0.326 Sum of Squared Residuals 21514 24163 208655 14932 df_m 123 89 124 85 LogL -58278 -69092 -629029 -46849 NOTES: ** p<0.01, * p<0.05 Coefficients in bold are significant at less than the 0.05 level . Robust standard errors in parentheses . PPIC.ORG Technical Appendices Academic Progress for English Learners 35 Table D5 shows regression results that control for newcomer supports and LTEL course availability. TABLE D5 Supplementary Results that Add LTEL and Newcomer Course Availability to the Earlier Models Regressor Variable CST ELA Z -Score CELDT Score CST Math Z -Score Annual GPA LAUSD SDUSD LAUSD SDUSD LAUSD SDUSD LAUSD SDUSD Any Newcomer Program*Never EL 0.000518 -0.00351 -0.00687 0.149** 0.00626 -0.0224 (0.00522) (0.0164) (0.0119) (0.0266) (0.00334) (0.0225) Any Newcomer Program*LAEL -0.0107 -0.0171 0.712 -8.425 -0.0266 0.166 0.0129* -0.0140 (0.00913) (0.0345) (1.134) (6.260) (0.0211) (0.174) (0.00589) (0.0515) Any Newcomer Program*LTEL -0.0103 -0.00309 -0.0112 3.797* -0.0332** 0.126** 0.00509 -0.0220 (0.00603) (0.0217) (0.653) (1.486) (0.00952) (0.0240) (0.00466) (0.0440) Any LTEL Course*Never EL 0.000557 (0.0250) Any LTEL Course*LAEL 7.571 -0.0721 (4.804) (0.0685) Any LTEL Course*LTEL 1.599 0.00618 (2.248) (0.0358) Observations 1,286,105 233,612 524,182 57,368 416,710 67,891 997,786 303,388 R-squared 0.711 0.727 0.725 0.633 0.681 0.673 0.887 0.670 r2_a 0.711 0.726 0.724 0.630 0.681 0.672 0.887 0.669 rss 343338 71248 9.670e+08 7.230e+07 116133 21590 93085 96418 df_m 151 91 130 77 135 54 151 87 LogL -975651 -192775 -2.715e+06 -286182 -325079 -57443 -232408 -256599 Robust standard errors in parentheses. ** p<0.01, * p<0.05 Coefficients in bold are significant at less th an the 0.05 level NOTE : Cells in gr ay indicate coefficients that could not be estimated, or coefficients that in the case of LAUSD were estimable but only on sma ll atypical samples, and are thus suppressed. PPIC.ORG Technical Appendices Academic Progress for English Learners 36 TABLE D6 Supplementary Results for SDUSD that Add the Ratio of ELs to EL Support Teachers to the Earlier Models in Table D 4A Regressor Variable CST ELA Z- Score CELDT Score CST Math Z- Score Annual GPA EL to ELST Ratio *Never EL -0.413 2.159* 0.223 (0.375) (0.938) (0.791) EL to ELST Ratio *LAEL -6.062 706.3 20.68 14.78* (5.150) (457.9) (19.54) (6.464) EL to ELST Ratio *LTEL 1.008 107.7 2.468 5.375* (0.985) (108.2) (1.476) (2.076) Observations 232,897 47,276 67,710 238,055 R-squared 0.727 0.628 0.672 0.674 r2_a 0.726 0.625 0.671 0.674 rss 70893 5.740e+07 21501 73953 df_m 82 69 49 75 LogL -191959 -234948 -57241 -198634 Robust standard errors in parentheses. ** p<0.01, * p<0.05 Coefficients in bold are significant at less than the 0.05 level NOTE : Cells in gr ay indicate coefficie nts that could not be estimated, because never EL students were not given the CELDT test . PPIC.ORG Technical Appendices Academic Progress for English Learners 37 Statistical Precision In evaluating the coefficients on the variables related to the language environment, the precision of the estimates matters. We can obtain the 95% confidence interval for any coefficient by multiplying the standard error (s.e.) by 1.96, so that the 95% confidence interval will be [estimate -1.96*s.e., estimate + 1.96*s.e.]. This analysis seems most important for our findings of “no effect.” The most striking “no effect” results are that for never EL students, the percentage of students who are EL is not nega tively related to any of the academic outcomes we model. The standard errors for this variable, shown in the first row of Table D 4A, are small. The minimum effect of raising the percent EL by 1 on the CST ELA growth for LAUSD is 0.168 - 1.96*0.0738=0.023. The minimum possible effects within the 95% confidence interval for San Diego is -0.004. Similar conclusions result when looking at the other outcomes for percentage EL for the other two linguistic groups, and for the variable measuring the percentage of all students speaking the same home language. A s noted in the main text, the precision of our estimates for the EL language homogeneity variable is lower. This could reflect the relatively smaller variation in this measure, especially in LAUSD. This inde x can theoretically range from 0 to 1, and Table 1 shows that the mean in LAUSD, depending on the linguistic group, ranges from 0.8 for never ELs to 0.89 for LTELs. That said, even with this lack of precision, when we claim that language homogeneity has a significant negative or positive relation to an outcome, we are 95% certain that the sign is correct. But we are unsure of the true size of the relation. Perhaps of greater concern is the many cases where language homogeneity is not significant, for example for math CST for LTELs and GPA models for Late Arriving ELs. To give one example, in the model of GPA for LAUSD, the coefficient and standard error for language homogeneity for LAELs are 0.0412 and 0.0278, meaning that the upper bound of the confidence interval is 0.095, and the lower bound is a small negative predicted effect. A nother factor to consider is whether any of our outcomes displays bunching of observations near a ceiling or floor. For example, in theory our result that LAELs’ CELDT scores rise more quickly than those of LTELs could arise if LTELs are already scoring near the maximum possible CELDT score. Conversely, if we find no significant effect of a given variable on the CST scores of LAELs, could it be because almost all LAELs have CST scores near the minimum possible. To test whether there is massive clumping of outcome values near a ceiling or floor, we can look at the distribution of proficiency levels on the CST for all groups and for ELs, for the CELDT as well. B ecause it is hard to show floor or ceiling effects for the CST using Z scores, we instead show the distribution of test scores across language groups using proficiency levels. Similarly for the CELDT we start by showing the distribution of students by performance levels. For annual GPA we divide students into four groups, based on whether GPA is less than 1, at least equal to 1 and less than 2, at least equal to 2 and less than 3, and at least equal to 3 and less than or equal to 4. PPIC.ORG Technical Appendices Academic Progress for English Learners 38 TABLE D7 The Percentage of Students by CST ELA Proficiency Level District Language Group Far Below Basic Below Basic Basic Proficient Advanced LAUSD Never EL 9.6 14.2 27.8 27.1 21.3 LAEL 49.2 32.3 14.7 2.9 0.5 LTEL 18.3 28.7 37.0 13.4 2.7 SDUSD Never EL 4.7 8.4 22.1 31.0 33.74 LAEL 38.6 32.7 21.3 6.1 1.4 LTEL 18.5 30.1 37.7 11.6 2.2 TABLE D8 The P ercentage of Students by CST Math Proficiency Level , Grades 6 and 7 Only District Language Group Far Below Basic Below Basic Basic Proficient Advanced LAUSD Never EL 8.1 20.5 26.0 27.1 18.4 LAEL 44.2 40.8 10.6 3.6 0.8 LTEL 15.5 38.4 30.5 13.1 2.4 SDUSD Never EL 3.6 12.2 23.1 34.8 26.3 LAEL 21.8 34.2 23.6 12.2 8.3 LTEL 12.2 34.1 34.3 16.5 3.0 TABLE D9 The P ercentage of Students by CELDT Performance Level District Language Group Beginning Earl y Intermediate Intermediate Earl y Advanced Advanced LAUSD LAEL 49.7 23.0 19.2 7.3 0.9 LTEL 0.0 10.2 43.7 38.2 8.0 SDUSD LAEL 49.8 21.1 19.4 8.6 1.1 LTEL 4.5 12.5 42.1 34.9 6.1 TABLE D10 The Percentage of Students by Annual GPA Level District Language Group 0-0.999 1-1.999 2-2.999 3-4 LAUSD Never EL 8.6 24.3 36.5 30.6 LAEL 12.5 26.7 34.5 26.4 LTEL 14.5 34.0 34.8 16.7 SDUSD Never EL 4.5 12.7 30.2 52.6 LAEL 9.2 20.7 33.6 36.4 LTEL 6.6 16.2 31.6 45.6 Our concern is that if a large percentage of students is in the top or bottom group, some of them could be achieving so high or so low that a change in their underlying performance would not lead to a change in their recorded achievement measure. For all but one case (CELDT) for LAELs, a majority of students in each language group are in the middle ranges of these measures. Thus for CST scores and GPA we find little evidence of a major problem of bunching. PPIC.ORG Technical Appendices Academic Progress for English Learners 39 The exception is the CELDT performance levels for LAELs. For instance in San Diego, 49.8% of LAELs are in the bottom (Beginning) group. We took a closer look at the underlying scale scores for the CELDT to look for bunching of students at the bottom possible score. The minimum scaled score is 248 in grades 6-8 and 251 in grades 9 -12, and the respective maximum allowed scores are 741 and 761. Very few LTELs have scores near the minimum or maximum. Figures D1, D2 show the histograms for LAEL and LTEL students in LAUSD, while Figures D3 and D4 show the corresponding information for SDUSD. For LAELs about 6 percent in SDUSD and 4 percent in LAUSD have CELDT scores within 10 points of the minimum of 248, as shown by the bottom bar in the histogram, so it is possible for a very small number of these students that the CELDT test would not detect actual gains in English proficiency if the student’s initial proficiency was below the bottom limit. Theoretically this could be a concern because for these students small increases in English proficiency might not be detectable on the test. But the proportion of students for whom this could be an issue is very small. FIGURE D1 Histogram of CELDT Total Scaled Scores for LAEL Students in LAUSD 0 2 4 6 8 10 Per ce nt 2 00 400 60 0 8 00 to _s cl PPIC.ORG Technical Appendices Academic Progress for English Learners 40 FIGURE D2 Histogram of CELDT Total Scaled Scores for LTEL Students in LAUSD FI GURE D3 Histogram of CELDT Total Scaled Scores for LAEL Students in SDUSD 0 2 4 6 8 Per ce nt 2 00 400 60 0 8 00 to _s cl 0 2 4 6 Percent 200 300 400 500 600 700 CELDT to scaled PPIC.ORG Technical Appendices Academic Progress for English Learners 41 FIGURE D4 Histogram of CELDT Total Scaled Scores for LTEL Students in SDUSD For the other outcomes, we find even less evidence of ceiling or floor effects. 0 2 4 6 8 10 Percent 200 400 600 800 CELDT to scaled The Public Policy Institute of California is dedicated to informing and improving public policy in California through independent, objective, nonpartisan research. Public Policy Institute of California 500 Washington Street, Suite 600 San Francisco, CA 94111 T: 415.291.4400 F: 415.291.4401 PPIC.ORG P PIC Sacramento Center Senator Office Building 1121 L Street, Suite 801 Sacramento, CA 95814 T: 916.440.1120 F: 916.440.1121" ["post_date_gmt"]=> string(19) "2019-01-23 18:07:10" ["comment_status"]=> string(6) "closed" ["ping_status"]=> string(6) "closed" ["post_password"]=> string(0) "" ["post_name"]=> string(16) "0119lhr-appendix" ["to_ping"]=> string(0) "" ["pinged"]=> string(0) "" ["post_modified"]=> string(19) "2019-01-23 10:07:47" ["post_modified_gmt"]=> string(19) "2019-01-23 18:07:47" ["post_content_filtered"]=> string(0) "" ["guid"]=> string(60) "https://www.ppic.org/wp-content/uploads/0119lhr-appendix.pdf" ["menu_order"]=> int(0) ["post_mime_type"]=> string(15) "application/pdf" ["comment_count"]=> string(1) "0" ["filter"]=> string(3) "raw" ["status"]=> string(7) "inherit" ["attachment_authors"]=> bool(false) }