Social Determinants of Health: Out-of-School-Time Academic Programs – Minimal Academic Content

The CPSTF findings for out-of-school-time academic programs (reading-focused, math-focused, general, minimal academic content) are based on evidence from a meta-analysis published in 2006 (Lauer et al., 35 studies, search period 1985 2003) combined with more recent evidence (25 studies, search period 2003-2011). Of the 35 studies included in the Lauer et al. meta-analysis, this review excluded three studies which reported only school grades, a relatively subjective measure of reading and math achievement. The CPSTF used evidence from an independent systematic review of summer school programs to confirm Community Guide review findings (Cooper et al., 2000).

The effectiveness evidence is based on a systematic review of all available studies, conducted on behalf of the CPSTF by a team of specialists in systematic review methods, and in research, practice, and policy related to the use of educational interventions for the promotion of health equity.

Overall, children from low-income and racial and ethnic minority populations in the United States have lower academic achievement levels than children from the higher-income and majority populations. As a long-term consequence, these children often grow to be adults with lower income levels and poorer health, perpetuating a “cycle of poverty” (Duncan et al., 1998). Out-of-school-time academic programs aim to interrupt this cycle by assisting children who are at risk for low academic achievement.

Detailed results from the systematic review are available in the CPSTF Finding and Rationale Statement [PDF – 315 kB].

• Out-of-school-time academic programs led to modest improvements in academic achievement, as measured by standardized achievement tests. The degree of effectiveness was largely dependent on program focus.
  • Minimal academic programs:
    • Reading achievement: 0.07 standard deviations (SD) (1 study)
    • Math achievement: 0.04 SD (1 study)

An economic review of this intervention was not conducted because the CPSTF did not have enough information to determine if the intervention works.

Applicability of this intervention across different settings and populations was not assessed because the CPSTF did not have enough information to determine if the intervention works.

The CPSTF identified several areas that have limited information. Additional research and evaluation could help answer the following questions and fill remaining gaps in the evidence base. (What are evidence gaps?)

• Which features of out-of-school-time academic programs contribute to their effectiveness?
• How does intervention effectiveness vary by the following?
  • Student characteristics (e.g., race or ethnicity, socioeconomic status)
  • Program focus
  • Degree of student participation in combination with program duration (short programs with full participation or long programs with minimal participation are likely to be less effective)
• What activities are used in the intervention and control groups?
• What are the long-term effects of out-of-school-time academic programs?
• How can implementation of, and participation in, out-of-school-time academic programs be improved?
• Transportation may be a challenge for children in low-income families. What are the best ways to address this barrier?
• With its increased academic focus, how effective are the 21st Century Community Learning Centers programs (James-Burdumy et al., 2007)?
• What are the actual costs of out-of-school-time academic programs?
• What are the cost variations of different program types used in different locations?
• What is the cost-benefit of out-of-school-time academic programs?

The following characteristics describe studies included in systematic reviews of out-of-school-time academic programs (reading-focused, math-focused, general, minimal academic content).

• All included studies were conducted in the United States.
• Of the evaluated programs, 37 (65%) were implemented in urban areas.
• Programs were implemented for elementary school students (28 programs; 49%), both elementary and middle school students (8 programs; 14%), middle school students (7 programs; 12%), middle and high school students (3; 5%), high school students (7 programs; 12%), or all students (4 programs; 7%).
• The studies evaluated similar numbers of summer (29) and after-school (28) programs.
• Study populations were predominantly racial/ethnic minorities (i.e., Black and Hispanic). Specifically, studies included majority Black students (25 studies; 43%), majority Hispanic (4 studies; 7%), majority non-White (unspecified, 2 studies; (4%), majority White (7 studies; 12%), mixed (4 studies; 7%), or did not report race/ethnicity (15 studies; 26%).
• Most studies evaluated majority low SES populations (42 studies; 74%).
• Studies included similar numbers of male and female students.
• Programs employed tutoring or individualized instruction as the instructional method (17 programs; 30%), group instruction (24 programs; 42%), both tutoring and group instruction (10 programs; 18%), or did not report didactic method (6 programs; 11%).
• Programs were reading-focused (23 programs; 40%), math-focused (7 programs; 12%), general (23 programs; 40%), or had minimal academic content (4 programs; 7%).

Knopf JA, Hahn RA, Proia KK, et al. Out-of-School-Time Academic Programs to Improve School Achievement: A Community Guide Health Equity Systematic Review. Journal of Public Health Management and Practice 2015;21(6):594–608.

Community Preventive Services Task Force. Out-of-School-Time Academic Programs Are Recommended to Improve Academic Achievement and Health Equity. Journal of Public Health Management and Practice 2015;21(6):209–12.

Hahn RA, Chattopadhyay SK. Linking studies to assess the life expectancy associated with eighth grade school achievement. Preventive Medicine Reports 2019;16(100980). doi:https://doi.org/10.1016/j.pmedr.2019.100980.

Effectiveness Review

Analytic Framework

When starting an effectiveness review, the systematic review team develops an analytic framework. The analytic framework illustrates how the intervention approach is thought to affect public health. It guides the search for evidence and may be used to summarize the evidence collected. The analytic framework often includes intermediate outcomes, potential effect modifiers, potential harms, and potential additional benefits.

Effectiveness Review

Summary Evidence Table
Contains evidence from reviews of these out-of-school-time academic programs: reading-focused, math-focused, general, minimal academic content

The Community Preventive Services Task Force findings for out-of-school-time academic programs (reading-focused, math-focused, general, minimal academic content) are based on evidence from a published meta-analysis and an updated search for newer evidence.

• 32 studies published between 1985-2003 (identified from: Lauer P, Motoko A, Wilkerson S. Out-of-school-time programs: a meta-analysis of effects for at-risk students. Review of Educational Research 2006;76:275-313.)
  • While the meta-analysis included 35 studies, three were excluded from the Community Guide systematic review because they reported only school grades, a relatively subjective measure of reading and math achievement (Cosden et al., 2001; Legro, 1990; Smeallie, 1997).
• 25 studies (from 24 publications) published between 2003-2011 (updated search)

The Task Force also reviewed evidence from an independent systematic review of summer school programs to confirm Community Guide review findings (Cooper et al., 2000).

The number of studies and publications do not always correspond (e.g., a publication may include several studies or one study may be explained in several publications).

Effectiveness Review

Cooper H, Charlton K, Valentine JC, Muhlenbruck L. Making the most of summer school: a meta-analytic and narrative review. Monographs of the Society for Research in Child Development 2000;65(1, Serial No. 260).

Cosden M, Morrison G, Albanese AL, Macias S. When homework is not home work: after-school programs for homework assistance. Educational Psychologist 2001;36(3):211 21.

Lauer P, Motoko A, Wilkerson S. Out-of-school-time programs: a meta-analysis of effects for at-risk students. Review of Educational Research 2006;76:275-313.

Legro DL. An evaluation of an after-school partnership program: the effects on young children’s performance (Doctoral dissertation, University of Houston). Dissertation Abstracts International 1990;52:02A.

Smeallie JE. An evaluation of an after-school tutorial and study skills program for middle school students at risk of academic failure (Doctoral dissertation, University of Maryland, College Park). Dissertation Abstracts International 1997;58:06A.

Studies from the Updated Search

*Black A, Somers M, Doolittle F, Unterman R, Grossman J. The evaluation of enhanced academic instruction in after-school programs: final report. Washington (DC): National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education; 2009.

Borman G, Dowling M. Longitudinal achievement effects of multiyear summer school: evidence from the Teach Baltimore randomized field trial. Educational Evaluation and Policy Analysis 2006;28(1):25-48.

Borman G, Goetz M, Dowling M. Halting the summer achievement slide: a randomized field trial of the KindergARTen summer camp. Journal of Education for Students Placed at Risk 2009;14(2):133-47.

Boulden W. Evaluation of the Kansas City LULAC National Education Service Center’s Young Reader’s Program. Children & Schools 2006;28(2):107-14.

Burgin J, Hughes G. Measuring the effectiveness of a summer literacy program for elementary students using writing samples. Research in the Schools 2008;15(2):55-64.

Cross A, Gottfredson D, Wilson D, Rorie M, Connell N. The impact of after-school programs on the routine activities of middle-school students: results from a randomized, controlled trial. Criminology & Public Policy 2009;8(2):391-412.

Denton C, Solari E, Ciancio D, Hecht S, Swank P. A pilot study of a kindergarten summer school reading program in high-poverty urban schools. Elementary School Journal 2010;110(4):423-39.

Dynarski M, James-Burdumy S, Moore M, Rosenberg L, Deke J, Mansfield W. When schools stay open late: the national evaluation of the 21st Century Community learning Centers program: new findings. Washington (DC): U.S. Department of Education, National Center for Education Evaluation and Regional Assistance; 2004.

Edmonds E, O’Donoghue C, Spano S, Algozzine R. Learning when school is out. Journal of Educational Research 2009;102(3):213-21.

Gottfredson D, Gerstenblith S, Soule D, Womer S, Lu S. Do after school programs reduce delinquency? Prevention Science 2004;5(4):253-66.

**Hanlon T, Simons B, O’Grady K, Carswell S, Callaman J. The effectiveness of an after-school program Targeting urban african american youth. Education and Urban Society 2009;1(42):96-118.

Heinrich C, Meyer R, Whitten G. Supplemental Education Services under No Child Left Behind: Who signs up, and what do they gain? Educational Evaluation and Policy Analysis 2010;32(2):273-98.

Huang D, Kim K, Cho J, Marshall A, Perez P. Keeping kids in school: a study examining the long-term impact of afterschool enrichment programs on students’ high school dropout rates. Journal of Contemporary Issues in Education 2011;6(1):4-23.

James-Burdumy S, Dynarski M, Deke J. When elementary schools stay open late: results from the national evaluation of the 21st Century Community Learning Centers Program. Educational Evaluation and Policy Analysis 2007;29(4):296 318.

Jenner E, Jenner L. Results from a first-year evaluation of academic impacts of an after-school program for at-risk students. Journal of Education for Students Placed at Risk 2007;12(2):213-37.

Kim J, Samson J, Fitzgerald R, Hartry A. A randomized experiment of a mixed-methods literacy intevention for struggling readers in grades 4-6: effects on word reading efficiency, reading comprehension and vocabulary, and oral reading fluency. Reading and Writing: An Interdisciplinary Journal 2010;23(9):1109-29.

Munoz M, Potter A, Ross S. Supplemental Educational Services as a consequence of the NCLB legislation: evaluating its impact on student achievement in a large urban district. Journal of Education for Students Placed at Risk 2008;13(1):1-25.

Myers D, Olsen R, Seftor N, Young J, Tuttle C. The impact of regular Upward Bound: results from the third follow-up data collection. Washington (DC): Mathematica Policy Research, U.S. Department of Education, Office of Under Secretary, Policy and Program Studies Service; 2007.

Olsen R, Seftor N, Silva T, Myers D, DesRoches D, Young J. Upward Bound Math-Science: program description and interim impact estimates. Washington (DC): Mathematica Policy Research, U.S. Department of Education , Office of Planning, Evaluation and Policy Development; 2007.

Ross S, Potter A, Paek J, McKay D, Sanders W, Ashton J. Implementation and outcomes of Supplemental Educational Services: The Tennessee state-wide evaluation study. Journal of Education for Students Placed at Risk 2008;2008(13):26-58.

Schacter J, Jo B. Learning when school is not in session: A reading summer day-camp intervention to improve the achievement of exiting first-grade students who are economically disadvantaged. Journal of Research in Reading 2005;28(2):158-69.

Schacter J. Preventing summer reading declines in children who are disadvantaged. Journal of Early Intervention 2003;26(1):47-58.

Schirm A, Stuart E, McKie A. The Quantum Opportunities Program demonstration: final impacts. Washington (DC): Mathmatica Policy Research, U.S. Department of Education; 2008.

Socias M, deSousa J, Le Floch K. Supplemental Educational Services and student achievement in waiver districts: Anchorage and Hillsborough. Washington (DC): U.S. Department of Education, Office of Planning, Evaluation and Policy Development, Policy and Program Studies Service; 2009.

Zimmer R, Gill B, Razquin P, et al. State and local implementation of the No Child Left Behind Act, volume I Title I school choice, Supplemental Educational Services, and student achievement. Washington (DC): U.S. Department of Education, Office of Planning, Evaluation and Policy Development, Policy and Program Studies Service; 2007.

*Two studies in one paper

**Excluded due to quality of execution scoring

• Logic Model: Promoting Health Equity Through Education Programs and Policies

The following outlines the search strategy used for reviews of these out-of-school-time academic programs: reading-focused, math-focused, general, minimal academic content.

Effectiveness Review

The Community Preventive Services Task Force finding is based on evidence from an existing meta-analysis published in 2006 (Lauer et al., 35 studies, search period January 1985- May 2003) and an updated search for newer studies (24 studies, search period May 2003- March 2011).

The full search strategy from the Lauer 2006 review is available online: http://www.sagepub.com/bjohnsonstudy/articles/Lauer.pdf .

The update review searched the following databases: ERIC, PubMed, Sociological Abstracts/Social Services Abstracts, and Psyc INFO. The databases searched covered publications in medical and social sciences, behavioral sciences, and education. The types of documents retrieved by the search included journal articles, books, book chapters, reports, conference papers, and theses.

Search terms and search strategies were adjusted to each database, based on controlled and uncontrolled vocabularies and search software. The review team also searched reference lists found in studies included from the updated search period. Once the literature search was completed, Community Guide staff reviewed the citations using the following set of pre-defined inclusion and exclusion criteria to narrow down the publications to be reviewed

Studies were included if they:

• Were primary research published in a journal, or government report
• Were published in English
• Assessed an intervention in a high-income country
• Met minimum quality criteria for study design and execution; the study design had to include a comparison (control) population or condition (pre)
• Evaluated an intervention meeting the definition of an out-of-school-time-academic program
• Evaluated a study population of K-12 students at risk for academic failure (low SES, racial or ethnic minority, or otherwise noted by author as at risk for low achievement)

Studies were not included if they:

• Evaluated an exclusively special population (e.g., with mental disability)
• Reported only school grade outcomes

Search Terms

Database: ERIC

Time period: 2003 March 2011

“weekend*” or “week-end*” or “before school” or “out-of-school” or “outside of school” or “after school” or afterschool or “vacation” or “summer school” or “after hours” or “academic summer” or “summer academic*” or “extended day” or “extended hours” or supplementary or “summer math” or “summer literacy” or “summer reading” or “summer program*” or “summer training” or “open late” or “community learning program*”

Limited to:

Education Level is Early Childhood Education or Elementary Education or Elementary Secondary Education or Grade 1 or Grade 2 or Grade 3 or Grade 4 or Grade 5 or Grade 6 or Grade 7 or Grade 8 or Grade 9 or Grade 10 or Grade 11 or Grade 12 or High School Equivalency Programs or High Schools or Intermediate Grades or Junior High Schools or Kindergarten or Middle Schools or Preschool Education or Primary Education or Secondary Education

Database: PubMed

Limits: 2003 March 2011

Languages: English, Spanish, Portuguese, German, French

Child: 0 -18 years

Title/Abstract: “academic summer*” or “summer academic*” or “weekend school” or “week-end school*” or “extended school” or (vacation and school) or (school* and “open late”) or “extended day” or “vacation program” or “summer camp*” or “after school” or “before school” or “supplementary tutoring” or “vacation program*” or “summer literacy” or “summer reading” or “summer math*” or tutoring or extracurricular

Databases: Sociological Abstracts/Social Services Abstracts

Limits: 2003 March 2011

“out of school” or “outside of school” or “after school” or “before school program*” or “before school tutoring” or extracurricular or “summer school” or “extended school” or “weekend school” or “week-end school” or “academic summer” or “summer academic*” or “summer reading” or “summer literacy” or “summer math” or “summer program*” or “academic camp*” or “extra school*” or tutoring or “supplemental education” or “supplementary education”

Database: PsycINFO

Time limit: 2003 March 2011

“out of school” or “outside of school” or “after school” or “before school program*” or “before school tutoring” or extracurricular or “summer school” or “extended school” or “weekend school” or “week-end school” or “academic summer” or “summer academic*” or “summer reading” or “summer literacy” or “summer math” or “summer program*” or “academic camp*” or “extra school*” or tutoring or “supplemental education” or “supplementary education”

Cooper H, Charlton K, Valentine JC, Muhlenbruck L. Making the most of summer school: a meta-analytic and narrative review. Monographs of the Society for Research in Child Development 2000;65(1, Serial No. 260).

Duncan GJ, Yeung WJ, Brooks-Gunn J, Smith JR. How much does childhood poverty affect the life changes of children? American Sociological Review 1998;63:404-23.

James-Burdumy S, Dynarski M, Deke J. When elementary schools stay open late: results from the national evaluation of the 21st Century Community Learning Centers Program. Educational Evaluation and Policy Analysis 2007;29(4):296 318.

Lauer P, Motoko A, Wilkerson S. Out-of-school-time programs: a meta-analysis of effects for at-risk students. Review of Educational Research 2006;76:275-313.

Despite the finding of insufficient evidence, the following are considerations for implementation drawn from studies included in the evidence reviews of out-of-school-time academic programs (reading-focused, math-focused, general, minimal academic content), the broader literature, and expert opinion.

• For many federal programs, oversight is the responsibility of the state, and compliance with program requirements and enforcement are commonly incomplete.
• Some school districts fail to notify parents about the availability of free programs, such as Supplemental Educational Services, and the programs are underused.
• Most out-of-school-time academic programs are voluntary and attendance may be especially low for students most in need.
• Transportation to and from programs may be an issue for potential out-of-school-time academic programs participants.
• Inadequate staff training and staff turnover can make programs inefficient.