The Community Preventive Services Task Force findings for out-of-school-time academic programs (reading-focused, math-focused, general, minimal academic content) are based on evidence from a published meta-analysis and an updated search for newer evidence.
- 32 studies published between 1985-2003 (identified from: Lauer P, Motoko A, Wilkerson S. Out-of-school-time programs: a meta-analysis of effects for at-risk students. Review of Educational Research 2006;76:275-313.)
- While the meta-analysis included 35 studies, three were excluded from the Community Guide systematic review because they reported only school grades, a relatively subjective measure of reading and math achievement (Cosden et al., 2001; Legro, 1990; Smeallie, 1997).
- 25 studies (from 24 publications) published between 2003-2011 (updated search)
The Task Force also reviewed evidence from an independent systematic review of summer school programs to confirm Community Guide review findings (Cooper et al., 2000).
The number of studies and publications do not always correspond (e.g., a publication may include several studies or one study may be explained in several publications).
Effectiveness Review
Cooper H, Charlton K, Valentine JC, Muhlenbruck L. Making the most of summer school: a meta-analytic and narrative review. Monographs of the Society for Research in Child Development 2000;65(1, Serial No. 260).
Cosden M, Morrison G, Albanese AL, Macias S. When homework is not home work: after-school programs for homework assistance. Educational Psychologist 2001;36(3):211–21.
Lauer P, Motoko A, Wilkerson S. Out-of-school-time programs: a meta-analysis of effects for at-risk students. Review of Educational Research 2006;76:275-313.
Legro DL. An evaluation of an after-school partnership program: the effects on young children's performance (Doctoral dissertation, University of Houston). Dissertation Abstracts International 1990;52:02A.
Smeallie JE. An evaluation of an after-school tutorial and study skills program for middle school students at risk of academic failure (Doctoral dissertation, University of Maryland, College Park). Dissertation Abstracts International 1997;58:06A.
Studies from the Updated Search
*Black A, Somers M, Doolittle F, Unterman R, Grossman J. The evaluation of enhanced academic instruction in after-school programs: final report. Washington (DC): National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education; 2009.
Borman G, Dowling M. Longitudinal achievement effects of multiyear summer school: evidence from the Teach Baltimore randomized field trial. Educational Evaluation and Policy Analysis 2006;28(1):25-48.
Borman G, Goetz M, Dowling M. Halting the summer achievement slide: a randomized field trial of the KindergARTen summer camp. Journal of Education for Students Placed at Risk 2009;14(2):133-47.
Boulden W. Evaluation of the Kansas City LULAC National Education Service Center's Young Reader's Program. Children & Schools 2006;28(2):107-14.
Burgin J, Hughes G. Measuring the effectiveness of a summer literacy program for elementary students using writing samples. Research in the Schools 2008;15(2):55-64.
Cross A, Gottfredson D, Wilson D, Rorie M, Connell N. The impact of after-school programs on the routine activities of middle-school students: results from a randomized, controlled trial. Criminology & Public Policy 2009;8(2):391-412.
Denton C, Solari E, Ciancio D, Hecht S, Swank P. A pilot study of a kindergarten summer school reading program in high-poverty urban schools. Elementary School Journal 2010;110(4):423-39.
Dynarski M, James-Burdumy S, Moore M, Rosenberg L, Deke J, Mansfield W. When schools stay open late: the national evaluation of the 21st Century Community learning Centers program: new findings. Washington (DC): U.S. Department of Education, National Center for Education Evaluation and Regional Assistance; 2004.
Edmonds E, O'Donoghue C, Spano S, Algozzine R. Learning when school is out. Journal of Educational Research 2009;102(3):213-21.
Gottfredson D, Gerstenblith S, Soule D, Womer S, Lu S. Do after school programs reduce delinquency? Prevention Science 2004;5(4):253-66.
**Hanlon T, Simons B, O'Grady K, Carswell S, Callaman J. The effectiveness of an after-school program Targeting urban african american youth. Education and Urban Society 2009;1(42):96-118.
Heinrich C, Meyer R, Whitten G. Supplemental Education Services under No Child Left Behind: Who signs up, and what do they gain? Educational Evaluation and Policy Analysis 2010;32(2):273-98.
Huang D, Kim K, Cho J, Marshall A, Perez P. Keeping kids in school: a study examining the long-term impact of afterschool enrichment programs on students' high school dropout rates. Journal of Contemporary Issues in Education 2011;6(1):4-23.
James-Burdumy S, Dynarski M, Deke J. When elementary schools stay open late: results from the national evaluation of the 21st Century Community Learning Centers Program. Educational Evaluation and Policy Analysis 2007;29(4):296–318.
Jenner E, Jenner L. Results from a first-year evaluation of academic impacts of an after-school program for at-risk students. Journal of Education for Students Placed at Risk 2007;12(2):213-37.
Kim J, Samson J, Fitzgerald R, Hartry A. A randomized experiment of a mixed-methods literacy intevention for struggling readers in grades 4-6: effects on word reading efficiency, reading comprehension and vocabulary, and oral reading fluency. Reading and Writing: An Interdisciplinary Journal 2010;23(9):1109-29.
Munoz M, Potter A, Ross S. Supplemental Educational Services as a consequence of the NCLB legislation: evaluating its impact on student achievement in a large urban district. Journal of Education for Students Placed at Risk 2008;13(1):1-25.
Myers D, Olsen R, Seftor N, Young J, Tuttle C. The impact of regular Upward Bound: results from the third follow-up data collection. Washington (DC): Mathematica Policy Research, U.S. Department of Education, Office of Under Secretary, Policy and Program Studies Service; 2007.
Olsen R, Seftor N, Silva T, Myers D, DesRoches D, Young J. Upward Bound Math-Science: program description and interim impact estimates. Washington (DC): Mathematica Policy Research, U.S. Department of Education , Office of Planning, Evaluation and Policy Development; 2007.
Ross S, Potter A, Paek J, McKay D, Sanders W, Ashton J. Implementation and outcomes of Supplemental Educational Services: The Tennessee state-wide evaluation study. Journal of Education for Students Placed at Risk 2008;2008(13):26-58.
Schacter J, Jo B. Learning when school is not in session: A reading summer day-camp intervention to improve the achievement of exiting first-grade students who are economically disadvantaged. Journal of Research in Reading 2005;28(2):158-69.
Schacter J. Preventing summer reading declines in children who are disadvantaged. Journal of Early Intervention 2003;26(1):47-58.
Schirm A, Stuart E, McKie A. The Quantum Opportunities Program demonstration: final impacts. Washington (DC): Mathmatica Policy Research, U.S. Department of Education; 2008.
Socias M, deSousa J, Le Floch K. Supplemental Educational Services and student achievement in waiver districts: Anchorage and Hillsborough. Washington (DC): U.S. Department of Education, Office of Planning, Evaluation and Policy Development, Policy and Program Studies Service; 2009.
Zimmer R, Gill B, Razquin P, et al. State and local implementation of the No Child Left Behind Act, volume I—Title I school choice, Supplemental Educational Services, and student achievement. Washington (DC): U.S. Department of Education, Office of Planning, Evaluation and Policy Development, Policy and Program Studies Service; 2007.
*Two studies in one paper
**Excluded due to quality of execution scoring
Economic Review
Borman G, Dowling M. Longitudinal achievement effects of multiyear summer school: evidence from the Teach Baltimore randomized field trial. Educ Eval Policy Anal. 2006;28(1):25-48.
Brown W, Frates S, Rudge I, Tradewell R. The Costs and Benefits of After School Programs: The Estimated Effects of the After School Education and Safety Program Act of 2002 September. Claremont, CA: The Rose Institute of Claremont-McKenna College; 2002.
Grossman J, Price M, Fellerath V. Multiple Choices After School: Findings from the Extended-Service Schools Initiative. Philadelphia, PA: Public/Private Ventures; 2002.
Grossman J, Lind C, Hayes C, McMaken J, Gersick A. The Cost of Quality Out-of-School Time Programs. Philadelphia, PA: Public/Private Ventures; 2009.
Gottfredson D, Cross A, Wilson D, Connell N, Rorie M. A Randomized Trial of the Effects of an Enhanced After-School Program for Middle-School Students. Washington, DC: Institute of Educational Sciences, National Center for Education Evaluation and Regional Assistance, US Department of Education; 2010.
Halpern R. After-School Programs for Low-Income Children: Promise and Challenges. The Future of Children 1999;9(2):81-95.
Herrera C, Arbreton A. A Report on the Experiences of Boys & Girls Clubs in Boston and New York City: Increasing Opportunities for Older Youth in After-School Programs. Philadelphia, PA: Public/Private Ventures; 2003.
Jacob B. Remedial education and student achievement: a regression discontinuity analysis. Rev Econ Stat 2004;86(1):226-44.
James-Burdumy S, Dynarski M, Moore M, Deke J, Mansfield W. When Schools Stay Open Late: The National Evaluation of the 21st Century Community Learning Centers Program Final Report. U.S. Department of Education / Institute of Education Sciences National Center for Education Evaluation and Regional Assistance (NCEE); 2005.
Maxfield M, Castner L. The Quantum Opportunity Program Demonstration: Implementation Findings. Washington, DC: Mathematica Policy Research Inc; 2003.
Proscio T, Whiting B. After-School Grows Up: How four Large American Cities Approach Scale and Quality in After-School Programs. After School Project of the Robert Wood Johnson Foundation. New York, NY: The After School Project; 2004.
Walker K, Arbreton A. After-School Pursuits: An Examination of Outcomes in the San Francisco Beacon Initiative. Philadelphia, PA: Public/Private Ventures; 2004.
The following outlines the search strategy used for reviews of these out-of-school-time academic programs: reading-focused, math-focused, general, minimal academic content.
Effectiveness Review
The Community Preventive Services Task Force finding is based on evidence from an existing meta-analysis published in 2006 (Lauer et al., 35 studies, search period January 1985- May 2003) and an updated search for newer studies (24 studies, search period May 2003- March 2011).
The full search strategy from the Lauer 2006 review is available online: http://www.sagepub.com/bjohnsonstudy/articles/Lauer.pdf 
[PDF - 555 KB] .
The update review searched the following databases: ERIC, PubMed, Sociological Abstracts/Social Services Abstracts, and Psyc INFO. The databases searched covered publications in medical and social sciences, behavioral sciences, and education. The types of documents retrieved by the search included journal articles, books, book chapters, reports, conference papers, and theses.
Search terms and search strategies were adjusted to each database, based on controlled and uncontrolled vocabularies and search software. The review team also searched reference lists found in studies included from the updated search period. Once the literature search was completed, Community Guide staff reviewed the citations using the following set of pre-defined inclusion and exclusion criteria to narrow down the publications to be reviewed
Studies were included if they:
- Were primary research published in a journal, or government report
- Were published in English
- Assessed an intervention in a high-income country
- Met minimum quality criteria for study design and execution; the study design had to include a comparison (control) population or condition (pre)
- Evaluated an intervention meeting the definition of an out-of-school-time-academic program
- Evaluated a study population of K-12 students at risk for academic failure (low SES, racial or ethnic minority, or otherwise noted by author as at risk for low achievement)
Studies were not included if they:
- Evaluated an exclusively special population (e.g., with mental disability)
- Reported only school grade outcomes
Search Terms
Database: ERIC
Time period: 2003 – March 2011
“weekend*” or “week-end*” or "before school" or "out-of-school" or “outside of school” or "after school" or afterschool or "vacation" or "summer school" or "after hours" or "academic summer" or “summer academic*” or "extended day" or "extended hours" or supplementary or "summer math" or "summer literacy" or "summer reading" or "summer program*" or "summer training" or "open late" or "community learning program*”
Limited to:
Education Level is Early Childhood Education or Elementary Education or Elementary Secondary Education or Grade 1 or Grade 2 or Grade 3 or Grade 4 or Grade 5 or Grade 6 or Grade 7 or Grade 8 or Grade 9 or Grade 10 or Grade 11 or Grade 12 or High School Equivalency Programs or High Schools or Intermediate Grades or Junior High Schools or Kindergarten or Middle Schools or Preschool Education or Primary Education or Secondary Education
Database: PubMed
Limits: 2003 – March 2011
Languages: English, Spanish, Portuguese, German, French
Child: 0 -18 years
Title/Abstract: “academic summer*” or “summer academic*” or “weekend school” or “week-end school*” or “extended school” or (vacation and school) or (school* and “open late”) or “extended day” or “vacation program” or “summer camp*” or “after school” or “before school” or “supplementary tutoring” or “vacation program*” or “summer literacy” or “summer reading” or “summer math*” or tutoring or extracurricular
Databases: Sociological Abstracts/Social Services Abstracts
Limits: 2003 – March 2011
“out of school” or “outside of school” or “after school” or “before school program*” or “before school tutoring” or extracurricular or “summer school” or “extended school” or “weekend school” or “week-end school” or “academic summer” or “summer academic*” or “summer reading” or “summer literacy” or “summer math” or “summer program*” or “academic camp*” or “extra school*” or tutoring or “supplemental education” or “supplementary education”
Database: PsycINFO
Time limit: 2003 – March 2011
“out of school” or “outside of school” or “after school” or “before school program*” or “before school tutoring” or extracurricular or “summer school” or “extended school” or “weekend school” or “week-end school” or “academic summer” or “summer academic*” or “summer reading” or “summer literacy” or “summer math” or “summer program*” or “academic camp*” or “extra school*” or tutoring or “supplemental education” or “supplementary education”
Economic Review
No content is available for this section.