Methods Manual – References and Appendix

Reference List

¹ O’Connor D, Green S, Higgins JP. Defining the Review Question and Developing Criteria for Including Studies. In: Cochrane Handbook for Systematic Reviews of Interventions; 2008. p. 81-94.

² Briss PA, Zaza S, Pappaioanou M, Fielding J, Wright-De Aguero L, et al. Developing an evidence-based Guide to Community Preventive Services–methods. American Journal of Preventive Medicine. 2000;18(1 Suppl):35-43.

³ Carande-Kulis VG, Maciosek MV, Briss PA, Teutsch SM, Zaza S, et al. Methods for systematic reviews of economic evaluations for the Guide to Community Preventive Services. Task Force on Community Preventive Services. American Journal of Preventive Medicine. 2000;18(1 Suppl):75-91.

⁴ Zaza S, Wright-De Aguero L, Briss PA, Truman BI, Hopkins DP, et al. Data collection instrument and procedure for systematic reviews in the Guide to Community Preventive Services. Task Force on Community Preventive Services. American Journal of Preventive Medicine. 2000;18(1 Suppl):44-74.

⁵ Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. British Medical Journal 2009;339:b2700.

⁶ Knopf JA, Finnie R, Peng Y, Hahn RA, Truman BI, et al. School-based health centers to advance health equity: a Community Guide systematic review. American Journal of Preventive Medicine. 2016;51(1):114-26.

⁷ Ledoux TA, Hingle MD, Baranowski T. Relationship of fruit and vegetable intake with adiposity: a systematic review. Obesity Reviews 2011 May;12(5):e143-e150. [doi: 10.1111/j.1467-789X.2010.00786.x].

⁸ Lock K, Pomerleau J, Causer L, Altmann DR, McKee M. The global burden of disease attributable to low consumption of fruit and vegetables: implications for the global strategy on diet. Bulletin of the World Health Organization. 2005 Feb;83(2):100-108.

⁹ Jacob V, Chattopadhyay SK, Reynolds JA, Hopkins DP, Morgan JA, et al. Economics of interventions to increase active travel to school: a Community Guide systematic review. American Journal of Preventive Medicine. 2021;60(1):e27-40.

Appendix

Appendix 1. Example of Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Flowchart from Review of Interventions to Support Healthier Foods and Beverages in Schools^F

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flowchart showing number of studies identified, reviewed in full text, reasons for exclusion, and total number of included studies from the review of Healthier Food and Beverage Interventions in Schools.

Appendix 2. Example of Evidence Table Template from School-Based Health Centers Review

Appendix 3. Example Body of Evidence Table from School-Based Health Clinics Review (N=48)

Appendix 4. Example Study Effect Estimate Display from CHWs for Diabetes Prevention Review^G

Appendix 5. CPSTF Evidence Decision Table (Revised 2017)*^H

*Based on “Translating Evidence of Effectiveness into Recommendations” from Briss 2000²

Appendix 6. CPSTF Options for Modifying Findings and Conclusions*^I

*This table expands on “Translating Evidence of Effectiveness into Recommendations” from Briss 2000²

Appendix 7. Detailed Evidence Table for Economic Review from School-Based Health Centers Review

Appendix 8. Decision Table for Conclusion on Cost-Benefit^J

*Woolf, Steven H. “A closer look at the economic argument for disease prevention.” JAMA 301.5 (2009): 536-538.

Appendix 9. Economic Decision Table for Cost-Effectiveness Finding^K

*Note the WHO guidelines suggest interventions are cost-effective when DALY is less than 3x annual GDP per capita. See World Health Organization. Macroeconomics and Health: Investing in Health for Economic Development: Report of the Commission on Macroeconomics and Health. 2001.

Appendix 10. Options in Case of Inconsistent Evidence

Given: A body of evidence which otherwise meets CPSTF Decision Table requirements for minimum quality and quantity of studies but with inconsistent results

Upgrade from No Finding to a Finding Statement

Based on:

The interquartile interval (IQI), as a summary of body of evidence, includes

• Net-Benefit values all positive (for cost-beneficial finding)
• Values which are all below or above the threshold (for cost-effective or not cost-effective finding)
• Quantity and quality of estimates when the IQI includes zero for cost-benefit analyses and the threshold value in case of cost-effectiveness analyses

Additional considerations to cost-effectiveness estimates that are closer to cut-off threshold or closer to lower or higher values in a range of thresholds

Other?

Accessibility Appendix

^A The four key components involved in determining and disseminating CPSTF findings include

1. The Community Preventive Services Task Force (CPSTF) is an independent, nonpartisan, nonfederal panel of public health and prevention experts that provides input into the review process and make recommendations on the effectiveness of systematically reviewed public health programs.
2. CPSTF Liaisons are experts in research, practice, and policy who help CPSTF to develop and disseminate materials designed to help targeted audiences use CPSTF recommendations and findings.
3. The Community Guide Program includes CDC staff and contractors who support CPSTF and its Liaisons, maintain The Community Guide website, conduct systematic reviews on prioritized public health topics, write and disseminate findings, and create and maintain partnerships with intended users.
4. The Community Guide is an online collection of evidence based CPSTF recommendations and findings, tools, and resources that communities can use to protect and improve their population health.

^B Ten Steps in the Community Guide Effectiveness Review Process include

1. Select Topic. Select a review topic based on CPSTF prioritization.
2. Create Coordination Team. Create a team of staff, CDC partners, CPSTF members, and subject matter experts.
3. Select Intervention. Select an intervention within the topic.
4. Conceptualize. Define the intervention, craft the analytic framework, and identify outcomes.
5. Search. Systematically search for and retrieve evidence.
6. Screen. Narrow retrieved set of papers.
7. Abstract. Evaluate studies for quality and collect information from papers.
8. Analyze. Analyze data from included studies.
9. Make CPSTF Recommendations. Translate evidence into CPSTF recommendations and findings.
10. Disseminate. Disseminate CPSTF findings and evidence gaps.

^C Conceptualization process elements include draft intervention definition, determine inclusion and exclusion criteria, develop research questions, develop analytic framework, and consider applicability factors.

^D The Three Stage Screening Process includes title screens, abstract screens, and full text screens. Each screening stage is more discriminating than the previous one, resulting in studies with greater relevancy.

^E Nine Steps in the Community Guide Economic Review Process include

1. Prioritize Economic Reviews. Select reviews based on CPSTF prioritization.
2. Create Coordination Team. Create a team of staff, CDC program partners, CPSTF members, and other subject matter experts.
3. Conceptualize Economic Review. Define the intervention, craft the economic analytic framework, and identify outcomes.
4. Search. Systematically search for and retrieve evidence.
5. Screen. Narrow retrieved set of papers.
6. Abstract. Evaluate studies for quality and collect information from papers.
7. Analyze. Analyze data from included studies.
8. Make CPSTF Economic Finding. Translate evidence into CPSTF economic findings.
9. Disseminate. Disseminate CPSTF findings and evidence gaps.

^F This example of a Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Flowchart shows how a total of 27,898 records for a review of Interventions to Support Healthier Foods and Beverages in Schools was reduced to 54 included studies. The flowchart shows the outcome of the number of records at each of 4 stages: identification, screening, eligibility, and the final number of included studies for the systematic review.

^G The scatterplot is an example of a study effect estimate display from the review of diabetes interventions including community health workers. The example includes results from 14 studies listing the author and study year along the y-axis. The change in effect estimate is listed along the x-axis. Studies of greatest suitability of design are grouped above the dotted horizontal line. Studies of least suitability of design are grouped below the dotted horizontal line.

^H This table displays options for CPSTF when rating the strength of evidence on effectiveness. The first column lists the options for rating the strength of the evidence. The following columns describe the requirements for study design, quality of execution, number of studies, overall assessment of study results, and overall assessment of the population health impact needed to reach a determination for the rating of strength of evidence.

CPSTF rates the strength of evidence strong when

• The body of evidence includes studies of greatest or moderate suitability of design.
• Greatest suitability of design and good quality of execution requires two or more studies, overall assessment of study results needs to be consistent, and overall assessment of population health impact needs to be meaningful.
• Moderate or a mix of greatest and moderate suitability of design and good quality of execution requires five or more studies, overall assessment of study results needs to be consistent, and overall assessment of population health impact needs to be meaningful.
• Greatest suitability of design and fair or a mix of fair and good quality of execution requires five or more studies, overall assessment of study results needs to be consistent, and overall assessment of population health impact needs to be meaningful.
• Studies meet criteria for sufficient but not strong body of evidence, but the magnitude of effect is substantial and supports upgrading the strength of evidence supporting CPSTF’s conclusion on effectiveness, and the overall health impact is large.

CPSTF rates the strength of evidence sufficient when

• The body of evidence includes studies of greatest, moderate, or least suitability of design.
• Greatest suitability of design and good quality of execution requires one study, overall assessment of study is not applicable, and overall assessment of population health impact needs to be meaningful.
• Moderate or a mix of greatest and moderate suitability of design and fair or a mix of far and good quality of execution requires three or more studies, overall assessment of study results needs to be consistent, and overall assessment of population health impact needs to be meaningful.
• Least or a mix of least and moderate or good suitability of design and fair or a mix of fair and good quality of execution requires five or more studies, overall assessment of study results needs to be consistent, and overall assessment of population health impact needs to be meaningful.
• Studies meet criteria for strong body of evidence, but CPSTF assessment finds one or more issues and therefore decides to downgrade the strength of the evidence to sufficient.

CPSTF rates the evidence insufficient when Identified evidence does not meet minimum requirements or combinations based on design suitability, quality of execution, or number of studies, or overall assessment is that study findings are inconsistent, or overall assessment is that studies demonstrate small or no effects.

^I This chart shows the reasons that CPSTF may choose to upgrade or downgrade a specific finding or conclusion.

CPSTF may choose to upgrade the strength of evidence rating from sufficient to strong when the included studies meet criteria for sufficient but not strong, body of evidence, and the magnitude of effect is meaningful and substantial in a population or public health context.

CPSTF may downgrade the strength of evidence rating from strong to sufficient when included studies meet the criteria for strong body of evidence, but the overall CPSTF conclusion incorporates one or more of the following concerns:

• Moderate, recurring flaws or gaps in study methods or reporting,
• Applicability findings or gaps,
• Concerns with link to health outcomes.

CPSTF may downgrade a conclusion to recommended against when

• There is adequate evidence of at least one important harm of meaningful impact on health in a population or public health context.
• There is adequate evidence that intervention would increase health inequity in the population to an unacceptable degree.
• Intervention studies provide either strong or sufficient evidence of no (or very small) effect on any of the recommendation outcomes.

CPSTF may downgrade a CPSTF finding to insufficient evidence when included studies meet criteria for strong or sufficient body of evidence, but overall CPSTF assessment incorporates at least one of the following concerns:

• Serious or recurring flaws or gaps in study methods or reporting,
• Applicability findings or gaps,
• Concerns with link to health outcomes
• Harms or equity concerns.

CPSTF may narrow the recommendation when appropriate subsets of the included studies have important differences in the evidence on effectiveness, such as

• Differential and meaningful findings on applicability,
• Differential evidence, or concerns on harms or equity.

^J This table displays options for CPSTF when making a cost-benefit finding. The first column lists the options for the finding. The following columns describe the requirements for quality of estimates, number of studies, overall assessment of study results, and criterion for determination needed to reach a finding.

CPSTF issues a finding that an intervention is cost-beneficial when

• Quality of estimates from included studies is good, number of studies is two or more, the overall assessment of the distribution of study results for the finding statement is consistent, and the benefits outweigh the costs.
• Quality of estimates from included studies is fair and good, number of studies is three or more, the overall assessment of the distribution of study results for the finding statement is consistent, and the benefits outweigh the costs.
• Quality of estimates from included studies is fair, number of studies is four or more, the overall assessment of the distribution of study results for the finding statement is consistent, and the benefits outweigh the costs.

CPSTF issues a no finding when identified evidence does not meet the requirement of minimum quality or combinations based on quality of estimates and number of studies, OR overall assessment is that study results are inconsistent, OR the costs outweigh the benefits.

^K This table displays options for CPSTF when making a cost-effectiveness finding. The first column lists the options for the finding. The following columns describe the requirements for quality of estimates, number of studies, overall assessment of study results, and criterion for determination needed to reach a finding.

CPSTF issues a finding that an intervention is cost-effective⁸ when

• Quality of estimates from included studies is good, number of studies is two or more, the overall assessment of the distribution of study results for the finding statement is consistent, and cost per QALY gained is less than or equal to $50,000 OR cost per DALY averted is less than or equal to annual GDP per capita.
• Quality of estimates from included studies is fair and good and number of studies is three or more, the overall assessment of the distribution of study results for the finding statement is consistent, and cost per QALY gained is less than or equal to $50,000 OR cost per DALY averted is less than or equal to annual GDP per capita.
• Quality of estimates from included studies is fair, number of studies is four or more, the overall assessment of the distribution of study results for the finding statement is consistent, and cost per QALY gained is less than or equal to $50,000 OR cost per DALY averted is less than or equal to annual GDP per capita.

CPSTF issues a finding that an intervention is not cost-effective when

• Quality of estimates from included studies is good, number of studies is two or more, the overall assessment of the distribution of study results for the finding statement is consistent, and cost per QALY gained is greater than $150,000 OR cost per DALY averted is greater than three times the annual GDP per capita.
• Quality of estimates from included studies is fair and good, number of studies is three or more, the overall assessment of the distribution of study results for the finding statement is consistent, and cost per QALY gained is greater than $150,000 OR cost per DALY averted is greater than three times the annual GDP per capita.
• Quality of estimates from included studies is fair, number of studies is four or more, the overall assessment of the distribution of study results for the finding statement is consistent, and cost per QALY gained is greater than $150,000 OR cost per DALY averted is greater than three times the annual GDP per capita.

CPSTF issues a no finding when identified evidence does not meet the requirement of minimum quality or combinations based on quality of estimates and number of studies, OR overall assessment is that study results are inconsistent, OR cost per QALY gained is between $50,000 and $150,000, OR cost per DALY averted is between the annual GDP per capital and three times the annual GDP per capita.

Printer Friendly Version

Printer friendly version of this portion of the methods manual [PDF – 389 KB]