Vaccination Programs: Provider Assessment and Feedback

This review was conducted on behalf of the CPSTF by a team of specialists in systematic review methods, and in research, practice and policy related to increasing appropriate vaccination.

The systematic review included 20 studies.

• Overall vaccination rates increased by a median of 9 percentage points (16 studies).
  • Provider assessment and feedback used alone increased vaccination rates by a median of 11 percentage points (7 study arms).
  • Provider assessment and feedback used in combination with additional interventions increased vaccination rates by a median of 6 percentage points (13 study arms).
• The relative change in vaccination rates showed a median increase of 18.0% (15 study arms).
• Overall vaccination rates among children increased by a median of 10.5 percentage points (8 studies with 10 study arms).
• One study that did not provide a common measurement of change in vaccination rates reported no intervention effect.

The economic evidence is based on three studies that looked at the childhood vaccination series (search period 1980 2012). Monetary values are reported in 2013 U.S. dollars.

• The cost to implement the intervention ranged from $0.22 to $4 per child per year.
• One study estimated the cost per additional vaccinated child at $80.
• The size of the intervention group ranged from 1,643 to 18,034 clients.
• Two studies were from the U.S. and one was from Canada.

• How effective are these interventions for adolescent populations?
• What is intervention effectiveness in communities with disparities in vaccination rates?
• What are effectiveness, utility, and cost-effectiveness of provider assessment and feedback interventions that use regional or state-level immunization information systems (IIS)?

Effectiveness Review

Analytic Framework

When starting an effectiveness review, the systematic review team develops an analytic framework. The analytic framework illustrates how the intervention approach is thought to affect public health. It guides the search for evidence and may be used to summarize the evidence collected. The analytic framework often includes intermediate outcomes, potential effect modifiers, potential harms, and potential additional benefits.

Economic Review

No content is available for this section.

Effectiveness Review

Summary Evidence Table – Effectiveness Review

Economic Review

No content is available for this section.

Effectiveness Review

Bordley WC, Chelminski A, Margolis PA, Kraus R, Szilagyi PG, Vann JJ. The effect of audit and feedback on immunization delivery: A systematic review. Am J Prev Med 2000; 18(4):343-50.

Borgiel AE, Williams JI, Davis DA, Dunn EV, Hobbs N, Hutchison B et al. Evaluating the effectiveness of 2 educational interventions in family practice. Can Med Assoc J 1999; 161(8):965-70.

Brousseau N, Sauvageau C, Ouakki M, Audet D, Kiely M, Couture C, Pare A, Deceuninck G. Feasibility and impact of providing feedback to vaccinating medical clinics: evaluating a public health intervention. BMC Public Health 2010;10:750.

Fairbrother G, Hanson KL, Friedman S, Butts GC. The impact of physician bonuses, enhanced fees, and feedback on childhood immunization coverage rates. American Journal of Public Health 1999;89:171-5.

Furey A, Robinson E, Young Y. Improving influenza immunisation coverage in 2000-2001: a baseline survey, review of the evidence and sharing of best practice. Communicable Disease & Public Health 2001;4(3):183-7.

Hambidge SJ, Davidson AJ, Phibbs SL, Chandramouli V, Zerbe G, LeBaron CW, et al. Strategies to improve immunization rates and well-child care in a disadvantaged population: a cluster randomized controlled trial. Archives of Pediatrics & Adolescent Medicine 2004; 158:162-9.

Hillman AL, Ripley K, Goldfarb N, Weiner J, Nuamah I, Lusk E. The use of physician financial incentives and feedback to improve pediatric preventive care in Medicaid managed care. Pediatrics 1999;104:931-5.

Humair JP, Buchs CR, Stalder H. Promoting influenza vaccination of elderly patients in primary care. Fam Pract 2002;19(4):383-9.

Kiefe CI, Allison JJ, Williams OD, Person SD, Weaver MT, Weissman NW. Improving quality improvement using achievable benchmarks for physician feedback: a randomized controlled trial. JAMA 2001;285:2871-9.

LeBaron CW, Mercer JT, Massoudi MS, Dini E, Stevenson J, Fischer WM et al. Changes in clinic vaccination coverage after institution of measurement and feedback in 4 states and 2 cities. Archives of Pediatrics & Adolescent Medicine 1999;153(8):879-86.

Massoudi MS, Walsh J, Stokley S, Rosenthal J, Stevenson J, Miljanovic B et al. Assessing immunization performance of private practitioners in Maine: impact of the assessment, feedback, incentives, and exchange strategy. Pediatrics 1999;103 (6 Pt 1):1218-23.

Melinkovich P, Hammer A, Staudenmaier A, Berg M. Improving pediatric immunization rates in a safety-net delivery system. Joint Commission Journal on Quality & Patient Safety 2007;33(4):205-10.

Nace DA, Hoffman EL, Resnick NM, Handler SM. Achieving and sustaining high rates of influenza immunization among long-term care staff. Journal of the American Medical Directors Association 2007;8(2):128-33.

Page D, Meires J, Dailey A. Factors influencing immunization status in primary care clinics. Family Medicine 2002;34(1):29-33.

Quinley JC, Shih A. Improving physician coverage of pneumococcal vaccine: a randomized trial of a telephone intervention. Journal of Community Health 2004;29:103-15.

Rhew DC, Glassman PA, Goetz MB. Improving pneumococcal vaccine rates. Nurse protocols versus clinical reminders. Journal of General Internal Medicine 1999;14:351-6.

Russell ML, Ferguson CA. Improving population influenza vaccine coverage through provider feedback and best practice identification. Canadian Journal of Public Health Revue Canadienne de Sante Publique 2001;92(5):345-6.

Rust CT, Sisk FA, Kuo AR, Smith J, Miller R, Sullivan KM. Impact of resident feedback on immunization outcomes. Archives of Pediatrics & Adolescent Medicine 1999;153:1165-9.

Sabnis SS, Pomeranz AJ, Amateau MM. The effect of education, feedback, and provider prompts on the rate of missed vaccine opportunities in a community health center. Clinical Pediatrics 2003;42 (2):147-51.

Taylor JA, Darden PM, Brooks DA, Hendricks JW, Baker AE, Wasserman RC. Practitioner policies and beliefs and practice immunization rates: A study from pediatric research in office settings and the National Medical Association. Pediatrics 2002;109(2 I):294-300.

Economic Review

Brousseau N, Sauvageau C, Ouakki M, et al. Feasibility and impact of providing feedback to vaccinating medical clinics: evaluating a public health intervention. BMC Public Health 2010;10(1):750. http://dx.doi.org/10.1186/1471-2458-10-750.

Fontanesi J, De Guire M, Kopald D, Holcomb K. The price of prevention Cost of recommended activities to improve immunizations. Am J Prev Med 2004;26(1):41-45. http://dx.doi.org/10.1016/j.amepre.2003.09.009.

LeBaron CW, Mercer JT, Massoudi MS, et al. Changes in clinic vaccination coverage after institution of measurement and feedback in 4 states and 2 cities. Arch Pediatr Adolesc Med 1999;153(8):879-886. http://dx.doi.org/10.1001/archpedi.153.8.879.

Effectiveness Review

The CPSTF findings are based on studies included in the original review (search period 1980-1997) combined with studies identified in the updated search (search period 1997- February 2012). Reference lists of articles reviewed as well as lists in review articles were also searched, and members of our coordination team were consulted for additional references.

Details of the original search (1980-1997)

The following five electronic databases were searched during the original review period of 1980 up to 1997: MEDLINE, Embase, Psychlit, CAB Health, and Sociological Abstracts. The team also reviewed reference lists in articles and consulted with immunization experts. To be included in the review, a study had to:

• have a publication date of 1980 1997;
• address universally recommended adult, adolescent, or childhood vaccinations;
• be a primary study rather than, for example, a guideline or review;
• take place in an industrialized country or countries;
• be written in English;
• meet the evidence review and Guide chapter development team’s definition of the interventions; provide information on one or more outcomes related to the analytic frameworks; and
• compare a group of persons who had been exposed to the intervention with a group who had not been exposed or who had been less exposed. In addition, we excluded studies with least suitable designs for two interventions (provider reminder/recall and client reminder/recall) where the literature was most extensive.

Details of the update search (1997- February 2012)

The team conducted a broad literature search to identify studies assessing the effectiveness of Vaccine Preventable Disease interventions in improving vaccination rates. The following nine databases were searched during the period of 1997 up to February 2012: CABI, CINAHL, The Cochrane Library, EMBASE, ERIC, MEDLINE, PSYCHINFO, Soci Abs and WOS. Reference lists of articles reviewed as well as lists in review articles were also searched, and subject matter experts consulted for additional references. To be included in the updated review, a study had to:

• have a publication date of 1997- February 2012;
• evaluate vaccinations with universal recommendations;
• meet the evidence review and Guide chapter development team’s definition of the interventions;
• be a primary research study with one or more outcomes related to the analytic frameworks;
• take place in an high income country or countries;
• be written in English
• compare a group of persons who had been exposed to the intervention with a group who had not been exposed or who had been less exposed. In addition, we excluded studies with least suitable designs for two interventions (provider reminder/recall and client reminder/recall) where the literature was most extensive

Search Terms

1. Immunization
2. Vaccination
3. Immunization Programs

Economic Review

The present review included studies that reported economic outcomes from the 2000 review (search period 1980-1997) combined with studies identified from updated searches (search period 1997- February 2012) within the standard medical and health-related research databases, Google Scholar, and databases specialized to economics and social sciences. The details of the two sets of searches are provided below.

Details of the Updated Search (1997- February 2012)

The team conducted a broad literature search to identify studies assessing interventions to improve vaccination rates. The following nine databases were searched during the period of 1997 up to February 2012: CABI, CINAHL, The Cochrane Library, EMBASE, ERIC, MEDLINE, PSYCHINFO, Soci Abs and WOS. In addition, Google Scholar and specialized databases (CRD-University of York: NHS EED, EconLit, and JSTOR), were also searched. Reference lists of articles reviewed as well as lists in review articles were also considered, and subject matter experts consulted for additional references.

Search Terms

• Immunization
• Vaccination
• Immunization Programs

To be included in the updated review, a study had to do the following:

• Have a publication date of 1997- February 2012
• Evaluate vaccinations with universal recommendations
• Meet the evidence review and Community Guide review team’s definition of the interventions
• Be a primary research study with one or more outcomes related to the analytic framework(s)
• Take place in a high income country or countries
• Be written in English; and
• Compare a group of persons who had been exposed to the intervention with a group who had not been exposed or who had been less exposed.

Details of the Original Search (1980-1997)

The following five electronic databases were searched during the original review period of 1980 up to 1997: MEDLINE, Embase, Psychlit, CAB Health, and Sociological Abstracts. The team also reviewed reference lists in articles and consulted with immunization experts. To be included, a study had to do the following:

• Have a publication date of 1980 1997
• Address universally recommended adult, adolescent, or childhood vaccinations
• Be a primary study rather than, for example, a guideline or review
• Take place in an industrialized country or countries
• Be written in English
• Meet the definition of the interventions
• Provide information on one or more outcomes related to the analytic frameworks; and
• Compare a group of persons who had been exposed to the intervention with a group who had not been exposed or who had been less exposed. In addition, we excluded studies with least suitable designs for two interventions (provider reminder/recall and client reminder/recall) where the literature was most extensive.

Potential barriers to use of this intervention include the following:

• Lack of an adequate information infrastructure
• Administrative burden on providers and systems
• An increasingly complex immunization schedule

Evidence-Based Cancer Control Programs (EBCCP)

Find programs from the EBCCP website that align with this systematic review. (What is EBCCP?)

Healthy People 2030

Healthy People 2030 includes the following objectives related to this CPSTF recommendation.

• Reduce the proportion of children who get no recommended vaccines by age 2 years — IID‑02
• Maintain the vaccination coverage level of 1 dose of the MMR vaccine in children by age 2 years — IID‑03
• Maintain the vaccination coverage level of 2 doses of the MMR vaccine for children in kindergarten — IID‑04
• Increase the coverage level of 4 doses of the DTaP vaccine in children by age 2 years — IID‑06
• Increase the proportion of people who get the flu vaccine every year — IID‑09
• Increase the proportion of adults age 19 years or older who get recommended vaccines — IID‑D03