FY 2022 EVALUATION PLAN

Department of Health and Human Services

2 
 
Contents 
Development of the HHS Evaluation Plan ............................................................................................................. 3 
Administration for Children and Families (ACF) .................................................................................................... 4 
Administration for Community Living (ACL) ........................................................................................................ 11 
Agency for Healthcare Research and Quality (AHRQ) ......................................................................................... 16 
Office of the Assistant Secretary for Planning and Evaluation (ASPE) ................................................................ 18 
Centers for Disease Control and Prevention (CDC) ............................................................................................. 27 
Centers for Medicare and Medicaid Services (CMS) ........................................................................................... 40 
Food and Drug Administration (FDA) .................................................................................................................. 47 
Indian Health Service (IHS) .................................................................................................................................. 50 
National Institutes of Health (NIH) ...................................................................................................................... 54 
Substance Abuse and Mental Health Services Administration (SAMHSA) .......................................................... 59 
 
 
 

Development of the HHS Evaluation Plan

The Foundations for Evidence-Based Policymaking Act of 2018 (Evidence Act) intends to improve

decision-making for federal programs and policy development by requiring a transparent, question-

driven approach to evidence development and analysis. As part of the Evidence Act, HHS is required

to submit “an evaluation plan describing activities the agency plans to conduct pursuant to [its

evidence-building plan].” The evaluation plan is designed to include significant evaluations and the

statute gives discretion to agencies to determine how to define significant. For purposes of this plan,

HHS has defined “significant” as evaluation activities that support answering questions from the HHS

Evidence-Building Plan.

In order to fulfill the various provisions of the Evidence Act, HHS created an implementation structure

to enact its different components. This implementation structure tasked the HHS Evidence and

Evaluation Council (the Council) with developing the HHS Evidence-Building and Evaluation Plans.

The Council predates the Evidence Act and is made up of senior evaluation staff and subject matter

experts from each agency within HHS. The Council meets monthly to address issues related to

evidence-building and evaluation policies or activities across HHS, with a recent focus on Evidence Act

implementation activities. A subcommittee of the Council developed the final process to collect the

information needed to develop the Evaluation Plan (the plan).

Nine operating divisions within HHS and one staff division developed evaluation plans based on a

central template and instructions, and included information on:

• Priority questions being examined by the agency;

• Programs to be analyzed;

• Relevant stakeholders;

• Data sources;

• Methods;

• Challenges and mitigation strategies; and

• Dissemination plans.

Each division cleared their plan through their agency leadership, and those plans have been compiled

here. More than 30 priority questions address a wide range of topic areas, including three main cross-

cutting topics: COVID-19 and pandemic response, the opioid crisis, and value-based care. Topics that

are cross-cutting are often due to different parts of HHS having different missions but working on the

same issue. This is true for the cross-cutting topics you see in the Evaluation Plans. These plans are

planned efforts that are subject to receiving appropriate approvals and resources, and they are subject

to change.

This plan lays out these priorities areas and analysis and evaluation activities related to each area that

will occur in FY 2022.

Administration for Children and Families (ACF)

Priority Question 1:

How do individual programs and approaches move Temporary Assistance for Needy Families (TANF)

recipients and other low-income individuals into jobs and help them retain employment?

Significant Evaluations that will address this question in FY22:

The Next Generation of Enhanced Employment Strategies Project and the Building Evidence on

Employment Strategies for Low-Income Families Project

Key Questions:

What is the effectiveness of select employment interventions designed to help individuals facing

complex challenges (e.g. physical and mental health conditions, criminal justice system involvement,

limited work skills and experience, current or foreseeable disabilities, opioid use disorder or other

substance use disorders) secure a pathway toward economic independence?

Background and Significance:

ACF seeks to understand and inform how TANF and other programs that serve TANF or TANF-eligible

populations can best support these individuals’ self-sufficiency and economic well-being. This body of

work builds on several decades of past research which has provided useful findings and identified

remaining questions. For example, the most successful training programs are generally inaccessible to

individuals with low literacy or numeracy levels or significant personal barriers. Interventions targeting

individuals who face complex barriers to employment, even when these programs boost employment

and earnings, typically leave most participants with low earnings or inconsistent employment.

Programs, Policies, Regulations, or Operations to be Analyzed or Evaluated:

These evaluations are meant to inform the TANF program and related employment-focused programs.

The TANF block grant provides funds to States, eligible territories, and tribes, which decide on the

design of the program, the type and amount of assistance payments to families, and the range of

other services to be provided. Approximately one million families nationwide received assistance

through TANF in an average month in 2018. The program supports a wide range of efforts to promote

family economic stability. For example, TANF grant dollars are used for programs that promote job

readiness through education and training; provide assistance with child care, transportation, or other

services that support employment activities; and improve services that prevent or address child abuse

and neglect.

Stakeholders:

The design and execution of this work has been and will be informed by stakeholder engagement such

as consultation with federal partners in HHS and other agencies, such as:

 The Assistant Secretary for Planning and Evaluation (ASPE)

 The Substance Abuse and Mental Health Services Administration (SAMHSA)

 Housing and Urban Development (HUD)/Office of Policy Development and Research (PD&R)

 Department of Labor (DOL):

o Chief Evaluation Office (CEO)

o Employment and Training Administration (ETA)

o Office of Disability Employment Policy (ODEP)

 US Department of Agriculture (USDA)/Food and Nutrition Service (FNS)

 Department of Education (ED)

o Institute of Education Services (IES)

o Office of Career, Technical, and Adult Education (OCTAE)

 Social Security Administration (SSA)/Office of Research, Demonstration, and Employment

Services (ORDES)

 Researchers and policy experts; and

 National organizations, such as the National Governors Association, the National Conference of

State Legislatures, and the American Public Human Services Association.

ACF gathered stakeholder views through conferences and meetings, such as the Research and

Evaluation Conference on Self-Sufficiency (RECS), National Association of Welfare Research and

Statistics (NAWRS) conference, Association for Public Policy Analysis and Management (APPAM)

conference, the Family Self-Sufficiency Research Technical Working Group, and the Federal

Employment, Training, and Education Working Group and periodic topic-specific meetings such as the

Next Steps for Employment & Training Research Roundtable; TANF and Tribal TANF Summits; and

surveys, focus groups, interviews, and other activities conducted as part of research and evaluation

studies.

These results may impact federal, state, and local policy-makers, program officials, and practitioners;

training and technical assistance providers; and TANF and potential TANF recipients.

Data Sources:

Data sources will include primary data collection through surveys of program participants and staff;

federal administrative data such as the National Directory of New Hires; state and local administrative

data such as TANF data and local program management information system data; and qualitative data

from interviews with program staff and participants, employers, community partners, and others; and

observations of program implementation.

Methods:

As evaluation sites are identified ACF will determine the most rigorous evaluation approach feasible,

focusing on random assignment designs where possible. The projects aim to conduct up to 22

evaluations, with the majority of these evaluations being experimental impact studies; a few may be

solely descriptive evaluations. Most evaluations will include an impact study to examine the

interventions’ impact on participants’ employment and earnings, and other outcomes of interest; an

implementation study to describe the design and operations of the selected interventions, and to

document the outcomes of participants served by the interventions; and a cost study to examine each

interventions’ sources of funding, use of resources to implement the approach, costs and benefits, and

sustainability.

Anticipated Challenges and Mitigation Strategies:

Challenges include availability and quality of administrative data; adequacy of outcome measures; mis-

match of annual funding vis-à-vis long-term evaluation timelines; and finding sites willing and able to

participate in evaluations. ACF will pursue the following mitigation strategies for these challenges:

 ACF proposed in the President’s FY 2021 budget to make multi-year funding available for

research and evaluation to better align funding and evaluation timelines.

 ACF will involve stakeholders to help identify promising and willing sites to participate in

evaluations, and work with potential sites to address their concerns and prepare them for

rigorous evaluation.

 ACF is helping state and local human services agencies build their capacity to engage in

research and evaluation activities.

Plan for Disseminating Results:

ACF will produce comprehensive research reports as well as shorter documents aimed at policy and

practitioner audiences. ACF will disseminate results through posting reports on the Internet; using

social media to alert potential audiences of the availability of results; presenting results at research,

policy, and practitioner conferences; briefing policy-makers and program officials; and submitting the

findings for review by the ACF-sponsored Pathways to Work Evidence Clearinghouse. Possible uses for

these findings include informing federal, state, and local policy-making as well as state and local

selection and design of services to help low-income people gain capacity to reduce dependency

though entering and succeeding in the labor market. ACF will archive data for secondary use.

Priority Question 2:

Who are the children and families served by Head Start and Early Head Start and how are they faring?

What services are provided by Head Start and Early Head Start programs? What is the quality of those

services and how do they support improved outcomes for children and families?

Significant Evaluations that will address this question in FY22:

Head Start Family and Child Experiences Survey (FACES)

Key Questions:

What are the characteristics, experiences, and development of Head Start children and families, and

the characteristics of the Head Start programs and staff who serve them?

Background and Significance:

The science of early childhood development demonstrates the importance of children’s earliest

experiences for long-term development and learning and highlights the potential for early care and

education (ECE) programs to help close the school readiness gap observed between low income

children and their more affluent peers. There is a large and growing body of evidence indicating high-

quality ECE programs can produce meaningful improvements in children’s language, literacy,

numeracy, and social-emotional development. Research further shows, however, that the quality of

ECE programs varies considerably. As such, ECE research has given extensive attention to identifying

the components of ECE programs that best improve children’s wellbeing and effective mechanisms for

enhancing quality. For over 50 years, Head Start research has contributed to this still growing research

base and provided valuable information not only for guiding program improvements in Head Start

itself, but also for the larger field of ECE.

Programs, Policies, Regulations, or Operations to be Analyzed:

Established in 1965, Head Start aims to promote the school readiness of children, ages three to five,

from low-income families by supporting the development of the whole child through high-quality,

comprehensive services. In 1994, the Early Head Start program was established to provide these same

comprehensive services to low-income families with infants and toddlers, as well as pregnant women.

Today, there are approximately 1,700 Head Start and Early Head Start grantees run by local public and

private non-profit and for-profit agencies throughout all 50 States, the District of Columbia, five

territories, and in tribal and migrant/seasonal farm-working communities.

Nearly one million children, birth to age five, are currently enrolled in Head Start and Early Head Start.

Children and their families receive services through a variety of models, including center- based, family

child care, and home visiting. Programs tailor their service models to the needs of the local community

and to be ethnically, culturally, and linguistically responsive to the families they serve. Children’s

growth and development is supported through individualized early learning experiences, health and

nutritional services, and supports for family wellbeing.

Head Start is authorized by the Improving Head Start for School Readiness Act of 2007. In fiscal year

2019, just over $10 billion were appropriated for Head Start and Early Head Start. ACF administers

these funds through grants to local agencies and provides oversight, policy direction, guidance,

technical assistance, and other supports for Head Start and Early Head Start grantees.

Stakeholders:

The design and execution of this work will be informed by consultation with stakeholders including

national, local, and non-profit Head Start administrators and staff, Head Start training and technical

assistance providers, Head Start curriculum and model developers, Federal partners in HHS and other

agencies, such as ASPE and ED/IES, researchers and policy experts, national organizations, such as the

National Head Start Association (NHSA) and the Society for the Research on Child Development

(SRCD), and partners in the broad array of community-based service systems that support children and

families. ACF gathers stakeholder views though conferences and meetings, such as the National

Research Conference on Early Childhood (NRCEC) and the Child Care and Early Education Policy

Research Consortium (CCEEPRC) Annual Meeting and Steering Committee; through ongoing

engagement with Head Start training and technical assistance networks; and through surveys, focus

groups, interviews, and other activities conducted as part of the research design process. These results

may impact federal, state, and local policy-makers, program officials, and practitioners; professional

development and/or technical assistance providers; Head Start administrators and staff; and families

who participate or might participate in Head Start.

Data Sources:

This study will collect data from a nationally representative sample of 3- to 4-year-old children and

their families, teachers, classrooms, centers, and programs in Head Start Regions I-X.

Methods:

This is a process and outcome study. Analyses will examine family characteristics; the growth in

children’s development, such as language, early literacy, numeracy, and social-emotional functioning;

classroom quality; teacher characteristics; and patterns of variation in outputs and outcomes related

to key variables such as setting (e.g. rural vs. urban), family characteristics (e.g. race and ethnicity),

teacher characteristics (e.g. credentials and motivation), and program characteristics (e.g. auspice).

Anticipated Challenges and Mitigation Strategies:

Challenges include mis-match of annual funding vis-à-vis long-term evaluation timelines; adequacy of

measures; and finding local programs willing and able to participate. ACF is pursuing the following

mitigation strategies for these challenges:

 ACF proposed in the President’s FY 2021 budget to make multi-year funding available for

research and evaluation to better align funding timelines with evaluation timelines.

 ACF will use the results of past projects to identify the best measures currently available, and will

use the FACES data to further refine measures.

 ACF will involve stakeholders in outreach to sites and work with sites to address their concerns.

Plan for Disseminating Results:

ACF will produce comprehensive research reports as well as shorter documents aimed at policy and

practitioner audiences. ACF will disseminate results through posting reports on the Internet; using

social media to alert potential audiences of the availability of results; presenting results at research,

policy, and practitioner conferences; and briefing policy-makers and program officials Possible uses for

these findings include informing federal, state, and local policy-making; training and technical

assistance efforts; as well as local Head Start program practice. ACF will archive data for secondary

use.

Priority Question 3:

What is the evidence of effectiveness of specific programs or services (e.g., mental health, substance

abuse prevention and treatment, and in-home parent skill-based programs and services; as well as

kinship navigator programs) in the domains of child safety, child permanency, child well-being and/or

adult well-being?

Significant Evaluations that will address this question in FY22:

Supporting Evidence Building in Child Welfare and Expanding Evidence on Replicable Recovery and

Reunification Interventions for Families (R3) project.

Key Questions:

What is the effectiveness of specific interventions for the child welfare population, especially programs and

services programs or services in the following areas: mental health prevention and treatment services;

substance abuse prevention and treatment services; in-home parent skill-based programs; kinship

navigator programs; and programs for transition-age youth?

Background and Significance:

Over the past several decades, research and evaluation in child welfare have increased significantly.

This large body of knowledge has shown that child maltreatment is a complex problem and associated

with multiple, interrelated risk and protective factors at individual, family, community, and contextual

levels. This research has also demonstrated that child abuse and neglect may have long-lasting and

cumulative effects on the well-being of children into adulthood. However, much still remains unknown

about why child maltreatment incidence may vary over time, across types of child abuse and neglect,

and across states or localities; the interplay of risk factors, protective factors, and child and family

outcomes; and the evidence of effectiveness for current and ongoing prevention and treatment

practices.

Programs, Policies, Regulations, or Operations to be Analyzed:

ACF provides matching federal funds to states, tribes, and communities to help them operate every

aspect of their child welfare (CW) systems. This includes the prevention of child abuse and neglect, the

support of permanent placements through adoption and subsidized guardianship, and the creation

and maintenance of information systems necessary to support these programs. ACF hopes to increase

the number of evidence-supported interventions for the child welfare population by conducting

rigorous evaluations and supporting the field in moving toward rigorous evaluations.

Stakeholders:

In 2019 ACF issued a public call for child welfare agencies and other interested parties to nominate

programs or services that they would like to be evaluated as part of the Supporting Evidence Building

in Child Welfare project. In 2020, stakeholders including child welfare agencies, parents with lived

experiences, and recovery support organizations were engaged as part of the Expanding Evidence on

Replicable Recovery and Reunification Interventions for Families project to help a) identify existing

family recovery and reunification interventions that use recovery coaches and b) elicit insights and

perspectives about the current state of the recovery coach field and the potential for a future impact

evaluation.

Data Sources:

Child welfare administrative data, interviews, and/or assessments with clients who are receiving

studied interventions and a comparison group of clients who are not getting the studied intervention;

interviews with managers, staff, or clients; and program data collection.

Methods:

For each studied intervention, the Supporting Evidence Building in Child Welfare project will conduct

an impact study to determine if the services provided lead to improved outcomes and an

implementation study to understand the client population, how the services are provided, service

components, and other details. Additionally, Phase I of Expanding Evidence on Replicable Recovery

and Reunification Interventions for Families project includes designing and implementing a feasibility

study to inform an impact evaluation of potentially replicable and scalable family recovery and

reunification interventions that use recovery coaches and are suitable for moving to the next level of

evidence. Phase II of the project is scheduled to begin in FY22, and the plans include conducting a pilot

study; an impact study with multiple follow-ups of participating families over a 5-year period; and an

implementation study.

Anticipated Challenges and Mitigation Strategies:

Challenges: Given the state of the field, it may be challenging to find a sufficient number of programs

ready and willing to participate in random assignment evaluations.

Mitigation Strategies: In 2019, ACF put out a public call and invited child welfare agencies and other

interested parties to nominate programs or services that they would like to be evaluated as part of this

project. In 2020, ACF sponsored an Evidence-Building Academy to increase child welfare

administrators’ and their partners’ capacity to do rigorous evaluations that provide critical information

on program effectiveness and meet the designs standards for clearinghouses reviewing programs and

services relevant to child welfare populations. ACF also conducted a systematic scan and assessed

several recovery coaching interventions for their readiness for replication and evaluation. Following

the assessment, three interventions with different levels of implementation requirements were

prioritized to allow replication by programs with varied capacities. In December 2020, ACF issued a

public call for interest for participation in a feasibility study that potentially comprises the three

interventions and informs a future impact evaluation of replicable and scalable recovery and

reunification interventions that utilize recovery coaches. The feasibility study will explore design

options for the impact evaluation that include analytic strategies capable of detecting effects while

conserving sample size.

Plan for Disseminating Results:

ACF will produce comprehensive research reports as well as shorter documents aimed at policy and

practitioner audiences. ACF will disseminate results through posting reports on the Internet and

journal articles; using social media to alert potential audiences of the availability of results; presenting

results at research, policy, and practitioner conferences; and briefing policy-makers and program

officials. Possible uses for these findings include informing federal, state, and local policy-making. ACF

will archive data for secondary use.

Administration for Community Living (ACL)

Priority Question 1:

What is the efficacy and effectiveness of ACL programs and initiatives?

To answer this priority question, ACL will focus on evidenced-based programs as required under the

Older Americans Act. ACL will conduct a fidelity evaluation to determine the effectiveness of ACL’s

implementation of these programs. While fidelity evaluation typically focusses on five dimensions

(adherence, exposure, quality of delivery, participant responsiveness, and program differentiation ), in

order to improve the effectiveness of ACL’s evidence-based programing the evaluation will focus on

the first three. The evaluation will compare the degree to which ACL funded programs adhere to the

guidelines associated with the evidence-based programs being implemented using the following

evaluation questions:

1. How do sites select appropriate evidence-based programs for their contexts (e.g., resources,

populations)?

2. To what extent do staff overseeing and implementing the programs fully understand and

comply with the implementation guidelines/instructions? For example,

a. Are facilitators trained in accordance with the guidelines of the relevant evidence-based

program(s)?

b. Are program resources in line with the guidelines of the relevant evidence-based

program(s)?

c. Are programs being implemented with the intended populations?

d. Is the amount of training and frequency sufficient?

e. Is the content provided completely and properly per the guidelines of the relevant

evidence-based program(s)?

3. While not recommended, what kinds of adaptations, if any, are grantees/sub-grantees

making to the evidence-based programs? And, why?

4. What can ACL do to support and encourage the proper use and implementation of evidence-

based programs?

Background and Significance:

Authorizing Legislation: Section 361 of the Older Americans Act (OAA) of 1965, as amended states that

“SEC. 361. (a) The Assistant Secretary shall carry out a program for making grants to States under State

plans approved under section 307 to provide evidence-based disease prevention and health

promotion services and information at multipurpose senior centers, at congregate meal sites, through

home delivered meals programs, or at other appropriate sites. In carrying out such program, the

Assistant Secretary shall consult with the Directors of the Centers for Disease Control and Prevention

and the National Institute on Aging. (b) The Assistant Secretary shall, to the extent possible, assure

that services provided by other community organizations and agencies are used to carry out the

provisions of this part.”

While ACL grant officers assure that the funds are being used to provide evidenced-based programs,

this evaluation is significant because it will help ACL ensure that those programs are being

implemented as intended so that older adults realize the full benefits of these programs.

Programs, Policies, Regulations, or Operations to be Analyzed or Evaluated:

The evaluation will focus on ACL’s Evidence-based health promotion programs, as defined in the Older

Americans Act (Title I section 102 (14)(D)) includes programs related to the prevention and mitigation

of the effects of chronic disease (including osteoporosis, hypertension, obesity, diabetes, and

cardiovascular disease), alcohol and substance misuse reduction, smoking cessation, weight loss and

control, stress management, falls prevention, physical activity, and improved nutrition.

Stakeholders:

This work was informed by consultation with ACL stakeholders and requirements under the Older

Americans Act. Stakeholders are ACL staff, grantees and sub-grantees, older adults and their families

and caregivers, and other members of the aging network.

Data Sources:

Primary data will be collected from staff of ACL-funded evidence-based programs using close-ended

surveys and structured interviews. Secondary data will be collected through structured reviews of the

State Plans that detail the evidence-based programs the grantees will provide as well as how they will

be implemented.

Methods:

A Qualitative methods will be used to code the data extracted from the State plans submitted by ACL

grantees and from interviews conducted with both ACL staff and grantee staff implementing evidence-

based programs. Themes will be identified, and the data will be categorized to provide responses to

the evaluation questions. Coding and theme identification will be conducted by at least two

individuals and compared for agreement. The analysis will triangulate across data sources and

analyses to obtain a comprehensive understanding and synthesis of the results and provide actionable

recommendations.

Anticipated Challenges and Mitigation Strategies:

Potential challenges in conducting this work may be the level of detail available in the State plans, and

getting participation from a representative sample of existing evidence-based programs funded

under the Older Americans Act. To mitigate this, ACL’s Office of Performance and Evaluation will work

closely with program staff and the ACL funded resource center which provides leadership, expert

guidance and resources to promote the value of, and increase access to, evidence-based chronic disease

self-management education programs (CDSME).

Plan for Disseminating Results:

The final deliverable for this fidelity evaluation is a report documenting the information collected and

providing clear, actionable recommendations for ensuring the effective use of evidence-based

programming. Recommendations will address what ACL and its grantees and sub-grantees can do to

improve the selection, implementation, and monitoring of evidence-based programming. The report

will also include tool(s) for use by ACL and its OAA state grantees to assess fidelity after this contract

ends. This report will be shared publicly on ACL’s website and information will be distributed to key

stakeholders, including grantees. Further, this project will create job aids and other technical

assistance materials, and briefings and webinars on the findings will also be presented for a variety of

stakeholders, both internal and external to the agency.

Priority Question 2:

What is the efficacy and effectiveness of ACL programs and initiatives? What are states funded by ACL

doing to enhance and promote aging and disability services and programs? What are the current

avenues for collaboration and coordination between the aging and disability networks that ACL works

with? How can ACL encourage better collaboration and coordination? What new research and

information are being generated, by both ACL and in the field, on aging and disability? How is ACL

implementing it, and how can ACL better implement it?

To answer these priority questions, ACL will conduct a process evaluation of the partnership between

federal, state, tribal and local agencies which. supports the work of those who provide assistance

to all older Americans including American Indian and Alaska Native (AI/AN) Elders and their

families nationwide (the Aging Network). The evaluation will focus on the inputs, activities, and

outputs of the Aging Network. The primary evaluation questions are:

1. How is the Aging Network structured and how does it operate at the local, state, and federal

levels including who the program serves, how it is staffed, and what data are collected about

activities and outcomes?

2. How does the Aging Network use resources to measure and improve the quality of services

available/provided? What is the role of the Network in identifying and responding to emerging

needs?

3. How do the various levels of the Aging Network work together, with whom do they partner,

and how do they work with those programs? For example, do they partner only with

organizations considered to be part of the Aging Network, or outside organizations as well?

4. How does the Aging Network measure successful practices and areas for improvement?

Background and Significance:

The Older Americans Act (OAA) was passed in 1965 with the goal of supporting older Americans and

helping them to live at home and in their communities independently and with dignity for as long as

possible. The OAA supports many programs and services, which as congregate and home-delivered

meals, in-home care, adult day care, caregiver support, elder abuse prevention, health and wellness

programs, transportation, and information and referral. These services are provided through the

national Aging Network, which is comprised of 56 State Units on Aging (SUA), 622 Area Agencies on

Aging (AAA), and more than 260 Title VI Native American aging programs. Additionally, thousands of

community organizations and volunteers support the aging network.

While the Aging Network is extensive, its processes and operations are not fully understood. Basing

policy and justifying expenditures on evidence is increasingly a priority in the federal government.

However, without a thorough understanding of the Aging Network, it is not feasible to accurately plan

to measure the return on the investment of the funds distributed through the network. Yet, the

increasing demands require rigorous and independent assessment of progress, efficiency and

effectiveness to ensure the most productive use of government funds for the best consumer

outcomes.

Programs, Policies, Regulations, or Operations to be Analyzed:

Through program evaluation contracts of Older Americans Act (OAA) programs, ACL seeks increased

understanding of how these programs are structured at the State and local levels and their progress

towards their goals and mission. The information will also aid in program refinement and

continuous improvement. The more productive ACL’s programs, the greater the number of older

adults will have access to a higher quality of life.

Stakeholders:

This work was informed by the needs of Aging Network stakeholders (e.g., advocacy organizations, State

Units on Aging, Area Agencies on Aging, Tribal grantees) to demonstrate the networks’ value to older

adults, funders, and policy makers.

Data Sources:

Data will be collected from:

1. Federal staff and national associations about their understanding of and role in the Aging

Network, interaction with other parts of the Aging Network, inter-organizational relationships,

and use/collection of program data.

2. State Units on Aging about their understanding of and role in the Aging Network, interaction

with other parts of the Aging Network, inter-organizational relationships, state mandates,

main activities, funding sources, feedback to and monitoring of local programs, and

use/collection of program data. It should also include a discussion of unique features of local

programs/structures, specifically interactions with AAAs and Native American aging programs

as well as federal entities.

3. Area Agencies on Aging about their understanding of and role in the Aging Network,

interaction with other parts of the Aging Network, inter-organizational relationships, state

mandates, main activities, funding sources, feedback to and monitoring of local programs, and

use/collection of program data. It should also include a discussion of unique features of local

programs/structures, specifically interactions with SUAs, Native American aging programs, and

federal entities.

4. Native American aging programs about their understanding of and role in the Aging Network,

interaction with other parts of the Aging Network, inter-organizational relationships, Tribal

mandates, main activities, funding sources, feedback to and monitoring of local programs, and

use/collection of program data. It should also include a discussion of unique features of local

programs, specifically interactions with SUAs, AAAs, and federal entities.

5. Other entities identified through the literature review and the other data collection efforts as

important parts of the Aging Network.

Methods:

The evaluation will include four steps:

 Step 1: Review literature to describe what is known of current Aging Network services and

outcomes, and to inform development of the data collection tools.

 Step 2: Design and conduct a web survey with a census of SUAs, AAAs, ADRCs, and tribal aging

agencies. The survey will contain a mix of open- and closed-ended questions to gather details

on the connections, process, and flow of information and funding within the network and any

barriers or gaps within it.

 Step 3: Analyze the survey findings and develop an interactive data visualization tool to

accurately and thoroughly describe network operations and connections. The tool will enable

drill-down for details by agency type, region, and connection, and will be customized based on

input from ACL and the TAP.

 Step 4: Design and conduct focused qualitative interviews with select survey respondents to

capture more in-depth insights on their collaboration across the network, program successes

and implementation barriers.

Data will be collected through a web survey with SUAs, AAAs, ADRCs and tribal aging agencies, in

addition to conducting in-depth interviews with a targeted set of network leaders. The web survey will

be a census of Aging Network agencies, offering ACL a representative and complete view of the

collaboration across agencies and ensuring the findings represent the geographic and cultural variation

in agency practices.

Anticipated Challenges and Mitigation Strategies:

Possible challenges in this work will be around scope of the evaluation and choosing a purposeful

sample as the aging network is vast. Other challenges may lie with access to and availability of

administrative data that allow ACL to determine measures for examining return on investment. The

information collected through this contract is intended to lay a foundation for a possible future

outcome evaluation and/or cost study. ACL will mitigate these challenges through contracting with an

outside organization with a proven track record with similar work.

Plan for Disseminating Results:

Findings from the Return on Investment study will be published on ACL’s website and information may

be shared with key stakeholders in the form of public presentations. In particular, ACL is looking to

produce a suite of highly visual products to be shared widely with stakeholders to better illustrate the

composition of the aging network and how it works.

Agency for Healthcare Research and Quality (AHRQ)

Priority Question 1:

How can AHRQ advance the uptake of healthcare research in healthcare settings? That is, how can

AHRQ advance the dissemination of evidence-based practices and foster their implementation within

care delivery settings?

Sub-question: How can TAKEheart, AHRQ’s Initiative to Increase Use of Cardiac Rehabilitation,

disseminate research-proven strategies for increasing hospitals’ referral and enrollment of

their patients in life-saving CR programs?

Background and Significance:

An important goal of AHRQ is to facilitate implementation of findings from patient centered outcomes

research (PCOR) into health care practice. Accordingly, to help improve cardiac rehabilitation rates,

the American Association of Cardiovascular and Pulmonary Rehabilitation/Million Hearts® Cardiac

Rehabilitation Collaborative developed a Cardiac Rehabilitation Change Package (CRCP) and

established a national goal of 70% participation in CR (up from 20-30%) by 2022 for eligible patients.

AHRQ’s TAKEheart initiative is designed to disseminate and implement broadly the strategies

described in the CRCP to hospitals nationwide to help achieve this goal.

Programs, Policies, Regulations, or Operations to be Analyzed:

TAKEheart, AHRQ’s Initiative to Increase Use of Cardiac Rehabilitation, uses Training, Awareness,

Knowledge and Engagement to help hospitals and health systems implement evidence-based

strategies to improve CR referral, enrollment, and retention.

Stakeholders:

Hospitals and healthcare systems, cardiologists, cardiac rehabilitation facilities, care coordinators and

discharge planners, patients, patient families, payers, policy-makers

Data Sources:

To answer each of the TAKEheart Evaluation questions, AHRQ will gather data from multiple

respondent groups as appropriate using both qualitative and quantitative methods. Specifically,

TAKEheart project leaders will collect data from Partner Hospitals and Learning Community hospitals

throughout the implementation process, including during TAKEheart registration and program on-

boarding, the keeping of a multi-part Implementation Log tool, Partner Hospital Action Plans, training

participation statistics, online surveys, interviews, and Partner Hospital-submitted intervention data,

among other data.

Methods:

To assess the implementation and effectiveness of these efforts, AHRQ will conduct an evaluation with

three key objectives, each of which includes a number of evaluation research questions. This mixed-

methods evaluation will use a hybrid implementation-effectiveness design, addressing the specific goal

of “testing [] an implementation strategy while observing and gathering information on the clinical

intervention’s impact on relevant outcomes,” including process outcomes such as referral to CR. The

evaluation is grounded in the RE-AIM framework, which addresses all five phases of the project: reach,

efficacy, adoption, implementation, and maintenance. Evaluation Objectives 1 and 2 are expressed in

general terms and mostly apply to both Partner Hospitals and Learning Community member hospitals;

questions under Objective 3 apply to Partner Hospitals.

 Objective 1: Assess the effectiveness of the TAKEheart program on engaging hospitals in

efforts to improve access to cardiac rehabilitation, increasing their staffs’ awareness of

evidence-based interventions, and increasing their understanding of how to implement new

practices.

 Objective 2: Assess the impact of TAKEheart on participating hospitals’ activities intended to

improve referral, enrollment and retention.

 Objective 3: Measure changes in CR referral, enrollment, and retention at Partner Hospitals.

Anticipated Challenges and Mitigation Strategies:

There could be a risk related to recruiting and enrolling participating hospitals due to the disruption of

normal healthcare delivery practices during the coronavirus pandemic. Should this occur, AHRQ could

adapt the timeline for Partner Hospitals’ participation to ensure sufficient numbers and modify the

plans for Learning Community hospitals to include topics related to CR referral, enrollment, and

participation during periods when CR activities are modified in response to the pandemic.

Plan for Disseminating Results:

AHRQ will disseminate findings and results with the aim of addressing both practice-based and

research/academic audiences through Web content, social media outreach, journal articles,

conference presentations, and targeted outreach to stakeholder groups (including the American

Association of Cardiovascular and Pulmonary Rehabilitation and their national meeting participants)

and policy-makers.

Office of the Assistant Secretary for Planning and Evaluation (ASPE)

Priority Question 1:

Which analyses conducted and/or coordinated within ASPE support development of evidence used in

response to COVID-19?

Background:

The emergence of coronavirus disease 2019 (COVID-19) has surfaced as an immediate and pressing

public health crisis. In response to the COVID-19 pandemic, the HHS Secretary has mobilized all

available HHS resources to combat the significant health threat posed by this virus.

COVID-19 is a novel human disease caused by a naturally arising, coronavirus, the severe acute

respiratory syndrome coronavirus 2 (SARS-CoV-2). The ease with which the virus spreads, coupled with

the ability of asymptomatic people to propagate transmission of the virus, has caused possibly the

most severe worldwide infectious disease pandemic of the modern age. Over 100,000 Americans died

from COVID-19 within the first four months of the pandemic, contributing to over 350,000 deaths

occurring worldwide in the same time period.

Evaluation or Analysis Activity:

Assess via analysis of administrative and survey data and policy analysis to:

1. Provide descriptive statistics, modeling, and quantitative and qualitative analyses of changes

related to COVID-19 and associated health outcomes.

2. Identify unmet data resource/tool and coordinating needs.

3. Identify best practices.

4. Provide recommendations on prioritization of new data resources, tools, and coordination

needs.

Programs, Policies, Regulations, or Operations to be Analyzed:

Analysis conducted or coordinated by ASPE to address COVID-19.

Stakeholders:

The initiatives outlined in this evaluation plan are designed in direct support of ASPE’s primary

stakeholders – the HHS Secretary and leadership team – in making the best possible evidence-based

policy decisions to advance progress for addressing COVID-19. In addition to ASPE stakeholders,

information may be solicited from Operating and Staff Division within HHS to better understand how

ASPE coordinated activities have supported the HHS Secretaries priorities on COVID-19.

Data Sources:

 Administrative data:

o Health care claims from relevant agencies, such as the Centers for Medicare &

Medicaid Services (CMS)

 Survey data

 Key informant interviews

Methods:

ASPE will utilize both quantitative and qualitative data collection and analysis strategies to inform the

development of policy research and analysis in support of COVID-19, including policy analysis, analyses

of survey and administrative data, modeling, key informant interview and other qualitative analyses,

and HHS-wide coordination.

Anticipated Challenges and Mitigation Strategies:

Given the expansive development of data resources and tools across the HHS and the on-going nature

of the COVID-19 pandemic, it may be challenging to identify all data sources and tools used for

evaluative and assessment purposes which were utilized to advance the HHS Secretaries priority for

addressing COVID-19. Data quality issues may be difficult to identify given that, in many cases the

primary use of the data and the reason for its creation, may not have been intended for evaluative

purposes. Response rate to the inventory and assessment review may fail to capture feedback from all

relevant ASPE stakeholders.

Mitigating these challenges will require a coordinated, transparent, collaborative process with all

relevant stakeholders. Given the potential complications related to the scope of this effort, analyses and

recommendations will focus on data resources, tools and needs which are likely to be of benefit across

ASPE or those ASPE units who are largely responsible for supporting COVID-19 efforts. Greater emphasis

will also be placed on determining recommendations critical for enhancing evidence building capacity to

aid decision making by HHS leadership for addressing the COVID-19 pandemic.

Plan for Disseminating Results:

As analyses are completed, they will be shared with ASPE leadership for review and comment. Key

findings will be summarized and shared with HHS leadership. As practicable, evaluation findings and

similar reports will be shared with other relevant stakeholders and will typically be published on the

ASPE website.

Priority Question 2:

How can ASPE’s analyses related to the opioid epidemic best support the Secretary’s decision- making to reduce

the overdose death rate and related morbidities?

Background:

The HHS Secretary has identified several priority areas, which includes the need to address the opioid

epidemic.

1. HHS has made addressing the opioid epidemic a top priority and is committed to decreasing opioid

overdoses, overall overdose mortality, and prevalence of opioid use disorder. Priority areas for

action include: Better addiction prevention, treatment, and recovery services

2. Better data

3. Better pain management

4. Better targeting of overdose reversal drugs

5. Better research

Evaluation or Analysis Activity:

Assess via analysis of administrative and survey data and policy analysis to:

1) Provide descriptive statistics, modeling, and quantitative and qualitative analyses of changes

related to improving morbidity and mortality related to opioid use disorder.

2) Identify unmet research and coordinating needs.

3) Identify, synthesize, and disseminate best practices.

4) Provide recommendations on prioritization of new data resources, tools, and coordination needs.

Programs, Policies, Regulations, or Operations to be Analyzed:

Analysis conducted or coordinated by ASPE to address the opioid crisis.

Stakeholders:

The initiatives outlined in this evaluation plan are designed in direct support of ASPE’s primary

stakeholders – the HHS Secretary and leadership team – in making the best possible evidence-based

policy decisions to advance progress for addressing opioid use and misuse. In addition to ASPE

stakeholders, information may be solicited from Operating and Staff Division within HHS to better

understand how ASPE coordinated activities have supported the HHS Secretaries priorities on the use

and misuse of opioids.

Data Sources:

 National Center for Health Statistics, National Vital Statistics System, Mortality File (Centers for

Disease Control and Prevention (CDC))

 Drug sales (ie. IQVIA)

 Administrative data:

o Health care claims (CMS)

 Survey data, such as:

o National Survey on Drug Use and Health (Substance Abuse and Mental Health Services

Administration (SAMHSA))

o National Survey on Substance Abuse Treatment Services (SAMHSA)

o Youth Risk Behavior Surveillance System (CDC)

 Key informant interviews

Methods:

ASPE will utilize both quantitative and qualitative data collection and analysis strategies to inform the

policy research and analysis in support of reducing the impacts of the opioid epidemic, including policy

analysis, analyses of survey and administrative data, modeling, key informant interview and other

qualitative analyses, and HHS-wide coordination bodies such as the Behavioral Health Coordinating

Council.

Anticipated Challenges and Mitigation Strategies:

Given the expansive development of data resources and tools across the HHS it may be challenging to

identify all data sources and tools used for evaluative and assessment purposes which were utilized to

advance the HHS Secretaries priority for addressing the opioid crisis. Data quality issues may be

difficult to identify given that, in many cases the primary use of the data and the reason for its

creation, may not have been intended for evaluative purposes. Response rate to the inventory and

assessment review may fail to capture feedback from all relevant ASPE stakeholders.

Mitigating these challenges will require a coordinated, transparent, collaborative process with all

relevant stakeholders. Given the potential complications related to the scope of this effort, analyses

and recommendations will focus on data resources, tools and needs which are likely to be of benefit

across ASPE. Greater emphasis will also be placed on determining recommendations critical for

enhancing evidence building capacity to aid decision making by HHS leadership for addressing the

opioid crisis.

Plan for Disseminating Results:

As analyses are completed, they will be shared with ASPE leadership for review and comment. Key

findings will be summarized and shared with HHS leadership. As practicable, evaluation findings and

similar reports will be shared with other relevant stakeholders and will typically be published on the

ASPE website.

Priority Question 3:

How can HHS programs and policies improve the economic and social well-being of Americans?

High priority questions that may be addressed depending on staffing and funding include:

 What are best practices in reducing federal silos at the local level and how can this inform

national economic mobility strategies?

 How do federal economic mobility programs work to together to promote, or hinder,

participant success? What program combinations are most successful in achieving economic

mobility and which may be inadvertently working against each other? How does this vary by

demographics?

 What is the comparative effectiveness of different programs and interventions in achieving

equity and economic mobility for vulnerable subpopulations?

 What improvements can be made to the safety net to better prepare for and respond to

significant multi-state events such as public health emergencies? What can be done to help

programs better evaluate actions taken in these situations?

 How are employers engaging with human services to better support their employees’ ability to

work?

 What emerging economic recovery strategies appear to be most successful and for whom?

What do states and local communities need from the federal government to best implement

them? Which strategies are best at addressing the compounding inequities introduced by the

pandemic?

 What role can HHS human services programs play in improving the effectiveness of COVID-19

vaccination among vulnerable communities and populations?

Background:

The HHS Secretary has identified several priority areas, which includes the need to address promote

economic resiliency.

HHS is committed to supporting the economic and social well-being of Americans, through fostering

opportunities and addressing social and economic challenges, especially for those individuals and

populations at high risk of adversity. Factors impacting economic mobility and well-being include

employment, child care and early education, involvement with the justice system, health and

substance use challenges, age, and disability. Further, economic and social well-being are influenced

by factors including access to healthy food, educational opportunities and outcomes, housing access,

and employment opportunities, and are also linked to health outcomes.

Evaluation or Analysis Activity:

Assess via analysis of administrative and survey data and policy analysis to:

1) Provide descriptive statistics, modeling, and quantitative and qualitative analyses to document

major new changes in the economy and the workforce that influence the success of human

services programs and policies.

2) Promote improved coordination of federal human services programs and data including through

leadership of the new Interagency Council on Economic Mobility and the Interagency Working

Group on Youth Programs.

3) Identify unmet needs for research and coordination.

4) Identify, synthesize, and disseminate results.

5) Provide recommendations on prioritization of new data resources, tools, and coordination needs.

Programs, Policies, Regulations, or Operations to be Analyzed:

Analysis conducted or coordinated by ASPE to promote economic resiliency, including human services

programs administered by the Administration for Children and Families, and other federal programs

that promote economic mobility and social wellbeing.

Stakeholders:

The initiatives outlined in this evaluation plan are designed in direct support of ASPE’s primary

stakeholders – the HHS Secretary and leadership team – in making the best possible evidence-based

policy decisions to advance progress for supporting economic resiliency. In addition to ASPE

stakeholders, information may be solicited from Operating and Staff Division within HHS and other

federal, state, and local partners, to better understand how ASPE coordinated activities have

supported the HHS Secretaries priority for enhancing economic resiliency.

Data Sources:

 Transfer Income Model, version 3 (TRIM3)

 Administrative data from:

o Administration for Children and Families (ACF)

o Administration for Community Living (ACL)

o Health Resources and Services Administration (HRSA)

o Indian Health Service (IHS)

o Office of the Assistant Secretary for Health (OASH)

o Substance Abuse and Mental Health Services Administration (SAMHSA)

o Centers for Disease Control and Prevention (CDC)

 Survey data:

o Current Population Survey

 Key informant interviews

Methods:

ASPE will utilize both quantitative and qualitative data collection and analysis strategies to inform the

development of policy research and analysis in support of economic resilience and well-being,

included policy analysis, analyses of survey and administrative data, modeling, key informant interview

and other qualitative analyses, and cross-program coordination including leadership of the new

Interagency Council on Economic Mobility and the longstanding Interagency Working Group on Youth

Programs.

Anticipated Challenges and Mitigation Strategies:

Strengthening the economic and social well-being of Americans requires the coordination and

engagement of divisions across HHS and the federal government, focusing on different areas of

economic resilience. Given that there are nearly 100 or more federal programs devoted to this goal, it

is challenging to either isolate the role of a specific program or policy, as well as to determine how

programs or policies interact to produce outcomes. To begin to mitigate this challenge, HHS is leading

a new interagency Council on Economic Mobility that will begin to set the stage for how to identify,

measure, and compare common outcomes across federal programs; and will continue to promote

common outcomes across youth-serving programs through leadership of the Interagency Working

Group on Youth Programs.

A second challenge is the rapidly changing nature of the economy and the workforce as a result of the

COVID-19 pandemic. Interventions, programs, or policies that proved successful or not in the previous

economy with very low unemployment and few risks of communicable disease may or may not

produce the same results now. To mitigate this challenge we will use modeling and other lessons from

past recessions and public health events to inform expectations and predictions about how to address

current challenges, as well as continually updating data assumptions, keeping in close contact with

stakeholders in the field. ASPE’s Transfer Income Model, version 3 (TRIM3) is an important tool. This

will also help address challenges inherent in using administrative and survey data, including time lag,

accuracy, and underreporting.

Plan for Disseminating Results:

As analyses are completed, they will be shared with ASPE leadership for review and comment. Key

findings will be summarized and shared with HHS leadership. As practicable, evaluation findings and

similar reports will be shared with other relevant stakeholders and will typically be published on the

ASPE website.

Priority Question 4:

How can ASPE’s analyses of drug development, regulation, manufacturing, marketing, distribution, and

markets improve HHS policymaking to support lower prices of prescription drugs?

Background:

Drugs and devices are integral to health care delivery, including the federal response to emerging and

reemerging infectious diseases; and HHS supports basic and applied research and provides regulatory

oversight, clinical guidelines, and insurance coverage and payment. However, HHS is not the only

player in these arenas. A better understanding of the incentives at each stage of development,

manufacturing and delivery, including pricing and insurance coverage, helps HHS to develop policies to

align incentives to ensure more efficient use of resources to meet HHS and national priorities.

Evaluation or Analysis Activity:

Assess via analysis of administrative and survey data and policy analysis to:

1) Provide descriptive statistics, modeling, and quantitative and qualitative analyses of enacted or

proposed changes to address the cost of drugs.

2) Identify unmet research and coordinating needs.

3) Identify best practices.

4) Provide recommendations on prioritization of new data resources, tools, and coordination needs.

Programs, Policies, Regulations, or Operations to be Analyzed:

Analysis conducted or coordinated by ASPE to address drug pricing issues and concerns.

Stakeholders:

The initiatives outlined in this evaluation plan are designed in direct support of ASPE’s primary

stakeholders – the HHS Secretary and leadership team – in making the best possible evidence-based

policy decisions to advance progress for addressing drug pricing. In addition to ASPE stakeholders,

information may be solicited from Operating and Staff Division within HHS to better understand how

ASPE coordinated activities have supported the HHS Secretaries priority on drug pricing.

Data Sources:

 Commercial and retail drug sales (IQVIA, Nielson retail data)

 Administrative data:

o Health care claims (CMS)

o Drug approval and drug production processes (FDA)

o Basic research and drug development data (NIH)

 Key informant interviews

Methods:

ASPE will utilize both quantitative and qualitative data collection and analysis strategies to inform the

development of policy research and analysis in support of drug pricing, including policy analysis,

analyses of cost data, modeling, qualitative analyses as needed, and relevant cross-agency

coordination.

Anticipated Challenges and Mitigation Strategies:

Drug and device markets are incredibly complex. Understanding the markets in order to improve HHS

policy will require coordination with FDA, NIH and other HHS agencies that play a role within the drug

market. Given the expansive development of data resources and tools across the HHS it may be

challenging to identify all data sources and tools used for evaluative and assessment purposes which

were utilized to advance the HHS Secretaries priority for addressing drug pricing. Data quality issues

may be difficult to identify given that, in many cases the primary use of the data and the reason for its

creation, may not have been intended for evaluative purposes. Response rate to the inventory and

assessment review may fail to capture feedback from all relevant ASPE stakeholders.

Mitigating these challenges will require a coordinated, transparent, collaborative process with all

relevant stakeholders. Given the potential complications related to the scope of this effort, analyses

and recommendations will focus on data resources, tools and needs which are likely to be of benefit

across ASPE or of the greatest benefit to those ASPE units which have a significant role in addressing

drug pricing. Greater emphasis will also be placed on determining recommendations critical for

enhancing evidence building capacity to aid decision making by HHS leadership for addressing drug

pricing.

Plan for Disseminating Results:

As analyses are completed, they will be shared with ASPE leadership for review and comment. Key

findings will be summarized and shared with HHS leadership. As practicable, evaluation findings and

similar reports will be shared with other relevant stakeholders and will typically be published on the

ASPE website.

Priority Question 5:

Value-Based Transformation: How can ASPE’s health policy analyses best support the Secretary’s

decision-making to transform our health care system into a patient-centered system that rewards high

quality care and addresses the patients’ full array of needs?

Background:

The HHS Secretary has identified several priority areas, which includes value-based transformation.

HHS has embraced value-based transformation as a strategy to achieve greater value for the nation’s

health dollar, as measured by quality outcomes and cost of care. While approaches to value-based

transformation vary, the general policy objectives are to move away from fee-for- service payments,

to pay for health care quality versus quantity of services provided, and to incentivize the provision of

patient-centered, coordinated care.

Evaluation or Analysis Activity:

Assess via analysis of administrative and survey data and policy analysis to:

1) Provide descriptive statistics, modeling, and quantitative and qualitative analyses of health care

system transformations designed to better improve the provisions of patient-centered,

coordinated care.

2) Identify unmet research and coordinating needs.

3) Identify best practices.

4) Provide recommendations on prioritization of new data resources, tools, and coordination needs.

Programs, Policies, Regulations, or Operations to be Analyzed:

Analysis conducted or coordinated by ASPE that support value-based transformation.

Stakeholders:

The initiatives outlined in this evaluation plan are designed in direct support of ASPE’s primary

stakeholders – the HHS Secretary and leadership team – in making the best possible evidence-based

policy decisions to advance progress on valued-based transformation. In addition to ASPE

stakeholders, information may be solicited from Operating and Staff Division within HHS to better

understand how ASPE coordinated activities have supported the HHS Secretaries priorities on this

priority issue.

Data Sources:

 Administrative data:

o Health care claims (CMS)

o Electronic health record data

 Sales and pricing data

 Survey data

 Key informant interviews 

Methods:

ASPE will utilize both quantitative and qualitative data collection and analysis strategies to inform the

development of policy research and analysis in support of value-based transformation, including policy

analysis, analyses of survey and administrative data, modeling, key informant interviews and other

qualitative analyses, and cross-program coordination, among others.

Anticipated Challenges and Mitigation Strategies:

Given the expansive development of data resources and tools across the HHS it may be challenging to

identify all data sources and tools used for evaluative and assessment purposes which were utilized to

advance the HHS Secretaries priority on value-based transformation. Data quality issues may be

difficult to identify given that, in many cases the primary use of the data and the reason for its

creation, may not have been intended for evaluative purposes. Response rate to the inventory and

assessment review may fail to capture feedback from all relevant ASPE stakeholders.

Mitigating these challenges will require a coordinated, transparent, collaborative process with all

relevant stakeholders. Given the potential complications related to the scope of this effort, analyses

and recommendations will focus on data resources, tools and needs which are likely to be of benefit

across ASPE or those units within ASPE which play significant role in value-based transformation

efforts. Greater emphasis will also be placed on determining recommendations critical for enhancing

evidence building capacity to aid decision making by HHS leadership on approaches for addressing

value-based transformation.

Plan for Disseminating Results:

As analyses are completed, they will be shared with ASPE leadership for review and comment. Key

findings will be summarized and shared with HHS leadership. As practicable, evaluation findings and

similar reports will be shared with other relevant stakeholders and will typically be published on the

ASPE website.

Centers for Disease Control and Prevention (CDC)

CDC commits to address health threats wherever they occur and save American lives by securing

global health, ensuring domestic preparedness, ending epidemics, and eliminating disease. CDC’s

strategy cascades from an ambitious aspiration to granular action plans and detailed measures of

success, with its foundational scientific work vital to its overall mission. Protecting America’s health

requires continuous improvement, and as a science-based, data-driven agency, CDC is well positioned

to strategically evaluate programs, use data to improve and protect the public’s health, eliminate

health disparities, and advance health equity.

CDC’s Draft Evaluation Plan consists of four focus areas aligned to CDC’s Strategic Priorities and Core

Capabilities, which are also reflected in other HHS strategic plans.

 Assuring Core Capabilities: Public Health Data Modernization

 Securing Global Health and America’s Preparedness: Pandemics

 Ending Epidemics:

1) Opioid Overdose Data to Action Cross-Site Evaluation

2) Emerging Substances

 Eliminating Disease: HIV/AIDS

This evaluation plan provides a framework for key questions and priority activities. CDC uses a

prospective evidence-building approach to innovate, test, evaluate and model strategies in order to

identify those that are most impactful, cost-effective, and feasible for achieving our public health

goals. As additional evidence is generated, some of these questions and approaches may shift. By

continuously building and assessing the evidence, CDC is better positioned to optimize our impact and

strategically drive informed decisions. This prospective generation of key evidence and ongoing data

evaluation is critical for data-driven policymaking.

Priority Question 1:

Assuring Core Capabilities: Public Health Data Modernization. How effective is CDC’s 10-year Public

Health Data Modernization Initiative (PHDMI) in enhancing the ability to characterize and predict

public health threats, make rapid data-driven public health decisions, and advance proven public

health prevention strategies, interventions, and solutions?

Sub-Questions: In the first-year plan, the evaluation questions will assess short-term outcomes that

focus on improving data and IT systems rather than on longer-term health impacts that depend on

these improvements.

 How has developing, adopting or implementing data standards improved the interoperability,

linkability, accessibility and usability of CDC’s and its partners’ data systems?

 How has developing and implementing enterprise-wide data and IT governance improved the

efficiency and effectiveness of CDC’s data and data systems?

Background and Significance:

CDC’s mission includes providing health information to protect against and respond to dangerous

health threats. In 2014, CDC embarked on a strategy to bring the agency’s public health data

surveillance capabilities into the 21st century. As the agency made progress, CDC determined that

modernization of all public health data and systems was needed and created the broader PHDMI. The

ultimate goal of PHDMI is to enhance public health surveillance and research and, thus, decision-

making to improve public health.

Programs, Policies, Regulations, or Operations to be Analyzed or Evaluated:

How has developing, adopting or implementing data standards improved the interoperability,

linkability, accessibility and usability of CDC’s and its partners’ data systems?

 Data standards can facilitate the rapid bi-directional transfer of data between different systems

and enable systems to be linked on common, standardized data elements. Data standards can

also make it easier to access, aggregate and report data from different systems to get a more

comprehensive picture of health (e.g., disease prevalence in different states) without the need for

time-consuming manual harmonization of disparate data elements representing the same

semantic concepts coming from different systems. The impact of data and IT systems adopting

and implementing data standards on interoperability, linkability, accessibility and usability can be

assessed.

How has developing and implementing enterprise-wide data and IT governance improved the

efficiency and effectiveness of CDC’s data and data systems?

 CDC’s IT and Data Governance Board (ITDG) has developed evaluation criteria for CDC new data

and IT systems investments as well as major modifications or cost increases to existing systems.

ITDG recommendations and funding decisions are documented in written memos, which can be

reviewed for evaluation purposes.

Stakeholders:

Stakeholders may include the business owners and customers of CDC’s and its partners’ data and IT

systems, including, but not limited to, healthcare providers, electronic health records and IT vendors,

standards development organizations, patients, STLT public health agencies and other federal

partners. In addition, different CDC programs support and use these data and systems.

Data Sources:

Potential data sources include the specific public health data and IT systems targeted for

modernization. Additional details will be provided once PHDMI investments have been finalized.

Methods:

The specific methods employed to answer year 1 questions have not yet been determined and will be

developed as part of year 1 activities.

Anticipated Challenges and Mitigation Strategies:

Optimizing data – to better characterize public health threats in order to implement quicker public

health responses and more effective interventions – has been an evolving process that has depended

on stakeholder engagement and good governance as well as technology solutions. However, the

COVID-19 pandemic has brought into sharp focus the need to greatly accelerate data modernization to

mount an effective public health response. The new Coronavirus Aid, Relief and Economic Security

(CARES) Act funding will greatly expedite CDC's ability to be successful in modernizing our public health

data and IT systems, through assessing and integrating innovations in data and IT standards and

technologies, and adopting best practices in data and IT stewardship. With these new resources, we

will intensify our focus on coordinating effectively with our public health and healthcare partners and

stakeholders who supply CDC with and use our public health data. The new resources CDC has

received for data modernization will be critical to mitigating historical and current challenges.

Plan for Disseminating Results:

The evaluation findings will be shared with the programs that are evaluated, as well as with CDC

leadership. The results will also be shared with relevant external stakeholders and partners. Additional

details of the dissemination plan will be developed once PHDMI investments are finalized.

Priority Question 2:

Securing Global Health and America’s Preparedness. Global and Domestic Preparedness: Pandemic

Contagions. How best can CDC improve readiness to characterize, mitigate and respond to pandemics,

including influenza, coronavirus, and other viruses with pandemic potential?

Sub-questions:

 How can CDC’s surveillance systems best provide information to monitor the spread and

intensity of pandemic disease in the US, promote understanding of disease severity and

monitor for virus changes?

 How can CDC increase capacity to produce data for forecasting spread and impact of

pandemics?

 What are new, effective ways for CDC to better monitor and evaluate vaccine safety and

effectiveness?

 What are new, effective ways for CDC to improve vaccine coverage?

 How can CDC increase capacity to characterize viruses and develop and evaluate candidate

vaccine viruses (CVVs)?

 What evidence and lessons that will be learned from the development and dissemination of a

pandemic vaccine for to improve future response plans?

Background and Significance:

Global pandemics have the potential to destabilize the world, a fact made acutely clear with the

current COVID-19 pandemic. Broadly, preparing for new and emerging pathogens with pandemic

potential requires continual engagement with USG agencies and public health partners including state

and local health department for review and enhancement of pandemic plans, exercises, response

capabilities, and resources. For this plan, however, CDC will focus on building the surveillance,

modeling and vaccine planning tools that are necessary for our agency’s efforts to characterize,

mitigate and respond to pandemic threats.

CDC relies on timely public health surveillance data to guide public health actions and inform response

to pandemics. Likewise, CDC and partners utilize models to inform pandemic responses, including

predicting the potential trajectory of virus spread at the national and state level, and projections

related to the potential impact of mitigation strategies on both health outcomes and associated

resource utilization. Vaccines are an important tool to mitigate the effects of a pandemic, and focused

efforts are needed to ensure that vaccines can be developed and manufactured quickly, are maximally

safe and effective, and reach the intended populations.

Programs, Policies, Regulations, or Operations to be Analyzed or Evaluated:

CDC’s surveillance plan is based on a combination of existing surveillance systems for influenza and

viral respiratory diseases syndromic data, case based and commercial lab reporting, ongoing and new

research platforms employed for COVID-19, and other systems designed to answer specific public

health questions. These systems will be used to produce data to understand the overall impact and

epidemic characteristics to inform future use of public health and medical resources. The COVID-19

pandemic is an opportunity to make positive and permanent changes to our methods of data

collection and integration. To improve data on vaccine effectiveness (VE), CDC will continue to use

existing networks to study VE in inpatient and outpatient settings. As CDC continues its strong record

of monitoring for vaccine safety, CDC will evaluate data from safety systems and identify new methods

for identifying potential safety concerns. Enhancing and improving robust, near real-time vaccine

safety monitoring systems, is essential to effectively and efficiently monitoring vaccines in the post-

approval period during large-scale vaccination programs in response to public health emergencies

involving infectious disease outbreaks. In order to increase vaccine coverage, CDC will continue to

increase access to vaccines and awareness of the safety and effectiveness of available and newly

developed vaccines. To improve the speed and availability of vaccines to combat a pandemic, CDC will

continue to increase the agency’s capacity to characterize influenza, coronavirus and other viruses to

evaluate CVVs, while also increasing throughput of CVV development and testing.

Stakeholders:

CDC works with public health partners at the state and local level, ministries of health around the

world, and the private sector to improve capacity to track respiratory illness, including those caused by

influenza, coronavirus, and other viruses. CDC engages with vaccine manufacturers and other federal

agencies (including the National Institutes of Health (NIH), the Food and Drug Administration (FDA),

Biomedical Advanced Research and Development Authority (BARDA), Department of Defense (DoD),

Department of State (DoS), and the US Agency for International Development (USAID)) in order to

protect Americans and populations around the world from the threat of a pandemic. Clinicians are

essential as vaccine providers and as trusted advisors in making vaccine recommendations. Hospitals

and academic institutions also play an important part in enrollment for and support of vaccine

effectiveness monitoring at sites across the United States. Academic institutions are also important

partners in the development of models and modeling methods. Other non-governmental partners are

critical to CDC’s efforts to promote the availability, safety and effectiveness of the U.S. vaccine supply

and vaccine recommendations.

Data Sources:

To improve the prevention and control of vaccine preventable diseases, CDC must enhance and

expand its data systems to provide critical information for public and private sector decisions about

disease spread, vaccine innovation, immunization recommendations as well as data on vaccine safety.

CDC employs multiple surveillance systems to understand and monitor disease spread in the US

(including ILINet, NSSP, NNDSS, COVID-NET and others); taken together, these systems create an

ongoing picture of the impact of virus spread and produce data to address the key questions for

directing and refining the US response. These data are also essential to developing models to support

public health preparedness and planning. CDC sponsors vaccine effectiveness networks (including U.S.

Flu VE Network, Hospitalized Adult Influenza Vaccine Effectiveness Network (HAIVEN), New Vaccine

Surveillance Network (NVSN), and SUPERNOVA, the surveillance platform of Veterans Affairs

hospitals); the data produced by these networks is essential to understanding how to best improve

disease prevention through vaccination. CDC monitors vaccine safety reporting systems (including

VAERS, CISA, VSD) to identify potential vaccine safety problems and conducts post-licensure vaccine

safety research through its vaccine safety networks to evaluate and identify potential vaccine safety

problems and further address ongoing vaccine safety concerns. CDC has supported development and

maintenance of state immunization information systems for tracking vaccination, and there are

opportunities for innovation to modernize these systems to achieve greater capabilities for monitoring

child and adult vaccination coverage. CDC’s data systems need to be modernized to be more reliable

and timelier to better track disease and characterize viruses in near-real time. See Assuring Core

Capabilities: Public Health Data Modernization for more information.

Methods:

Surveillance and Modeling:

 Use multiple surveillance systems and epidemiology networks for situational awareness, to

understand impact, forecast disease spread and characterize infection and apply these methods

for future pandemic planning

 Assess models of transmission to assess the potential impact of different intervention strategies

Vaccine Effectiveness:

 Maintain strong capacity to enroll patients in Vaccine Effectiveness systems, including U.S. Flu VE

Network, HAIVEN, NVSN, and SUPERNOVA.

 Partner with existing ICU networks in order to assess VE against influenza–associated PICU and

ICU admissions, as well as those associated with COVID-19 and other respiratory viruses.

Vaccine Safety:

 Expand analysis of data from safety monitoring systems to identify potential adverse events.

Specific methods will be determined

Vaccine Coverage:

 Use immunization information system data to pinpoint areas of low vaccination coverage

 Leverage diverse data sources to identify and protect communities at risk

 Expand resources for working with local communities

Virus Characterization and Vaccine Virus Development:

 Monitor pathogen genetics that may be associated with changes in transmission, disease

severity, diagnostic test design or medical countermeasure design

 Further develop bioinformatics tools to select optimal antigenically advanced viruses for CVV

development and analysis

 Improve synthetic genomics and reverse genetics systems to engineer optimal CVVs

 Develop state-of-the-art technologies and expand existing strategies to conduct virus

neutralization assays

Anticipated Challenges and Mitigation Strategies:

CDC’s programs have been crucial to advances in preparing for and addressing pandemics through

prevention, control, vaccination and treatment. Looking forward, this work will require:

 More broadly effective vaccines that can be made more quickly. The global infrastructure to

produce and distribute pandemic vaccines also must be improved

 Better worldwide human and zoonotic surveillance of viruses with pandemic potential, including

shared specimens and data to be used in vaccine development

 Modernized public health data systems to produce timely, complete, and accurate data for

decision makers without burdening providers and local health departments

 Improved ability for CDC and State Health Departments to access and integrate data sources (i.e.,

epidemiology and laboratory information, vaccine coverage, impact on healthcare systems) to

improve federal, state, and local decision making

 More complete case data; data may be incomplete both in terms of the numbers of cases

reported and the data reported for each case

 Bioinformatics tools to select optimal viruses for CVV development and analysis

 Enhanced systems for monitoring vaccine safety and effectiveness

 Making strides in advance molecular detection and expanding genomics activities

 Rapid development and safety testing for CVV

 Ongoing commitment and communication on vaccine safety, along with robust vaccine safety

monitoring systems, to maintain public confidence and trust in vaccination

The current COVID-19 pandemic may impact each of these strategies and activities due to changes in

health-seeking behavior, vaccine acceptance and other factors.

Plan for Disseminating Results:

While specific dissemination plans are still to be determined, CDC will utilize journal publications and

relationships with existing stakeholders (including public health departments, clinicians, vaccine

manufacturers and other US Government agencies) to share data and recommendations to improve

decision making. CDC will also focus on sharing data and information with the general public through

its website and media engagement.

Priority Question 3a:

Ending Epidemics: Opioid Overdoses – Overdose Data to Action Cross-Site Evaluation. Among CDC’s

funded activities to reduce opioid overdose and misuse, what strategies, or combination of strategies,

are associated with reducing US drug overdose mortality?

Sub-Question:

How and to what extent are Overdose Data to Action (OD2A) funded recipients using overdose

data to inform prevention activities? Which strategies, or combinations of strategies, appear to

have the desired programmatic effect?

Background and Significance:

The United States is amid a drug overdose epidemic. While the number of drug overdose deaths

decreased by 4% from 2017 to 2018, the number of drug overdose deaths was four times higher in

2018 than in 1999. Nearly 70% of the 67,367 deaths in 2018 involved an opioid (prescription and illicit).

From 2017 to 2018, there were significant changes in opioid-involved death rates: Opioid-involved

death rates decreased by 2%; prescription opioid-involved death rates decreased by 13.5%; and

heroin-involved death rates decreased by 4%.

From 1999–2018, almost 450,000 people died from an overdose involving any opioid, including

prescription and illicit opioids. This rise in opioid overdose deaths can be understood in three distinct

waves. The first wave began with increased prescribing of opioids in the 1990s, with overdose deaths

involving prescription opioids (natural and semi-synthetic opioids and methadone) increasing since at

least 1999. The second wave began in 2010, with rapid increases in overdose deaths involving heroin.

The third wave began in 2013, with significant increases in overdose deaths involving synthetic

opioids, particularly those involving illicitly manufactured fentanyl.

The market for illicitly

manufactured fentanyl continues to change, and it can be found in combination with heroin,

counterfeit pills, and cocaine.

Multidisciplinary collaboration is essential for success in preventing opioid overdose deaths. Medical

personnel, emergency departments, first responders, public safety officials, behavioral health and

substance use treatment providers, community-based organizations, public health, and members of

the community all bring awareness, resources, and expertise to address this complex and fast-moving

epidemic. CDC is fighting the opioid overdose epidemic by supporting states and communities as they

work to identify outbreaks, collect data, respond to overdoses, implement effective public health

strategies, and connect people to care and treatment.

Programs, Policies, Regulations, or Operations to be Analyzed or Evaluated:

The Overdose Data to Action (OD2A) cooperative agreement funds 66 health departments – 47 state

health departments, 16 city or county health departments, and 3 district and/or U.S. territories over a

three-year period beginning in September 2019. OD2A focuses on the complex and changing nature of

the drug overdose epidemic and highlights the need for an interdisciplinary, comprehensive, and

cohesive public health approach. Funds awarded as part of this agreement will support state,

territorial, county, and city health departments in obtaining high quality, more comprehensive, and

timelier data on overdose morbidity and mortality and using those data to inform prevention and

response efforts.

Stakeholders:

For the OD2A cross-site evaluation, the stakeholders are:

 Funded and unfunded state, county, and local health department staff members working in

prevention, surveillance, program implementation and management/leadership

 Practitioners working in non-profit or public sector on harm reduction, substance abuse

prevention and treatment

 Researchers and evaluators

 Public health and public safety partners

 CDC staff

Data Sources:

The evaluation will use existing program and secondary data and some primary data collection in the

form of interviews and focus groups to address gaps and add context.

Methods:

A mixed method, quasi-experimental design is being used to assess program implementation progress

and measure and explain OD2A’s impact on several key short-, medium-, and long-term outcomes.

Qualitative data will be analyzed using thematic analysis to identify patterns and trends among

recipients and their implemented activities. Quantitative data will be analyzed using descriptive

statistics and trend analyses. A mixed-methods approach will also be taken to synthesize and

triangulate data to gain a deeper understanding of OD2A activities.

Anticipated Challenges and Mitigation Strategies:

One challenge for the OD2A cross-site evaluation could be the variability in state and jurisdictional

capacity to collect and report high quality standardized data, especially where the demands of COVID-

19 impact data collection and reporting. CDC will mitigate this challenge by providing guidance and

technical assistance on reporting requirements including the type and quality of data to be reported by

funded jurisdictions. CDC scientists will also review data submitted on a quarterly or annual basis and

provided feedback and technical assistance to those jurisdictions who require more assistance to

report.

Plan for Disseminating Results:

The OD2A evaluation will result in a variety of dissemination products created over the three-year

period and will include: two evaluation briefs, a white paper, three annual reports, and a manuscript.

Products will be tailored to specific audiences like public health practitioners, providers, insurers,

and/or CDC staff and leadership.

Priority Question 3b:

Ending Epidemics: Opioids – Emerging Substances. Among CDC’s funded activities to reduce opioid

overdose and misuse, what strategies, or combination of strategies, are associated with reducing US

drug overdose mortality?

Sub-Questions:

 What are risk factors for early use initiation and escalation of use?

 How do risk and protective factors and trajectories for use, misuse, drug use disorder, and

overdose differ among prescription opioids, illicit opioids, and emerging drug trends, such as

poly-substances and stimulants?

 What norms and behaviors impact substance use, misuse, overdose, or related health and

behavioral outcomes? What are the best strategies to communicate about risk for these

outcomes to the public?

 What is the effectiveness of new or innovative prevention approaches for stimulant use and

overdose that have not been evaluated, including those designed to address those at greatest

risk?

 How can we translate and evaluate interventions from other settings and for other outcomes,

such as those for opioid use disorder, to address stimulant use and other emerging drugs?

Background and Significance:

The U.S.’s prescription drug overdose crisis has evolved in recent years into an epidemic involving

overdose deaths from heroin, synthetic opioids, and now, poly-drug use involving both stimulants and

opioids. Risk and protective factors for drug overdose exist at multiple levels of the social ecology. Our

understanding of the risk and protective factors for drug overdose is complicated by the intricate and

evolving nature of the drug overdose epidemic. Changes in the drug supply, mixing of drugs with or

without the knowledge of the person using those drugs, and polysubstance use are factors that

contribute to the complexity of the drug overdose landscape and challenge our ability to identify and

address risk and protective factors for drug use and overdose. Addressing the drug overdose epidemic

will require a better understanding of the unique risk and protective factors for the multiple

trajectories and combinations of drug use, use disorder, and overdose. It will also require a better

understanding of how current evidence-based strategies for opioid use disorder can be adapted or

implemented for treatment of stimulant use disorder and other emerging drugs.

Programs, Policies, Regulations, or Operations to be Analyzed or Evaluated:

This work will be implemented through extramural research grants. In the future, research findings

from this work will help inform CDC’s programmatic work and prevention interventions for emerging

substances in the field. This work and input from stakeholders can help inform our next drug overdose-

related state cooperative agreement. Interventions for which a strong evidence base for

implementation has been established are particularly important to informing our state cooperative

agreements. For example, we know currently that medications for opioid use disorder are effective

treatments. The work outlined here may provide additional insight into what opioid use disorder

treatment strategies can be adapted to stimulant use disorder and other emerging drugs.

Stakeholders:

 Funded and unfunded state, county, and local health department staff members working in

prevention, surveillance, program implementation, and management/leadership

 Practitioners working in non-profit or public sector on harm reduction, substance abuse

prevention, and treatment

 Clinicians and health care providers (especially for the linkage to care priority question)

 Researchers and evaluators

 Public health and public safety partners

 CDC staff

Data Sources:

Data sources for this work will include those gathered by extramural researchers and investigators

through new research grants as well as a variety of sources for intramural research (possibly to include

the National Syndromic Surveillance Program).

Methods:

This work will be implemented through extramural research grants. The specific methods will be

determined by funded researchers.

Anticipated Challenges and Mitigation Strategies:

It is important to ensure that CDC receives relevant and comprehensive applications to conduct

extramural research addressing CDC’s priority question and emerging substances sub-questions for

this evidence-building plan. To support this, CDC will widely disseminate and promote this opportunity

widely.

Plan for Disseminating Results:

The findings from this work will be disseminated through a variety of channels, including:

1. Peer-reviewed journal articles

2. State cooperative agreements

3. Communication efforts such as Division of Overdose Prevention’s website and active

dissemination to stakeholders

4. Tools and resources for NCIPC Division of Overdose Prevention recipients and partners

Other dissemination efforts will likely take place by funded grantees conducting this work. This may

include peer-reviewed journal articles; poster presentations; and talks given at conferences and other

meetings.

Priority Question 4:

Eliminating Disease: HIV/AIDS. How do CDC’s efforts contribute to achieving the goals of the End the

HIV Epidemic (EHE) initiative?

Sub-questions:

 To what extent did the 57 jurisdictions implement all the strategies of EHE?

o To what extent did PS20-2010 funded health departments implement the strategies

described in their PS20-2010 application?

o What were the primary challenges to implementing the strategies by PS20-2010 funded

health departments?

o How can implementing the strategies be facilitated or accelerated?

 What are the differences in outcomes between jurisdictions in these areas?

o HIV testing, early HIV diagnosis, PrEP coverage, linkage to HIV care?

o How do outcomes vary by geographic area and other contextual and individual

characteristics?

 How are HIV-related health disparities being addressed? To what extent are other diseases

(COVID-19) and overlapping epidemics (STDs, opioid and drug use) challenging the

implementation and success of EHE?

Background and Significance:

New HIV infections in the U.S. have been stagnant for the last few years, despite advances in HIV

prevention and control. HHS has embarked on a new initiative called Ending the HIV Epidemic: A Plan

for America, to accelerate implementation of known, effective interventions to prevent and control

new HIV infections—focusing on four key strategies: Diagnose, Treat, Prevent, and Respond. The

priority question focuses on whether these resources can be marshalled effectively to meet these

ambitious new goals.

Programs, Policies, Regulations, or Operations to be Analyzed or Evaluated:

As described above, this evaluation will focus on CDC’s role in the EHE initiative. The initiative aims to

reduce new HIV infections in the U.S. by 90% by 2030. Ending the HIV Epidemic leverages critical

scientific advances in HIV prevention, diagnosis, treatment, and outbreak response by coordinating the

highly successful programs, resources, and infrastructure of many HHS agencies and offices.

Stakeholders:

The HHS Office of the Assistant Secretary for Health (OASH) is coordinating this cross-agency initiative.

CDC is one of several collaborating agencies for this initiative—other agencies include Health

Resources and Services Administration (HRSA), Indian Health Service (IHS), National Institutes of

Health (NIH), OASH, Substance Abuse and Mental Health Services Administration (SAMHSA).

Stakeholders will also include the Presidential Advisory Council on HIV/AIDS (PACHA), the CDC/HRSA

Advisory Committee on HIV, Viral Hepatitis and STD Prevention and Treatment (CHAC), the National

Alliance of State and Territorial AIDS Directors (NASTAD), the National Association of County and City

Health Officials (NACHHO), the Association of State and Territorial Health Officials (ASTHO), as well as

state and local health departments, community-based organizations, and other HIV/AIDS

organizations.

Community involvement is crucial to the EHE initiative and remains a requirement to develop the EHE

plans. CDC leadership traveled the country to hear input from communities and foster partnerships

critical to the success of the initiative. New EHE funds enable jurisdictions to accelerate and continue

this community engagement and partnership work. CDC Director Dr. Redfield, personally visited 38 of

the 57 EHE jurisdictions. Collectively, in 2019, EHE leadership from across HHS visited nearly all of the

57 jurisdictions for phase one of the initiative.

Data Sources:

Sub-questions:

 To what extent did the 57 jurisdictions implement all of the strategies of EHE?

o Data Sources: PS20-2010 EPMP, Annual Progress Reports, End of Year Reports

 How much do key outcomes vary in priority areas for EHE?

o Data Sources:

o National HIV Surveillance System (NHSS) = HIV diagnosis, early HIV diagnosis,

PrEP coverage, viral suppression

o NHSS = linkage to HIV care

o National HIV Prevention Program Monitoring & Evaluation (NHM&E) = Key

Programmatic Cross-jurisdiction measures (TBD)

o Evaluation and Performance Measurement Plan (EPMP), Annual Progress

Reports to provide context for programmatic outcomes

 How are HIV-related disparities being addressed?

o Data Sources:

o NHSS = HIV diagnosis, early HIV diagnosis, PrEP coverage, viral suppression

o NHSS = linkage to HIV care

o NHM&E = Demographic characteristics for key population groups

 To what extent are other diseases (COVID-19) and overlapping epidemics (STDs, opioid and

drug use) challenging the implementation and success of EHE?

o Data Source: Joint Monitoring Team calls, Annual Progress Reports, End of Year

Reports

Methods:

CDC will assess the progress towards EHE overall goals by using the data collected for the six EHE core

indicators. Since most activities supported by CDC’s main EHE implementation NOFO will ultimately

feed into these indicators, programmatic outcomes will be monitored and evaluated based on selected

key cross-jurisdictional and jurisdiction-specific measures. Contextual information from the EPMP and

progress reports will be assessed to monitor the programmatic outcomes and performance of PS20-

2010 funded health departments.

Anticipated Challenges and Mitigation Strategies:

The six core indicators are population based and will reflect the collective efforts of all the HHS agencies

and other agencies funded to implement EHE. These national indicators will not solely measure CDC’s

contribution. While the indicators will tell us what is happening, they won’t tell us why which can be

very important to inform best practices in implementation. That is a challenge which stakeholders may

not understand when viewing results on the EHE dashboard.

Given the need to implement and make accessible HIV prevention and care services in new and/or

resource scarce areas, we can anticipate challenges to include but not limited to availability of viable

organizations to support provision of required services, staff recruitment and training, transportation

and accessibility, coordination and collaboration between organizations and health departments and

data collection and information sharing.

For monitoring and evaluating the programmatic outcomes and performance, there may be questions

related to indicator definitions, variations in implementing the activities across jurisdictions, and other

local contextual factors. CDC will provide ongoing technical assistance through Joint Monitoring Team

calls.

Plan for Disseminating Results:

As described above, HHS will develop the AHEAD website which will provide communities, decision

makers, stakeholders, jurisdictions, and the federal government with the data necessary to make

programmatic decisions on EHE progress. The dashboard will include CDC-reported data on six core

indicators. It will promote transparency in data sharing and help to monitor movement towards the EHE

goal of ending the HIV epidemic in the U.S. by 2030. This site will also serve as a primary site to share

information about EHE progress, interventions, and success stories. CDC will also provide success

stories and progress with the EHE jurisdictions on the CDC EHE website.

Centers for Medicare and Medicaid Services (CMS)

Priority Questions:

How can CMS improve the quality and affordability of health care for all Americans? How does CMS

drive American health care towards payment for value, not volume? How can CMS lower the rate of

growth in America’s health care spending?

Focus Area 1:

Ensuring safety and quality – This effort addresses CMS goals of improving the quality and safety of

health care for all Americans by implementing the Meaningful Measures initiative; this initiative

anchors quality measurement and reporting across care settings and drives towards better health

outcomes in the populations served. Within the Meaningful Measures framework, CMS has and is

developing cost measures to include as a part of a drive toward value.

Background and Significance:

CMS, through its multiple programs and levers, is leading the transition to value based care, which

includes a focus on highest quality at lowest cost. CMS has multiple quality reporting and payment

programs to measure and incentivize organizations to provide high quality, and at the same time make

this information transparent in public reporting that provides consumers with information to make

best health care choices and therefore focus on high value care. Another domain of measurement is

cost and efficiency. Finally, CMS promotes price transparency, as well as the development of

innovative payment models that are discussed in focus area 3. Quality health care is a high priority for

CMS.

CMS implements many quality initiatives, including statutorily mandated quality reporting and value-

based purchasing programs, to assure value (cost and quality) through accountability, patient and

stakeholder engagement, and transparency. The CMS value based incentive programs utilize quality

measurement across multiple sites of care, from hospitals, to ambulatory care, to post-acute care, and

in specialty areas such as kidney disease, to drive improvements in quality. The measures used in these

programs are selected utilizing the framework of CMS’ Meaningful Measures initiative, which focuses

on specific domains, and supports transparency, burden reduction, and a transition to digital measures

and increased patient reported care. Measures and programs are evaluated at least annually through

extensive stakeholder feedback, technical expert panels, and evaluation of trends as further detailed

below.

In 2017, CMS developed the Meaningful Measures framework to focus on strategic priorities, identify

gap areas, align measures, and ensure high impact of measures used in these value based programs

which promote accountability. CMS regularly evaluates its current measure set in the various

programs to identify gaps where new measures are needed and, remove measures that are no longer

necessary.

An example of the Meaningful Measures in action is the 2018 CMS National Assessment of Quality

Measures Reports, statutorily required every three years under section 1890A(a)(6) of the Social

Security Act. In this report, the CMS Quality Priorities, which align with the Meaningful Measure

domains, were used to frame the report, and high performing measures called Key Indicators, or

groups of measures, were used to evaluate the quality and efficiency impact of the use of measures,

including patient impact and cost-avoided analyses, national surveys, measure performance trends,

and disparity analyses.

Programs, Policies, Regulations, or Operations to be Analyzed or Evaluated:

Meaningful Measures

Stakeholders:

The Meaningful Measures Initiative was developed collaboratively with the input from a wide variety

of stakeholders that include patients, clinicians, providers, and specialists. It also draws on prior work

performed by the: Health Care Payment Learning and Action Network, other Agencies, the National

Quality Forum, and the National Academies of Medicine. It includes perspectives from patient

representatives, clinicians and providers, measure developers, and other experts such as the Core

Quality Measures Collaborative. These extensive collaborations and stakeholder outreach provide

ongoing evaluation and feedback on quality measures and programs in an iterative feedback loop.

Additionally, CMS engages in a wide variety of education and listening sessions on topics related to

quality. The Measures Management Website , (https://www.cms.gov/Medicare/Quality-Initiatives-

Patient-Assessment-Instruments/MMS/MMS-Content-Page) averages over 50,000 visits a year, with

over half being first time users each month, due to the continued stakeholder engagement through

the Measures Management System (MMS). CMS holds public webinars, where sessions are frequently

attended by thousands of participants, plus an additional 1,840 views on the CMS YouTube page. CMS

circulates MMS newsletters to provide updates on new, innovative enhancements to the systems,

upcoming webinars, opportunities for Technical Evaluation Panels (TEP), calls for public comment, and

other measure-related content to over 90,000 recipients. And finally, CMS hosts 8 compare sites

where data for Meaningful Measures is displayed by setting, allowing patients and advocates the

ability to make an educated choice for their health care needs which ensures the quality and

affordability for all Americans is achieved. Significant stakeholder input is sought for the development

and testing of these sites.

Data Sources:

Data for these meaningful measures are derived from a variety of data sources including data from

electronic health records, claims, registries, surveys, and charts. CMS has also developed an evaluation

tool for quality measures - the Quality Measure Index (QMI), which is in beta testing, and is envisioned

as a transparent and reliable scoring instrument for quality measures based on standardized

definitions of quantifiable measure characteristics. Capable of producing repeatable results yet

adaptable to evolving priorities, the index provides capabilities that are unique among current

assessment tools used in decision-making for evaluating an individual quality measure. To date, Merit-

based Incentive Payment System (MIPS) measures are undergoing evaluation using the QMI with

recommendations to follow.

CMS is promoting digital quality measures, with the goal of fully digital measures. CMS is continuing to

work on developing more Application Program Interfaces (API) for quality data submission,

incentivizing use of interoperable electronic registries, harmonizing measures across registries,

working to give more timely and actionable feedback to providers, and working across the agency on

the use of artificial intelligence to predict outcomes. Working towards these goals will require a shift

towards digital measurement, a top priority for the Meaningful Measures Initiative moving forward.

Digital measures will lead to an expansion of quality data available for evaluation derived directly from

the electronic medical record.

Methods:

CMS measures health care quality in many domains including health outcomes, important clinical

processes, patient safety, efficient use of resources, health care costs, care coordination, patient and

consumer engagement, population and public health, and adherence to clinical guidelines. CMS’s

Meaningful Measures Initiative unites strategic efforts to reduce the burden of quality measure

reporting with a rigorous and comprehensive approach to identify and adopt measures that are the

most critical to providing high quality care and driving better patient outcomes at lower costs.

Specifically, CMS is actively working to encourage the use of parsimonious measure sets, develop more

timely feedback reports on performance based on data, and to further prioritize more all- payer,

patient-centric, population-based outcome measures, which allow a more targeted approach to

determining if this work leads to better outcomes at lower costs. As an initial step, with the support of

government contractors and federal stakeholders, CMS is prioritizing the development and use of

electronic clinical quality measures, improved electronic infrastructure, harmonized measures across

public and private quality reporting, and targeted efforts to address rural health concerns, health

inequities, population health, and patient reported outcomes in order to make better comparisons

across the healthcare ecosystem and over time.

CMS assesses the utility of specific measures for potential use in quality reporting programs and

selects measures that are meaningful to patients, actionable for providers with minimal burden of

implementation, and likely to improve health outcomes. All measures are evaluated using a Pre-

Rulemaking and Rulemaking multi-stakeholder review to solicit feedback. In addition, most all

measures used in CMS programs go through the process for National Quality Form (NQF)

endorsement, a rigorous and well established process to evaluate the impact, feasibility, validity, and

reliability of measures, including evaluation by the Scientific Methods Panel, a group of highly skilled

statisticians who are experts in measure evaluation. Measures to be used by CMS in programs also

must be evaluated by the Measure Application Partnership, a consensus based group of experts,

convened through the National Quality Forum, to evaluate and make recommendations to CMS on the

appropriateness of using measures in its programs. Finally, as noted earlier, all measures are

developed using technical expert panels who are skilled in their respective areas, to provide feedback

during development.

The impact of health and value of this program is the statutorily mandated submission of a triennial

Impact Assessment Report to Congress. CMS employs a comprehensive methodology to evaluate the

quality and efficiency impact of the use of endorsed measures in CMS reporting programs, including

patient impact and cost- avoided analyses, national surveys of quality leaders in hospitals and nursing

homes, and measure performance trends and disparity analyses. As defined in this report, Key

Indicators are measures or groups of measures used to gauge performance on aspects of six

Meaningful Measure priority domains: Patient Safety, Person and Family Engagement, Care

Coordination, Effective Treatment, Healthy Living, and Affordable Care. In the future, CMS is also

developing our analytics infrastructure to further evaluate the impact of each quality measure and

program.

Anticipated Challenges and Mitigation Strategies:

As part of CMS’s Meaningful Measures work, more focus is placed on new measure development from

Electronic Health Records (EHRs), such as electronic Clinical Quality Measures (eCQMs). CMS currently

uses many eCQMs in its various programs but the data are not publicly reported. When evaluating any

program or area, it will be important to be aware of these other measures since data may not be

publicly available for evaluation efforts at this time, but may be in the future.

Plan for Disseminating Results:

Every 3 years CMS is required by law to assess the impact of using endorsed quality measures in the

programs and initiatives we administer and to post the results. The quality measures are assessed over

time using traditional time series analysis. Subgroup analyses are conducted, including the use of

demographic and provider characteristics to determine if there are disparities in improvement based

upon any factors. The 2018 Impact Assessment Report organized measure analyses under our 6 Quality

Strategy priorities, which fully align with the Health Care Quality Priorities in the Meaningful Measures

framework.

In addition, every year CMS is required by law to publish Measure Development Plan Annual Reports

which provide updates on the status of previously identified gaps to support measure development

and progress made in the development of quality measures to support the Quality Payment Program.

These annual reports and related measure environmental scans and gap analysis work required by

section 102 of the Medicare Access and Children’s Health Insurance Program (CHIP) Reauthorization

Act of 2015 (MACRA) support the CMS Quality Measure Development Plan which is a strategic

framework to help develop and improve clinician-level quality measures, point out the known

measurement and performance gaps, and recommend prioritized approaches to close those gaps.

In addition to the Report to Congress, CMS presents these works at numerous quality conferences

nationally, including the CMS Quality Conference held annualy in February-March.

Focus Area 2:

The opioid epidemic – This effort addresses CMS’ goals of improving the quality of health care for all

Americans by expanding access to evidence-based opioid use disorder treatment to Medicaid

beneficiaries. This program has the potential to improve the quality care, reduce death, and reduce

costs related to opioid deaths and complications.

Background and Significance:

Substance use disorders (SUD) impact the lives of millions of Americans including individuals enrolled in the

Medicaid program. They occur when the recurrent use of alcohol and/or drugs causes clinically significant

impairment, including health problems, disability, and failure to meet major responsibilities at work,

school, and/or home. An estimated 2.1 million people had an opioid use disorder (OUD), which includes 1.7

million people with a prescription pain reliever use disorder and 0.7 million people with a heroin use

disorder. Increasing Medicaid provider capacity is essential to expanding access to SUD treatment for

Medicaid beneficiaries. The need for SUD providers with expertise and skills to provide SUD prevention,

early detection, and whole-person treatment and recovery services is evident.

The Substance Use-Disorder Prevention that Promotes Opioid Recovery and Treatment for Patients

and Communities (SUPPORT) Act requires CMS, in consultation with SAMHSA and AHRQ, to conduct a

54-month demonstration project to increase the treatment capacity of Medicaid providers to deliver

substance use disorder (SUD) treatment and recovery services. The demonstration project is

comprised of two phases: an 18-month planning grants awarded to at least 10 states ($50 million

aggregate); and a 36-month demonstrations with up to 5 states that received planning grants.

In September 2019, CMS awarded approximately $48.5 million to 15 state Medicaid agencies. The

planning grants were awarded as cooperative agreements, which will support greater engagement of

grantees by CMS throughout the period of performance.

Programs, Policies, Regulations, or Operations to be Analyzed:

Fighting the Opioid Epidemic

Stakeholders:

CMS consulted with the Assistant Secretary for Mental Health and Substance Use (SAMHSA) and the

Director of the Agency for Healthcare Research and Quality (AHRQ) as appropriate, in developing the

demonstration program. Prior to the planning grant application deadline, CMS hosted two

teleconferences or webinars for states to provide details about the Demonstration Project and to

answer questions. A third specific webinar was conducted for the 37 states with AI/AN populations,

Indian health care providers, and AI/AN Medicaid beneficiaries.

Data Sources:

Applicants must provide a plan for gathering and timely reporting data to CMS, including Quarterly

and Annual Progress Reporting and data related to milestones. CMS expects to use this information in

developing its three statutorily required reports: an initial report; an interim report; and a final report.

In addition, AHRQ may use this information in developing its statutorily required report to Congress on

the experiences of states awarded planning grants and of states conducting demonstration projects.

Methods:

CMS expects to award one support contract for evaluation. Planning grantees will be required to work

with the national evaluation contractor and participate in all evaluation activities including the

collection of data and reporting of grant activities. This includes completion of a semi-annual CMS

report detailing implementation progress, challenges, barriers, solutions, outputs and outcomes.

States may also choose to conduct their own independent evaluation to assist in the establishment of

a formative learning process and/or to serve as the interface between the grant applicant and the CMS

national evaluation contractor. The grant applicant and its evaluation contractor (if the grant applicant

chooses to engage one) will be required to cooperate with CMS and the national evaluation

contractor, including in the use of standard data sets such as the Transformed Medicaid Statistical

Information System (T-MSIS). The national evaluation will include an analysis of the impact, outcomes,

and processes of the grant program, including identifying the elements that were critical to successful

expansion of SUD treatment and recovery service capacity, as well as the barriers that impede states’

success.

The evaluation will include both qualitative and quantitative methods to (1) assess whether the

demonstration achieved its stated goals, (2) describe the strengths and limitations of the

demonstration project, and (3) propose a plan for the sustainability of the project. The evaluation will

also address the impact of the states’ initiatives to expand the number, capacity, and willingness of

providers to deliver SUD treatment and recovery services. In addition, the evaluation will assess the

impact of the states’ initiatives on the number of individuals enrolled in Medicaid who receive SUD

treatment or recovery services under the demonstration and on the outcomes of Medicaid

beneficiaries with SUD.

The states’ results will be assessed related to: (1) impact, benefits, barriers, and outcomes; (2)

evidence of improved coordination and efficiencies in the SUD treatment and recovery system; (3)

experiences of the beneficiaries and providers; and (4) SUD treatment and recovery services access

and utilization.

Anticipated Challenges and Mitigation Strategies:

While the demonstration focuses on increasing the treatment capacity of providers participating under

the state Medicaid program to provide SUD treatment or recovery services, we recognize that Medicaid

providers’ willingness to treat individuals with SUD may also be a barrier to treatment. There are many

providers who could provide treatment for SUD but may be unwilling to do so. A 2017 ASPE Report

indicates that, although providers recognize the dangers of the opioid epidemic, many providers do not

recognize OUD in their patient population, have significant misunderstanding about treating SUD, and

may believe MAT is a replacement addiction treatment rather than a treatment.

In addition, there is geographic variability in the training, qualifications, and licensure of SUD

treatment providers. States and localities have varying requirements with respect to clinical staffing in

SUD treatment programs. SUD licenses and certificates tend to be state-specific. States may also have

additional specific continuing education course requirements for credentialing and licensing. The

implications of the geographic variability can limit the ability to determine the impact of SUD

treatment on outcome and any analysis will take this into account.

Plan for Disseminating Results:

SAMHSA and AHRQ will consult with CMS on three statutorily required reports to Congress: an initial

report; an interim report; and a final report. In addition, AHRQ, in consultation with CMS, will submit

to Congress a statutorily required summary of experiences of states awarded planning grants and of

states conducting demonstration projects. All reports will be made public.

Focus Area 3:

Innovative payment models – This effort addresses CMS’ goal of driving the American health system

toward paying for value instead of volume.

Background and Significance:

CMS is addressing this priority through our payment models targeting the transition to value-based

care. CMS is committed to using our program authorities to create innovative payment models that

move our healthcare system towards value by rewarding quality, innovation, and improved health

outcomes. Under these models, providers can accept higher levels of risk, and also new financial

arrangements to allow them to enter into value-based agreements and be accountable for patients’

outcomes, rather than merely volume of services. We believe that the quality of health care

individuals receive should not simply be defined by the amount of services an individual receives but

instead should reflect the value and benefit patients receive from their health care experiences.

Programs, Policies, Regulations, or Operations to be Analyzed:

Innovative Payment Models

Stakeholders:

Innovative payment models impact the entire health care delivery system and all payers. Therefore,

the stakeholder community is broad. While not necessarily an exhaustive list, stakeholders include

Medicare and Medicaid beneficiaries and their advocates; CMS leadership and staff and other Federal

agencies; Congress; health plan representatives; all providers, practitioners, and allied health

professionals; health and legal policy experts; professional societies; and state officials and other

insurers.

Data Sources:

Evaluations of innovative payment models use multiple data sources including existing data such as

Medicare and Medicaid claims and encounter data, plan bids and other CMS plan and provider data.

Additionally, evaluations may also include the collection of new primary data such as site visits to

participants, key stakeholder interviews, and interviews with or surveys of beneficiaries, and

caregivers as appropriate. The evaluation will also leverage program data collected for purposes of

running the model that includes reported care delivery functioning.

Methods:

Evaluations of innovative payment models use a mixed-methods approach to assess both impact and

implementation experience. The impact component measures the degree to which the model

improved key outcomes for the target beneficiaries including lower total cost and improved quality of

care. The typical method is to use multivariate regression techniques to assess the change over time in

the outcome of interest for the model participants relative to a well-constructed comparison group

not exposed to the model intervention. The implementation component will describe how the model

was implemented, assessing barriers and facilitators to change. A synthesis of these two components

will provide insight into what worked and why, as well as how best to scale, if successful.

Anticipated Challenges and Mitigation Strategies:

These evaluations often need to contend with a number of challenges. First, it is common for a model

to include multiple components which may require separate impact and implementation designs for

each component. Second, because these models are implemented in a real life situation they generally

include few formal requirements and are very flexible. The evaluation will need to include robust

primary data collection to be able to adequately describe how the model was implemented and what

factors contribute to success. Finally, without random assignment, establishing a rigorous and fair

counterfactual is challenging. The evaluations use state-of-the-art quasi-experimental techniques,

including propensity score matching or weighting and difference-in- differences analysis design.

Plan for Disseminating Results:

The results will be disseminated utilizing multiple approaches including reports, manuscripts,

webinars, and conference presentations. At a minimum, the official evaluation report will be posted

on the CMS web page along with Findings at a Glance and CMS Perspective documents. The Findings

at a Glance is a 2-page document that provides the high level findings suitable for policy makers. The

CMS Perspective provides an insight of how CMS is applying the evaluation findings to the current

model and to any current or future related efforts. Webinars may be used to present findings to model

participants and other stakeholders. Findings may also be presented at annual research conferences

by CMS staff and/or the CMS contractors conducting the evaluation.

Food and Drug Administration (FDA)

Priority Question 1:

To what extent has The SUPPORT (Substance Use-Disorder Prevention that Promotes Opioid Recovery

and Treatment [SUPPORT] for Patients and Communities) Act enabled improvements to the

International Mail Facilities (IMFs) and advanced efforts to stop illegal and unsafe drugs from being

imported into the United States?

Background and Significance:

Mail entering the U.S. from abroad arrives at a U.S. Postal Service (USPS) sorting facility (International

Mail Facility or IMF). Upon initial screening at arrival, USPS sends packages to U.S. Customs and Border

Protection (CBP) for examination. CBP then refers FDA-regulated products to the FDA for entry into

the regulatory process and or a criminal referral. There are currently nine IMF locations across the

U.S., with one location in Florida, Hawaii, Illinois, New Jersey, New York, Puerto Rico, U.S. Virgin Islands

respectively and two locations in California.

At the IMFs, the FDA provides front line defense against illegal, illicit, unapproved, counterfeit and

potentially dangerous drugs and other violative FDA-regulated products from entering the U.S. via

international mail. The FDA's integrated team composed of civil investigators, special agents, and

forensic scientists are responsible for monitoring mail importations of FDA-regulated products by

conducting comprehensive examinations of packages that have been referred by CBP. The FDA

examines and determines admissibility of packages containing FDA-regulated products that are

imported through the IMF. If a package is found or suspected to contain a controlled substance during

the FDA's screening, it may be referred to CBP for an admissibility determination. In fiscal year 2017,

the IMFs received an estimated 275 million packages. Of these, the FDA estimated that approximately

9% of them contained drugs of some kind. In FY19, the FDA screened approximately 45,000 packages

after increasing staffing to full capacity at the IMFs. Previously, the FDA was inspecting between

10,000-20,000 packages annually. With additional staffing and physical resources, we hope to increase

that number to 100,000 packages per year.

The SUPPORT Act was enacted on October 24, 2018. This new law grants FDA additional import

authorities that FDA believes will meaningfully advance efforts to stop illegal and unsafe drugs from

being imported into the United States. For example, the SUPPORT Act includes:

 Improvements to the International Mail Facilities (IMFs);

 Authority to treat an FDA-regulated article as a drug if that article contains an active ingredient

that is found in an FDA-approved drug or licensed biologic, and the ingredient presents a

significant public health concern;

 Authority for FDA to debar people who have been convicted of a felony involving illegal

importation of drugs or controlled substances, or who have engaged in a pattern of illegally

importing controlled substances or certain adulterated or misbranded drugs; and

 Authority for FDA to treat any imported drugs as illegal from a person who has engaged in a

pattern of importing adulterated or misbranded drugs if the shipments are from the same

manufacturer, distributor, or importer.

Evaluation or Analysis Activity:

FDA will evaluate the extent to which the SUPPORT Act has enabled improvements to the

International Mail Facilities (IMFs) and advanced efforts to stop illegal and unsafe drugs from being

imported into the United States.

 This new law grants FDA additional import authorities that FDA believes will meaningfully

advance efforts to stop illegal and unsafe drugs from being imported into the United States

 Improvements to the International Mail Facilities (IMFs) where FDA conducts its mission

related work.

Stakeholders:

Stakeholder engagement:

 Internal and External Communication plans, including engagement with Industry, Partner

Government Agencies, and Trade Associations.

 Internal and External Government Working Groups, for example the Commercial Customs

Operations Advisory Committee (COAC) and Center Work Groups.

Impacts:

 International Trade Community

 Industry, especially those trying to protect their brand

 American public consumers

 American public health system (for example: care providers, hospitals, other care facilities)

Data Sources:

 United State Postal Service (USPS)-GBS System

 U.S. Customs and Border Protection Systems

 FDA/ORA IT data sources (for example, OASIS, SERIO)

 Health Fraud and OCI databases 

Methods:

A mixed method data collection approach will be used to capture and assess if the SUPPORT Act has

achieved its intended outcomes. FDA will rely most heavily on quantitative data collection and

analysis, including the identification and analysis of outcome measures to assess short-, medium-, and

long-term effectiveness. When necessary, qualitative data will be collected and analyzed to provide

additional context to effectiveness measures, to assess SUPPORT Act implementation consistency, and

to identify organizational efficiencies after SUPPORT Act implementation. Quantitative outcome

measures will, in the short-term, be baselined and analyzed using descriptive statistics. Over the long-

term, quantitative outcome measures and qualitative data analysis will be used to identify trends and

patterns for continued improvements.

Anticipated Challenges and Mitigation Strategies:

Data Sharing: Currently, USPS, CBP, and FDA use their own data systems that maybe redundant or

going into independent, duplicative systems to track packages. The data in these systems store

information about where the mail originated, mail contents and destination information. FDA is

working with USPS and CBP on a unified, one-system approach.

Plan for Disseminating Results:

Results will be communicated to external stakeholders:

 Internal and External Communication plans for stakeholders, including Industry, Partner

Government Agencies, and Trade Associations. Will use external outlets including Web Sites,

Press Releases, and Outreach Events, and Social Media.

 Internal and External Government Working Groups, for example the Commercial Customs

Operations Advisory Committee (COAC) and Center Work Groups.

Indian Health Service (IHS)

Priority Question 1:

To what extent did the Tribal Injury Prevention Cooperative Agreement Program (TIPCAP) grantees

meet their goals and objectives?

Background:

The purpose of this IHS cooperative agreement is to address the disparity in injury rates by

encouraging Tribes to implement focused, community-based injury prevention programs and projects

using evidence-based strategies. The IHS Injury Prevention Program (IPP) categorizes injuries by intent

and type. Unintentional injury types are falls, burns, drowning, poisoning, and motor vehicle related

injuries. Intentional injury types are suicide and violence related injuries, and are a leading cause of

death in Indian country.

Evaluation or Analysis Activity:

 Analysis of semi-annual grantee reports – both program implementation and data collection.

Data collection includes both primary and secondary (via local partners) data related to

injuries.

 Analysis of grantee survey related to how current implementation compares to both each

individual grantee’s goals and overall IPP goals.

 Technical assistance is provided to Tribes to assist the analyzing their own program data. 

Programs, Policies, Regulations, or Operations to be Analyzed:

The IHS TIPCAP, operating since 1997, enables tribes to address injuries in communities. In 2017, a

summary report titled 20 Years of TIPCAP was generated. The current 5-year funding cycle ends in 2020.

Stakeholders:

At the local/Tribal level, program partners’ sources of data become key stakeholders. As such, each

grantee varies the focus of the type(s) of injury they may focus on. For example, for a seat belt

program, partners might include: police, emergency departments, health departments, tribal leaders,

or any related coalition.

Data Sources:

 Primary data related to the specific evidence-based program implemented to address the

focus of the local program.

 Primary and secondary data sources include implementation level and health outcome data.

Methods:

Utilize quantitative and qualitative methods. Need to have increased efforts to develop more standard

data to be collected across grantees via drop-down lists for evidence-based programs appropriate for

grantee foci and sub-components. Technical assistance is also provided to assure accurate data is

collected and reported.

Anticipated Challenges and Mitigation Strategies:

There are inherent limitations to self-reported data. There are also limitations to access secondary

data for specific types of injuries. Finally, the various foci among grantees limits the ability to compare

and aggregate data across entire cohorts.

By requiring grantees to have an evaluation plan in place before program implementation, programs

are able to at least look back and compare expectations versus actual results regarding both evidence-

based implementation and program results. Program usage and injury rates are also analyzed for

trends over time. Currently the program is exploring avenues to compare grantee data to state or

national data and formalizing local secondary data access. At the end of the grant cycle final reports

might not be submitted due to various reasons. The contracted technical monitor will provide report

assistance to grantees to address challenges related to final report completion and submittal.

Plan for Disseminating Results:

Technical assistance is provided to Tribes to assist the analyzing their own data. Summaries, per

program type, are disseminated across sites. Local dissemination plans are developed as part of each

grantee’s work plan before implementation. These identify destinations and follow a one- page profile

of each progress and areas still in need. These reports are part of each grantees’ annual report, and

shared within IHS, Area Officers, and other (federal/local) partners. These summaries also identify

challenging implementation or data collection issues. Program addresses these challenges during

either grant continuation applications and/or future Notice of Funding Opportunities.

Priority Question 2:

How could telehealth reframe access to care and quality of care within IHS? Identify the real and

potential costs and benefits of telehealth within IHS.

Background:

The Indian Health Service (IHS) has increased efforts to expand access to care through the integration

of telemedicine with community-based services. Maximizing the use of modern technology in support

of healthcare services facilitates high-quality, secure, and interoperable health information technology

(IT) systems; promotes recruitment and retention of healthcare professionals; and increases access to

care in remote and isolated locations and reduces patient transportation costs by making services

available through telehealth. Increasing the number of telehealth encounters will serve to expand and

increase delivery of health care services to meet the current need and demand due to the COVID-19

pandemic.

Evaluation or Analysis Activity:

Evaluate the adoption of the Telehealth Program, examine the success of telehealth campaign and

promotion. Evaluate patient and provider experience.

Programs, Policies, Regulations, or Operations to be Analyzed:

1) Conflicting guidelines for prescribing controlled substances via telehealth across state lines.

2) Access barriers to In-Person care continue to exist.

3) Identify accessibility of high speed internet to allow for telehealth services.

4) Analysis of Extension for Community healthcare Outcomes (ECHO) programs that provide

telehealth support to increase clinical capacity through case-based learning.

5) Analysis of non-clinical barriers, i.e., medication acquisition and linkage of patients to care,

shifting with new clinical guidelines, insurer requirements and presence of ancillary programs

such as medication-assisted therapy for opioid use.

6) Analysis needed to determine whether telehealth sessions have had a measurable impact on

navigating these barriers.

7) Impact on facility third party revenue by utilizing telehealth services.

Stakeholders:

All IHS Areas that implement the Telehealth Program, IHS headquarters, IHS Health care facilities,

Office of Information Technology (OIT) with IHS departments within the IHS.

Data Sources:

 The percent of healthcare facilities that offer telehealth services within each of the twelve IHS

Areas that report into the National Data Warehouse (NDW).

 The number service codes (representing the service types) offered through telehealth,

Specialty care access.

 The percent of telehealth visits among AI/AN who received greater than 1 visit within one year

from date of initial visit.

 Provider recruitment, and retention and cost savings for providers using telehealth, hazard pay,

premium pay and other recruitment and retention incentives may be eliminated if providers

can work from ADS. We would review staffing levels and shortages over time.

 Data to be extracted from the Indian Health Service electronic health records NDW will allow a

completed interpretation of the data to focus on the specific sites that submit data.

 Chart data could review: No show rates, retention rates, consultation clinics over period of

time.

 Primary data collection from patients and providers. 

Methods:

 Mixed methods will be used, combining quantitative data collection as well as qualitative

methods like focus groups and key informant interviews.

 Number of visits that support several IHS GPRA measures including the IHS Mental Health.

GPRA measure which provides the percent of youth ages 12-17 screened for depression and

the percent of adults ages 18 and over screened for depression.

 The percent of patients screened who received a follow-up visit with a Behavioral Health

provider code.

 Percent of outpatient services provided for tele behavioral Health over all behavioral health

clinical settings (expected increase by 10% post FY 2021 baseline year).

Anticipated Challenges and Mitigation Strategies:

Prior to COVID-19, IHS offered limited tele-behavioral services to address disparities experienced

related to suicide, substance abuse (including opioid misuse and methamphetamine abuse), intimate

partner violence, trauma (including Historical Trauma), and Fetal Alcohol Spectrum Disorders (FASD).

Despite this obvious need, access to behavioral health services is often limited due to a number of

reasons, including difficulty recruiting qualified providers, lack of housing, lack of employment

opportunities for partners, and difficulties retaining staff.

Due to COVID-19 and the expansion of telehealth services, IHS has increased efforts to accurately

capture telehealth visits. This work is consistently improving our data collection efforts, and we

anticipate preliminary data analysis will allow us to compare the number of patient visits and the

number of hours of service.

Plan for Disseminating Results:

Evaluation results to be disseminated to health care facilities which implement the telehealth

program, to health care providers and IHS Area offices. Disseminating results will enable further

expansion of program as well as program improvement.

National Institutes of Health (NIH)

Priority Question 1:

How can NIH strengthen its capacity to assess* its research programs?

(*The words “assess” and “assessment” in this plan refer to evaluation and other analytical approaches

that are used to determine the effectiveness of NIH’s programs, policies, and operations.)

Focus Area 1:

Data and Tool Needs

Background:

Accurate, timely, and targeted evidence is essential for informed decision-making about effective and

efficient use of resources. NIH’s capacity to conduct evaluations and other assessments has grown in

the last decade with greater availability of data (primarily in NIH’s main administrative database

IMPACII) and the development of tools to query such data. However, anecdotal information suggests

that the capacity to effectively utilize these tools varies across NIH, and currently available data

resources and tools may not fully address NIH’s evaluation and other assessment needs. Additionally,

it is often unclear within the NIH community which data resources and tools are best suited for

addressing specific types of questions.

Evaluation or Analysis Activity:

Assess the utility of existing NIH data resources and tools for conducting evaluations and other

assessments and identify data resource/tool needs.

Programs, Policies, Regulations, or Operations to be Analyzed:

NIH and its Institutes, Centers, and Offices (ICOs) data resources and tools used for evaluation and

other assessment activities.

Stakeholders:

The NIH Office of the Director as well as NIH’s 27 institutes and centers are all stakeholders in this

effort. NIH staff to be surveyed include those who are:

 responsible for stewardship of data resources

 developing tools to query and analyze data resources

 utilizing existing data resources and tools for evaluative and analytic purposes 

Data Sources:

Data sources include existing documentation on NIH data tools/resources, evaluation final reports,

and qualitative input from NIH Staff.

Methods:

Initial steps involve developing an inventory of existing NIH data resources and tools used for the

purpose of NIH evaluations and other assessments. The data source and tool inventory will be shared

with the ICOs for review and for soliciting additional comment to determine how broadly these

resources and tools are used across the NIH. This review of the inventory will help highlight

the utility of existing NIH data resources and tools for conducting evaluation and other assessments at

the NIH. The information gathered in the assessment will also be utilized to identify best practices on

which tools are best fit for which purposes/assessment questions. Feedback from NIH stakeholders is

also intended to help inform the identification of data quality concerns in existing resources and help

inform the prioritization of new data resource and tool needs. This approach will not only support the

NIH response to requirements in Title I of the Evidence Act, the inventory can also be leveraged by NIH

to help respond to the requirement for a catalog of data repositories in Title II of the Evidence Act.

Second, an environmental scan (spanning no more than 5 years) of technical reports of NIH

evaluations and other assessments, which utilize these data resources and tools, will be conducted in

order to better understand the impacts of evaluations and other assessments derived from their use.

Lastly, qualitative data from interviews with ICO leadership, which are part of the larger NIH Evidence

Building Plan, will serve as additional data to inform which data resources and tools are the best fit for

which purposes/assessment questions.

Anticipated Challenges and Mitigation Strategies:

Given the expansive development of data resources and tools across the NIH, it may be challenging to

identify all data resources and tools used for evaluative purposes at NIH. Identifying evaluations and

other assessments may prove challenging given the sheer breadth of NIH staff who may have initiated

evaluative efforts. Though it is straightforward to identify data quality issues, other than the rare

circumstance in which ground truth is unknown or poorly defined, addressing those issues can be

more challenging, often because the primary use of the data and the reason for its creation is not for

evaluative purposes. Response rate to the inventory review may be insufficient and it may fail to

capture feedback from all relevant NIH stakeholders.

Mitigating these challenges will require a coordinated, transparent, collaborative process with all

relevant stakeholders. Given the potential complications related to the scope of this effort, analyses

and recommendations will focus on data resources, tools, and needs which are likely to be of benefit

across the NIH enterprise. Greater emphasis will be placed on determining recommendations critical

for enhancing evidence building capacity to aid decision-making by NIH leadership.

Plan for Disseminating Results:

After the assessments and surveys are completed, analyzed, and summarized, the findings will be

shared with the offices responsible for providing the data and tools and with the NIH Planning and

Evaluation Officers Committee. The Committee’s Evaluation Subcommittee will use the results to

develop a final report of findings, priorities, and recommendations for NIH leadership. Findings from

this effort can also be leveraged and utilized to inform the development of a new knowledge base to

further enhance existing NIH evaluation and other assessment resources.

Focus Area 2:

Workforce

Background:

Accurate, timely, and targeted evidence is essential for informed decision-making about effective and

efficient use of resources. Strengthening NIH’s evidence-building capacity is a critical step toward

providing decision-makers with high-quality information about the implementation, outputs, and

outcomes of NIH programs, policies, and operations.

A foundational pillar of NIH’s evidence-building capacity is having – and maintaining – an adequate

pool of skilled and experienced evaluators and analysts whose work can influence the quality and

utility of evaluations and other assessments across NIH. NIH is a large organization, composed of 27

separate institutes and centers and several offices within the Office of the NIH Director. Recent

surveys of NIH staff, one focusing on evaluation and the other on research portfolio analysis,

suggested that evidence-building capacity varies significantly across ICOs. Informal discussions within

the NIH Planning and Evaluation community revealed that some ICOs have devoted substantial

resources to evaluation and/or analytical activities, while others have not due to limited resources and

competing priorities. For instance, some ICOs have established dedicated units with trained specialists

whose primary function is to conduct evaluations and other assessments, while others utilize

analytical staff in policy-focused units or rely on evaluators and analysts in program units. Such

differences do not necessarily mean some ICOs are more (or less) equipped at generating evidence for

decision-making, although anecdotal information suggests that in some cases fewer resources do

affect the number and/or quality of evaluations and other assessments that ICOs undertake.

The activity described below represents NIH’s initial plan for developing a full and nuanced

understanding of its evaluation and analytical workforce and addressing any gaps and limitations that

might prevent the agency from meeting its evidence-building goals.

Evaluation or Analysis Activity:

To set a baseline for conducting a formative evaluation of NIH's workforce capacity, determine the

staffing levels and competencies currently available at NIH for assessing trans-NIH and ICO-specific

programs, policies, and operations and whether they are sufficient to meet NIH's evidence-building

goals.

Programs, Policies, Regulations, or Operations to be Analyzed:

The workforce capacity within each ICO and for NIH as a whole to generate high-quality evidence about

the effectiveness and impact of NIH programs, policies, and operations.

Stakeholders:

NIH leadership and directors/chiefs who oversee planning and evaluation units within their ICOs.

Data Sources:

 Environmental scan of competency models developed by federal agencies and professional

associations for program evaluation and other types of assessment commonly done by NIH (e.g.,

research portfolio analysis, bibliometric analysis).

 A self-administered survey designed to solicit from ICOs specific information about the staff

who are responsible for designing, managing, and/or conducting evaluations and other

assessments.

 Post-survey follow-up interviews may be conducted with selected ICOs to ensure that

information being gathered from all ICOs is as complete and consistent as possible.

Methods:

First, NIH will conduct an environmental scan of competency models (e.g., American Evaluation

Association’s Evaluator Competencies), and use the findings to define the skill and knowledge

requirements most relevant for assessing NIH programs, policies, and operations. Second, NIH will

develop a self-administered survey to gather information from ICOs regarding the number,

qualifications (in relation to the skill and knowledge requirements identified), and roles and

responsibilities of their evaluation and analytical personnel. Finally, based on the survey responses,

follow-up interviews may be conducted with selected ICOs to ensure that complete and consistent

information is being gathered from all ICOs. If follow-up interviews are conducted, interview responses

will be summarized in writing and then sent to the interviewees for review. Based on information

gathered from the survey and any follow-up interviews, NIH will prepare a profile of assessment-

related staffing and staff competencies for each ICO and for NIH at-large.

Anticipated Challenges and Mitigation Strategies:

Low response rate is an anticipated challenge. Some ICOs may not participate in the survey due to the

burden associated with gathering information to prepare an ICO-wide response or due to internal

concerns that they might not compare favorably with other ICOs. Involvement of the Evaluation

Subcommittee of the NIH Planning and Evaluation Officers Committee, whose membership covers

more than 20 ICs, will be critical in creating a survey that will generate useful information and not be

overly burdensome, determining the ideal time of the year to launch the survey and request

documents, and engaging in outreach efforts to increase participation.

Plan for Disseminating Results:

Each ICO will receive a copy of its own profile and an aggregated profile for NIH at-large. The NIH

Planning and Evaluation Officers Committee will use the information gathered, in conjunction with the

evidence-building needs and priorities identified by ICO leadership, to develop a proposal laying out

how NIH might build toward the staffing levels and competencies required to support the agency’s

short- and long-term evidence-building goals.

Focus Area 3:

NIH Leadership’s Evidence-Building Needs

Background:

Accurate, timely, and targeted evidence is essential for informed decision-making about effective and

efficient use of resources. Strengthening NIH’s evidence-building capacity is a critical step towards

providing decision-makers with high-quality information about the implementation, outputs, and

outcomes of NIH programs, policies, and operations.

Recent surveys of NIH staff about their assessment activities and needs were not targeted towards

leadership and therefore did not provide a complete picture of how evidence informs leadership’s

decision-making. The surveys have provided helpful but incomplete information about their

Institutes’, Centers’, and Offices’ (ICOs’) assessment priorities, the value their leadership places on

evidence from assessment activities and how such evidence is used to inform their leadership’s

decision-making. A better understanding of how decision-makers use data, evidence, and other

information in their decision-making processes is needed. Insight from the decision-makers will ensure

that data and evidence generated will be targeted, timely, and useful.

Evaluation or Analysis Activity:

Engage NIH and its ICOs’ leadership to determine how they define, prioritize, value, and utilize

evidence (including the types of evidence they use and would like to have) to inform their decision-

making and identify their evaluation and other assessment priorities.

Programs, Policies, Regulations, or Operations to be Analyzed:

Use of and needs for evidence-based decision-making at the leadership level of NIH.

Stakeholders:

NIH plans to build on HHS’s Evidence Act implementation and capacity assessment efforts to discern

NIH and ICO leadership’s understanding and use of evidence in their decision-making as well as their

current and future assessment priorities.

Data Sources:

Qualitative data will be collected from NIH and ICO leadership through structured interviews.

Methods:

NIH will first develop an interview guide to capture detailed information on NIH and ICO

leadership’s assessment priorities and use of evidence, including, but not limited to: how do they view,

value, and use evidence in the decision-making process; what do evaluations and other assessments

offer them; how have they used evaluation and other assessment findings in the past; what challenges

have they encountered with past use of evidence; and what data/evidence would be most useful to

them going forward.

Planning and Evaluation Officers affiliated with every NIH ICO will be asked to assist with engaging

their leadership to increase participation. The interview responses will be analyzed and summarized.

Follow-up interviews and focus groups may be conducted as needed.

Anticipated Challenges and Mitigation Strategies:

Response rate is an anticipated challenge. Leadership may not participate due to competing priorities,

existing workload, and/or scheduling conflicts. Involvement of the Evaluation Subcommittee of the

NIH Planning and Evaluation Officers Committee, whose membership covers more than 20 ICs, will be

critical in engaging leadership and other outreach efforts to increase participation.

Plan for Disseminating Results:

After the interviews are conducted and the responses are analyzed and summarized, the aggregate

findings will be shared with the NIH Planning and Evaluation

Officers Committee and the interviewees. The Committee’s Evaluation Subcommittee will use the

results to design a systematic and strategic approach to build and improve on NIH’s existing

assessment capacity. It is expected that findings will also inform the next iteration of NIH’s Evidence

Building Plan.

Substance Abuse and Mental Health Services Administration (SAMHSA)

Priority Question 1:

How will SAMHSA collect, analyze, and disseminate data to inform policies, programs, and practices?

Background and Significance:

The SAMHSA Strategic Plan FY2019-FY2023 outlines five priority areas with goals and measurable

objectives that provide a roadmap to carry out the vision and mission of the agency. For each priority

area, an overarching goal and series of measurable objectives track progress. Priority #4, Improving

Data Collection, Analysis, Dissemination, and Program and Policy Evaluation is most relevant to the

information being requested for the Evidence Act annual evaluation plan.

Programs, Policies, Regulations, or Operations to be Analyzed or Evaluated:

SAMHSA has strengthened its data collection, outcomes, evaluation, and quality support efforts to

enhance health care and health systems integration; identify and address mental and substance use

disorder-related disparities; identify what works; and strengthen and expand the provision of

evidence-based behavioral health services for all Americans. Such performance-based efforts will be

conducted by SAMHSA, along with federal, state, territorial, tribal, and community partners, and will

directly improve the delivery of services, promote awareness, and inform the development of policy

and programmatic initiatives. Data collections have also been streamlined to address respondent

burden.

Stakeholders:

Stakeholders include, but are not limited to: federal, state, territorial, tribal and community partners,

policy makers, researchers, health professionals, service providers, and grantee and grantee

populations.

Data Sources:

 Drug Abuse Warning Network (DAWN) survey: DAWN is a nationwide public health

surveillance system that provides early warning information on substance use-involved

hospital emergency department visits, with a focus on the nation's opioid crisis.

 National Survey on Drug Use and Health (NSDUH):- NSDUH is the primary source of

nationally representative statistical information on the prevalence of substance use and

mental illness in the U.S. The NSDUH generates statistical estimates at the national, state,

and sub-state levels on a variety of key substance use and mental health-related measures.

 National Survey of Substance Abuse Treatment Services (N-SSATS): N-SSATS collects data

on the location and characteristics of substance abuse treatment facilities, and is used to

update the SAMHSA online Behavioral Health Treatment Services Locator.

 National Mental Health Services Survey (N-MHSS): N-MHSS collects data on the location

and characteristics of mental health treatment facilities, and is used to update the SAMHSA

online Behavioral Health Treatment Services Locator.

 Treatment Episode Data Set (TEDS): TEDS collects demographic and substance use

characteristics of treatment admissions to and discharges from publicly funded substance

abuse treatment facilities.

 Mental Disorder Prevalence Study (MDPS): MDPS is a household and non-household data

collection effort to capture prevalence estimates for certain mental health conditions (e.g.,

schizophrenia, bipolar disorder).

 Mental Health Client Level Data (MH-CLD) data collection: MH-CLD collects administrative

data on clients receiving care at state-funded mental health treatment facilities.

 SAMHSA’s Performance Accountability and Reporting System (SPARS): SPARS captures real-

time data from SAMHSA discretionary grant programs, in order to monitor the progress,

impact, and effectiveness of SAMHSA programs.

Methods:

SAMHSA’s Goal under Priority #4, Improving Data Collection, Analysis, Dissemination, and Program

and Policy Evaluation is to expand and improve the data collection, analysis, evaluation, and

dissemination of information related to mental and substance use disorders and receipt of services for

these conditions, in order to inform policy and programmatic efforts, assess the effectiveness and

quality of services, and determine the impacts of policies, programs, and practices. The measurable

objectives associated with this Priority and Goal define the agency’s approach, and are as follows:

 Objective 4.1: Develop consistent data collection strategies to identify and track mental health

and substance use needs across the nation

 Objective 4.2: Ensure that all SAMHSA programs are evaluated in a robust, timely, and high-

quality manner

 Objective 4.3: Promote access to and use of the nation's substance use and mental health data

and conduct program and policy evaluations, and use the results to advance the adoption of

evidence-based policies, programs, and practices.

Anticipated Challenges and Mitigation Strategies:

Due to COVID-19, some of the SAMSHA key household data collections have been delayed. These

include the National Survey on Drug Use and Health and the Mental Health Prevalence Study.

NSDUH is expected to resume field data collection in the summer of 2020. SAMHSA is exploring

options for other data collection approaches (e.g., web-based survey), as alternatives to household

data collection. The MDPS has expanded data collection to be done virtually, in order to acquire data

safely.

Plan for Disseminating Results:

SAMHSA disseminates key national annual reports, evidence-based practice guidebooks and

evaluation of data via presentations throughout the year. In addition, SAMHSA administers the

Substance Abuse and Mental Health Data Archive (SAMHDA), SAMHSA’s platform for providing access

to public-use and restricted-data assembled from its national mental health and substance use data

collections.