2018 Federal Standard of Excellence
Resources
Did the agency invest at least 1% of program funds in evaluations in FY18? (Examples: Impact studies; implementation studies; rapid cycle evaluations; evaluation technical assistance, and capacity-building)
Score
7
7
Administration for Children and Families (HHS)
- In FY18, ACF plans to spend approximately $165 million on research and evaluation, research and evaluation technical assistance, and research and evaluation capacity-building, representing 0.3% of ACF’s $58.6 billion budget in FY18 (in addition to investments in evaluations by ACF grantees). The amount of ACF’s spending on evaluation is largely determined by Congress.
Score
9
9
Administration for Community Living
- ACL spent approximately $16 million on evaluation in FY18, representing .88% of the agency’s $1.8 billion FY18 budget.
- The bulk of these funds are based on a set-aside required under the Older Americans Act. As specified in Title II, section 206 of the Older Americans Act, “From the total amount appropriated for each fiscal year to carry out title III, the Secretary may use such sums as may be necessary, but not to exceed 1⁄2 of 1 percent of such amount, for purposes of conducting evaluations under this section, either directly or through grants or contracts.”
Score
8
8
Corporation for National and Community Service
- CNCS plans to spend a total of $6.55 million, representing .65% of CNCS’s $1 billion budget in FY18, on evaluation and evaluation capacity building activities (R&E evaluation and program funds combined), including:
- $8,500,000 of FY18 AmeriCorps funds have been awarded in Commission Investment Fund grants, of which approximately one third will be used for building commission capacity to support grantees (formula grantees) in the areas of evaluation and performance measurement;
- $2,050,000 of FY18 Senior Corps funds will be spent on evaluation and evidence-building activities (representing 1 percent of Senior Corps’ $202.11 million FY18 budget); and
- $1,671,766 of FY18 evaluation funds have been awarded for AmeriCorps grantee evaluation capacity building.
- In addition to these program evaluation and evidence capacity building investments, CNCS will invest close to $1.5 million in research grants that fund studies designed to systematically assess the outcomes and impacts of national service programs and volunteerism. These studies rely on large Federal survey and administrative data sets to statistically examine relationships between nonprofit organizations that rely on volunteers and other civil society organizations and various community outcomes (e.g., health and well-being). Furthermore, these studies examine individual outcomes such as the long-term economic outcomes of volunteering (using longitudinal survey data) or the effects of culturally relevant curriculum on the civic behavior of participants over time. Finally, these research grants fund studies that examine the role of civic participation in identifying and implementing solutions for community issues like opioid use, disaster recovery, academic success, and equitable economic development.
Score
10
10
Millennium Challenge Corporation
- MCC’s FY18 investment in monitoring and evaluation to date is $15.1 million, which amounts to 5.1% of Compact spending for FY18 ($293.6 million). This investment exceeds the proportion of FY17 spending in which MCC invested over $21.9 million in M&E, roughly 4.2% of Compact spending for FY17 ($516.1 million).
Score
0
0
Substance Abuse and Mental Health Services Administration
- RFA was unable to determine the amount of resources SAMHSA invested in evaluations in FY18.²
- ²RFA was unable to determine the amount of resources SAMHSA invested in evaluations in FY18 for criterion #3. Therefore, to tally a final score, RFA scored criterion #3 a zero and reduced the denominator from 100 to 90 points.
Score
10
10
U.S. Agency for International Development
- In FY17, USAID missions and offices reported completing 161 evaluations with resources totaling approximately $42 million. In addition, they were actively managing another 286 ongoing evaluations, many that span more than one year, with total ongoing evaluation budgets estimated to reach almost $210 million. Overall spending on evaluations completed or ongoing in FY17 ($252 million) represents about 1.4% of USAID’s $17.6 billion FY17 program budget.
- This amount does not include the budget for the Office of Learning, Evaluation, and Research which primarily focuses on monitoring, evaluation, and learning capacity building and technical assistance ($8.9 million in FY17) or the investment in the Demographic and Health Surveys (DHS) ($189 million total in FY13-FY18) or surveys funded by other sector programs that often make up some of the underlying data used in many evaluations.
Score
6
6
U.S. Department of Education
- Overall spending on evaluation (about $38 million in FY18) represents .08% of ED’s $46.1 billion discretionary budget (without Pell Grants) minus administrative funds in FY18. This total reflects a targeted definition of program funds dedicated to evaluation, including impact studies and implementation studies. It is important to note that the timing of evaluation projects and the type of research projects proposed by the field results in year-to-year fluctuations in this amount, and does not reflect a change in ED’s commitment to evaluation.
- While some evaluation funding – such as that for Special Education Studies and Evaluations – is appropriated to IES ($10.8 million in FY18), most evaluations are supported by funds appropriated to ED programs. The Evidence Planning Group (EPG) described above supports program staff that run evidence-based grant competitions and monitor evidence-based grant projects, advises Department leadership and staff on how evidence can be used to improve Department programs, and provides support to staff in the use of evidence.
- The Education Innovation and Research (EIR) and the Supporting Effective Educator Development (SEED) programs utilize technical assistance contractors that support grantees in the design and implementation of their project-level evaluations, as well as the development of evaluation resources that are now being shared publicly on the NCEE, What Works Clearinghouse, and program websites. In FY17, using funds appropriated in previous fiscal years, about $2.8 million was dedicated to this evaluation technical assistance for EIR grantees, and about $700,000 was dedicated to supporting SEED grantees.
- In addition to the narrowly defined figure above, ED invests in evaluation by supporting states and school districts so that they can conduct studies of their own education policies and programs. For example, IES runs annual grant competitions to support researcher-practitioner partnerships between state and local education officials and research institutions, including a new program for low-cost, short duration evaluations. IES also awarded approximately $24 million in the Statewide Longitudinal Data Systems grant program across 16 States to support efforts related to (1) increasing use of data for decision making; (2) conducting training on data use, data tools, or accessing data and reporting systems; and (3) utilizing research and analysis results. Finally, the Regional Education Laboratories (RELs) provide extensive technical assistance on evaluation and support research alliances that conduct implementation and impact studies on education policies and programs in ten geographic regions of the U.S., covering all states, territories, and the District of Columbia. Over $55 million was appropriated for the RELs in FY18.
- ED emphasizes evaluation and the building of evidence in a number of its grant programs through requirements that grantees conduct or participate in evaluations. This emphasis on evaluation continues even for programs supported by an existing evidence base, as the evaluation design for these programs looks at impact with new settings, different populations, and project-specific implementation. In many instances, during grant competitions, this evaluation criterion is reviewed and scored by evaluation experts familiar with the What Works Clearinghouse standards, ensuring proposed evaluation plans are of the appropriate design and rigor. Additionally, a number of OII programs dedicate resources to evaluation technical assistance and program level evaluations.
- IES and OII grantees are expected to make the results of their evaluations public through Education Resources Information Clearinghouse (ERIC) and other grant-funded dissemination activities. In addition, all impact evaluations funded by IES and OII are reviewed by the What Works Clearinghouse (WWC), which plays a major role in summarizing and disseminating findings from the most rigorous studies to ED and the broader field.
- ED has an opportunity to increase its annual investment in program evaluation through the reauthorized ESEA pooled evaluation authority, which allows the Department to use funds appropriated to ESEA programs to evaluate any ESEA program included in the biennial evaluation plan prepared by IES. In addition, the ESEA permits ED to reserve up to .5 percent of program funding for evaluation activities.
Score
6
6
U.S. Dept. of Housing & Urban Development
- In FY18, HUD plans to spend $89 million on evaluations, representing 0.17% of HUD’s $51 billion discretionary budget (minus salary and expenses) in FY18.
- For FY18, Congress appropriated $89 million for the Office of Policy Development and Research’s (PD&R’s) Research & Technology account, including $50 million for core research activities; up to $14 million for research, evaluations, and demonstrations; and not less than $25 million for technical assistance. The total represents an FY18 investment in evaluations and evidence amounting to 0.17 percent of HUD’s $51 billion gross discretionary budget authority, net of salaries and expenses, for FY18. The funding for core research is used primarily for the American Housing Survey, other surveys, and data acquisition that support evaluation of HUD’s mission activities in domains such as affordable housing and housing finance.
- PD&R’s FY18 appropriation of $24 million for Salaries and Expenses also supports evidence in the form of PD&R’s in-house research and evaluation program, economic analyses, data linkage initiatives, and management of housing surveys and contract research and evaluation.
- Additionally, PD&R leverages public resources by issuing data licenses and by establishing Research Partnerships with private-sector partners who contribute at least 50 percent of project resources and undertake research that contribute to HUD’s objectives.
Score
7
7
U.S. Department of Labor
- In FY18, DOL’s CEO invested approximately $21 million in evaluation and evidence building activities. This represents approximately 0.20% of DOL’s FY18 discretionary budget for agency programmatic appropriations minus salaries and expenses.
- This amount only represents the dollars that are directly appropriated or transferred to CEO. Additionally, many DOL evaluations and research studies are supported by funds appropriated to DOL programs and/or are carried out by other offices within DOL. In fact, in addition to CEO, most agencies and program offices conduct and support evaluation activities with their own dollars. For example, ETA funds evaluations and provides evaluation technical assistance and capacity building activities to states to help them meet evaluation and reporting requirements under the Workforce Innovation and Opportunity Act (WIOA).ETA continues funding and technical assistance to states under the Workforce Data Quality Grant Initiative (WDQI) to link earnings and workforce data with education data longitudinally (for example in in FY18, ETA will award approximately $6.0M for WDQI grants). ETA and DOL’s Veterans’ Employment and Training Service (VETS) have also modified state workforce program reporting system requirements to include data items for a larger set of grant programs, which will improve access to administrative data for evaluation and performance management purposes.Further, several DOL agencies also have separate evaluation appropriations. DOL studies funded through individual agencies and program offices also coordinate with DOL’s CEO.
- In many areas where DOL is undertaking evaluation activities, the evaluation budget far exceeds 1% of the budget for the program. For example, the budgets for the evaluations of a number of recent grant programs, such as the America’s Promise grant evaluation and the Reentry Grant Evaluation, are between 3% and 5% of the programmatic budget.
- The Administration’s FY14-FY17 budget requests recommended allowing the U.S. Secretary of Labor to set aside up to 1% of all operating agencies’ budgets for evaluations, coordinated by CEO. In FYs 2012-2015, Congress authorized the Secretary to set aside up to 0.5% of these funds for evaluations, in addition to the separate evaluation funds that exist in many DOL agencies. In FY16-FY18, Congress authorized DOL to set aside up to 0.75% of operating agency budgets for use by CEO for evaluations.