2018 Federal Standard of Excellence
Repurpose for Results
In FY18, did the agency shift funds away from or within any practice, program, or policy that consistently failed to achieve desired outcomes? (Examples: Requiring low-performing grantees to re-compete for funding; removing ineffective interventions from allowable use of grant funds; proposing the elimination of ineffective programs through annual budget requests)
Score
8
8
Administration for Children and Families (HHS)
- The Head Start Designation Renewal System requires Head Start ($9.9 billion in FY18) grantees to compete for grants moving forward if they failed to meet criteria related to service quality, licensing and operations, and fiscal and internal controls. The 2007 Head Start Reauthorization Act made all Head Start grants renewable, five-year grants. At the end of each five-year term, grantees that are running high-quality programs will have their grants renewed. But grantees that fall short of standards are now required to compete to renew grants. Grantees whose ratings on any of the three domains of the Classroom Assessment Scoring System, an assessment of adult:child interactions linked to improved outcomes, fall below a certain threshold, or in the lowest 10 percent of grantees, must also compete.
- ACF, in collaboration with the HHS Health Resources and Services Administration, has established criteria for evidence of effectiveness of home visiting models, and oversees the Home Visiting Evidence of Effectiveness Review (HomVEE), which determines whether models have evidence of effectiveness. To date HomVEE has reviewed evidence on 45 home visiting models, determining 20 of these to have evidence of effectiveness. Grantees must use at least 75% of their federal home visiting funds to implement one or more of these models.
- As noted in the response to criterion one, OPRE engages in ongoing collaboration with program office staff and leadership to interpret research and evaluation findings and to identify their implications for programmatic and policy decisions, including encouraging or requiring recipients of grants to use effective (and not ineffective) practices.
Score
6
6
Administration for Community Living
- ACL released two new funding announcements in FY18 exploring methods for potentially redirecting funds from lower performing programs to higher performing ones:
- The Paralysis Resource Center State Pilot Program is part of an effort to ensure program efficiency and to test two approaches for making sub-awards to community-based organizations that provide long-term services and supports to people with paralysis, their families, and their support networks. Outcomes from the pilot will help ACL assess the most effective and efficient ways to make such sub-awards and will determine how ACL funds this effort going forward.
- Under OPE’s overarching analytic support contract, ACL is developing a tool to help grant officers more easily monitor the degree to which Chronic Disease Self-Management and Falls Prevention grantees are meeting their ACL approved program completion targets. ACL staff will use this information to either fully release or restrict grant funds over the life of the multi-year awards.
- The Innovations in Nutrition Programs and Services grants will be monitored using a new tool that allows grant officers to more clearly determine grantees’ progress towards meeting their targeted service levels and therefore restrict or withhold funding based on that progress.
- Based on a review of programmatic purpose and outcomes, ACL’s FY18 budget request proposed to:
- In part, based on work with researchers which was disseminated through a 2014 webinar series, ACL consolidated three small programs that serve people affected by Alzheimer’s Disease and Related Dementias programs into a single Alzheimer’s Disease Program, funded at the same total level. This will increase programmatic flexibility and efficiency and enable grantees to better meet the needs of people affected by these illnesses.
- Based on a lack of evidence regarding the unique benefits of these programs, ACL proposed to devolve program responsibilities to states or local communities for the Limb Loss Resource Center and Paralysis Resource Center. The mission and activities carried out by these programs are duplicative of other Federal efforts. Savings from eliminating these redundant programs total $3 million and $8 million, respectively.
- ACL has implemented a quality review system (QRS) for developmental disability programs under ACL/AIDD. The QRS uses a three-tiered model to review program compliance, outcomes (i.e., evidence), and fiscal operations and use review results to target and coordinate technical assistance. The first tier is annual standardized review. The second tier is standardized, in-depth review involving a team of reviewers. These reviews are conducted on a periodic basis. Tier three is customized monitoring for programs that ACL has significant concerns regarding terms of compliance and performance. ACL continues development of a formula grant monitoring framework for Older Americans Act Title III and VII state formula grants. The framework combines assessments of a grantee’s progress toward program goals and objectives with identification of risk or instances of fraud, waste, and abuse.These reviews allow ACL, if warranted, to restrict grant funding based on findings of insufficient evidence of performance.
Score
5
5
Corporation for National and Community Service
- According to CNCS policy, Americorps considers past performance (e.g., meeting performance targets) as part of its criteria for making funding decisions. This assessment is in addition to the evaluation of the applicant’s eligibility for funding or the quality of its application on the basis of the Selection Criteria. Results from this assessment inform funding decisions every year for every grant competition. In evaluating programmatic performance, CNCS considers the following for applicants that are current formula and competitive grantees submitting applications for the same program model:
- Grant progress reports – attainment of Performance Measures;
- Enrollment and retention;
- Compliance with 30-day enrollment and exit requirements in the AmeriCorps portal;
- Site visit or other monitoring findings (if applicable);
- Significant opportunities and/or risks of the grantee related to national service; and
- Commission Rank.
- For example, in FY18, due to ongoing program design and performance issues, AmeriCorps provided ongoing and directive feedback to a long-standing grantee that it would face significant hurdles to securing future CNCS funding. The organization decided against submitting a grant application, and CNCS was able to repurpose almost $3 million to fund interventions implemented by other stronger, more impactful grantees.
- The AmeriCorps NCCC program reallocated significant mandatory and discretionary program resources in FY18 (for savings totaling an estimated $1,872,000) to optimize the program’s utilization of facilities and maintain responsiveness to community needs (e.g., teams located near areas frequently affected by natural disasters and more members residing in fewer federal facilities), to maximize the cost per unit (e.g., member), and to maintain the quality and effectiveness of service in the community. NCCC outsourced recruitment, created lodging efficiencies, consolidated member support services (e.g., administrative intake), and closed a campus that had the highest region cost per unit. Some of these savings (an estimated $400,000) will be invested in research and evaluation activities in FY18. These efficiency measures have a long term estimated savings of $4 million for reinvestment in program priorities.
- The AmeriCorps VISTA program started reallocating member training resources in FY16, moving from more resource-intensive, in-person training to virtual training. By the end of FY18, all member orientation training will be virtual. Member survey data indicate consistently positive results over this time period with regard to the utility of the training for a successful service experience, with over two thirds of respondents rating training and resources received from AmeriCorps as “excellent” or “good.” Preparing VISTA members to impact their communities is now achieved in a more cost-effective and efficient manner, and allows more flexible onboarding for the thousands of nonprofits hosting VISTAs around the country. VISTA is also collaborating with NCCC on a consolidated member support pilot, to merge support functions across the VISTA and NCCC programs to achieve efficiencies. VISTA also created a new Data Analyst position to standardize data tools and create new internal controls through automated flags and reports. Savings are being reinvested in targeted skill-based trainings for VISTA members and supervisors and evaluation activities.
- The AmeriCorps NCCC program tracks five key performance indicators: (1) alignment of NCCC teams with state identified priorities; (2) in-kind contributions from project sponsor organizations and communities; (3) employee viewpoint trends; (4) member graduation rates; and (5) number of alumni remaining in the community post-graduation. A pilot was initiated in 2014 to determine how the program might pursue increasing its effectiveness while decreasing costs. A comparison of key performance indicators (e.g., number of service hours, number of projects and sponsors, member attrition) was made between two classes that served and graduated from the program prior to the pilot and three classes following pilot implementation. Findings from the pilot demonstrated that costs could be reduced while maintaining the same level of community service and increasing member retention in the program. The revised program model has since been implemented in 3 of 4 program locations with the third occurring in FY18.
Score
8
8
Millennium Challenge Corporation
- MCC has an established Policy on Suspension and Termination that lays out the reasons for which MCC may suspend or terminate assistance to partner countries. Assistance may be suspended or terminated, in whole or in part, if a country:1) engages in activities contrary to the national security interests of the US, 2) engages in a pattern of actions inconsistent with the MCC’s eligibility criteria, or 3) fails to adhere to its responsibilities under a compact or threshold grant, or related agreement. Evidence of failing to achieve desired outcomes is included in category two. Such actions may be evidenced by, among other things:
- A decline in performance on the indicators used to determine eligibility;
- A decline in performance not yet reflected in the indicators used to determine eligibility; or
- Actions by the country which are determined to be contrary to sound performance in the areas assessed for eligibility for Assistance, and which together evidence an overall decline in the country’s commitment to the eligibility criteria.
- MCC has terminated a compact partnership, in part or in full, seven times out of 35 compacts approved to date, and has suspended partner country eligibility (both compact and threshold) four times, most recently seen with the suspension of Tanzania in March 2016 due to actions contrary to MCC’s eligibility criteria. MCC’s Policy on Suspension and Termination also allows MCC to reinstate eligibility when countries demonstrate a clear policy reversal, a remediation of MCC’s concerns, and an obvious commitment to MCC’s eligibility indicators, including achieving desired results. For example, in early 2012, MCC suspended Malawi’s Compact prior to Entry into Force as MCC determined that the Government of Malawi had engaged in a pattern of actions inconsistent with MCC’s eligibility criteria. Thereafter, the new Government of Malawi took a number of decisive steps to improve the democratic rights environment and reverse the negative economic policy trends of concern to MCC, which led to a reinstatement of eligibility for assistance in mid-2012. MCC’s model uses objective, evidence-based eligibility criteria to determine whether a country is pursuing the policies necessary to support poverty reduction through economic growth. For countries that meet these criteria, MCC provides large investments to capitalize on the country’s policy environment to spur economic growth and reduce poverty. When countries fail to maintain such policies or take actions that undermine these policies, they can have a negative impact on the ability of MCC’s programs to achieve their desired results.
- In a number of cases, MCC has repurposed investments based on real-time evidence. In MCC’s first compact with Morocco, the Government of Morocco proposed the Enterprise Support Project to address two of its critical economic priorities: reduce high unemployment among young graduates and encourage a more entrepreneurial culture. The project was designed to be carried out in two phases, with continuation of the second phase subject to positive results from an impact evaluation of the first phase. The pilot phase was completed in March 2012; although it met its implementation targets and showed promising trends, the impact evaluation did not show statistically significant impacts. The revised economic rate of return did not justify scaling up the project for a second phase. MCC did not continue with a second phase and the project was closed in May 2012. In MCC’s compact with Lesotho, MCC cancelled the Automated Clearing House Sub-Activity within the Private Sector Development Project after monitoring data determined it would not accomplish the economic growth and poverty reduction outcomes envisioned during compact development. The remaining $600,000 in the sub-activity was transferred to the Debit Smart Card Sub-Activity, which targeted expanding financial services to people living in remote areas of Lesotho. In Tanzania, the $32 million Non-Revenue Water Activity was re-scoped after the final design estimates on two of the activity’s infrastructure investments indicated higher costs that would significantly impact their economic rates of return. As a result, $13.2 million was reallocated to the Lower Ruvu Plant Expansion Activity, $9.6 million to the Morogoro Water Supply Activity, and $400,000 for other environmental and social activities. In all of these country examples, the funding is either reallocated to activities with continued evidence of results or returned to MCC for investment in future programming.
- MCC also consistently monitors the progress of compact programs and their evaluations across sectors, using the learning from this evidence to make changes to MCC’s portfolio. For example, MCC undertook a review of its portfolio investments in roads in an attempt to better design, implement, and evaluate road investments. Through evidence collected across 16 compacts with road projects, MCC uncovered seven key lessons including the need to prioritize and select projects based on a road network analysis, to standardize content and quality of road data collection across road projects, and to consider cost and the potential for learning in determining how road projects are evaluated. This body of evidence and analysis was published in November 2017. The lessons from this analysis are being applied to road projects in compacts in Cote d’Ivoire and Nepal as MCC roads investments see a shift toward increased maintenance investments. Critically, the evidence also pointed to MCC shifting how it undertakes road evaluations which led to a new request and re-bid for proposals for MCC’s roads evaluations based on new guidelines and principles. These changes in future practice are especially important to highlight as MCC operates in a five-year timeframe, often investing in large infrastructure projects. Because of the difficulty in changing implementation in such a limited timeframe given the types of projects MCC finances, MCC’s focus on shifting funds from ineffective programs and policies is often demonstrated in future compact and threshold program investment decisions.
Score
1
1
Substance Abuse and Mental Health Services Administration
- SAMHSA’s FY19 budget request proposed eliminating nine programs (totaling $279.6 million), but none of them for reasons of failing to achieve desired outcomes.
- In January 2018, SAMHSA announced it would shift resources away from the National Registry of Evidence-based Programs and Practices (NREPP) toward targeted technical assistance and training for implementing evidence-based practices. The reasoning was that NREPP had skewed presentation of evidence-based interventions, which did not address the spectrum of needs of those living with serious mental illness and substance use disorders.” SAMHSA publicly confirmed that it terminated the contract for the organization running NREPP.
- The SAMHSA budget provides performance information along with budget information which Congress can use to determine funding levels. Each year the program Centers review grantees within each program, project, or activity in terms of performance and financial management, when funding decisions are made for continuation funding. It is up to each Center to determine the factors that go into decisions related to continued funding based on guidance from the Office of Financial Management, Division of Grants Management. To the extent that costs are reduced for continuation funding, those funds can be repurposed to fund new grantees or to provide additional contract support for those grantees. In FY 2017, SAMHSA underwent a stringent review process for all funding requests utilizing both program and fiscal performance. During this process, SAMHSA utilized $51 million in unspent funding from existing grantees to fund new programs and activities.
- CBHSQ staff conducted a summer evaluation inventory in the summer of 2016, requesting that program staff from the Centers provide information related to how their evaluation findings inform the next iteration of their programs and/or new evaluation activities. For the most part, program staff indicated that evaluation findings were used to improve the next round of funding opportunity announcements and thus grantee implementation of program.
Score
8
8
U.S. Agency for International Development
- USAID’s updated operational policy for planning and implementing country programs has incorporated a set of tools and practices called Collaborating, Learning, and Adapting (CLA), that include designing adaptable activities that build in feedback loops; using flexible implementing mechanisms; and adopting a management approach that includes consulting with partners about how implementation is evolving and what changes need to be made. Through the Program Cycle, USAID encourages managing projects and activities adaptively, responding to rigorous data and evidence and shifting design and/or implementation accordingly.
- USAID uses rigorous evaluations to maximize its investments. A recent independent study found that 71 percent of USAID evaluations have been used to modify and/or design USAID projects. Below are a few recent examples where USAID has shifted funds and/or programming decisions based on performance:
- Serbia | Democracy & Governance – An evaluation completed six years after the end of a community development and civic engagement project found that increasing citizen engagement in public policy and government responsiveness requires simultaneous targeting of citizens, the government, and private and civil sectors. Findings led to the launch of new activities focused on the rule of law, media strengthening, and business competitiveness incorporating the evaluation’s recommended multifaceted approach.
- El Salvador | Education – An evaluation of an education project revealed the need to improve effectiveness by better matching interventions with individual target school and community needs to better serve out-of-school youth. As a result, the project changed how it is working with the Ministry of Education. It was also modified to address barriers to gender equality and social inclusion, to increase access to and quality of secondary-level education, and improve educational opportunities for school dropouts.
- Cambodia | Agriculture – An impact evaluation showed significant increases in participating farmers’ gross commercial horticulture income and cropped area, but not in returns to land or economic productivity. Going forward, USAID agricultural programming in Cambodia will help organize farmers into cooperatives and strengthen farmer-buyer linkages to improve production and overall household income.
- Afghanistan | Economic Growth – A mid-term evaluation of a program that enables women to increase their participation in the formal economy found that 98% of interns placed in jobs report working in a women-friendly workplace, although it is too soon to show whether the program has helped businesses increase profits or hire more women. Going forward, the program plans to raise the age cap of job placement candidates, hold trade fairs featuring women-to-women networking, and better support microenterprises.
- Ethiopia | Health – An evaluation of a project to improve the lives of vulnerable children and their families showed a significant improvement in educational attainment, health status, economic status, access to food and nutrition, and safer shelter. Future USAID efforts will further strengthen the capacities of regional and local government and civil society to serve these children and their families even better.
- BalanceD MERL: One of the MERLIN mechanisms, BalanceD, is ending early because evidence showed that it was being used for traditional M&E activities (e.g. survey data collection and M&E system set-up) rather than anything innovative. The decision to end the program early was made after two MOUs requested traditional support from BalanceD MERL, and no further interest for testing innovations using BalanceD was found.
- USAID’s Securing Water for Food: A Grand Challenge for Development (SWFF) selected the highest potential water-for-food innovations and is providing grant funds and ongoing assistance to support business development. SWFF starts as a competition, but the winners must continually show results to receive a new tranche of funding. To move forward, grantees must achieve technical and financial milestones, such as increased crop yields and total product sales. Of the 38 total awardees, twenty-three received Year 2 funding; fifteen did not, because they did not meet the target number of end-users/customers in a cost-effective way and because their model was not deemed sustainable without USAID funding. By using milestone-based funding, SWFF has helped over 3.6 million farmers and other customers grow more than 4 million metric tons of food and reduced water consumption in agriculture by more than 11.4 billion liters of water compared to traditional practices.For every $1,000 spent by the SWFF program, SWFF innovators have impacted 156 customers and end users, produced 282 tons of crops, reduced water consumption by more than 832,000 liters, improved water management on 86 hectares of agricultural land, and generated more than $200 in sales. In addition, SWFF innovators have formed more than 300partnerships and secured more than $16 million in leveraged funding.
Score
7
7
U.S. Department of Education
- ED seeks to shift program funds to support more effective practices by prioritizing the use of evidence as a requirement when applying for a competitive grant. For ED’s grant competitions where there is data about current or past grantees, or where new evidence has emerged independent of grantee activities, ED typically reviews such data to shape the design of future grant competitions.
- Additionally, ED uses evidence in competitive programs to encourage the field to shift away from less effective practices and toward more effective practices. For example, ESSA’s EIR program supports the creation, development, implementation, replication, and scaling up of evidence-based, field-initiated innovations designed to improve student achievement and attainment for high-need students. IES released The Investing in Innovation Fund: Summary of 67 Evaluations, which can be used to inform efforts to move to more effective practices. The Department is exploring the results to determine what lessons learned can be applied to other programs.
- The President’s FY19 Budget eliminates, streamlines or reduces 39 discretionary programs that duplicate other programs, are ineffective, or are supported with state, local, or private funds. Major eliminations and reductions in the FY19 Budget include:
- Supporting Effective Instruction State grants (Title II-A), a savings of $2.3 billion. The program is proposed for elimination because the program lacks evidence of improving student outcomes (pp. C-17 – C-19 of the FY19 budget request). It also duplicates other ESEA program funds that may be used for professional development.
- 21st Century Community Learning Centers program, a savings of $1.2 billion. The program lacks strong evidence of meeting its objectives, such as improving student achievement. Based on program performance data from the 2014-2015 school year, more than half of program participants had no improvement in their math and English grades and nearly 60 percent of participants attended centers for fewer than 30 days (see p. C-22 in the FY19 budget request).
- In addition, the President’s Budget proposes to streamline and consolidate programs to achieve management efficiencies, focus federal investments on activities supported by evidence, and reduce the federal role in education.
- In the previous administration, ED worked with Congress to eliminate 50 programs, saving more than $1.2 billion, including programs like Even Start (see pp. A-72 – A-73) (-$66.5 million in FY11) and Mentoring Grants (see p. G-31) (-$47.3 million in FY10), which the Department recommended eliminating out of concern based on evidence.
Score
7
7
U.S. Dept. of Housing & Urban Development
- HUD’s FY17 budget request included a new formula for funding Housing Choice Voucher Administrative Fees that shifts funding away from inappropriately compensated public housing agencies and increases overall funding according to evidence about actual costs of maintaining a high-performing voucher program. (See here for more info.)
- HUD’s FY17 budget request sought a $11 billion shift (pp. 8–9) of resources toward housing vouchers for homeless families based on the rigorous experimental analysis of 4 service options in the Family Options study.
- HUD’s FY18 budget request sought to eliminate funding for Community Development Block Grants. A 2005 PD&R evaluation had shown that targeting of CDBG resources toward communities with greater needs would be greatly enhanced by any of four alternatives to the 1978 statutory formula, but such improvements have not been authorized. An earlier 1995 evaluation found that although CDBG had made a contribution to community development, the neighborhood interventions generally were ad hoc rather than well-coordinated and strategic.
Score
6
6
U.S. Department of Labor
- DOL’s evidence-based strategy is focused on program performance improvement and expansion of strategies and programs on which there is evidence of positive impact from rigorous evaluations. DOL uses both evaluation results and program performance measures to make decisions about future funding.
- Since 2014, DOL has closed three Job Corps centers based on chronic low performance. Closure of underperforming centers allows DOL to shift limited program dollars to centers that will better serve students by providing the training and credentials they need to achieve positive employment and educational outcomes. In a Federal Register notice published in August of 2017, DOL announced a center closure and the methodology used for selecting centers for closure.
- Discretionary grant performance is closely monitored and has been used to take corrective action and make decisions about continued funding. For example, Youthbuild grant funding is based heavily on past performance. Organizations that have previously received and completed a YouthBuild grant award receive points based on past performance demonstrated, totaling 28 points (almost 30% of their score). This effectively weeds out low performing grantees from winning future awards. (For more information, see the Grant Funding Announcement).
- Additionally, DOL uses evidence in competitive grant programs to encourage the field to shift away from less effective practices and toward more effective practices. For example, recent grant programs such as the Reentry Grant Program use a tiered evidence framework to require grantees to use evidence based approaches and not choose approaches that have been found to be ineffective. This supports the creation, development, implementation, replication, and scaling up of evidence-based practices designed to improve outcomes, and dis-incentivizes grantees from using approaches not backed by evidence or that have been found to be ineffective.
- DOL’s FY19 budget request prioritizes programs with demonstrated evidence (e.g., it proposes investing $200 million in apprenticeships, a proven strategy) and proposes reductions to unproven strategies (e.g., it proposes shifting dollars in Job Corps away from serving younger youth for whom past studies have not found positive long term impacts, to focus the program more on the older youth for whom the program has been shown to be more effective).