Did the agency have a senior staff member(s) with the authority, staff, and budget to evaluate its major programs and inform policy decisions affecting them in FY18? (Example: Chief Evaluation Officer)

• The Corporation for National and Community Services’s (CNCS) Office of Research and Evaluation Director (R&E) oversees the development of social science research designed to measure the impact of CNCS programs and shape policy decisions; encourage a culture of performance and accountability in national and community service programs; provide information on volunteering, civic engagement, and volunteer management in nonprofit organizations; and assist in the development and assessment of new initiatives and demonstration projects. The R&E Director, who is overseeing R&E’s $4 million budget and a staff of nine in FY17, reports directly to the CNCS Chief of Staff and is a member of CNCS’s Leadership Team and Policy Council. The R&E Director also meets regularly with CNCS Program Directors to identify areas where evidence can be generated and used for various decisions.
• The R&E Director meets annually with all CNCS program offices to identify priorities and negotiate which pools of funds are need to support the year’s priorities. The FY17 plan was developed through a series of formal and informal conversations.

Did the agency have an evaluation policy, evaluation plan, and research/learning agenda(s) and did it publicly release the findings of all completed evaluations in FY18?

• CNCS considers its mission a set of empirical questions to be tested. As such, the agency uses the following core set of questions to organize its evidence building strategy each year:
  1. How do CNCS programs affect the individuals who serve (e.g., national service members and volunteers)?
  2. How do CNCS programs affect the individuals served by grantee and sponsor organizations (e.g., “beneficiaries”)?
  3. How do CNCS programs contribute to the civic health of communities?
  4. How can CNCS programs be used most effectively by communities to solve local challenges?
• A comprehensive portfolio of research projects has been built to address these questions. As findings emerge, future studies are designed to continuously build the agency’s evidence base. The CNCS Office of Research & Evaluation relies on scholarship in relevant fields of academic study; a variety of research and program evaluation approaches including field, experimental, and survey research; multiple data sources including internal and external administrative data; and different statistical analytic methods.
• The agency’s evidence-building strategy is updated annually based on input from agency leadership as well as from emerging evidence from completed studies. This agenda is reflected in the CNCS Congressional Budget Justifications each year (see Fiscal Year 2016 pp. 55-56; Fiscal Year 2017 pp. 5-6, 55-56; and Fiscal Year 2018 p. 3). CNCS’s R&E coordinates the agency’s learning agenda, which includes building its evidence base and facilitating the use of evaluation to inform important decisions. To this end, the office conducts research and evaluation on CNCS service programs; helps build the capacity of agency-funded partners to conduct and understand evaluations; and facilitates evidence-based and evidence-informed grant-making.
• A report synthesizing findings from FY16 and early FY17 research and evaluation studies conducted by or sponsored by CNCS may be found here. More generally, CNCS creates four types of reports for public release: research reports produced directly by research and evaluation staff, research conducted by third party research firms and overseen by research and evaluation staff, reports produced by CNCS-funded research grantees (see research competition for more information), and evaluation reports submitted by CNCS-funded program grantees. All reports completed and cleared internally are posted to the Evidence Exchange, an electronic repository for reports. This virtual repository was launched in September 2015. Quarterly analytics for new products created, number of reports posted, page views, and users are provided by our contractor.
• In FY16, CNCS developed Evaluation Core Curriculum Courses which are presented to its grantees through a webinar series and is available on the CNCS website along with other evaluation resources. The courses are designed to help grantees and other stakeholders easily access materials to aid in conducting or managing program evaluations. R&E staff supported workshops using these materials for Senior Corps grantees in July 2018 and AmeriCorps grantees in September 2018. In addition, according to an internal evaluation CNCS conducted with State Commissions regarding their use of Commission Investment Fund grants to improve their ability to conduct high quality performance measurement and evaluation, having these CNCS resources facilitated implementation of the grant. As one commission explained, “One thing that definitely kept things running smoothly is that the resources—the two core curriculum courses, the performance measure and evaluation—having those already developed, ready to go…not having to develop new things from scratch and just being able to go directly to this is what theory of change is…having that ready to go was also really helpful in moving along.”

Did the agency invest at least 1% of program funds in evaluations in FY18? (Examples: Impact studies; implementation studies; rapid cycle evaluations; evaluation technical assistance, and capacity-building)

• CNCS plans to spend a total of $6.55 million, representing .65% of CNCS’s $1 billion budget in FY18, on evaluation and evaluation capacity building activities (R&E evaluation and program funds combined), including:
  • $8,500,000 of FY18 AmeriCorps funds have been awarded in Commission Investment Fund grants, of which approximately one third will be used for building commission capacity to support grantees (formula grantees) in the areas of evaluation and performance measurement;
  • $2,050,000 of FY18 Senior Corps funds will be spent on evaluation and evidence-building activities (representing 1 percent of Senior Corps’ $202.11 million FY18 budget); and
  • $1,671,766 of FY18 evaluation funds have been awarded for AmeriCorps grantee evaluation capacity building.
• In addition to these program evaluation and evidence capacity building investments, CNCS will invest close to $1.5 million in research grants that fund studies designed to systematically assess the outcomes and impacts of national service programs and volunteerism. These studies rely on large Federal survey and administrative data sets to statistically examine relationships between nonprofit organizations that rely on volunteers and other civil society organizations and various community outcomes (e.g., health and well-being). Furthermore, these studies examine individual outcomes such as the long-term economic outcomes of volunteering (using longitudinal survey data) or the effects of culturally relevant curriculum on the civic behavior of participants over time. Finally, these research grants fund studies that examine the role of civic participation in identifying and implementing solutions for community issues like opioid use, disaster recovery, academic success, and equitable economic development.

Did the agency implement a performance management system with clear and prioritized outcome-focused goals and aligned program objectives and measures, and did it frequently collect, analyze, and use data and evidence to improve outcomes, return on investment, and other dimensions of performance in FY18? (Example: Performance stat systems)

• CNCS has a focused set of Agency-Wide Priority Measures derived from the 2011-2015 Strategic Plan. Every CNCS Program contributes to the Agency-Wide Priority Measures. There are also specific grantee/sponsor measures that roll up into the Agency-Wide Priority Measures, which can be found in the Agency-Wide Priority Measures chart. Grantees are required to select at least one national performance measure, and they are required to report performance measures data annually. CNCS encourages grantees to use these measure for continuous program improvement. CNCS uses the agency-wide priority measures to assess its own progress toward attaining the goals and objectives of its strategic plan.
• The CNCS Chief of Staff has asked the agency’s Management Team of 18 Directors to identify a core business improvement to initiate in FY18 to advance the goal of strengthening core business functions (identified in the agency Transformation and Sustainability Plan). Each Director will be responsible for the project’s outcome, impacts, timeline, and metrics for determining success.
• The Senior Corps program has invested over $1 million in various management analyses that have been and will be used to inform operations, programming, and its research agenda moving forward. Focus groups with the field were conducted by an independent contractor to improve performance measurement, recruitment practices, volunteer incentive policies, partnership development, and information sharing among offices. Case studies were also funded (and are currently underway) to better understand various dimensions of program implementation. A contract will also be awarded this fiscal year to comprehensively assess the quality of program administrative data and its potential uses for performance management.
• The AmeriCorps NCCC program tracks five key performance indicators: (1) alignment of NCCC teams with state identified priorities, (2) in-kind contributions from project sponsor organizations and communities, (3) employee viewpoint trends, (4) member graduation rates, and (5) number of alumni remaining in the community post-graduation. A pilot was initiated in 2014 to determine how the program might pursue increasing its effectiveness while decreasing costs. A comparison of key performance indicators (e.g., number of service hours, number of projects and sponsors, member attrition) was made between two classes that served and graduated from the program prior to the pilot and three classes following pilot implementation. Findings from the pilot demonstrated that costs could be reduced while maintaining the same level of community service and increasing member retention in the program. The revised program model has since been implemented in three of four program locations with the third occurring in FY18.

Did the agency use a common evidence framework, guidelines, or standards to inform its research and funding decisions and did it disseminate and promote the use of evidence-based interventions through a user-friendly tool in FY18? (Example: What Works Clearinghouses)

• CNCS adapted the evidence framework used by its Social Innovation Fund and the Investing in Innovation Fund at ED in order to include it as part of the AmeriCorps State and National program’s FY16, FY17, and FY18 grant competitions.
• In March 2015, CNCS released Phase one of the CNCS Evidence Exchange, a virtual repository of reports intended to help CNCS grantees and other interested stakeholders find information about evidence- and research- based national service and social innovation programs. Phase one includes a database of single study reports with some additional descriptive information about the study, as well as a systematic review of the national service evidence base. Phase two in FY16 added studies as grantees completed their independent evaluations and submitted reports to CNCS. In FY17, CNCS focused on disseminating final reports as studies were completed and ensuring that the functionality of the site made the information as accessible as possible. In FY18, CNCS focused on enhancing the search function as more reports have been added.

Did the agency have staff, policies, and processes in place that encouraged innovation to improve the impact of its programs in FY18? (Examples: Prizes and challenges; behavioral science trials; innovation labs/accelerators; performance partnership pilots; demonstration projects or waivers with strong evaluation requirements)

•  CNCS awarded four AmeriCorps Evidence-Based Planning grants (an investment of $237,183 in evaluation funds) in FY18 (see grant announcements here). These one-year planning grants were awarded to encourage the identification of a new evidence-based program or practice and develop a strategy for integrating national service into the effective model. In addition, awardees will develop an evaluation plan to assess the innovation should it be funded in future grant competitions. R&E staff is conducting a process evaluation of these grantees to systematically assess the successes and challenges of implementing the grants.
• CNCS awarded seven AmeriCorps Evidence-Based Planning grants (an investment of $500,000 in evaluation funds) in FY17 (p. 5 and p. 20 of the FY18 AmeriCorps State and National Grants notice of funding opportunity). Of the seven Evidence-Based Planning grantees in 2017, twoapplied for competitive 2018 implementation grants, and one applicant was awarded. One grantee is planning to apply for a 2019 implementation grant. In addition, two of the four commission subgrantees applied for 2018 formula funding, and both applications were awarded.
• Lessons learned from Year 1 include the following: 1) Planning grants need a full year of planning time and R&E support. Currently they receive about three months of support before they must submit their application for the next implementation grant; and 2) the implicit expectation should be that a 2017 planning grantee will apply only for the 2019 funding cycle if a decision is made to implement and evaluate the intervention.
• CNCS anticipates awarding continuation funds for Commission Investment Fund grants in FY18. Overall, a similar percentage of these grants (roughly one third) will focus on building the capacity of State Commissions and their grantees to collect and use performance and evaluation data. R&E staff are completing a process evaluation of these grantees that will identify the successes and challenges of implementing these grants.

Did the agency use evidence of effectiveness when allocating funds from its five largest competitive grant programs in FY18? (Examples: Tiered-evidence frameworks; evidence-based funding set-asides; priority preference points or other preference scoring; Pay for Success provisions)

• • CNCS is operating two competitive grant programs in FY18: (1) the AmeriCorps State and National program (excluding State formula grant funds) ($244,064,965 million); and (2) Senior Corps RSVP program ($49 million). (The Social Innovation Fund (SIF) grants were integrated into the AmeriCorps State and National program. CNCS requested $50 million for SIF in FY17, but Congress eliminated funding for this evidence-based program in FY17 and again in FY18.)
  • CNCS’s AmeriCorps State and National Grants Program (excluding State formula grant funds), application (pp. 14-17) allocated up to 36 points out of 100 to organizations that submit applications supported by performance and evaluation data in FY18. Specifically, up to 24 points can be assigned to applications with theories of change supported by relevant research literature, program performance data, or program evaluation data; and up to 12 points can be assigned for an applicant’s incoming level of evidence, the quality of the evidence, and the applicant’s evaluation capacity. These categories of evidence are modeled closely on the levels of evidence defined in the Social Innovation Fund (see note above). An additional 8 points could be earned by demonstrating a culture of learning (e.g., collecting and using information for learning and decision making). In sum, 44 of 100 points are earned through demonstrating quality data, rigorous evidence, and the use of this information for continuous improvement and decision-making. The percentage of grant dollars allocated to strong, moderate, preliminary, and no evidence categories shifted between FY17 and FY18 (see chart below), such that more FY18 grant dollars were awarded to applicants with strong levels of evidence for proposed interventions, and fewer grant dollars were awarded to applicants with little to no evidence of effectiveness.

  Percentage of competitive AmeriCorps grant funds that support evidence-based projects	FY16	FY17	FY18
  Strong	20%	18%	26%
  Moderate	14%	11%	11%
  Preliminary	44%	45%	34%
  No Evidence	22%	26%	29%
  Total	100%	100%	100%

  • CNCS and the VA are outcome payers for the Veterans Coordinated Approach to Recovery and Employment (CARE) Pay for Success Project. The intervention is the Individual Placement and Support program, an evidence-based approach to supportive employment. The evidence base consists of 25 published RCTs and is also currently under review by the Laura and John Arnold Foundation for inclusion in their Moving the Needle Initiative. This project will initiate service delivery in the Summer 2018 with outcome payments beginning at 18 months and continuing in months 24, 30, and 42 of the project. CNCS funded this project through forward-funded resources from the Social Innovation Fund (see note above).
  • CNCS will initiate a process evaluation in FY18 as part of its Scaling Effective National Service Solutions initiative (initiated in FY16). An independent contractor reviewed grantee evaluation reports and scaling plans submitted in FY15, FY16, and FY17, and determined which interventions had sufficient evidence to warrant scaling as well as variation in scaling goals and experiences. Three grantees were selected for a process evaluation to systematically assess their scaling experiences. Findings from the evaluation will be used to inform grantmaking and provide guidance for how and when to scale effective service interventions.
  • CNCS published and began implementing its Transformation and Sustainability Plan in FY18. One of the six goals included in this plan is prioritizing evidence-based interventions. Specifically, “CNCS will further refine the intervention models it funds based on evidence and demonstrated success, while maintaining the flexibility to support statutory and Administration priorities. CNCS will also continue to support innovative interventions and approaches based upon preliminary evidence in order to build grantee capacity, foster innovation, and meet evolving community needs, and will evaluate these interventions to learn more about whether they are effective.”

Did the agency use evidence of effectiveness when allocating funds from its five largest non-competitive grant programs in FY18? (Examples: Evidence-based funding set-asides; requirements to invest funds in evidence-based activities; Pay for Success provisions)

• CNCS operates one formula grant program in FY18, the AmeriCorps State formula grants program ($137,453,226). CNCS also operates four direct grant programs in FY18: (1) AmeriCorps National Civilian Community Corps (NCCC) ($32 million); (2) AmeriCorps VISTA ($92 million); (3) Senior Corps Foster Grandparents ($108 million); and (4) Senior Corps Senior Companion Program ($46 million).
• In FY18, the Senior Corps Foster Grandparents and Senior Companion programs embedded evidence into their grant renewal processes by offering supplemental funding (“augmentation grants”) to grantees interested in deploying volunteers to serve in evidence-based programs and providing evaluation data on implementation fidelity, including outcomes. A total of $1.2 million is allocated for FY18 augmentation grants.
• NCCC will invest savings from efficiency measures into research and evidence activities, and VISTA will invest savings from reallocating member training dollars into evaluation projects –
  • NCCC plans to invest $1.2 million in FY18 in a contract that will allow CNCS to conduct three studies focusing on retention of members, leadership development among members attributable to their service experience, and an analysis of community impacts based on years of data on projects conducted by teams nationwide. NCCC will also invest approximately $300,000 to support three research grants that will examine civic engagement following natural disasters in relation to community resiliency with respect to sustainable environments.
  • VISTA will contribute approximately $50,000 to five research grants in FY18 that will study equitable development issues. CNCS is also working with the program to develop a comparative case study that will allow it to follow up in a sample of communities to assess the longer-term effects of volunteer mobilization, partnership development, and resources leveraged to sustain project work and achieve community impacts.

In FY18, did the agency shift funds away from or within any practice, program, or policy that consistently failed to achieve desired outcomes? (Examples: Requiring low-performing grantees to re-compete for funding; removing ineffective interventions from allowable use of grant funds; proposing the elimination of ineffective programs through annual budget requests)

• According to CNCS policy, Americorps considers past performance (e.g., meeting performance targets) as part of its criteria for making funding decisions. This assessment is in addition to the evaluation of the applicant’s eligibility for funding or the quality of its application on the basis of the Selection Criteria. Results from this assessment inform funding decisions every year for every grant competition. In evaluating programmatic performance, CNCS considers the following for applicants that are current formula and competitive grantees submitting applications for the same program model:
  • Grant progress reports – attainment of Performance Measures;
  • Enrollment and retention;
  • Compliance with 30-day enrollment and exit requirements in the AmeriCorps portal;
  • Site visit or other monitoring findings (if applicable);
  • Significant opportunities and/or risks of the grantee related to national service; and
  • Commission Rank.
• For example, in FY18, due to ongoing program design and performance issues, AmeriCorps provided ongoing and directive feedback to a long-standing grantee that it would face significant hurdles to securing future CNCS funding. The organization decided against submitting a grant application, and CNCS was able to repurpose almost $3 million to fund interventions implemented by other stronger, more impactful grantees.
• The AmeriCorps NCCC program reallocated significant mandatory and discretionary program resources in FY18 (for savings totaling an estimated $1,872,000) to optimize the program’s utilization of facilities and maintain responsiveness to community needs (e.g., teams located near areas frequently affected by natural disasters and more members residing in fewer federal facilities), to maximize the cost per unit (e.g., member), and to maintain the quality and effectiveness of service in the community. NCCC outsourced recruitment, created lodging efficiencies, consolidated member support services (e.g., administrative intake), and closed a campus that had the highest region cost per unit. Some of these savings (an estimated $400,000) will be invested in research and evaluation activities in FY18. These efficiency measures have a long term estimated savings of $4 million for reinvestment in program priorities.
• The AmeriCorps VISTA program started reallocating member training resources in FY16, moving from more resource-intensive, in-person training to virtual training. By the end of FY18, all member orientation training will be virtual. Member survey data indicate consistently positive results over this time period with regard to the utility of the training for a successful service experience, with over two thirds of respondents rating training and resources received from AmeriCorps as “excellent” or “good.” Preparing VISTA members to impact their communities is now achieved in a more cost-effective and efficient manner, and allows more flexible onboarding for the thousands of nonprofits hosting VISTAs around the country. VISTA is also collaborating with NCCC on a consolidated member support pilot, to merge support functions across the VISTA and NCCC programs to achieve efficiencies. VISTA also created a new Data Analyst position to standardize data tools and create new internal controls through automated flags and reports. Savings are being reinvested in targeted skill-based trainings for VISTA members and supervisors and evaluation activities.
• The AmeriCorps NCCC program tracks five key performance indicators: (1) alignment of NCCC teams with state identified priorities; (2) in-kind contributions from project sponsor organizations and communities; (3) employee viewpoint trends; (4) member graduation rates; and (5) number of alumni remaining in the community post-graduation. A pilot was initiated in 2014 to determine how the program might pursue increasing its effectiveness while decreasing costs. A comparison of key performance indicators (e.g., number of service hours, number of projects and sponsors, member attrition) was made between two classes that served and graduated from the program prior to the pilot and three classes following pilot implementation. Findings from the pilot demonstrated that costs could be reduced while maintaining the same level of community service and increasing member retention in the program. The revised program model has since been implemented in three of four program locations with the third occurring in FY18.

• CNCS awarded a contract in June 2018 to support the agency’s ability to review and synthesize research findings from program evaluations conducted by grantees on a continuous basis. The ability to independently assess the quality and rigor of evaluation studies as they are submitted to the agency is a critical capacity and pillar of building evidence infrastructure. In the past, this function was siloed within individual programs, but now one contract will support all agency programs – creating important efficiencies and consistencies for assessing evidence. Existing review and synthesis findings (see R&E’s State of the Evidence Report to see the numbers behind national service and practical benefits of volunteering) and their use in FY19 grantmaking and management are under discussion.