Did the agency have a senior staff member(s) with the authority, staff, and budget to evaluate its major programs and inform policy decisions affecting them in FY18? (Example: Chief Evaluation Officer)

• The Administration for Community Living (ACL) is led by the Administrator who oversees five major units: the Administration on Aging (AoA), Administration on Disabilities (AoD), Center for Integrated Programs (CIP), Center for Policy and Evaluation (CPE), and the National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR).
• CPE houses the Office for Performance and Evaluation (OPE). OPE is the primary office that oversees evaluation efforts within ACL. OPE is led by a Director who is a GS-15, senior career civil servant, who reports directly to the Director of CPE who reports directly to ACL’s Administrator. In addition to the Director, there are five other full time staff overseeing 17 evaluation-related contracts. OPE’s Federal Fiscal 2018 budget for evaluation and performance management was approximately $10 million dollars. This budget represents dedicated funding authorized through legislation and is administered by the Director of OPE.
• Evaluation and performance information is communicated to leadership to support policy decisions through weekly meetings between the Director of OPE and CPE, weekly meetings between the Director of CPE and the Principal Deputy Administrator and Acting Commissioner on Disabilities, and quarterly meetings between the Director of OPE and the Principal Deputy Administrator and Acting Commissioner on Disabilities.
• Under an overarching analytic support contract, ACL’s program offices are able to transfer funds to OPE, which allows OPE staff to provide oversight and guidance regarding evaluation design and implementation, performance measurement development, and the interpretation of data for decision-making. This approach promotes coordination and allows close partnerships between programmatic experts and the evaluation experts in OPE to help programmatic staff develop needed evidence to support policy and funding decisions.
• ACL’s evaluation policy describes ACL’s commitment to conducting rigorous, relevant evaluations and to using evidence from evaluations to inform policy decisions and practice across the agency. It indicates ACL’s interest in conducting outcome-focused evaluations for all ACL programs, and in promoting rigor, relevance, transparency, independence, and ethics in the conduct of evaluations. This policy gives the director of OPE the authority to approve the design of evaluation projects and analysis plans; and the authority to approve, release and disseminate evaluation reports. ACL’s evaluation policy “applies to all ACL-sponsored evaluations” and states that “OPE and program offices will work in partnership to inform potential applicants, program providers, administrators, policy-makers and funders through disseminating evidence from ACL-sponsored and other good quality evaluations.”
• At the start of each budget cycle, the Director of ACL’s Center for Policy and Evaluation sends ACL center Directors a one-page document titled “Policy Review of ACL Funding Opportunity Announcements (FOAs).” As part of this review process, OPE staff review Funding Opportunity Announcements (FOA) to determine: “Does the FOA provide enough detailed information to help applicants think critically about the measures they should include in their applications to effectively measure their progress towards meeting the goals outlined in the FOA?” OPE staff then make specific recommendations to program staff to improve language in their FOAs around performance measurement and evaluation to help ensure that program staff have evidence to inform future funding and policy decisions.
• While OPE directly oversees most of ACL’s evaluations, ACL’s, NIDILRR conducts its own evaluations (NIDILRR External Evaluation and NIDILRR Performance and Evaluation). Coordination between OPE staff and NIDILRR evaluation staff occurs through participation of NIDILRR evaluation staff in bi-weekly OPE staff meetings.

Did the agency have an evaluation policy, evaluation plan, and research/learning agenda(s) and did it publicly release the findings of all completed evaluations in FY18?

• ACL has an agency-wide evaluation policy that reconfirms ACL’s commitment to conducting rigorous, relevant evaluations and to using evidence from evaluations to inform policy and practice. The evaluation policy addresses how ACL promotes coordination between evaluation staff and policymakers as well as stressing the importance of the involvement of policymakers in the development of evaluation questions. ACL’s evaluation policy stipulates that “ACL will release evaluation results regardless of the findings. Evaluation reports will describe the methods used, including strengths and weaknesses, and discuss the generalizability of the findings. Evaluation reports will present comprehensive results, including favorable, unfavorable, and null findings. ACL will release evaluation results timely – usually within six months of a report’s completion.”
• All completed evaluation reports are posted on the ACL website (see also NIDILRR External Evaluation). Authorizing legislation for ACL programs also specify that evaluation be conducted and that the results will be made available to the public (e.g., Older Americans Act Title II, Section 206, Developmental Disabilities Assistance and Bill of Rights Act (DD Act) Title II Section 210, Workforce Innovation and Opportunity Act Chapter III subtitle D Section 169, Elder Justice Act Part II Section 2044).
• For an evaluation plan, ACL’s Office of Performance and Evaluation (OPE) submits a concept paper to the Principal Deputy Administrator and Acting Commissioner on Disabilities outlining proposed evaluation activities for each upcoming year. This plan reflects conversations between OPE staff, Agency leadership, and program staff regarding policy priorities. It describes how OPE will allocate it resources to answer identified evaluation questions, and to provide sound evidence regarding how well programs are meeting their stated goals as well as recommendations for program improvement.
• While ACL has not completed a formal agency-wide learning agenda, ACL has a process for developing Center-specific learning agendas that will form the basis for an eventual agency-wide learning agenda to be completed and released in FY19. The process involves annual reviews witheach ACL Center to support the generation and use of evaluation findings to inform agency strategies and decision making. Specifically, a series of interviews with Center Directors is conducted immediately prior to the development of Center funding proposals and include discussion of the most important questions that need to be answered in order to improve program implementation and performance; ways to strategically prioritize these questions given the level of current understanding, available resources, feasibility, and other considerations; appropriate tools and methods to answer each question; and approaches for information dissemination that are accessible and useful to ACL leadership. ACL anticipates piloting this process in late fall of 2018.
• The Long-Range Plan of the National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR) publishes a five-year agenda that will advance its research efforts (i.e. learning agenda).

Did the agency invest at least 1% of program funds in evaluations in FY18? (Examples: Impact studies; implementation studies; rapid cycle evaluations; evaluation technical assistance, and capacity-building)

• ACL spent approximately $16 million on evaluation in FY18, representing .88% of the agency’s $1.8 billion FY18 budget.
• The bulk of these funds are based on a set-aside required under the Older Americans Act. As specified in Title II, section 206 of the Older Americans Act, “From the total amount appropriated for each fiscal year to carry out title III, the Secretary may use such sums as may be necessary, but not to exceed 1⁄2 of 1 percent of such amount, for purposes of conducting evaluations under this section, either directly or through grants or contracts.”

Did the agency implement a performance management system with clear and prioritized outcome-focused goals and aligned program objectives and measures, and did it frequently collect, analyze, and use data and evidence to improve outcomes, return on investment, and other dimensions of performance in FY18? (Example: Performance stat systems)

• OPE is responsible for performance management, which includes approximately 25 output and outcome measures reported in annual budget justifications. These results are reviewed annually by ACL leadership. OPE, in coordination with ACL’s Center for Management and Budget, leads the ACL’s internal Continuous Process Improvement (CPI) Program which supports agencies in efforts to gain operational efficiencies and improve performance.
• ACL’s performance strategy presents a high-level approach to the planning, conduct, and implementation of performance management. The strategypresents a high-level approach to the planning, conduct, and implementation of performance management and represents ACL’s commitment to providing rigorous, relevant, and transparent performance data highlighting all the programs and initiatives ACL supports. This strategy describes (p. 2) how ACL’s ”performance data is reported and tracked (1) to monitor the administration’s progress towards achieving our departmental and agency strategic goals, objectives, and priorities (2) to support ACL’s budget justifications; and (3) to monitor program performance and support improvement.” ACL developsand maintainsa repository of high quality and robust performance data on all of ACL programs and business lines to demonstrate the impact of programs and services. And ACL, as described under Goal 4 (p. 1), works to “Encourage the utilization of the performance strategy and data in policy and practice to enhance planning and decision-making enabling ACL to easily track goals, objectives and performance across the agency.” Implementation of this strategy, in combination with the center-specific learning agenda process and NIDILLR’s Long-Range Plan (Described in criterion two), contributes to ACL’s development of an Agency-wide learning agenda.
• Current information about ACL performance is available in reports to Congress and Congressional budget justifications. The 2018 justification provides the overview of ACL’s performance management approach:
  • With its aging programs, ACL focuses on three categories of performance measures: (1) improving consumer outcomes; (2) effectively targeting services to vulnerable populations; and (3) improving efficiency. Each measure is representative of activities across the Aging Services Program budget, and progress toward achievement is tracked using a number of indicators.
  • ACL has implemented a quality review system (QRS) for developmental disability programs under ACL’s/AIDD. The QRS uses a three-tiered model to review program compliance, outcomes, and fiscal operations and use review results to target and coordinate technical assistance. The first tier is annual standardized review. The second tier is standardized, in-depth review involving a team of reviewers conducted on a periodic basis. Tier three is customized monitoring for programs for which ACL has significant concerns in terms of compliance and performance.
• ACL continues development of a formula grant monitoring framework for Older Americans Act Title III and VII state formula grants. The framework combines assessments of grantee’s progress toward program goals and objectives with identification of risk or instances of fraud, waste and abuse.
• There is a rigorous process in which each office within ACL develops Program Funding Plan Memoranda which detail the proposed discretionary grant and procurement activities for the office and justify each proposed activity consistent with ACL’s mission and performance measures. Senior leadership has established processes for use of performance data for management decision-making, including a quarterly discretionary dashboard, bi-weekly reports for the Administrator/Assistant Secretary, quarterly reviews of operating budgets, quarterly managers’ meetings and bi-weekly center director meetings.
• NIDILRR has developed a data and performance management program to measure progress and track outcomes of competitive grant recipients. The program has posted and made available annual performance data from grantees to all NIDIILRR staff from 2007 to 2013. A public summary can be accessed online.
• Annually, all ACL grantees report their performance data which is used to inform agency budget justifications, publicly available reports, and for the delivery of technical assistance to support continuous program improvement. For example:
  • NIDILRR uses a web-based platform. Additionally, grantees are required to complete the final performance report module within 90 days of the grant’s end-date. This report captures overarching, summative results and insights gained from the performance period which grant officers feed back to grantees as part of its continuous quality improvement approach.
  • Many of ACL’s Disability Program grantees use a system called ACLReporting to report on performance annually or semiannually.
  • Older American Act Title III and VII grantees use as system called NAPIS/CARDS for annual performance reporting. In FY 2018 ACL piloted a new web-based reporting system to improve the user experience and include additional data security and verification tools. These performance data, as well as data for the OAA Title VI program and data from a national survey which are used for performance reporting, are available in the Aging Integrated Database.
  • SMP/ SHIP programs have complex and standalone data systems. Their performance is reported to ACL and is used to provide more focused TA and support for the programs. Data from both programs is publicized in a number of ways. Each quarter, all State Health Insurance Assistance Programs (SHIP) get a report from ACL to show how they are doing with a county by county breakdown with quality Likert scores and quantitative analysis of their reports. Project officers use this to help states understand how they are doing and where they can improve.

• ACL makes the majority of its administrative and survey datasets publicly available through the following systems:
  • The National Adult Maltreatment Reporting System provides national data on the exploitation and abuse of older adults and adults with disabilities. ACL has developed a best practices and policies for data submission and specifications.
  • The Aging Integrated Database (AGID) is an integrated, user-friendly online data system, including data on ACL programs and Census bureau data.
  • The Burn Model System (BMS) provides a comprehensive and longitudinal record of health and community outcomes of burn survivors with more severe injuries. It is the only project that collects long-term outcomes on both pediatric and adult patients to better understand the relation between the injury, acute care, rehabilitation, and long-term functioning of people with burn injury.
  • NIDILRR’s Traumatic Brain Injury Model Systems National Data and Statistical Center (TBINDSC) advances medical rehabilitation by increasing the rigor and efficiency of scientific efforts to longitudinally assess the experience of individuals with traumatic brain injury (TBI).
  • The National Spinal Cord Injury Statistical Center (NSCISC) supports and directs the collection, management and analysis of the world’s largest and longest spinal cord injury research database.
• The Office of Information Resources Management manages ACL information and technology services, including providing IT governance and managing network security and privacy responsibilities. Prior to the collection of data, all programs must complete Privacy Impact Assessments specifying how performance and evaluation data will be secured and its privacy protected.
• ACL has an internal policy requiring that all evaluation contracts include the following language: “The contractor shall develop an IT Security Plan and conduct related security assessments in accordance with the Federal Information Security Management Act (FISMA). The IT Security Plan must ensure the integrity, confidentiality, when appropriate, and availability of all data collected on behalf of the Federal government. All records that are the property of the Federal government must be maintained in accordance with HHS policies and procedures, and National Archives and Records Administration (NARA) disposition schedules. The Contractor shall provide a certification statement concerning the proper maintenance of all records to the COR at the beginning of the contract. The Contractor shall discuss the disposition of records with the COR and obtain COR approval before any records are disposed. The Contractor shall notify the COR within 24 hours concerning any loss of data integrity, any unauthorized disclosure of data, or any misuse of data.” Contractor staff are required to sign confidentiality agreements prior to accessing any ACL data.
• The majority of ACL’s data is aggregated at the grantee (e.g., State) level, limiting the ability to link those data to other datasets. For individual evaluation projects, for which individual-level data are collected, ACL has had success using administrative data sets from other federal agencies.
  • Specifically, as part of the Older Americans Act Nutrition Services Program (NSP) evaluation, ACL’s contractor, Mathematica, used Medicare claims and enrollment data to construct outcome measures and define Medicare beneficiary characteristics such as hierarchical condition category (HCC) scores, the original reason for an individual’s Medicare eligibility, whether the individual had dual enrollment in Medicare and Medicaid, and whether the individual had chronic conditions.
  • To describe NSP participants’ geographic access to food, the contractor used residential address information for each respondent in the outcomes survey, data from the Census Bureau, and address data for food retailers from the U.S. Department of Agriculture (USDA). Using this information, the research team calculated measures of geographic access to food and determined whether a respondent lived in an urban or rural area.
  • The research team used data from the American Community Survey to obtain local-area population characteristics to better describe the communities in which meal participants lived.
• ACL collects administrative data from grantees to improve its programs and the capacity of service providers:
  • In 2014, ACL developed a Dementia Capability Assessment Tool in support of their programs to expand dementia capability in communities. In 2017 the tool was translated into an onlineformat, making it possible to analyze each program’s progress toward dementia capability in its entirety or broken down by sector.
  • VD-HCBS data is used to understand the effects of the program and to provide targeted TA to network locations that are having issues with the VA program.
  • AIDD is building the capacity of state developmental disabilities agencies to gather vital information on service outcomes through the National Data Measurement Project and the adoption of the National Core Indicators (NCI) as the uniform dataset. The NCI framework comprises over 100 key outcome indicators that are designed to gather valid and reliable data across five broad domains: individual outcomes; family outcomes; health, welfare, and rights; staff stability; and system performance.

Did the agency use a common evidence framework, guidelines, or standards to inform its research and funding decisions and did it disseminate and promote the use of evidence-based interventions through a user-friendly tool in FY18? (Example: What Works Clearinghouses)

• The National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR) uses a stages of research framework (SORF) to classify and describe its funded grants and research projects within the grants. The four stages of SORF include: exploration and discovery, intervention development, intervention efficacy, and scale-up evaluation. Using SORF, NIDILRR gains insight into what is known and unknown about a problem; whether it is time to develop interventions to address a particular problem; whether it is to time test the efficacy of interventions; and whether it is time to “scale-up” interventions for broader use.
• The Older Americans Act (OAA) requires the use of evidence-based programming in Title III-D-funded activities, Disease Prevention and Health Promotion Services. In response, ACL developed a definition of the term evidence-based, and created a website containing links to a range of resources for evidence-based programs. This is a common evidence framework used for OAA-funded activities for this particular program.
• While ACL does not endorse specific programs, it does publish intervention summaries of aging and disability evidence-based programs and practices that include: general information about the intervention; a description of the research outcomes reviewed; ratings of the quality of research and readiness for dissemination; a list of studies and materials reviewed; information about the translation of the intervention to additional settings and populations; and, contact information to obtain more information about implementation or research.The information provided was modeled after the Substance Abuse and Mental Health Services Administration’s (SAMHSA) National Registry of Evidence-Based Programs and Practices (NREPP).
• ACL works through its resource centers to help grantees use evidence to drive improvements in outcomes for older adults and individuals with disabilities.
• For example, with funding from ACL, the National Resource Centers at NCOA, in collaboration with the Evidence-Based Leadership Council, lead an innovative vetting process to increase the number of programs available to ACL’s aging network that meet the Title III-D evidence-based criteria. This process has resulted in adding six new health promotion programs and three new programs for preventing falls. Another round of program reviews is currently underway.
• To support the use of evidence-based and evidence-informed programming, service providers can find out about evidence-based programs that serve people with dementia and their caregivers by consulting a white paper drafted with funds from ACL – Translating Innovation to Impact: Evidence-based interventions to support people with Alzheimer’s disease and their Caregivers at home and in their Communities.
• The Alzheimer’s Disease Supportive Services Program (ADSSP) funds competitive grants limited to States to expand the availability of evidence-based services that support persons with Alzheimer’s disease and related dementias (ADRD) and their family caregivers; and to create state-wide, person-centered, dementia-capable home and community-based service (HCBS) systems. Half of the funding must be used to provide direct services.
• In 2015, ACL’s Aging and Disability Evidence-Based Programs and Practices initiative developed a toolkit, currently available on the ACL website, to help the public learn about evidence-based programs and interventions for implementation at the community-level and to promote their utilization.
• The Model Systems Knowledge Translation Center (MSKTC) has worked with NIDILRR’s Model Systems grantees to develop and publish a variety of evidence-based factsheets about living with spinal cord injury, traumatic brain injury, or burn injury – written in language that all users can read and understand.
• ACL’s Alzheimer’s Disease Supportive Services Program (ADSSP) encourages the translation of dementia specific interventions shown to improve the health and well-being of persons with Alzheimer’s disease and related dementia and/or their caregivers and implemented into communities. Since the introduction of evidence-based programming into the agencies Alzheimer’s and dementia programs more than 20 evidence-based interventions have been introduced and continue to be translated to states and communities.  ACL’s requirement for inclusion of dementia specific evidence-based interventions is demonstrated in the 2018 funding opportunity announcement entitled Alzheimer’s Disease Programs to States and Communities.
• To build the evidence base, ACL funded cooperative agreements for the development and testing of model approaches towards coordinated and comprehensive systems for enhancing and assuring the independence, integration, safety, health, and well-being of individuals with intellectual and developmental disabilities living in the community. In FY18, ACL implemented an evaluation of these model programs to determine whether the models implemented across the sites impact the quality of life of individuals with developmental disabilities as follows. Information from the evaluation will be used to inform future ACL funding announcements and the training and technical assistance provided by ACL to communities serving individuals with intellectual and developmental disabilities to ensure the integration of evidence into ACL’s programming. While the evaluation is not yet complete, initial findings about what works were integrated into the requirements of the funding announcement for the FY18 award cycle.

Did the agency have staff, policies, and processes in place that encouraged innovation to improve the impact of its programs in FY18? (Examples: Prizes and challenges; behavioral science trials; innovation labs/accelerators; performance partnership pilots; demonstration projects or waivers with strong evaluation requirements)

• ACL has historically used innovation dollars provided under Title IV of OAA as a means of testing new approaches to service delivery and developing replicable models that could then be embedded into core programs. One example of how this approach has been used effectively is the Title III-D Preventive Health Services Program. Since 2003, ACL has tested evidence-based models for delivering health services in community-based settings across a variety of areas, including Chronic Disease Self-Management, Falls Prevention, Enhanced Fitness, and Medication Management. As these models proved more successful than traditional approaches, states increasingly shifted their funding to provide greater support to evidence-based approaches.
• In FY18, ACL released a new grant opportunity from the National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR) under the Small Business Innovation Research Program (SBIR), Phase I. An estimated 10 grants will be awarded to small businesses. The purpose of the SBIR program is to stimulate technological innovation in the private sector, strengthen the role of small business in meeting federal research or research and development needs, and improve the return on investment from federally-funded research for economic and social benefits to the nation.
• In FY16, ACL established the Elder Justice Innovation Grants program to increase knowledge about effective prevention and intervention of abuse, neglect, and exploitation of older adults, native elders, adults with disabilities, people who self-neglect, and guardianship abuse. In FY18, ACL monitored the second year of two-year grants awarded to five nonprofit organizations in FY17 totaling $2.2 million.
• In 2015, ACL awarded a grant, Advancing Person-Centered, Trauma-Informed Supportive Services for Holocaust Survivors, to develop and implement person-centered, trauma-informed supportive services for Holocaust survivors living in the United States. In FY18, the program worked with one grantee and more than 50 community-based sub-grantees to develop innovative interventions that seek to minimize the risk of re-traumatization by receiving supportive services. To date, more than 40 new models of care have been developed.
• In FY18, ACL will award grants under the Alzheimer’s Disease Programs to States and Communities (HHS-2018-ACL-AOA-ADPI-0307). The ACL’s Alzheimer’s Disease Programs Initiative (ADPI) is informed by the work and recommendations of National Alzheimer’s Project Act Advisory Committee. The ADPI is implemented with authority contained within Title IV of the Older Americans Act and is designed to pilot dementia-capable HCBS programs to states and communities, evaluate program outcomes and use program outcome data to garner support to sustain successful initiatives beyond the federal program period.
• NIDILRR participates in the Small Business Innovation Research program to improve the lives of people with disabilities through research and development of innovative products generated by small businesses, and to increase the commercial application of NIDILRR-supported research results and development products.
• ACL funds resource centers to promote innovation, such as the Center for Assistive Technology Act Data Assistance. The Innovations in Nutrition Programs and Services program tests innovative and promising practices that move members of the aging services network towards evidenced-based practices that enhance the quality and effectiveness of nutrition services programs.

• In FY18, the five largest competitive grants programs are: (1) National Institute on Disability, Independent Living, and Rehabilitation Research ($95 million); (2) Independent Living ($78 million); (3) University Centers for Excellence in Developmental Disabilities (UCEDD) ($39 million); (4)Medicare Improvements for Patients and Providers Act Programs (MIPPA) ($38 million); and (5) Alzheimer’s Disease Program ($18 Million).
• ACL grant awards are made, in part, based on the clarity and nature of proposed outcomes and whether the proposed project evaluation reflects a thoughtful and well-designed approach that will be able to successfully measure whether or not the project has achieved its proposed outcome(s); includes the qualitative and/or quantitative methods necessary to reliably measure outcomes; and is designed to capture “lessons learned” from the overall effort that might be of use to others, especially those who might be interested in replicating the project. To the extent that grantees had completed similar work in the past, they are asked to demonstrate in their funding applications that those efforts were successful. Further, grantees are required to submit data through the ACL Reporting tool. The data are reviewed and, as needed, technical assistance is provided to grantees.
• At the start of each budget cycle, ACL’s Center for Policy and Evaluation sends ACL centers a one-page document titled “Policy Review of ACL Funding Opportunity Announcements (FOAs).” As part of this review process, OPE staff review Funding Opportunity Announcements to determine “Does the FOA provide enough detailed information to help applicants think critically about the measures they should include in their applications to effectively measure their progress towards meeting the goals outlined in the FOA?” OPE staff make specific recommendations to program staff to improve language around performance measurement and evaluation. The degree to which applications provide detailed information about their expected outcomes and how they will measure those outcomes informs funding decisions.
• The NIDILRR evaluation was conducted by the National Academy of Sciences (NAS) and was released in 2012. A ten-year evaluation plan was developed based on the NAS evaluation. The plan includes a set of research questions aimed at assessing the effectiveness and efficiency of NIDILRR’s operations as well as the quality and impacts of NIDILRR-funded activities and products. Implementation is on-going and helps to guide funding priorities and decisions by ensuring that they are more closely tied to the evidence about promising practices produced by prior NIDILRR grantees.
• In FY18 NIDILRR released a competitive grant with the express purpose of building evidence of the effectiveness of exercise interventions or programs for improving and sustaining health and health related quality of life (HRQOL) among people with disabilities, and to determine the extent to which improved HRQOL is related to improved community participation outcomes.
• The Long-Range Plan of the National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR) describes ways in which “NIDILRR proposes to support competitions that build on prior investments that resulted in evidence of efficacy and effectiveness. These competitions will provide funding for further development and testing of practices and interventions in additional settings, or among new populations of people with disabilities. These efforts may support translational research to develop practical strategies for ensuring more widespread use of new evidence-based findings in the area of disability and rehabilitation research and development.”
• Independent Living (IL) programs include a mix of formula and discretionary grants. The Centers for Independent Living (CILs) Program provides 354 discretionary grants to centers that are consumer-controlled, community-based, cross-disability, nonresidential, private nonprofit agencies who provide IL services. To continue receiving CIL program funding, eligible centers must provide evidence that they have previously had an impact on the goals and objectives for this funding including:
  • Promotion of the IL philosophy;
  • Provision of IL services on a cross-disability basis;
  • Support for the development and achievement of IL goals chosen by the consumer;
  • Efforts to increase the availability of quality community options for IL;
  • Provision of IL core services and, as appropriate, a combination of any other IL service;
  • Building community capacity to meet the needs of individuals with significant disabilities; and
  • Resource development activities to secure other funding sources.
• UCEDDs are a nationwide network of independent but interlinked centers, representing an expansive national resource for addressing issues, finding solutions, and advancing research related to the needs of individuals with developmental disabilities and their families. Applications are also reviewed based on their description of current or previous relevant experience and/or the record of the project team in preparing cogent and useful reports, publications, and other products.
• MIPPA funds are awarded to State grantees and to theNational Center for Benefits Outreach and Enrollment. To continue funding without restrictions, State grantees are required to submit state plans that ACL staff review for the specific strategies that grantees will employ to enhance efforts through statewide and local coalition building focused on intensified outreach activities to help beneficiaries likely to be eligible for the Low Income Subsidy program (LIS), Medicare Savings Program (MSP), Medicare Prescription Drug Coverage (Part D) and in assisting beneficiaries in applying for benefits. The plans also require that States reflect successes achieved to date and direct their efforts to enhance and expand their MIPPA outreach activities. The National Center applicants must describe the rationale for using the particular intervention, including factors such as evidence of intervention effectiveness.
• For Alzheimer’s Disease Programs Initiative (ADPI) funding, “If the applicant has held an ADSSP grant between 2011 and 2017, they must explain the work of their previous dementia systems project.” Applicants must also “describe the rationale for using the particular intervention, including factors such as: “lessons learned” for similar projects previously tested in your community, or in other areas of the country; and factors in the larger environment that have created the “right conditions” for the intervention (e.g., existing social or economic factors that you’ll be able to take advantage of, etc.)” ACL promotes evidence building in this area through activities such as a 2017 research summit on dementia care. The goal of the research summit was to identify what is known and what new knowledge is needed in order to accelerate the development, evaluation, translation, implementation, and scaling up of comprehensive care, services, and supports for persons with dementia, families, and other caregivers. The summit focused on research needed to improve quality of care and outcomes across care settings, including quality of life and the lived experience of persons with dementia and their caregivers. Grant applicants and awardees are encouraged to use such information to inform their programming.
• Although not one of its five largest competitive grants, ACL awarded a cooperative agreement titled “Living Well-Model Approaches for Enhancing the Quality, Effectiveness and Monitoring of Home and Community Based Services for Individuals with Developmental Disabilities” to design, identify, and implement a range of evidence based practices and/or innovative strategies focusing on improving services in the community that support people with developmental disabilities living in the community or those moving to the community from a more restrictive setting. Successful applicants are required to demonstrate expertise and experience in improving quality outcomes for individuals with developmental disabilities.

Did the agency use evidence of effectiveness when allocating funds from its five largest non-competitive grant programs in FY18? (Examples: Evidence-based funding set-asides; requirements to invest funds in evidence-based activities; Pay for Success provisions)

• In FY18, the five largest non-competitive grants programs are: (1) Nutrition Services ($835 million, which includes both congregate meals (e.g., served at senior centers) and home delivered meals); (2) Home and Community-Based Supportive Services ($348 million); (3) Family Caregiver Support Services ($150 million); (4) Partnerships for Innovation, Inclusion and Independence ($45 million); and (5) Developmental Disabilities – Protection and Advocacy ($39 million).
• Eligibility and funding levels for mandatory (formula) grants are based on specific legislation. While non-competitive grants are mandatory (formula) grants for ongoingprograms, which require no application or competition, ACL regional staff work closely with awardees to monitor performance including through review and approval of annual performance data and through the provision of technical assistance both on an as-needed basis and through regularly scheduled grantee meetings
• ACL’s three largest non-competitive grant programs (Nutrition Services, Home and Community-Based Supportive Services, and Family Caregiver Support Services) are funded under the Older American’s Act and, therefore, are required:
  • to participate in periodic evaluations that build the evidence base. OPE is currently conducting evaluations of the Nutrition Services program (Title III-C) and the Family Caregiver support Services program (Title III-E); and
  • to submit state plans that use uniform procedures for determining need for services which should guide the plan for delivering those services under the grants. This section also requires that State grantees conduct periodic evaluations of, and public hearings on, activities and projects carried out in the State including evaluations of the effectiveness of services provided to individuals with greatest economic need, greatest social need, or disabilities; and
  • to provide technical assistance about the use of evidence-based programs and model programs, which ACL does through its resource centers. For example, resource centers can help meal providers to ensure that meals provided through these programs fulfill the standards set by the current Dietary Guidelines for Americans and which are required under the Older Americans Act.
• These three programs also require applicants to submit State plans, which must be approved prior to funding awards. The State plans are assessed for the degree to which applicants have:
  • Documented tangible outcomes achieved as a result of previous state long-term care reform efforts; and
  • Translated activities, data (i.e., evidence), and outcomes into proven best practices which can be used to leverage additional funding.
• The PIII program was proposed in FY 2018 as a new program. ACL proposes to work with Congress on the development of authorization language for this new program including methods for using evidence to allocate and monitor funding.
• While not one of ACL’s largest non-competitive grant programs, the Health Promotion program (part of ACL’s Preventative Health Services funded at $1.8 million) requires that funds be spent on evidence-based programs. This change followed a decade of progress by the aging services network to move efforts toward implementing disease prevention and health promotion programs that are based on scientific evidence and demonstrated to improve the health of older adults. The federal FY12 Congressional appropriations law included, for the first time, an evidence-based requirement.

In FY18, did the agency shift funds away from or within any practice, program, or policy that consistently failed to achieve desired outcomes? (Examples: Requiring low-performing grantees to re-compete for funding; removing ineffective interventions from allowable use of grant funds; proposing the elimination of ineffective programs through annual budget requests)

• ACL released two new funding announcements in FY18 exploring methods for potentially redirecting funds from lower performing programs to higher performing ones:
  • The Paralysis Resource Center State Pilot Program is part of an effort to ensure program efficiency and to test two approaches for making sub-awards to community-based organizations that provide long-term services and supports to people with paralysis, their families, and their support networks. Outcomes from the pilot will help ACL assess the most effective and efficient ways to make such sub-awards and will determine how ACL funds this effort going forward.
  • Under OPE’s overarching analytic support contract, ACL is developing a tool to help grant officers more easily monitor the degree to which Chronic Disease Self-Management and Falls Prevention grantees are meeting their ACL approved program completion targets. ACL staff will use this information to either fully release or restrict grant funds over the life of the multi-year awards.
  • The Innovations in Nutrition Programs and Services grants will be monitored using a new tool that allows grant officers to more clearly determine grantees’ progress towards meeting their targeted service levels and therefore restrict or withhold funding based on that progress.
• Based on a review of programmatic purpose and outcomes, ACL’s FY18 budget request proposed to:
  • In part, based on work with researchers which was disseminated through a 2014 webinar series, ACL consolidated three small programs that serve people affected by Alzheimer’s Disease and Related Dementias programs into a single Alzheimer’s Disease Program, funded at the same total level. This will increase programmatic flexibility and efficiency and enable grantees to better meet the needs of people affected by these illnesses.
  • Based on a lack of evidence regarding the unique benefits of these programs, ACL proposed to devolve program responsibilities to states or local communities for the Limb Loss Resource Center and Paralysis Resource Center. The mission and activities carried out by these programs are duplicative of other Federal efforts. Savings from eliminating these redundant programs total $3 million and $8 million, respectively.
• ACL has implemented a quality review system (QRS) for developmental disability programs under ACL/AIDD. The QRS uses a three-tiered model to review program compliance, outcomes (i.e., evidence), and fiscal operations and use review results to target and coordinate technical assistance. The first tier is annual standardized review. The second tier is standardized, in-depth review involving a team of reviewers. These reviews are conducted on a periodic basis. Tier three is customized monitoring for programs that ACL has significant concerns regarding terms of compliance and performance. ACL continues development of a formula grant monitoring framework for Older Americans Act Title III and VII state formula grants. The framework combines assessments of a grantee’s progress toward program goals and objectives with identification of risk or instances of fraud, waste, and abuse.These reviews allow ACL, if warranted, to restrict grant funding based on findings of insufficient evidence of performance.

• The Developmental Disabilities Protection and Advocacy program (PADD) establishes and maintains a system to protect the legal and human rights of all persons with developmental disabilities. The annual performance measure of the percentage of individuals who have their complaint of abuse, neglect, discrimination, or other human or civil rights corrected compared to the total assisted demonstrates the rate of successful benefits accruing from the program. The rate of success has been consistently over 80 percent and trending upward since FY11. In FY15, the target was exceeded with nearly 87 percent of consumers having their complaint corrected. While funding is required under statute, as specified in criterion two in its Paperwork Reduction Act application to the Office of Management and Budget, ACL uses evidence submitted through the annual performance reporting process to “gauge for program accomplishments against program objectives for purposes of identifying continuing challenges and formulating technical assistance and management support provided to P&A systems.”