2018 Invest in What Works State Standard of Excellence
U.S. Department of Housing & Urban Development
Did the agency have a senior staff member(s) with the authority, staff, and budget to evaluate its major programs and inform policy decisions affecting them in FY18? (Example: Chief Evaluation Officer)
- The United States Department of Housing and Urban Development’s (USHUD or HUD) Office of Policy Development & Research (PD&R) informs HUD’s policy development and implementation by conducting, supporting, and sharing research, surveys, demonstrations, program evaluations, and best practices. PD&R achieves this mission through three interrelated core functions: (1) collecting and analyzing national housing market data (including with the Census Bureau); (2) conducting research, program evaluations, and demonstrations; and (3) providing policy advice and analytic support to the HUD Secretary and program offices.
- PD&R is led by an Assistant Secretary who oversees six offices, about 139 staff including a team of field economists that work in HUD’s 10 regional offices across the country, and a budget of $89 million in FY18. The Assistant Secretary ensures that evidence informs policy development through frequent personal engagement with other principal staff, the Secretary, and external policy officials including Congress, speeches to policy audiences, sponsorship of public research briefings, and policy implications memoranda.
- PD&R staff is integral to departmental working groups focused on program-specific or cross-cutting policy development and initiatives. As part of such engagement, PD&R ensures that deliberations are informed by program evaluations and the research literature and conducts policy studies, regulatory impact analyses, and special-purpose analyses of administrative data or external data to answer specific policy questions that arise. As part of ongoing research in FY18, PD&R continues to undertake evaluations of the programs that account for the vast majority of HUD’s outlays, including evaluations of key outcomes for rental assistance programs (Moving to Work and Moving to Work Expansion, Family Self Sufficiency, Rental Assistance Demonstration, Rent Reform), homeless assistance programs (Family Unification Program), and the Housing Counseling program.
- PD&R regularly engages with each HUD program office to ensure that metrics, evaluations, and evidence inform program design, budgeting, and implementation. Periodic meetings enable PD&R to inform program offices about evaluation progress, and program offices to share knowledge with PD&R about emerging needs for research, evaluation, and demonstrations to advance program policy. Such collaboration has ensured that major policy changes have developed through rigorously evaluated program demonstrations that include interim reports to help shape program design.
Evaluation & Research
Did the agency have an evaluation policy, evaluation plan, and research/learning agenda(s) and did it publicly release the findings of all completed evaluations in FY18?
- PD&R has published a Program Evaluation policy that establishes core principles and practices of PD&R’s evaluation and research activities. The six core principles are rigor, relevance, transparency, independence, ethics, and technical innovation. In FY18, PD&R undertook an internal review of the principles and compliance to assess whether modifications are needed. The review highlighted points of tension in the application of standards, but PD&R found that the standards themselves did not require amendment.
- PD&R’s evaluation policy guides HUD’s research planning efforts, known as research roadmapping. Key features of research roadmapping include reaching out to internal and external stakeholders through a participatory approach; making research planning systematic, iterative, and transparent; driving a learning agenda by focusing on research questions that are timely, forward-looking, policy-relevant, and leverage HUD’s comparative advantages and partnership opportunities; and aligning research with HUD’s strategic goals and areas of special focus. HUD also employs its role as convener to help establish frameworks for evidence, metrics, and future research. In FY18, PD&R staff is collaborating on an assessment of processes and procedures for identifying and executing in-house research projects at 15 federal agencies. This work will develop an approach to assess practice maturity and identify lessons to strengthen the value of PD&R’s in-house research efforts.
- HUD’s original “Research Roadmap FY14-FY18” and “Research Roadmap: 2017 Update” constitute the core of HUD’s learning agenda.The roadmaps are strategic, long-term (five-year) plans for priority program evaluations and research to be pursued given a sufficiently robust level of funding. On the basis of the learning agenda and additional policy questions that emerge, HUD also develops annual evaluation plans that identify specific research priorities. Actual research activities are substantially determined by Congressional funding and guidance.
- PD&R’s Program Evaluation Policy (p. 87950) is to publish and disseminate all evaluations that meet standards of methodological rigor in a timely fashion. Additionally, PD&R includes language in research and evaluation contracts that allows researchers to independently publish results, even without HUD approval, after not more than six months. PD&R has occasionally declined to publish reports that fell short of standards for methodological rigor. Completed evaluations and research are summarized in HUD’s Annual Performance Report (pp. 123–131) at the end of each fiscal year, and research reports are posted on PD&R’s website, HUDUSER.gov.
Did the agency invest at least 1% of program funds in evaluations in FY18? (Examples: Impact studies; implementation studies; rapid cycle evaluations; evaluation technical assistance, and capacity-building)
- In FY18, HUD plans to spend $89 million on evaluations, representing 0.17% of HUD’s $51 billion discretionary budget (minus salary and expenses) in FY18.
- For FY18, Congress appropriated $89 million for the Office of Policy Development and Research’s (PD&R’s) Research & Technology account, including $50 million for core research activities; up to $14 million for research, evaluations, and demonstrations; and not less than $25 million for technical assistance. The total represents an FY18 investment in evaluations and evidence amounting to 0.17 percent of HUD’s $51 billion gross discretionary budget authority, net of salaries and expenses, for FY18. The funding for core research is used primarily for the American Housing Survey, other surveys, and data acquisition that support evaluation of HUD’s mission activities in domains such as affordable housing and housing finance.
- PD&R’s FY18 appropriation of $24 million for Salaries and Expenses also supports evidence in the form of PD&R’s in-house research and evaluation program, economic analyses, data linkage initiatives, and management of housing surveys and contract research and evaluation.
- Additionally, PD&R leverages public resources by issuing data licenses and by establishing Research Partnerships with private-sector partners who contribute at least 50 percent of project resources and undertake research that contribute to HUD’s objectives.
Performance Management / Continuous Improvement
Did the agency implement a performance management system with clear and prioritized outcome-focused goals and aligned program objectives and measures, and did it frequently collect, analyze, and use data and evidence to improve outcomes, return on investment, and other dimensions of performance in FY18? (Example: Performance stat systems)
- HUD complies with federal strategic planning and performance management requirements, which include quarterly reporting on Agency Priority Goals on Performance.gov. HUD documents alignment between strategic goals and supporting objectives and performance metrics in the Annual Performance Plan and Annual Performance Report, and identifies the staff assigned lead responsibility for each objective.
- HUD has launched an effort called “Prescription for HUD,” (see mention here) through which senior staff reviews quarterly data demonstrating progress toward Secretarial priorities and Agency Priority Goals. Prescription for HUD supersedes the original HUDstat approach to performance management.
- In FY17, HUD launched a pilot of “Standards for Success,” a new standardized data collection and reporting framework for discretionary grant programs. The framework is intended to enable grant activities to be driven by coordinated outcomes and assessed using return on investment metrics. The framework is helping to standardize data elements, measures, definitions, metrics, and reporting periods; align programmatic data elements and measures with higher-level agency priority goals and objectives; and strengthen online reporting through record-level reports for greater analysis and responsiveness of programs. In the pilot’s first year, three HUD programs participated with a subset of their grants. The pilot is building an evidence base for scaling up Standards for Success as the common reporting framework for all discretionary grant programs.
Did the agency collect, analyze, share, and use high-quality administrative and survey data – consistent with strong privacy protections – to improve (or help other entities improve) federal, state, and local programs in FY18? (Examples: Model data-sharing agreements or data-licensing agreements; data tagging and documentation; data standardization; open data policies)
- HUD has an ambitious open data program. The HUDUSER.gov web portal provides researchers, practitioners, and the public with PD&R datasets including the American Housing Survey, HUD median family income limits and Fair Market Rents, and Picture of Subsidized Households tabulations of administrative tenant records that cross program silos and provide summary statistics at multiple geographic levels. HUD’s eGIS portal provides geo-identified versions of these datasets, administrative data, and other datasets to support public analysis of housing and community development issues related to multiple programs and policy domains using GIS tools. HUD sponsors custom tabulations of American Community Survey data that make standard adjustments of household incomes and units for household size to enable researchers and practitioners to analyze state and local housing needs. HUD provides researchers with microdata from experimental program demonstrations and research initiatives on topics such as housing discrimination, the HUD-insured multifamily housing stock, and the public housing population. To help users identify which data are useful to them, reference guides identify datasets and characterize their relevance and usefulness for research in designated categories. HUD continues its partnership with the Census Bureau to enhance public access to the American Housing Survey with a custom table creator, administrative data linkages to break down data silos, infographics to summarize results, and stronger data privacy controls.
- PD&R has authority to enter into cooperative agreements with research organizations, including both funded Research Partnerships and unfunded Data License Agreements, to support innovative research projects that leverage HUD’s data assets and inform HUD’s policies and programs. A dedicated subject-matter expert is available to answer questions for those seeking a data license. Data licensing protocols ensure that confidential information is protected.
- PD&R partnered with the National Center for Health Statistics (NCHS) at the U.S. Centers for Disease Control to link HUD administrative data for assisted renters with respondents to two national health surveys and made the linked data available to researchers to begin building a picture of tenant health issues. In FY18, the data linkage is being extended to include survey and administrative data for 1999 through 2016. Data access is provided through the NCHS research data centers to ensure that confidential information is protected.
- HUD is involved in a wide array of data-sharing agreements described under Data Infrastructure in the Roadmap Update (pp. 52–56). Notably, HUD and the Census Bureau have entered into an interagency agreement for the Bureau’s Center for Administrative Records Research and Applications (CARRA) to link data from HUD’s tenant databases and randomized control trials with the Bureau’s survey data collection and other administrative data collected under the privacy protections of its Title 13 authority. These RCT datasets are the first intervention data added to Federal Statistical RDCs by any federal agency, and strict protocols ensure that confidential information is protected.
Common Evidence Standards / What Works Designations
Did the agency use a common evidence framework, guidelines, or standards to inform its research and funding decisions and did it disseminate and promote the use of evidence-based interventions through a user-friendly tool in FY18? (Example: What Works Clearinghouses)
- PD&R provides the public, policymakers, and practitioners with evidence of “what works” primarily through HUD USER, a portal and web store for program evaluations, case studies, and policy analysis and research; the Regulatory Barriers Clearinghouse; and through initiatives such as Innovation of the Day, Sustainable Construction Methods in Indian Country, and the Consumer’s Guide to Energy-Efficient and Healthy Homes. This content is designed to provide current policy information, elevate effective practices, and synthesize data and other evidence in accessible formats such as Evidence Matters. Through these resources, researchers and practitioners can see the full breadth of work on a given topic (e.g., rigorous established evidence, case studies of what’s worked in the field, and new innovations currently being explored) to inform their work.
- In FY17, HUD developed and piloted a new standardized data collection and reporting framework for its discretionary grant programs called Standards for Success. The framework consists of a repository of data elements that participating programs use in their grant reporting. The repository of data elements establishes common definitions and measures across programs for greater analysis and coordination of services. HUD designed and made available an online data collection and reporting tool for those grants participating in the pilot. This tool enables grantees to directly enter the applicable data elements as well as upload files. All data are consolidated for record and analysis. HUD’s broader adoption of this framework would establish a common basis for determining grant performance, return on investment, and funding decisions.
Did the agency have staff, policies, and processes in place that encouraged innovation to improve the impact of its programs in FY18? (Examples: Prizes and challenges; behavioral science trials; innovation labs/accelerators; performance partnership pilots; demonstration projects or waivers with strong evaluation requirements)
- HUD’s FY19 budget request (p. 48-4) seeks to establish an Office of Innovation that will focus on testing and validating solutions to state, local, and federal housing and community development problems, modeled on the Innovation, Design, Entrepreneurship, Action (IDEA) Lab at HHS. The Office of Innovation will comprise three main components: a Building Technology component that addresses the need for resilient housing in disaster prone areas and affordable housing nationwide by identifying, evaluating, and incentivizing the adoption of resilient and affordable housing construction methods and materials; an Internal Innovation component that will facilitate innovation workshops for small HUD teams to improve the way their program or office works; and an Open Innovation Component that will engage the expertise, methods and tools of the private sector such as open data, crowdsourcing, challenges and prizes, and entrepreneurs in residence.
- PD&R is conducting numerous random-assignment program demonstrations to test new, innovative program models, as described in PD&R’s biennial report and online: the Family Self-Sufficiency Demonstration, First-Time Homebuyer Education and Counseling Demonstration, Pre-Purchase Homeownership Counseling Demonstration, Support and Services at Home (SASH) Demonstration for elderly households, Supportive Services Demonstration for health services in elderly housing, Rent Reform Demonstration, Rental Assistance Demonstration, and the Small Area Fair Market Rent Demonstration. The evaluation reports for such demonstrations frequently include baseline, interim, final, and long-term follow-up reports to keep policymakers apprised of emerging evidence. HUD’s largest program, Housing Choice Vouchers, grew out of the landmark Housing Allowance Demonstration of the 1970s and is shaped by the Moving to Opportunity demonstration.
- PD&R houses the Office of International and Philanthropic Innovation, and administers five types of Secretary’s Awards to encourage excellence: Public-Philanthropic Partnerships, Opportunity and Empowerment, Healthy Homes, Historic Preservation, and Housing and Community Design. The competitions are judged by juries of professionals and bring visibility to the nation’s most compelling, innovative solutions for addressing housing and community development challenges.
- PD&R sponsors an Innovation in Affordable Housing Competition to engage multidisciplinary teams of graduate students in addressing a specific housing problem developed by an actual public housing agency. The competition increases the nation’s future human capacity to address the affordable housing crisis by exposing future designers, administrators, and policymakers to real-world challenges of a specific legal and community context, with their proposals to be evaluated by an expert jury.
- HUD’s Office of Strategic Planning and Management completed a Data Vision Initiative to identify opportunities for HUD to collect more data about impacts. The initiative resulted in recommendations for specific metrics that HUD programs could begin to collect (such as reasons tenants exit HUD-assisted housing).
Use of Evidence in Five Largest Competitive Grant Programs
Did the agency use evidence of effectiveness when allocating funds from its five largest competitive grant programs in FY18? (Examples: Tiered-evidence frameworks; evidence-based funding set-asides; priority preference points or other preference scoring; Pay for Success provisions)
- In FY18 HUD’s largest competitive programs are: (1) Continuum of Care ($1.9 billion); (2) CHOICE Neighborhoods Implementation ($145 million); (3) Lead Based Paint Hazard Reduction ($130 million); (4) Section 202 ($105 million); and (5) Section 811 ($82 million).
- Competitive grants in the Continuum of Care program account for most HUD grant resources in FY18, and serve homeless populations by providing permanent supportive housing and rapid rehousing services. The Continuum of Care program awards preference points based on reporting of system performance measures focused on outcomes. The FY17 NOFA allocated $1.6 billion using a 200-point scale. The 200 points were awarded for various features, many of which included evidence of effectiveness:
- Up to 49 points for system performance, including –
- up to 10 points to CoCs that demonstrate an overall reduction of at least 5 percent in the number of individuals and families who experience homelessness;
- up to 3 points to CoCs that demonstrate how they are working to reduce the number of individuals and families who become homeless for the first time, with maximum points awarded to CoCs that demonstrate a reduction in the number of first-time homeless;
- up to 11 points to CoCs that reduce the length of time individuals and families remain homeless and specifically describe how they will reduce the length of time individuals and families remain homeless;
- up to 9 points to CoCs that demonstrate an increase in the rate in which individuals and families move to permanent housing destinations or continue to reside in permanent housing projects;
- up to 6 points to CoCs that reduce the extent to which individuals and families leaving homelessness experience additional spells of homelessness;
- up to 4 points to CoCs that increase program participants’ incomes from employment and non-employment cash sources; and
- Up to 60 points for performance and strategic planning, including –
- up to 15 points to CoCs for demonstrating the extent to which they are ending chronic homelessness;
- up to 3 points to CoCs that demonstrate the total number of homeless households with children and youth has decreased;
- up to 8 points to CoCs that demonstrate a decrease in the total number of homeless veterans in the CoC; and
- up to 4 points to CoCs that demonstrate the total number of homeless veterans has decreased.
- Up to 49 points for system performance, including –
- The FY18 CHOICE Neighborhoods Implementation Grants program’s scoring criteria accounts for how strong the evidence base is for each applicant’s selected strategies. The 102 points possible are awarded for factors of capacity, need, strategy, leverage, and soundness of approach (pp. 54–56). Included in these factors are points for evidence-based criteria, including 3 points for evidence-based public safety approaches (p. 65), 2 points for evidence-based early learning programs (p. 72), and 2 points for high-quality school-based or out-of-school education programs (p. 72).
- The Lead Based Paint Hazard Reduction program scores applicants based on their ability to monitor performance and specifically asks for statistics on elevated blood lead incidence/prevalence. Of 102 points possible, 10 points are awarded for the applicant’s performance history (p. 38), 5 points for effective use of funds (p. 40), and 20 points for evidence of applicant need (pp. 41–43).
- Additionally, all HUD-funded programs require recipients to submit, not less than annually, a report documenting achievement of outcomes under the purpose of the program and the work plan in the award agreement for accountability purposes and to build evidence of effective practices in the field.
- HUD and the U.S. Department of Justice have a partnership to demonstrate the effectiveness of a Pay for Success financing approach. Demonstration grants require implementing Pay for Success financing to reduce homelessness and prisoner recidivism by providing permanent supportive housing using the “housing first” model. HUD is conducting an evaluation of the efficacy and cost effectiveness of the Pay for Success approach.
Use of Evidence in Five Largest Non-Competitive Grant Programs
Did the agency use evidence of effectiveness when allocating funds from its five largest non-competitive grant programs in FY18? (Examples: Evidence-based funding set-asides; requirements to invest funds in evidence-based activities; Pay for Success provisions)
- HUD’s budget contains three large formula grant programs for public housing authorities (PHAs): (1) the Public Housing Operating Fund ($4.6 billion in FY18); (2) the Public Housing Capital Grants ($2.8 billion); and (3) Housing Choice Voucher (HCV) Administrative Fees ($1.8 billion). Another three formula grant programs serve cities or tribes: (1) Community Development Block Grant Entitlement/Non-Entitlement ($3.3 billion in FY18); (2) HOME Investment Partnerships ($1.4 billion); and (3) Native American Housing Block Grants ($0.8 billion).
- Although the funding formulas are prescribed in statute, evaluation-based evidence is central to each program. HUD used evidence from a 2015 Administrative Fee study of the costs that high-performing PHAs incur in administering a HCV program to propose a new FY17 approach for funding Administrative Fees while strengthening PHA incentives to improve HCV outcomes by providing tenant mobility counseling.
- HUD’s funding of public housing is being radically shifted through the evidence-based Rental Assistance Demonstration (RAD), which enables accessing private capital to address the $26 billion backlog of capital needs funding. Based on demonstrated success of RAD, for FY19 HUD proposed removing the cap on the number of public housing developments to be converted to Section 8 contracts. HUD is also conducting a Rent Reform demonstration and a Moving To Work (MTW) demonstration to test efficiencies of changing rent rules and effects on tenant outcomes.
- HUD conducted an extensive assessment of Native American, Alaska Native, and Native Hawaiian housing needs to strengthen the evidence base for the formula programs.
Repurpose for Results
In FY18, did the agency shift funds away from or within any practice, program, or policy that consistently failed to achieve desired outcomes? (Examples: Requiring low-performing grantees to re-compete for funding; removing ineffective interventions from allowable use of grant funds; proposing the elimination of ineffective programs through annual budget requests)
- HUD’s FY17 budget request included a new formula for funding Housing Choice Voucher Administrative Fees that shifts funding away from inappropriately compensated public housing agencies and increases overall funding according to evidence about actual costs of maintaining a high-performing voucher program. (See here for more info.)
- HUD’s FY17 budget request sought a $11 billion shift (pp. 8–9) of resources toward housing vouchers for homeless families based on the rigorous experimental analysis of 4 service options in the Family Options study.
- HUD’s FY18 budget request sought to eliminate funding for Community Development Block Grants. A 2005 PD&R evaluation had shown that targeting of CDBG resources toward communities with greater needs would be greatly enhanced by any of four alternatives to the 1978 statutory formula, but such improvements have not been authorized. An earlier 1995 evaluation found that although CDBG had made a contribution to community development, the neighborhood interventions generally were ad hoc rather than well-coordinated and strategic.