Performance Management in Human Services

Vol. 1, No. 9 October 1997

Performance Management in Human Services

by Jessica Yates

Background

Performance management systems have become increasingly relevant to governmental and private organizations involved in the provision of children and family services. The Personal Responsibility and Work Opportunity Reconciliation Act of 1996 (PROWRA), as amended by the Balanced Budget Act of 1997, provides for rewards or penalties for states’ performance, such as having welfare recipients participating in work activities and preventing out-of-wedlock births. While the federal government and other stakeholders are still working on a formula for determining what level of performance qualifies a state for bonus funding under the Temporary Assistance for Needy Families and Welfare-to-Work programs, it is clear that state and local governments will be expected to ensure that their information management systems can collect and report additional performance data.

Other related programs are already experiencing the challenges of awarding incentive funds based on performance. Congress is considering legislation to revise the incentive formula for child support programs, which currently bases its bonus dollars on a state program’s cost-effectiveness (program dollars spent per $1 support collected). The legislation arose from concerns that cost-effectiveness is too limited a measure for assessing a state child support program’s performance; other factors, such as paternity establishment rates, also are important.

Since 1982, the Job Training Partnership Act (JTPA) programs have required states and local service agencies to use client outcome measures, and have provided for incentives and sanctions accordingly. An August 1996 National Governors’ Association survey of 37 states found that 20 were establishing performance standards for their entire workforce development systems (NGA, 1997). In contrast, the former welfare program (Aid to Families With Dependent Children – AFDC) mandated performance reporting on process accuracy – the quality control system; no federal outcome measures for AFDC’s work program (JOBS) were ever developed. The General Accounting Office argued that while most states had established some performance measures for JOBS, a lack of outcome data contributed to the program’s insufficient focus on employment and to poor linkages with the business community (GAO, 1994 and 1995).

Under welfare reform, state and local governments are taking the lead in managing for performance. For example, Ohio is giving counties block grants to administer welfare reform; counties that exceed performance standards will earn extra funding. In addition, state agencies, local governments and community organizations involved in human services delivery are using performance information to stimulate strategic planning or to provide feedback and accountability to state legislatures or the public. Jurisdictions increasingly are using that information for contracting, budgeting and other operations. Lessons learned thus far indicate that both policy-setting and service delivery organizations can benefit from performance management practices, especially as part of a strategic plan for improving programs or services.

Policy Issues

In a recent "best practices" study of the public and private sectors, the National Performance Review (NPR) defined performance management as "the use of performance measurement information to help set agreed-upon performance goals, allocate and prioritize resources, inform managers to either confirm or change current policy or program directions to meet those goals, and report on the success in meeting those goals." (Performance management and related terms are defined differently by various experts; jurisdictions need to select and use a consistent set of definitions in designing performance management systems.) Developing a performance management system is more a process than an event, and often evolves over several years. While no single model will meet every organization’s needs, NPR and other observers have described key issues that must be addressed in developing effective performance management systems.

Developing appropriate performance goals. Taking the first step – setting performance goals – is one of the hardest parts of developing a performance management system, according to a General Accounting Office (GAO) survey of federal managers (GAO, 1997). At the state and local levels, jurisdictions need to decide who will set those goals and how the goals will be articulated. For example, surveys and forums can be used to get a sense of citizens’ priorities. Jurisdictions also can convene agency task forces or focus groups with invited stakeholders. The goal-setters need to understand what are realistic objectives and how to establish standards that will motivate agencies or service providers to improve. Some form of baseline information – such as how well the agency or program has historically performed, or how comparable agencies or programs perform – will be needed to set appropriate goals.

Jurisdictions with successful performance management systems generally have a strategic plan that integrates performance goals from across programs and services into one vision. These objectives represent a community’s or state’s blueprint for well-being, and program or agency goals feed into the larger plan. Prioritizing goals can help address the problem of inconsistent or conflicting program objectives too.

Selecting measures for those goals. Performance measures serve as indicators of how well the organization is doing in achieving its goals and can help quantify the "value added" of a program, which reflects what the extent of a societal condition or problem would be without the program in place. Welfare and related programs often have relied on measures of input (e.g., number of caseworkers), output (e.g., number of individuals completing job readiness classes or error rates in determining eligibility) and efficiency (e.g., program dollars spent per $1 child support collected). However, outcome measures (e.g., percentage of welfare caseload staying in jobs for six months) usually have a more direct relationship to the established performance goals.

Many potential client outcome measures are long-term, and therefore are not practical to use in a performance management system. Instead, jurisdictions often look to "interim" measures that, while not a perfect picture of a program’s success, offer a glimpse at a few aspects of performance. For example, 30-day, 60-day and 90-day job retention rates for welfare recipients are commonly used indicators of performance; longitudinal data about recipients’ employment would give better information on outcomes but is difficult to collect. Jurisdictions also can consider how to control for economic fluctuations and other external factors that could detract from the reliability of performance measures. These types of adjustments are made in JTPA programs.

Selecting the type and number of measures will vary according to the program goal, but most experienced agencies and outside observers recommend choosing a few relatively simple measures for a goal. A complex measurement system is difficult to communicate to legislators and the public, and increases the possibility that the measures conflict with each other or with strategic goals. However, advocates of performance management acknowledge that a range of measures may be necessary when they are tied to financial incentives or sanctions, and relying on too few measures could be perceived as inequitable to service providers who may have strengths not demonstrated in the selected measures. Multiple measures also can help prevent perverse incentives created by any single measure that might encourage "creaming" (serving only easier cases) or other behavior contrary to established performance goals.

Obtaining data to measure performance. Jurisdictions often identify additional data elements necessary to use the full capacity of their performance management systems. Those charged with implementing a system should explore the various pools of data already being collected, such as program administrative data, census data, program results reports and any research information. Data sources may be outside the government. For example, Case Western University’s Center on Urban Poverty and Social Change compiles community data from roughly 20 sources into a publicly-accessible database for the Cleveland, Ohio, area. (For more information, call (216) 368-6946 or see http://povertycenter.cwru.edu/cando.htm.)

As with any data, reliability and timeliness are issues. In a performance management system, consistency of data definitions is also very important to ensure that all individuals involved in data collection are gathering comparable information. Policy-makers will need to decide what should be the cost of performance measurement – including the cost of data collection and processing – and balance those costs against the gains in efficiency and effectiveness of programs that can be generated by the performance management system.

Bringing your information management systems up to speed. In the initial phases of performance management, agencies generally must rely on data accessible through existing federal, state and local information management systems. But because these systems often were designed to meet specific, process-oriented program requirements, many jurisdictions find that those systems cannot easily track client outcomes or measure progress toward goals with inter-agency or multi-program implications.

Some governments initially try to maintain existing systems and establish a separate computer system for performance purposes (GAO, 1994). While this approach may require fewer immediate investments, dual reporting places extra burdens on agencies and may limit the potential uses for performance information. Other states implement a comprehensive system (e.g. the Texas Strategic Planning and Budgeting System) to integrate program reporting that meets federal/state requirements and the needs of their performance management system (GAO, 1994). When making systems changes to be able to meet PRWORA’s requirements, states may have an opportunity to include data elements or reporting formats that would support their performance management systems.

Using information in performance management systems. In addition to meeting program reporting requirements and gaining access to incentive funds, jurisdictions also use performance information to revise measures and update strategic plans. Often, periodic surveys of citizens’ priorities and satisfaction with current services are part of this feedback cycle. Performance information also helps agencies make program improvements and management decisions. For example, agencies that are considering contracting out services can use such data in preparing preliminary cost-benefit analyses, comparing public and private sector performance, setting performance standards for a contractor, and evaluating a contractor’s performance.

Performance data also can be used to allocate resources through program budgeting and even staff compensation. NGA’s 1996 survey identified 10 of 37 responding states that were planning to use outcome-based or performance-based budgeting for their workforce development systems. Some organizations are starting to use more sophisticated accounting methods, such as activity-based costing, that more clearly reflect how much a given performance outcome actually costs. Another technique is to categorize current or proposed agency activities as "value-adding" versus "non-value adding" to help measure how much an activity helps an agency to meet its performance goals.

Building consensus around performance management. The idea of investing time and resources into such management systems often must be "sold" to the leadership of an agency, other collaborating agencies and state or local funding sources. Experienced jurisdictions have found that managers and policy-makers often need seminars or workshops on performance issues before goals can be set. This education may help prevent problems that some jurisdictions have when performance management is "oversold," and legislators expect significant monetary savings or program improvements in a short period of time.

Moving toward performance management systems also affects the organizational culture within agencies. Front-line staff may be accustomed to focusing on process checks, and training in both performance concepts and corresponding information technology systems may be needed. A jurisdiction could be faced with strong staff concerns if performance information will be used in program budgets, staff compensation or privatization decisions.

Research Findings

Research on performance management in human services tends to take a "how-to" approach and therefore focuses on state and local innovations and "lessons learned." NPR, GAO, The Finance Project, the Harvard Family Research Project, the Center for Accountability and Performance at the American Society for Public Administration, the Alliance for Redesigning Government and others have documented such examples. The "Publications" section of this paper references some of this research.

Some literature has examined the extent to which outcome measurement correlates to actual performance gains. An Upjohn Institute for Employment Research paper suggested that JTPA standards and incentives contributed to better performance and higher earnings by at least some populations (Bartik, 1996). Research by Abt Associates also found that JTPA measures for participant outcomes were valid predictors of earnings, and to a lesser extent, reductions in welfare dependency (Zornitsky and Rubin, 1988). In contrast, the U.S. Department of Health and Human Services (HHS) elected not to propose national outcome measures for JOBS after reviewing Manpower Demonstration Research Corporation (MDRC) research that showed how the job placement rates and earnings gains used as outcome measures in California’s GAIN program did not always correlate to the program’s added value (the extent to which program participants moved into jobs and off welfare compared to control groups) (HHS, 1994).

In addition, researchers have studied whether outcome measures encourage service providers to "cream," or focus on easier-to-serve clients who are more likely to achieve better outcomes, at the expense of clients with multiple barriers. For example, much of the research discussed above has concluded that the design of JTPA’s measures give providers an incentive to cream. Equity concerns were prominent in HHS’ decision, defended in a 1994 report to Congress, to not recommend outcome measures for JOBS. However, the MDRC and Abt Associates research suggests that performance data that is adjusted to keep local economic differences, client characteristics and other factors from skewing outcome results often shows a stronger relationship to a program’s value added, and if understood by service providers, could offer a disincentive to under-serve more difficult clients (Bartik, 1996).

Innovative Practices

State and local governments with experience in performance management usually have a range of noteworthy components to their systems. Most of the following examples highlight certain components, rather than the entirety, of the jurisdictions’ systems. Most of these jurisdictions also publish performance reports.

Fairfax County, Virginia: The county published its first human services performance budget this year to better examine the effectiveness of county activities. It complements but does not replace a traditional line-item budget. The performance budget outlined nine "community challenges" describing conditions that need to be addressed, such as "alcohol, drug, mental and physical health issues." Goals and measures for programs across six county agencies were grouped under each challenge. For example, the health "challenge" includes a family planning goal of having 98% of clients pregnancy-free after one year, regardless of which agency is providing that program. Measures that lacked baseline data were expressed as inputs or outputs. The performance budget will not allocate resources among programs and agencies until more data is collected and existing measures are refined.

Performance management also will be applied to services provided by private entities. Already, the county’s job development and placement contract permits the vendor to earn incentive payments based on the quality of job placement and a client’s wages. Services for the aging also are being provided through performance-based contracts, and the county is moving to pay homeless shelters according to client outcomes. The county’s competitive community funding pool requires applicants to specify planned outcomes in their contract proposals. Through a consultant, the county is now helping the community-based organizations establish better measures for client outcomes. That performance information may eventually be used to help make award decisions, but for now will not trigger incentives or penalties for the organizations. Contact: Marguerite Keily, Office of Systems Support, (703) 324-7533, for performance budgeting information; Brenda Gardiner, Office of Administration, (703) 324-7852, for performance contracting information.

Florida: Florida has several state-wide performance initiatives. Beginning in 1994, the state established a vocational education performance incentive funding pool with state general revenues and lottery funds to encourage the development of training programs for high wage/high skill/high demand jobs, increase participation by public assistance recipients, and increase the number of students completing training programs and being placed in jobs. Community colleges and school districts’ technical centers could opt to participate by placing a portion of their state vocational funding "at risk." Participants could earn back those funds and additional financial rewards when students achieved certain levels of performance in classes for targeted occupations, with more funding awarded if designated populations were served.

The first program year (1994/1995), 20 of 28 community colleges and 27 of 35 school district technical centers opted into the incentive pool. That year, two of the school districts failed to meet the performance levels needed to earn back their set asides and incentives, but all 20 colleges qualified for the bonus funds. In the next two years, additional colleges and school districts volunteered to participate, and all earned back their set asides and received some incentive funding. Beginning July 1, 1997, participation became mandatory. However, the program will soon be replaced by a larger performance-based system. State legislation was enacted this session to require that starting July 1, 1998, all vocational education funding must be awarded according to community colleges’ and school districts’ student outcomes. Contact: Bunny Hanley, Jobs and Education Partnership, at (904) 921-1119; also see http://www.state.fl.us/commerce/public_html/html/dochelp/enterpri/jobs.html and http://mailer.fsu.edu/~spap/research/gapp/gapp.html.

Georgia: Using federal child support incentive funds and other sources, the state child support agency had a performance compensation system from 1988 to 1994 that rewarded staff teams for meeting established goals measured by each team’s percentage of welfare cases with child support orders, collections on welfare cases and collections on non-welfare cases. Goals were set every six months according to historical and projected collection patterns. Individual staff could earn $500 per six months if their teams met the goals, and extra rewards if they collected even more child support on welfare cases (since the federal incentive formula rewards states more for these collections). To encourage individuals to pull their weight and maintain quality service, staff were ineligible for the awards if they: were absent for more than 20 workdays during the six-month period, had not met minimum standards on his or her most recent performance evaluation, or had been reprimanded for abusive actions toward a client. The initiative was terminated when a state-wide system was implemented for giving staff pay increases related to performance evaluations. Contact: Cindy Moss, Child Support Enforcement, Department of Human Resources, (404) 371-7333.

Minnesota: The Minnesota Milestones initiative began in 1991 with the development of performance goals and indicators through legislative and citizen input. The executive and legislative branches have used performance information to set a context for policy-making and to justify budget requests, but the Milestones are not used systematically for resource allocation. The coordinating agency, Minnesota Planning, is currently revising indicators so they are more relevant to local governments. The state has facilitated local collaboratives around family services and children’s mental health to help local agencies and community-based organizations work together and focus on client outcomes in their programs.

In a related project, the state Department of Finance is helping Itasca County to develop an outcome-based reporting system for roughly 20 children and family development programs funded with federal, state and local dollars. The system is based on performance goals and measures created over the past two years by the local collaboratives. It integrates about eight data elements (including expenditures and population served) from categorical programs and reports the information in terms of whether the services provided meet established goals. For example, child care information from several different programs would be aggregated and reported by linking that information with child care performance objectives. While it is not a results-based budgeting system and does not eliminate required categorical grant reporting, it does help feed local information into state-level performance reports and can be used to assess community needs. The system lays the groundwork for individual agencies and community organizations to eventually adopt accounting methods that calculate the true cost of achieving certain outcomes. The system also could work in conjunction with future common client intake systems used for all assistance programs, generating client-specific information about need, services and outcomes. Contact: Susan Roth, Minnesota Planning, at (612) 297-3279, Thomas Rice, Department of Finance, at (612) 296-6661; also see http://www.mnplan.state.mn.us/index.html.

Oregon: The state implemented its Oregon Benchmarks initiative with legislative leadership and support in reaction to concerns that the state’s prosperity was slipping. The state aimed to involve all stakeholders, including its citizens, in setting goals and priorities for its strategic plan. In 1994, Oregon extended the reach of its performance management system by approaching the federal government for permission to combine categorical programs and reporting in exchange for improved outcomes. A federal interagency agreement has waived many restrictive regulations and permitted greater flexibility in spending federal dollars, both of which were deemed necessary for an outcome-based approach to proceed. Contact: The Oregon Option at (503) 986-0244. Also see http://www.econ.state.or.us/opb/OR_OPT/index.htm.

United Way of America: Responding to public demands for information on how charitable contributions are used, United Way of America has established a resource network to assist local United Ways in implementing systems for measuring local programs’ performance. The network has published a variety of guides to help community-based, non-profits identify goals, select measures, collect data and use their findings. Seven local United Ways are taking further steps, such as using performance information to allocate funding to community agencies. Contact: United Way of America, (703) 836-7100. Also see http://www.unitedway.org/outcomes/.

For More Information . . .

RESOURCE CONTACTS

Alliance for Redesigning Government, (202) 347-3190 or see http://www.clearlake.ibm.com/Alliance.

Arizona State University School of Social Work, Dr. Peter Kettner, (602) 965-5918, and Dr. Robert Moroney, (602) 965-1275.

Center for Accountability and Performance at the American Society for Public Administration, (202) 393-7878 or see http://www.aspanet.org/cap/index.htm.

Columbia University School of Social Work, Dr. Lawrence Martin, (212) 854-2735.

Harvard Family Research Project, (617) 495-9108 or see http://hugse1.harvard.edu/~hfrp/index.html.

Institution for Educational Leadership, (202) 822-8405 or see http://www.iel.org.

Johns Hopkins University, Institute for Policy Studies, Dr. Burt Barnow, (410) 516-5388.

National Center for Public Productivity, Rutgers University, Dr. Marc Holzer, (973) 353-1351, and Stuart Grifel, (973) 353-5624, or see http://newark.rutgers.edu/~ncpp/.

National Performance Review, (202) 632-0150 or see http://www.npr.gov/.

Performance Benchmarking Center, (813) 622-8484 or see http://www.pbcenter.com.

The Finance Project, (202) 628-4200 or see http://www.financeproject.org.

The Rensselaerville Institute, (518) 797-3783 or see http://www.crisny.org/not-for-profit/thetute/INDEX.HTM.

United Way of America, (703) 836-7100 or see http://www.unitedway.org/outcomes/

PUBLICATIONS

A Guide to Developing and Using Performance Measures in Results-Based Budgeting, by Mark Friedman, for The Finance Project, May 1997. (202) 628-4200 or see http://www.financeproject.org/measures.html.

A Guide to Selecting Results and Indicators: Implementing Results-Based Budgeting, by Atelia Melaville, for The Finance Project, May 1997. (202) 628-4200 or see http://www.financeproject.org/indicators.html.

Accountability for Performance: Measurement and Monitoring in Local Government, edited by David N. Ammons, International City/County Management Association, 1995. (800) 745-8780 or see http://www.icma.org.

Caring for Our Children: Labor’s Role in Human Services Reform, by Marcia Calicchia and Laura Ginsburg, Public Employee Department, AFL-CIO, 1996. (202) 393-2820.

Challenges to Data Capacity for Outcome-Based Accountability, by Nancy Dunton, "The Evaluation Exchange: Emerging Strategies in Evaluating Child and Family Services," Vol. 2, No. 1, 1996. (617) 495-9108 or see http://hugse1.harvard.edu/~hfrp/eval/issue3/dunton.html.

Contracting for Performance in Welfare Reform, by Jessica Yates, Welfare Information Network, August 1997. (202) 628-5790 or see http://www.welfareinfo.org/jessicacontract.htm.

Designing and Managing Programs: An Effectiveness Based Approach, by Peter Kettner, Robert Moroney and Lawrence Martin, Sage Publications, 1990. (805) 499-9774 or see http://www.sagepub.com.

Establishing a Performance Management System for Targeted Welfare Programs, by Jeffrey Zornitsky and Mary Rubin of Abt Associates, for the National Commission for Employment Policy, August 1988. (301) 913-0500.

Focus on Client Outcomes: A Guidebook for Results-Oriented Human Services, by the Community Services Division, Minnesota Department of Human Services, March 1996. (612) 296-7031.

Follow-Up Analyses of the National JTPA Study: Final Report, by Larry Orr, Burt Barnow, Robert Lerman and Erik Beecroft, Abt Associates, for the U.S. Department of Labor, July 1997. (202) 219-7664.

From Quality Control to Quality Improvement in AFDC and Medicaid, edited by Fredrica D. Kramer, National Academy of Sciences, 1987. (800) 624-6242.

Managing for Results: Analytic Challenges in Measuring Performance, U.S. General Accounting Office (GAO), May 30, 1997. GAO/HEHS/GGD-97-138. (202) 512-6000 or see http://www.gao.gov/reports.htm.

Managing for Results: State Experiences Provide Insights for Federal Management Reforms, GAO, Dec. 21, 1994. GAO/GGD-95-22. (202) 512-6000 or search at http://thorplus.lib.purdue.edu:8100/gpo/GPOAccess.cgi.

Measuring the Performance of Human Services Programs, by Lawrence Martin and Peter Kettner, Sage Publications, 1996. (805) 499-9774 or see http://www.sagepub.com.

Measuring Program Outcomes: A Practical Approach, by the United Way of America, 1996. (703) 836-7100 or see http://www.unitedway.org/outcomes/.

Outcomes and Performance – Glossary, by the Alliance for Redesigning Government. (202) 347-3190 or see http://www.clearlake.ibm.com/Alliance/clusters/op/glossary.6.html.

Reaching Public Goals: Managing Government for Results, by NPR, October 1996. (202) 632-0150 or see http://www.npr.gov/library/papers/bkgrd/cover.html.

Replacing Welfare: Options for the Future, by Keon Chi, Council of State Governments, Solutions, Vol. 5, No. 2, 1997. (800) 800-1910.

Report to Congress: Recommendations on Performance Standards for the JOBS Program, U.S. Department of Health and Human Services, Sept. 30, 1994. (202) 401-9215.

Resource Guide of Results-Based Accountability Efforts, by the Harvard Family Research Project, August 1996. (617) 495-9108 or see http://hugse1.harvard.edu/~hfrp/releases/rba/efforts/.

Restructuring and Reinventing State Workforce Development Systems, by Jill Hyland, "Stateline," National Governors’ Association, Jan. 21, 1997. (202) 624-5300.

The New Oregon Trail: Accountability for Results, Special Report #7, The Policy Exchange: The Institute for Educational Leadership, 1996. (202) 822-8405 or see http://aspe.os.dhhs.gov/progsys/oregon/newtrail/ieldoc.htm.

The Oregon Option: Early Lessons from a Performance Partnership on Building Results-Driven Accountability, by Barbara Dyer, for the Alliance for Redesigning Government, July 1996. (202) 347-3190 or see http://aspe.os.dhhs.gov/progsys/oregon/lessons.htm.

The Status of Research and Indicators on Nonprofit Performance in Human Services, by Martha Taylor Greenway, United Way of America, August 1996. (703) 836-7100 or see http://www.unitedway.org/outcomes/ispaper.html.

Toward Results-Oriented Intergovernmental Systems: An Historical Look at the Development of the Oregon Option Benchmarks, by Mark G. Popovich, for the Alliance for Redesigning Government, July 1996. (202) 347-3190 or see http://aspe.os.dhhs.gov/progsys/oregon/history/intro.htm.

Using Performance Indicators to Improve the Effectiveness of Welfare-to-Work Programs, by Timothy J. Bartik, Working Paper 95-36. W.E. Upjohn Institute for Employment Research. 1995. (616) 343-5541 or see http://www.upjohninst.org/publications/wp/95-36.pdf.

Welfare to Work: Current AFDC Program Not Sufficiently Focused on Employment, by GAO, Dec. 19, 1994. GAO/HEHS-95-28. (202) 512-6000 or search at http://thorplus.lib.purdue.edu:8100/gpo/GPOAccess.cgi.

Welfare to Work: Measuring Outcomes for JOBS Participants, by GAO, April 17, 1995. GAO/HEHS-95-86. (202) 512-6000 or search at http://thorplus.lib.purdue.edu:8100/gpo/GPOAccess.cgi.

The author wishes to thank Erik Beecroft, Gail Christopher, Mark Friedman and the many other individuals who assisted in the compilation of this material.

WIN Web Site Expands

WIN’s extensive web site is growing! Among the recent additions are:

links to state Temporary Assistance for Needy Families plans

information on the new Welfare to Work and Child Health programs

a web page on tribal issues related to welfare reform

a new page on welfare sanctions

access to all WIN Issue Notes and Resources for Welfare Decisions

Visit Our Site Today – www.welfareinfo.org

Other promising practices, publications and research related

to performance management can be found in WIN’s clearinghouse.

Contact WIN or see our web site:

www.welfareinfo.org