Keeping Track: National Disability Status and Program Performance Indicators

Letter of Transmittal

April 21, 2008

The President
The White House
Washington, D.C. 20500

Dear Mr. President:

The National Council on Disability (NCD) is pleased to present you with Keeping Track: National Disability Status and Program Performance Indicators. This report is the result of a year-long effort by NCD to identify valid federal data and to describe the status of the U.S. population of Americans with disabilities.

During the past 30 years, advocates, policymakers, and a variety of public and private organizations have undertaken significant efforts to improve the lives of people with disabilities, culminating in the passage or improvement of legislation such as the Americans with Disabilities Act (ADA), various sections of the Rehabilitation Act, the Individuals with Disabilities Education Act, the Ticket to Work and Work Incentives Improvement Act, and others. Notwithstanding these various policies, little effort and progress has been made to measure and reflect upon the overall effectiveness and performance of these laws and policies, and their impacts on the quality of life for people with disabilities.

In 2005, the Government Accountability Office (GAO) conducted a review of 200 federal programs located in 20 agencies that served individuals with disabilities. It identified the need to transform many of the programs it reviewed to keep pace with the changing expectations and challenges of the 21st century. In addition, most participants at a 2007 GAO forum on modernizing disability programs agreed that multiple indicators were needed to measure the success of disability programs and that these measures should include not only economic measures such as income and employment, but quality of life measures as well.

Keeping Track: National Disability Status and Program Performance Indicators includes a set of statistical social indicators that NCD believes are currently able to measure the progress of people with disabilities in important areas of their life, over time. The report includes 18 indicators determined by stakeholders to measure “quality of life” using both objective and subjective measures. The indicators span a variety of life domains, including employment, education, health status and health care, financial status and security, leisure and recreation, personal relationships, and crime and safety. Collectively they will create a holistic representation of the lives of people with disabilities.

This report also lays out a roadmap for the federal government to expand the national disability indicator set, as well as a mechanism for installing the set into a key national indicator system. This report is consistent with the overall policy direction and progress advanced by the Administration through the New Freedom Initiative toward the goal of full participation for people with disabilities. NCD looks forward to working with key stakeholders in the federal government in the implementation of the recommendations in this report.

Sincerely,

John R. Vaughn
Chairperson

(The same letter of transmittal was sent to the President Pro Tempore of the U.S. Senate and the Speaker of the U.S. House of Representatives.)

National Council on Disability Members and Staff

Members

John R. Vaughn, Chairperson
Chad Colley, Vice Chairperson
Patricia Pound, Vice Chairperson

Milton Aponte, J.D.
Victoria Ray Carlson
Robert R. Davila, Ph.D.
Graham Hill
Marylyn Howe
Young Woo Kang, Ph.D.
Kathleen Martinez
Lisa Mattheiss
Lonnie Moore
Anne M. Rader
Cynthia Wainscott
Linda Wetters

Staff

Michael C. Collins, Executive Director
Martin Gould, Ed.D., Director of Research and Technology
Mark S. Quigley, Director of External Affairs
Julie Carroll, Senior Attorney Advisor
Joan M. Durocher, Senior Attorney Advisor
Sharon M. Lisa Grubb, Special Assistant to the Executive Director
Geraldine Drake Hawkins, Ph.D., Senior Program Analyst
Brenda Bratton, Executive Assistant
Stacey S. Brown, Staff Assistant
Carla Nelson, Secretary

Acknowledgments

The National Council on Disability (NCD) wishes to express its gratitude to the following people who helped draft this report: Nanette Goodman, Bonnie O'Day, and Allison Roche from the Cornell University Institute for Policy Research.

NCD also expresses its appreciation to the following individuals and organizations who gave freely of their time and expertise: Susan Daniels, Consultant, SM Daniels and Associates, Washington, DC; David Gray, Associate Professor of Occupational Therapy, Washington University School of Medicine, St. Louis, MO; Gerry Hendershot, Consultant on Disability and Health Statistics, Washington, DC; Andrew Houtenville, Senior Research Associate, Employment and Disability Institute, Cornell University, Ithaca, NY; Corinne Kirchner, Consultant, New York, NY; David Ruth, Consultant, Charlottesville, VA, Paraquad, St. Louis, MO; Help Yourself Inc., Washington, DC; American Association of People with Disabilities, Washington, DC; Laurie Mitchell Employment Center, Alexandria ,VA; and the Consortium of Citizens with Disabilities, Washington, DC.

TABLE OF CONTENTS

• Executive Summary

• Chapter 1: Introduction

  1. Need for the National Disability Performance and Indicators Project
  2. Current Measurement Systems
  3. Organization of the Report

• Chapter 2: Framework for Discussion of Social Reporting Systems

  1. Concepts and Definitions
  2. Brief History of Social Reporting Initiatives in the United States
  3. Objectives of Societal Indicator Systems
  4. Quality-of-Life Measures and Social Indicators

• Chapter 3: Existing and Emerging Societal Indicators

  1. Key National Indicators Initiative
  2. The Canadian Experience
  3. Topical Indicator Systems
  4. Conclusions and Recommendations

• Chapter 4: The Status of People with Disabilities

  1. General Limitations of Existing Data
  2. Identifying Concepts That Are Important to Measure
  3. Domains and Data
  4. Conclusions and Recommendations

• Chapter 5: Key Disability Indicators

  1. Introduction
  2. Desirable Features of the Disability Topical Indicators
  3. The Indicators

• Chapter 6: Program Performance Indicators

  1. Existing Program Performance Measurement Systems
  2. Integrating Quality of Life Concepts into Program Performance Measures
  3. Conclusions and Recommendations

• Chapter 7: Conclusions and Recommendations

• References

• Appendix

Additional resources are available at http://www.ncd.gov/newsroom/publications/index.htm

APPENDIX A.           Review of Societal Indicator Systems
APPENDIX B.           Background and Methodology for Focus Groups
APPENDIX C.           National Data Collection Instruments
APPENDIX D.           Limitations of Existing Data
APPENDIX E.           Data on Individuals with Disabilities
APPENDIX F.           Program Assessment Rating Tool Outcome Measures
APPENDIX G.           National Core Indicators
APPENDIX H.           References

Return to Table of Contents

Executive Summary

During the past few years, policy makers and researchers have attempted to measure the status of people with disabilities compared with others in our society. They want to know if the lives of people with disabilities have improved since the passage of legislation such as the Americans with Disabilities Act (ADA), the Rehabilitation Act, the Individuals with Disabilities Education Act, and the Ticket to Work and Work Incentives Improvement Act. In addition to measuring overall status, various efforts are now being undertaken to measure the effectiveness and performance of these laws and programs, and their impacts on people with disabilities.

Measuring the status of people with disabilities and the extent to which programs are assisting their clients to achieve better outcomes is important for many reasons. The number of people with disabilities in the U.S. is large and growing: 49.7 million non-institutionalized people in the U.S. have disabilities, and about 21.5 million of these are working-age individuals (U.S. Census Bureau 2003, reported in Iezzoni and O’Day 2006; StatsRRTC 2005). These numbers are likely to increase with the aging of baby boomers. People who acquire significant physical disabilities in early life are living longer than in prior years largely because of medical advancements. Measuring the status of people with disabilities is also important because the limited data we have indicate that the employment rate, level of household income, and educational attainment are low, and the poverty and obesity rates among people with disabilities are high compared to their peers without disabilities. Program evaluations, such as those conducted by the Government Accountability Office (GAO) and the Office of Management and Budget (OMB) also show that disability programs could be improved (GAO 2005; OMB 2004).

There is much we do not know about the lives of people with disabilities. Currently, the statistics informing the policy debate are predominantly economic, such as employment and household income. Such statistics are helpful, but paint only part of the picture. Other quality of life dimensions are substantially overlooked. Although some surveys cover topics that are related to some aspects of well-being, such as income, assets, or health insurance status, they do not necessarily reflect other aspects of well-being as they would be defined by the target population.

This report describes what we know about the status of people with disabilities in the U.S., and examines current data to assess the extent to which they meaningfully measure the well-being of people with disabilities. While much useful data on the status of people with disabilities exists, significant knowledge gaps hamper efforts to improve their well-being. Additionally, social indicator systems currently used or being developed either underemphasize or do not include people with disabilities. No comprehensive report exists describing the status of people with disabilities.

This report proposes a set of statistical indicators to annually measure the status of working-age people with disabilities and compare them to working-age people without disabilities in the United States. This report focuses on the working-age population of individuals with disabilities, defined as those ages 21 to 64, because many federal programs focus on employment, making indicators for the working-age population particularly relevant. Additionally, the life circumstances of working-age people with disabilities differ enough from the circumstances of children, youth and seniors, that separate indicators are warranted; indeed, indicator systems already exist for children and seniors. The topical indicator set, which can be produced relatively quickly,
can provide a foundation for the social indicator systems being developed, and stimulate public awareness about the status of people with disabilities and the need
for improvement.

This report discusses disability as a socio-demographic characteristic. This may not reflect the full experience of disability because disability is an interaction between an individual with an impairment and the environment, rather than merely a personal characteristic. The report uses this language because it allows discussion of the disparity between people with and without disabilities in measures of well-being.

Social Indicators

Social indicator systems measure the progress of society as a whole on a given set of valued characteristics. Indicator systems can measure attributes of people, institutions, the economy, and the physical environment (GAO 2005). Existing indicator systems generally cover a particular domain, such as health or education, a particular population such as children or the elderly, or cover a particular locality, such as a city or county. In 2003, the GAO, in cooperation with the National Academies, convened a group of national leaders that ultimately began the Key National Indicators Initiative (KNII) to develop a comprehensive indicator system for the country as a whole. The goal of the initiative is to provide measures that will help Americans assess the nation’s progress in key areas, such as economic well-being, health status, and the environment. After several years of research at the request of Congress, the GAO recommended in November 2006, as part of a report on oversight priorities, that Key National Indicators be pursued.

The designers of the KNII system are using the wealth of existing data from a variety of public and private sources to develop a website with data on a set of key indicators that cover a wide range of topics from education and business to health and the environment. Unfortunately, the surveys used to collect data on many of these topics do not contain adequate measures to define disability, or meaningful measures for program outcomes. The range of different disability definitions and multiple uses of the term “disability” in laws, policies, programs, and data collection instruments make comprehensive data on people with disabilities extremely difficult to compile, which likely contributes to the fact that there is no comprehensive document that reports on the status of people with disabilities in the U.S. This lack of information limits the ability to monitor and evaluate the impact of laws, policies and programs on people with disabilities, which in turn undermines the nation’s ability to make informed programmatic and funding decisions.

This report presents a brief topical indicator set based upon available data to track key indicators over time for people with disabilities. NCD will present current data to populate these indicators in each of its Annual Progress Reports. NCD’s Progress Report offers an annual opportunity to feature the topical indicator set, and thereby stimulate interest among politicians, policy makers, the press, and the public to focus on how people with disabilities are or are not making progress as compared to people without disabilities, highlighting their continuing needs. These indicators will also provide a foundation for engaging with the KNII as it conducts its planned national civic outreach over the next two years. The proposed topical indicator set includes the following:

Table E.1: Topical Indicator Set

Recommendation 1: NCD recommends that the National Institute on Disability and Rehabilitation Research (NIDRR) establish and fund a coalition of disability policy makers and advocates to: 1) develop a fuller set of indicators that are important to people with disabilities, building on the indicators outlined in this report (see Chapter 5); and 2) ensure that disability is included as a demographic subgroup as the KNII is developed. The KNII offers an important opportunity to integrate disability into a larger national indicator system. When completed, the KNII will offer individuals who are looking for disability data relatively easy access to existing data. It will highlight the importance of including disability as a subgroup in analyzing the relative status and progress of the population and highlight gaps in data about people with disabilities.

Recommendation 2: Promote a standard set of disability questions. Some important federal surveys have no disability measures. When measures are included, they vary across surveys, often yielding inconsistent and confusing results. A common core of disability questions on all federal surveys would improve comparability and improve the national discourse about disability programs and policy.

The definition should identify people “who, because of their functional limitations, are at risk for the loss or restriction of opportunities to take part in the normal life of the community on an equal level with others” (Altman 2006). Questions aiming to identify this group should characterize people by the difficulties they have in undertaking basic activities, like walking, seeing, and self-care—and not questions asking if they have a condition which limits their ability to participate in society, for example to work or attend school. The analysis can then determine to what extent people with functional limitations are excluded from society. It is also important that the questions ask about functioning without assistive devices, since the presence of assistive devices can assist people with functional limitations gain access to society.

Based on ideas developed by the Washington City Group (Madans et al. 2004, Altman 2006, Mont 2007) and Stapleton et al. (forthcoming), we propose that the set of questions used to operationalize this definition meet the following criteria:

• Be reliable and valid when self-reported;
• Include the smallest number of questions possible to capture the needed concepts;
• Be comparable across different national cultures;
• Focus on functional limitations;
• Not include the word “disability” because it can be interpreted differently based on age and other factors;
• Include scaled responses to allow the researcher to identify functioning at different levels;
• Identify the length of time the person has had the impairment; and
• Not preclude including other measures of disability for other purposes.

Including a set of questions meeting these criteria on all national surveys will allow us to compare the well-being of people with and without disabilities in each of the domains covered by the surveys, and to monitor changes over time in response to changes in environmental factors such as the economy and public policy. It would also ensure that data available for the general population and reported for other socio-demographic groups is also available by disability status. A short common set of questions also would allow for comparisons across surveys.

Recommendation 3: Fully disseminate disability data. Federal agencies and other organizations that conduct national surveys, such as the US Census Bureau of Labor Statistics and the US Department of Health and Human Services Administration on Healthcare Research and Quality, should provide comparisons of people with and without disabilities in their aggregated data reports and should, where sufficient data exists, offer comparisons of people with disability by gender, race, and other socio-demographic characteristics.

The Status of People with Disabilities

Some existing surveys probe deeply into certain life domains and are useful for measuring the well-being of people with disabilities. Using these data, we identify what is known about the status of people with disabilities. To find out what is important, we conducted six focus groups with individuals with disabilities, and one focus group with service providers. We asked participants what they considered to be important areas in which to assess quality of life, and what was important in each area. Based on their input, we identified 11 high priority domains or areas of life: employment, education, health, financial means, community participation, leisure and recreation, political participation, transportation, housing, personal relationships, and technology. Focus groups participants also identified 13 crosscutting individual and community dimensions that would contribute to a high quality of life.

Individual dimensions included the following:

Choice: To have the ability to select from various, acceptable options in important life areas, for example: the ability to choose whether and where to work, attend school, or participate in civic or community activities; the ability to select among various health care providers, insurance companies, and personal assistants; having options regarding where and with whom to live and how to spend money; and having the choice to marry and have children.

Spontaneity: To have the ability to make last minute plans to attend community events, go out with friends, or visit family. This requires accessible and readily available transportation and personal assistance, the ability to use a cell phone, community accessibility, and housing constructed to meet universal design requirements.

Aspirations: To have dreams and set and pursue goals for oneself in life areas such as employment, education, health care, or political participation.

Empowerment: To have: a sense of self esteem; the ability to advocate for oneself; understanding and acceptance of disability; knowledge about legal rights, including legislation such as ADA; a voice in programs that affect one’s life; and control
over finances.

Quality: To be able to: engage in employment that one finds meaningful; access timely and high caliber health care irrespective of insurance coverage; have adequate financial resources to afford to participate in community activities; have accessible, reliable transportation; and foster close, egalitarian relationships with friends and family.

Financial means: To have the ability to: purchase high quality services, including transportation, housing, and personal assistance; obtain a job that pays a decent wage with good benefits; and afford purchases required for leisure time activities.

Overall satisfaction: This category represents overall satisfaction in each domain; fulfillment in employment or education; ability to obtain high-quality health care; safe, affordable, accessible housing; participation in community events of one’s choosing; and having close relationships with friends and family.

Community dimensions included the following:

Inclusion and integration: Participating in employment, education, recreation and religious activities along with people without disabilities; inclusion in social activities with co-workers; integrated housing and transportation; inclusion in exercise and fitness activities; having a feeling of belonging or acceptance; and having relationships with people with and without disabilities.

Assistance and support: Having access to help to meet various needs for independence, where “independence” means the opportunity to live outside externally- controlled settings, notably institutions; having assistance in making choices; getting informal support from family and friends to enable independence; and having access to needed coaching and support services in education and employment.

Public attitudes: Promoting appropriate expectations by members of the general public applied to people with disabilities in employment, education and community participation; experiencing high comfort level by members of the general public with people with all types of disabilities; having appropriate portrayals of disability in the media; increasing public regard of disability as a normal part of life; and being treated with dignity and respect.

Accessibility: Having physical and programmatic accessibility; usability of facilities; usable off-the-shelf technology by people with disabilities.

Non-discrimination: Experiencing fair treatment in employment, housing, education, health care, public and private agencies and organizations, lending and voting. This category is similar to, but goes beyond, public attitudes to encompass a legal-based definition of discrimination.

Safety/Risk: Safety includes such items as housing located in safe areas and back-up system for personal assistance services; risk includes the ability to work without fear of benefit loss.

With this feedback from focus group participants, we reviewed the existing national data in each of the 11 domains to report on what is known about the status of people with disabilities. While there is some data in each of the domains identified by the focus group participants, serious gaps exist in what the nation can measure using current data. Most questions related to the crosscutting concepts cannot be addressed with the data that are currently available.

Program Performance Measures

Program performance indicators, or indicators used for “performance measurement,” have a narrower purpose than the societal indicator systems we have been discussing. While topical and comprehensive indicator systems attempt to capture the state of society, program performance indicators are used by agencies, Congress, and OMB for ongoing monitoring and reporting of program accomplishments, particularly progress toward pre-established goals.

Most current program performance indicators measure the impact of programs on participants. They generally do not measure the number of people who could benefit from the program but who do not participate either because they do not meet eligibility requirements, are unaware that the program exists, or do not use the program because of other entry barriers such as inaccessibility or excessive red tape.

A second limitation of program performance indicators is that they do not account for the complex interaction between the program and other factors, such as the economy or other public programs. For example, the percentage of vocational rehabilitation clients obtaining competitive jobs at closure partly depends on the quality of the services delivered, but also depends on the characteristics of the users, the availability of jobs in the economy, and the barriers and disincentives to work created by Social Security Disability Insurance and other cash and medical benefits.

A third limitation is that programs serving a broader population tend to report the outcomes by all program enrollees, rather than by different demographic subsets of enrollees, and do not allow a program to determine the specific impact on people with disabilities. Outcome indicators for programs focusing on people with disabilities, particularly employment programs, are evaluated based on a narrow range of indicators that do not adequately capture the program’s impact on the quality of people’s lives.

Recommendation 4: Administrative records of all means-tested programs should include a disability indicator. Programs that serve individuals with disabilities, such as One-Stop Employment Centers, and TANF, should collect data on the number of individuals with disabilities who use their programs and compare outcomes between program users with and without disabilities. We recommend that the Interagency Committee on Disability Research (ICDR) develop a workgroup to establish criteria on which the indicator is based.

Recommendation 5: Expand the Job Training Common Indicators. NCD should ask the Department of Labor to explore options within its administrative data collection system to add questions to the Job Training Common Indicators that more adequately capture concepts important to the focus groups, including choice in job; whether the job uses the employee’s full talents and abilities; whether the wage is appropriate given their qualifications; the extent to which they are satisfied with job conditions (including place, facility, co-workers, schedule requirements, accommodations, and opportunities for advancement); and whether they have meaningful opportunities to make choices about the conditions of their work.

Recommendation 6: Agencies should consider the effects of programs on non-participants. Agencies should include participation rates for eligible individuals (or potentially eligible individuals) and measures of well-being for those who are denied services in their GPRA and PART outcome measures. Improved survey data on people with disabilities would help support such measures.

Return to Table of Contents

Chapter 1: Introduction

A.    Need for the National Disability Performance and Indicators Project

During the past few years, policy makers and researchers have attempted to measure the status of people with disabilities as compared to others in our society. They want to know if the lives of people with disabilities have improved since the passage of legislation such as the Americans with Disabilities Act (ADA), the Rehabilitation Act, the Individuals with Disabilities Education Act, and the Ticket to Work and Work Incentives Improvement Act. In addition to measuring overall status, various efforts are now being undertaken to measure the effectiveness and performance of these laws and programs and their impacts on people with disabilities.

Measuring the status of people with disabilities and whether programs are truly
assisting their clients to achieve better outcomes is important for many reasons. First, about one-fifth of the non-institutionalized population or 49.7 million people report disabilities (U.S. Census Bureau 2003 reported in Iezzoni and O’Day 2006). Among those of working age, about 21.5 million or 13 percent have some type of disability. (See Exhibit 1.1, which presents disability by type, age, education and race among the working age population.)

The disability prevalence rate increases with age. As shown in Exhibit 1.1, seven percent of people ages 21–29 have a disability compared to 26 percent of people ages 60–64. These rates increase even more dramatically for the population over age 65: 30 percent of people ages 65–74 have a disability, and more than two-thirds (69%) of people over age 85 have some type of disability. Because the numbers of elders will grow substantially in coming decades, the number of individuals with disabilities is likely to increase. People who acquire significant physical disabilities in early life are living longer than in prior years largely because of medical advancements. Healthy aging with a disability has become an important clinical consideration and research topic, as persons with such conditions as cerebral palsy, polio, and spina bifida increasingly live into their seventh decade and beyond (Iezzoni and O’Day 2006).

Measuring the status of people with disabilities is also important because the limited data we have indicate that, in many ways, the status of people with disabilities has changed little. They remain below their non-disabled counterparts in many key areas where “more is better,” such as employment, household income, and educational attainment, and above their counterparts in other areas where a lower rate is preferable, such as poverty and material hardship. Other disparities include excess weight, reduced physical activity, increased stress, and less frequent mammograms for women over age 55 with disabilities (Iezzoni et al. 2001; Weil et al. 2002).

Program evaluations also suggest the need for improvement. The Government Accountability Office (GAO) has recently conducted a review of 200 federal programs located in 20 agencies that serve individuals with disabilities. It identified the need to transform many of the programs it reviewed to keep pace with the changing expectations and challenges of the 21st century. In addition to significant program redesign, it recommends looking across programs to determine the effectiveness of disability programs as a whole (GAO 2005). Such a review requires an examination of the entire population of those with disabilities, not just those who receive services from the individual programs.

The Office of Management and Budget (OMB) also sees need for improvement among federal programs serving people with disabilities. Through its Program Assessment Rating Tool (PART), it continues to mandate outcome measurement as a part of its government wide program review. It assesses whether agencies focus on achievable outcomes based on long-term goals and benchmarks, and mandates that agencies develop a plan of improvement to address deficits. Many of the federal programs that serve people with disabilities show need for improvement (OMB 2004).

Lack of progress on key indicators for people with disabilities may reflect two problems. One is that programs and services for people with disabilities have not kept pace with the changing expectations and aspirations of people with disabilities and the new realities of the 21st century. The other is inadequacies in our measurement and data systems. Although some existing national surveys probe deeply into certain areas and could be useful in measuring the well-being and progress of people with disabilities, they have several notable limitations, including inadequate definitions of disability and failure to measure aspects of life that are very important to people with disabilities. According to the National Council on Disability (NCD), policy issues on the forefront of the disability agenda, such as long-term care services and employment, require relevant and routinely collected measures, accurate data, sophisticated analyses, and both broad and well-targeted dissemination (NCD 1998).

There is much we do not know about the lives of people with disabilities. Currently, the statistics informing the policy debate are predominantly economic, such as employment and household income. Such statistics are helpful, but paint only part of the picture. Other dimensions of quality of life are substantially overlooked. Current survey questions do not provide a holistic picture of the disability experience. Although some surveys cover topics that are related to some aspects of well-being, such as income, assets, or health insurance status, they do not necessarily reflect other aspects of well-being as they would be defined by the target population.

Statistics can be a powerful tool for shaping the public debate. Whether viewed as “keeping score” or as a “report card,” regular reporting of numbers that can be compared over time is a technique for accountability that is likely to stimulate interest among politicians, policy makers, the press, and the public. The purpose of this report is to develop a set of statistical indicators to annually measure the status of working-age people with disabilities in the United States. NCD wishes to describe the status of working-age people with disabilities in our society and determine whether current data meaningfully measures their well-being. More specifically, the purposes of this report are to:

• Describe the status of people with disabilities in the U.S. based on indicators that measure their quality of life;
• Identify gaps in knowledge;
• Identify indicators of well-being that can be used to compare people with and without disabilities; and
• Recommend strategies to insure that the federal government has adequate and appropriate information to measure the success of its programs and policies and to track progress of people with disabilities over time.

This report focuses on the working-age population of individuals with disabilities, defined as those ages 21 to 64. We are focusing on this population for several reasons: First, many federal programs focus on employment, making indicators for the working-age population particularly relevant. Second, the life circumstances of working-age people with disabilities differ enough from the circumstances of children, youth and seniors, that separate indicators are warranted; indeed, indicator systems already exist for children and seniors. (See Appendix A for a fuller discussion of these indicators.) Third, including seniors or children may impact the interpretation of the results. For example, including seniors, all of whom are covered by Medicare, in an indicator of insurance coverage, may mask the differential in levels of private health insurance coverage between working-age people with and without disabilities.

This report refers to disability as a socio-demographic characteristic. This may not reflect the full experience of disability because disability is an interaction between an individual with an impairment and the environment, rather than merely a personal characteristic. The report uses this language because it allows discussion of the disparity between people with and without disabilities in measures of well-being. It is the approach used to measure the disparity among racial groups. Race, which is consistently used as a socio-demographic characteristic, is also a social construction (Wilson 2003) and can be viewed as an interaction between personal characteristic and social response; we attribute social value to biological characteristics by labeling an individual with a racial identity. This report explores some of these definitional
issues throughout.

We begin with a brief review of problems with current measurement systems.

B.     Current Measurement Systems

No comprehensive indicator system that describes the social conditions of the U.S. population exists, but interest in developing such a social indicator system is growing. (See Chapter 3.)

Social indicator systems are one of the chief mechanisms for assessing group status in our society. Although social data exist on many important aspects of daily life, there is no comprehensive indicator system at the federal level that describes our social condition (Miringoff et al. 2003). The official portrait of our nation’s prosperity focuses on narrowly-defined economic indicators such as Gross Domestic Product, the rate of inflation, the unemployment rate, income per capita, and poverty rates. These economic indicators give us a concrete sense of how we are doing as a nation, but important aspects of life, such as health status, psychological well-being, environmental factors, and others also deserve serious attention.

Social indicator systems measure the progress of society as a whole on a given set of valued characteristics. Indicator systems can measure attributes of people, institutions, the economy, and the physical environment (GAO 2005). Indicators are a reflection of many factors, such as the scarcity of natural resources relative to the size of the population, the state of knowledge and technology, and the structure of governance. Indicators also reflect the actions of federal, state, and local governments, a wide array of for-profit and not-for-profit entities, and individuals.

Existing indicator systems generally cover a particular domain, such as health or education, or cover a particular locality, such as a city or county. In 2003, the GAO, in cooperation with the National Academies, convened a group of national leaders that ultimately began the Key National Indicators Initiative (KNII) to develop a comprehensive indicator system for the country as a whole. The goal of the initiative is to provide measures that will help Americans assess the nation’s progress in key areas, such as economic well-being, health status, and the environment. After several years of research at the request of Congress, the GAO recommended in November 2006, as part of a report on oversight priorities, that Key National Indicators be pursued.

Social data currently collected are based on surveys without adequate disability measures. No comprehensive report exists on the status of individuals with disabilities in America. Government agencies or private organizations that serve particular segments of our society use national data to assess and report on the status of particular groups. For example the Federal Interagency Forum on Aging-Related Statistics produces Older Americans, Key Indicators of Well-Being. The Urban League annually publishes The State of Black America to focus upon the status of African Americans in the U.S. compared to society as a whole. Kids Count, a project of the Annie E. Casey Foundation, tracks the status of children in the U.S., providing policymakers and citizens with benchmarks of child well-being. These reports are a shorthand way to provide a picture of the group’s status and to educate the general public about social issues affecting particular groups. They can call attention to particular issues, but policy makers and administrators will need more complete data to make informed programmatic and funding decisions.

Unfortunately, many of the surveys used to collect social data do not contain adequate measures to define disability, or meaningful measures for program outcomes. Some existing surveys probe deeply into certain life domains and appear to be useful in measuring the well-being of people with disabilities in some domains. For example, the Cornell University Rehabilitation Research and Training Center on Disability Statistics (StatsRRTC) uses the American Community Survey (ACS) to develop “Status Reports” that provide demographic and economic indicators in several areas, including disability prevalence, employment (full and part-time), personal and household income, poverty rate, and education (www.disabilitystatistics.org ). But the ACS is constrained by the conceptual and methodological limitations of its disability questions, and because the other ACS questions cannot support a complete, holistic set of indicators. Existing social measurement systems generally do not include measures that would enable us to identify people with disabilities in a consistent and meaningful fashion, or to evaluate their quality of life. The range of different disability definitions and multiple uses of the term “disability” in laws, policies, programs, and data collection instruments make comprehensive data on people with disabilities extremely difficult to compile. This contributes to the fact that there is no comprehensive document that reports on the status of people with disabilities in the U.S., similar to those mentioned above. The lack of data limits that ability to monitor and evaluate the impact of laws, policies and programs on people with disabilities, which in turn undermines our nation’s ability to make informed programmatic and funding decisions.

Existing data on people with disabilities is focused on health and economic measures and tells us little about other things that people with disabilities think are important.

Currently available data provide information on some health and economic indicators, for example, obesity, employment, income, and assets. However, we know little about how people with disabilities perceive the quality of their lives or their social environments. Are their lives meaningful and rewarding? Are they content with the quality of their lives? Do they feel accepted by their communities? To what extent do they feel their opportunities for social participation are equal to those of their peers without disabilities? What is their perception of programs, such as those that provide rehabilitation, housing, health care and income support? These are questions for which little information is available to answer them.

Although this report focuses on outcome measurement to support indicator systems for working-age people with disabilities, many of the limitations we find are common to other indicator systems. In particular, they often do not take the perspective of the target group—how their lives are actually lived and how they perceive their own well-being.

Limitations on outcome measurement for people with disabilities apply in other contexts as well. Most importantly, programs serving people with disabilities are evaluated based on outcome measures that often poorly reflect outcomes important to their clients. For instance, a program’s success might be based on an employment placement rate relative to a benchmark (Projects with Industry, Department of Labor Customized Employment Grants); the percent who are employed for at least 90 days (vocational rehabilitation), increases in client earnings (Medicaid Buy-In programs); or reduced expenditures (Medicaid Home and Community-Based Waiver programs). These do not adequately capture the program’s impact on the quality of people’s lives, and are therefore insufficient to assess the true value of the program.

Given the inadequacies of existing indicators as just described, we can anticipate that the search for a workable indicator system for persons with disabilities will entail considering complex concepts, and ultimately will require some compromises toward practical recommendations. The next section outlines how this report tackles
those challenges.

C.     Organization of the Report

Chapter Two sets the context for review of performance measurement systems, by providing a framework for discussing disability and social indicator systems. The chapter also provides a brief history of social indicator systems and a discussion of the role social indicator and program evaluation systems can play in measuring quality
of life.

Chapter Three reviews existing and emerging social indicator systems. Because no comprehensive national system yet exists, it focuses on the Key National Indicators Initiative (KNII). It also highlights lessons learned from a similar initiative in Canada. Chapter Three also reviews topical and population-specific systems related to health, education, aging, and race.

Chapter Four reviews available data on the status of people with disabilities and identifies gaps in our current knowledge. To identify knowledge gaps, we conducted six focus groups with individuals with disabilities, and one focus group with service providers. We asked them what they considered to be important areas in which to assess quality of life. Based on their input, we identified 11 high priority domains or areas of life, including employment, education, health, financial means, community participation, leisure and recreation, political participation, transportation, housing, personal relationships, and technology. Focus group participants commented on
what aspects of each area were important. With this feedback, we reviewed existing data in each of the 11 domains to report what is known about the status of people
with disabilities.

Chapter Five proposes a set of topical indicators to be used to compare people with and without disabilities in important life domains over time. These 18 indicators, along with the most recent data to populate them, provide a snapshot of the status of people with disabilities and a foundation for engaging with the KNII discussed in Chapter Three.

Chapter Six discusses the use of indicators in measuring the performance of federal programs. The chapter describes the Program Assessment Rating Tool (PART), the primary government-wide effort to measure program outcomes, as well as other program evaluation initiatives. We review several initiatives based on measures that assess the quality of life for program participants, including the National Core Indicators Project, the Supported Work Indicators, and others. Most of these indicators have focused on evaluating specific providers but they can be valuable in development of broader program evaluation measures.

Chapter Seven lists the recommendations that arise out of review, both those related to national indicator systems and those related to other data collection efforts.

Appendices A–G provide additional information on many of the issues covered in
this report.

*Types of disability total more than “any disability” because an individual may have more than one type of disability.

Source: Unpublished statistics generated by A. Houtenville using 2005 ACS data.

Return to Table of Contents

Chapter 2: Framework for Discussion of Social Reporting Systems

The purpose of this chapter is to establish the context and provide a framework for a discussion of performance measurement systems. We begin with key definitions and a brief history of social measurement systems. We discuss the purpose of social indicator systems and the role social indicator and program measurement systems can play in measuring the status of the population.

A.     Concepts and Definitions

In this report, we use the following key concepts and definitions:

• Indicator: a quantitative measure of economic, environmental, social, and
  cultural condition.
• Indicator system: an organized effort to assemble and disseminate indicators.
• Topical indicator system: brings together indicators on a related set of issues, such as health or transportation.
• Comprehensive key indicator system: a collection of indicators on a broad range of economic, environmental, social, and cultural issues.

Indicators and indicator systems are the building blocks to measure quality of life and evaluate program effectiveness. They are used to measure three types of outcomes:

• Individual Outcomes: Individual outcomes express the impact of a particular intervention on a particular individual. For example, an employment outcome might express the impact of vocational rehabilitation (VR) services on a VR client. Clinical outcomes express the impact of a particular intervention or course of medical treatment, such as physical or speech therapy, drug regimens, surgery, or drug treatment, on an individual. A measure of clinical outcome might be the number of pneumonia hospitalizations after a public immunization campaign. Program and societal outcomes reflect individual outcomes. This report focuses on indicators of interest at the federal level such as program outcomes and social indicators and does not review clinical outcome systems.
• Program Outcomes: Program performance and outcome indicators focus on the effect of a particular government program on its users or those who meet program eligibility requirements. They are used for ongoing monitoring and reporting of program accomplishments, particularly progress toward pre-established goals. For example, an outcome indicator of a state’s vocational rehabilitation program collects data on how many clients find employment, whether the employment is full or part-time, the wage rate, whether health insurance or other benefits are provided, and the amount of funds spent per case.
• Societal Outcomes:Social indicator systems measure the progress of society as a whole on a given set of indicators; e.g., economic, education, health, or home ownership. Social indicator systems can answer vital questions including, How are we doing as a country, as a state, or as a locality? How do we compare to others and how do specific subgroups compare to the population as a whole? Such indicators help us assess the position and progress of an entire jurisdiction or group, not just the performance of any one program.

Both societal indicators and program performance indicators play important roles in ensuring a responsive and accountable government. Social indicators measure societal progress; program performance indicators measure the impact of specific programs on people’s lives. But the distinction is sometimes blurry. A social indicator answers a “societal outcome” question, such as “Do people have the ability to travel where and when they want to go using transportation that is affordable?” For people with disabilities, many factors play into the answer—the nature of the individual’s disability, their personal family resources, available public transportation, affordable vehicle modifications for people with physical impairments, available personal assistants or volunteer drivers, mobility training for people who are blind or have cognitive disabilities, and availability of assistive technologies. No single agency is responsible for ensuring that all of these needs are met.

Program indicators focus on a particular program or group of programs and measure the effect of the program on users’ lives. In the above example, an evaluation of a paratransit program might focus on the question “Is the program sufficiently flexible to permit spontaneous travel?” or “Do the paratransit and fixed-route transit services together provide adequate coverage throughout the transit district?” Only one program is being evaluated and, in general, the program will have its primary, and perhaps
only, impact on its participants. In both cases, measures that indicate quality should
be included.

In the examples above, both the societal indicator and the program performance indicator measure affordability, availability, and spontaneity in transportation. Quality in a specific transportation program contributes to quality transportation in the jurisdiction as a whole primarily, if not exclusively, through its impact on participants. Even when government programs are designed to address a certain issue for people with disabilities (such as improving access to transportation), they do not benefit all people with disabilities, because only a subset of the disability population uses them. As a result, a strong federal law or program may have only a small direct impact on the societal measure.

At the same time, however, such a program can have indirect consequences for society that go beyond the services provided to participants. For instance, the availability of transportation facilitates employment, shopping and participation in many other activities, and increased presence of people with disabilities in social settings can influence societal attitudes in ways that might have significant impacts on society. Such indirect impacts are captured in societal measures, but it is rarely possible to separate the indirect impacts from the total impact.

B.     Brief History of Social Reporting Initiatives in the United States

There is a long history of efforts by leading democracies throughout the world to develop and sustain indicator systems. In the United States, efforts to use social indicators to provide a picture of our nation’s well-being arose during periodic crises: the rapid changes of the Progressive Era, the poverty of the Great Depression, and the social movements of the 1960s and early 1970s (GAO 2003).

Early in the twentieth century, problems of poverty and child labor raised concerns about the lack of a consistent monitoring system for the well-being of children and youth. To address this problem, the Children’s Bureau compiled a Handbook of Federal Statistics on Children, a compendium of data on infant mortality, birth rates, poverty, and other child-oriented indicators. This publication represented the first effort by the federal government to bring together social information from a variety of sources throughout the country and became an early model for social reporting in the U.S. (Miringoff et al. 2003).

During the 1930s, as the Great Depression deepened, federal mechanisms were established to gather social data and monitor social conditions. New Deal agencies began to coordinate data previously collected by the states. These agencies initiated and refined comprehensive national surveys in areas such as labor and health and began to document the nation’s social problems.

Social reporting received little attention until the 1960s, when Congress and the Administration wished to explore whether the technology used for space exploration contributed to the rise of new social problems. The Commission responsible for this investigation discovered that little of the critical information needed to assess American life was available. In 1966, President Lyndon Johnson directed the Department of Health, Education, and Welfare to explore “ways to improve the nation’s ability to chart its social progress.” The resulting study, Toward a Social Report, was viewed as a “preliminary step toward the evolution of a regular system of social reporting.” The study considered the types of indicators necessary for regular reporting, including measures of health and illness, social mobility, the environment, income and poverty, public order and safety, learning, science, and art (Miringoff et al. 2003).

After Congressional efforts to adopt a multifaceted indicator system failed to pass, the Bureau of the Census published Social Indicators, a less ambitious statistical report. Follow-up reports appeared in 1973, 1976, and 1980 but, the series was discontinued by the Reagan Administration. Today, reports on specific issue areas, such as crime, health, education, the elderly, and children are published regularly. There have been no further efforts at comprehensive, regular social reporting, but past endeavors have helped to shape the developments that are underway today.

C.     Objectives of Societal Indicator Systems

Societal indicator systems can achieve several objectives:

• Measure Progress: Indicators can measure progress toward national outcomes, assess conditions and trends, and help communicate complex issues. They can help to assess the overall position and progress of our nation in key areas, frame strategic issues, support public choices and enhance accountability (Walker 2007).
• Identify Needs: Because so many government departments are involved in disability programs (GAO 2005), societal indicators would provide a “snapshot” of how the entire system is performing. This should help decision makers assess where adjustments, new programs or greater coordination is needed. For example, policy makers in Canada have found that their societal indicators on disabilities have drawn attention to the fragmented nature of disability programs across many departments and governments (Bennet et al. 2001).
• Ground Debate: Indicators can provide government decision makers with transparent and relatively objective points of reference against which to carry out public debate on the goals of programs and services (Bennet et al. 2001). They are also a powerful tool for shaping the public debate. Currently, the statistics that inform policy discussions are predominantly economic. Such statistics are helpful, but paint only part of the picture. Proposed new indicators could be used to expand the debate by focusing attention on other dimensions of the quality of life.

The chief functions of broad societal indicators are to monitor movement relative to societal goals and to identify areas for which additional research or government attention is warranted. It is important to recognize however, that societal indicators are “indicative” of the relative status of different groups of people. They are not intended to imply causality. For example, data show that women with disabilities are less likely to have mammograms than women without disabilities (Iezzoni et al. 2001). But what causes this disparity? It might be that many mammogram machines are not physically accessible or that provider attitudes discourage use. It might be that disability is highly correlated with low income and low income leads to lower use of preventive care or lack of affordable, accessible transportation to travel to appointments. It might be that women with disabilities are more likely than others to rely on Medicaid, and there are too few Medicaid providers. Each of these possibilities leads to a different policy intervention. The social indicator is a signal that there is a problem, but additional research is needed to identify the cause.

D.     Quality-of-Life Measures and Social Indicators

The primary purpose of this report is to identify quantifiable measures that represent a person’s quality of life (QOL). Various disciplines approach and define QOL, and its closely related topics of socioeconomic well-being and happiness, differently.

 Philosophers have been debating the essential qualities of happiness for millennia. One school of thought believes that true happiness lay in the devotion to intellectual thought and the rejection of worldly goods (Russell 1945). Aristotle, on the other hand, thought that happiness lay in virtue (Russell 1945). The notion of happiness is subjective and everyone must find their own path to “happiness.”

We define QOL as “having the means and wherewithal to pursue happiness—however defined by each person.” This definition is consistent with the thinking of Thomas Jefferson, who included “life, liberty and the pursuit of happiness” in the Declaration of Independence, as well as with the conclusions of QOL researchers Diener and Suh (1997), and our focus group findings. When we asked people with disabilities to define “quality of life” or to answer the question, “What makes you happy?” they consistently mentioned concepts such as choice and empowerment. They wanted adequate income in order to be able to choose goods and services that would make them happy. Thinking about choice in housing, some wanted a place that would allow pets; some wanted to choose their roommate; some valued safety above all else. Consistently though, they all valued choice as critical to their quality of life.

Happiness and quality of life are often associated with the presence of favorable circumstances external to the individual, such as economic stability, a supportive family, access to health care to maximize health, as well as social, cultural and environmental conditions. These concepts are also influenced by internal factors such as attitudes and perspectives on personal circumstances, feelings of empowerment, and other intangibles that are difficult to quantify.

Social Indicators are used to quantify quality of life. While some social indicators may address factors internal to the individual, we adopt the more common approach (JCCI 2006) and focus mostly on the external environment, which is more measurable and is affected by public policies.

QOL indicators fall into two groups:

• Objective measures that represent social facts independent of personal evaluations, such as level of education or use of health care; and
• Subjective measures which are based on the individual’s perception and evaluation of social conditions, such as life satisfaction or perception of justice.

Objective measures, subjective well-being measures, and traditional economic indices are needed in unison to understand human quality of life, and to make informed decisions about public policy and funding (Diener and Suh 1997). Because objective measures are easier to gather than are individuals’ subjective perceptions of their social environment, comparing objective measures across subpopulations and over time is relatively straightforward. However, decisions involved in selecting and measuring the variables are not value-free, which limits the objectivity of the data.

The cost of collecting objective measures varies across measures. As a result, the choice of measures collected does not necessarily reflect their importance to society. For instance, financial transactions are much easier to document than how individuals use their time. As a result, outcome indicators based on financial transactions dominate the literature on social well-being, and statistics on time use (e.g., hours of unpaid work, time spent on self care, or visiting health care providers) are rarely reported, even though they are regarded as important by many people.

An individual’s sense of well-being is far more complex than can be quantified by even a well chosen set of objective measures based on external circumstances. Subjective measures are based on the notion that if a person experiences his or her life as good and desirable, it is assumed to be so (Diener and Suh 1997). Thus, it is appropriate to ask directly about factors such as feelings of joy, pleasure, contentment, and life satisfaction. Subjective well-being is not a state of simply being hedonistically happy. The central elements of well-being, a sense of satisfaction with one’s life and positive affective experiences, are derived from the context of one’s most important values, goals and expectations.

Despite the impression that “subjective” connotes lesser scientific credibility, studies have shown that subjective well-being measures possess adequate validity. For example, data on the Australian Unity Wellbeing Index, an index of subjective measures, has been collected since 2001. Recent studies have found that it is a valid, reliable, and sensitive instrument (Cummins et al. 2005). Despite this credibility, most measures of quality of life and well-being focus predominantly on objective rather than subjective measures.

In summary, this chapter has established a framework for the discussion presented in the remainder of the report, sorting out some useful distinctions between societal outcomes and related phenomena, specifically: individual outcomes; program evaluations; and quality-of-life assessments. We have drawn attention to objective and subjective aspects of outcomes, and measures that correspond to those aspects. This chapter also reviewed the early history of attempts at comprehensive social indicator systems, and the objectives sought by further work in that arena. The next chapter turns to current efforts to develop a social reporting system, along with some well known topical indicator systems.

Return to Table of Contents

Chapter 3: Existing and Emerging Societal Indicators

Many democratic nations, including Canada, Australia, South Africa and almost all European countries, are operating comprehensive key indicator systems and have been doing so for years. Even though the U.S. government uses literally hundreds of data collection instruments (surveys and administrative records) and spends over $4 billion per year collecting data, the data are not easy to find and use. The Federal Interagency Council on Statistical Policy provides a portal to data from the websites of over 100 federal agencies, including program participation data and survey data, accessible at http://www.fedstats.gov/ . This site links to a vast amount of data which, taken together, must be described as “disjointed,” with each data source using different definitions of such key concepts as disability and race. The system is difficult to navigate even for experienced researchers and policy makers.

There have been several attempts to rationalize this mass of data. For example, the Census Bureau produces the annual Statistical Abstract of the United States and the Bush administration presents selected economic and social indicators in the “White House Briefing Room” accessible at http://www.whitehouse.gov/fsbr/ssbr.html . The Briefing Room social indicators refer to crime, demography, education, and health.

The notion of disability is substantially absent in both of these compendia. In general, the briefing rooms present data for the population as a whole rather than by different demographic characteristics, and the Statistical Abstract generally includes disability as a health outcome rather than as a socio-demographic characteristic. That is, while it includes the number of people with disabilities by age group by state it does not include other characteristics of the population with disabilities, such as educational, economic or health status.

Hundreds of U.S. state and local jurisdictions have drawn from the substantial body of general population data collected at federal, state, and local levels to assemble indicators into an easily accessible tool such as a website or chartbook. Some of these indicator systems measure progress toward specific goals or benchmarks; others provide more general information about the economic, environmental, social, and cultural conditions of a jurisdiction (GAO 2004).

In addition to these broad-based collections of indicators, there are numerous topical indicator systems that are national in scale. Such systems focus on specific topics such as health care or education, or specific populations such as the elderly, children, or African Americans.

There is, however, no national indicator set that provides the public with information about the nation’s progress on a range of issues. In order to fill this gap, the Key National Indicators Initiative (KNII) undertook the challenge of developing and disseminating a set of key indicators that would encompass economic, social, cultural, and environmental outcomes. The initiative has grown to include diverse leaders from government, business, research, the media, and the nonprofit sector (GAO 2004). The KNII has recently formed a non-profit organization and is currently planning to release the first public version of its website in 2009. In this chapter, we discuss the KNII and several existing topical indicator systems with an eye toward identifying whether the best strategy for monitoring the status and progress of people with disabilities is to integrate disability into the KNII, to develop a separate topical indicator system focused on disability, or both.

A.     Key National Indicators Initiative

In 2003, GAO, in cooperation with the National Academies, convened a forum of leaders from different sectors of the economy to discuss the need for and potential of a national indicator set. Since then, those individuals and institutions that chose to develop the KNII have taken a methodical approach to establishing a “credible, trusted source of unbiased, nonpartisan, data” (Hoenig 2007b). They have done extensive research on existing indicator systems (GAO 2004), engaged with diverse stakeholders, gained financial support from private foundations, established a governance structure, developed a prototype demonstration website and begun the design of a civic engagement process to help select key issues and indicators. GAO has recommended to Congress that it consider contributing funds to the initiative through a public-private partnership (GAO 2005b).

Over the past two years, the KNII has been pursuing a series of targeted efforts—under the auspices of the National Academies—aimed at testing effective and efficient ways of choosing topics and indicators as well as disseminating them in a useful manner via web-based information systems. Their first round of work on issues and indicators produced 22 topical areas and a pool of over 900 candidate indicators. Through a civic engagement process which is now being designed in detail, these topical areas will be reassessed and a smaller set of key indicators chosen to display on a website. Some examples of potential population indicators include:

For each indicator ultimately chosen, the website would provide a description of the indicator, overall statistics, breakdowns over time, geography and by demographic subgroup (where that data is available), description and limitation of the data source, and links to the source and related indicators and commentary.

To choose issues and indicators for the public release of its website, the KNII will use research, surveys, and a civic engagement process to bring together both diverse groups of Americans and experts from a variety of government agencies, research institutions, associations, non-profits, the media, and businesses. They are currently soliciting input from a broad range of groups. There has been, to date, only minor input from the disability community.

First, the goal of the KNII is to assemble information, not to collect it. Thus the indicators will be based on available data. However, the KNII is committed to highlighting the need for additional data collection for important indicators that cannot be measured with existing data (Hoenig 2007b). Second, the set of indicators is national and intended to drive decision making; thus it must be comprehensive and it must integrate the links
and interactions between component measures (GAO 2004). Third, the indicators focus on current status and past trends and not on modeling or projecting into the future (Hoenig 2007b).

To be successful at measuring the status and progress of people with disabilities, the KNII will have to be attentive to several issues. First, key indicators will need to reflect the issues that are particularly important to people with disabilities. For example, there will need to be measures of affordable housing, reliable and accessible transportation, or community accessibility. Second, since the KNII plans to present many of the population-based indicators by race, gender, and other socio-demographic characteristics, disability status must also be one of those characteristics. Even if the KNII does include indicators to address the first issue, there are significant challenges in addressing the second issue, including:

• Many of the data sources the KNII might use do not include disability measures;
• Even in those sources that include disability measures, the measures are limited in very significant ways; and
• The definition of disability is not consistent across surveys.

Only with the full engagement of the disability community can these issues be turned into opportunities through KNII that could increase the amount of knowledge available to the nation on people with disabilities.

B.     The Canadian Experience

For the past six years, the Treasury Board of Canada has published annual Performance reports designed to provide a “whole-of-government” view of how individual departments and agencies contribute to broad outcomes in four areas—economic, social, international, and government affairs (Treasury Board of Canada 2006). It is designed as a companion document to 88 separate Departmental Performance Reports, which links each indicator with the government departments that can affect it.

While the Treasury Board developed its indicators, a coalition of public, private for-profit and not-for profit organizations, led by the Canadian Policy Research Network (CPRN) developed a related but separate indicator initiative. The CPRN set of indicators is designed to “measure what matters” (Atkinson Charitable Foundation 2007). The architects of the project note that although there are a variety of social welfare indicator systems in Canada, the question of which aspects of well-being should be monitored remains a “flashpoint of controversy” (Michalski 2002). There has been no effort to integrate the Treasury Board and the CPRN indicators.

Based on the concept that to be legitimate, societal indicators should reflect what matters most to the members of a community or a nation, the CPRN organized 40 “dialogue discussions” with nearly 350 participants in nine provinces across Canada. The participants reviewed background materials on quality of life issues and indicators, engaged in three-hour dialogue discussions, and identified their priorities for defining quality of life. These priorities serve as the basis for selecting the indicators (Michalski 2002). The final indicators are listed in Appendix A.

Most recently, the group has undertaken an initiative to aggregate the indicators into a single composite measure—the Canadian Index of Wellbeing (CIW) which they hope will be used in conjunction with the Gross Domestic Product (GDP) to measure growth in Canada. Unlike the GDP which is a single measure, the CIW is an aggregation of measures. There is still considerable disagreement over the appropriate methodology that should be used to create a statistic that is “comprehensive enough to satisfy the statisticians and policymakers but simple enough to be understood by the general public” (Laidlaw 2007).

C.     Topical Indicator Systems

Topical indicator systems in the US focus on a variety of issues including the business cycle (the Conference Board Business Indicators), the environment (e.g., the State of the Nation’s Ecosystems, Environmental Protection Indicators for California and many others), education (e.g., State Education Indicators, National Science Foundation Science and Engineering Indicators), and health (the U.S. Department of Health and Human Services, Healthy People 2010).

In addition, a variety of groups, both public and private, have developed topical indicator systems focused on particular populations. For example:

• Children—The Annie E. Casey Foundation’s Kids Count and America’s Children by the Federal Interagency Forum on Child and Family Statistics;
• The Elderly—Older Americans: Key Indicators of Well-Being from Federal Interagency Forum on Aging-Related Statistics;
• African Americans—The National Urban League’s State of Black America; and
• People with Disabilities—the NOD/Harris Survey and Healthy People 2010, Focus Area Six.

We review each of these topical indicator systems in Appendix A.

The topical indicator systems that focus on African Americans, children, and the elderly do not generally identify people with disability as a subpopulation. For example, the compendia that present information on children presents the data by race, ethnicity, gender and age (where appropriate) but not by disability status. The one exception is the Older Americans chartbook, but even here, disability is presented as a health indicator rather than a subgroup characteristic. Thus, the project is limited to tracking the prevalence of disability in the aging population. It does not track other socioeconomic and health measures (such as poverty, housing expenditures, participation in the labor market, health care expenditures, or source of payment for health care) by disability status.

NOD in cooperation with the Harris Poll, has conducted five surveys between 1986 and 2004 to measure the participation of people with disabilities in American life. The survey asks dozens of questions, many of which change each time the survey is administered. However, NOD/Harris developed ten key indicators that are monitored over time. This set of indicators is potentially very useful because it includes objective and subjective measures and it captures some of the issues identified as important by our focus groups. NOD compares the value of the indicator over time for people with disabilities and the gap between the value for people with and without disabilities. However the indicators have significant limitations. Among others, the sample size of the survey is too small to detect important changes in the value of the indicator over time unless they are very large. (See Appendix A for more information on NOD indicators.)

Unlike the NOD/Harris indicators, the data for the “Healthy People 2010” indicators are drawn from data sources that are collected as part of ongoing federal surveys and administrative data. This approach has the advantage of presenting data that are based on larger sample sizes and collected regularly, but the breadth of the indicators is limited by the type of questions asked in existing surveys.

Neither existing indicator system for people with disabilities covers the breadth of issues that our focus groups indicated were important. For example, neither addresses the domains of housing, financial status and security or technology. Nor do they address the dimensions that are important to people with disabilities such as choice, spontaneity, and aspirations.

While the topical indicator systems for African Americans, children, and the elderly do not integrate disability status into their reports, they provide valuable insights about how to develop a successful topical indicator system.

Develop a coalition: Although indicator sets can be developed by governmental or non-governmental entities (e.g., The Annie E. Casey Foundation, Child Trends, the National Urban League, and the NOD), they all rely on coalitions of stakeholders to develop and disseminate the product.

Select indicators parsimoniously to provide adequate coverage of the subject with as few indicators as possible: All the topical indicator systems present a wide variety of indicators on a website or in a chartbook. The systems that receive the most attention from the media, such as Kids Count, the State of Black America and the NOD/Harris Survey of Americans with Disabilities, focus on fewer than 20
core indicators.

Avoid composite indicators: Only one indicator system, the State of Black America, includes a composite measure of well-being. It allows the National Urban League to report the status of African Americans is 73.3 percent in 2007, when compared to their white counterparts, considered as 100 percent, up from 73.0 percent in 2005 and 2006. Readers cannot identify whether the change of less than half of one percentage point represents any real change, or what type of change it might represent. The National Urban League does report sub-indices and highlights bellwether individual indicators such as unemployment, poverty, homeownership and incarceration rates (National Urban League 2007). Two well-known composite indicators of quality of life—Disability Adjusted Life Years (DALYs) and Quality Adjusted Life Years (QALYs)—illustrate one type of danger in producing a composite indicator. These measures are based on the medical model of disability and neglect the value of other outcomes for people with disabilities. (See Appendix A for a fuller discussion of these issues.)

Highlight limitations in existing data: Both the Federal Interagency Forum on Child and Family Statistics and the Federal Interagency Forum on Aging-Related Statistics include a section in their reports highlighting measures that should be developed and data limitations that need to be addressed in order to obtain a complete picture of the population (GAO 2004).

D.     Conclusions and Recommendations

Recommendation 1: NCD recommends that the National Institute on Disability and Rehabilitation Research (NIDRR) establish and fund a coalition of disability policy-makers and advocates to 1) develop indicators that are important to people with disabilities, building on the indicators outlined in this report (see Chapter 5); and 2) ensure that disability is included as a subgroup characteristic as the KNII is developed. The KNII offers an important opportunity to integrate disability into a larger national indicator system, which is important for several reasons:

• When completed, the KNII will offer individuals who are looking for disability data relatively easy access to the data;
• The KNII can highlight the importance of including disability as a subgroup in analyzing the relative status and progress of different populations and disparities between populations;
• The KNII offers the ability to highlight gaps in data about people with disabilities; and
• The KNII can be sustainable.

Chapters Two and Three of this report painted the landscape of existing and emerging social indicator systems. Although there are some topical indicator systems in discrete areas such as healthcare or on specific populations such as children, there are no comprehensive national quality of life systems in the U.S. The KNII effort shows promise but thus far has lacked meaningful involvement of disability researchers and advocates. If a comprehensive social indicator system is to include measurement of the status of people with disabilities in a meaningful way, existing surveys sponsored by government agencies must include adequate and consistent disability measures. Within this context, Chapter Four reviews available data on the status of people with disabilities and identifies gaps in our current knowledge.

Return to Table of Contents

Chapter 4: The Status of People with Disabilities

In this chapter, we discuss some of the limitations of existing data that should be kept in mind when reviewing statistics on disability, including problems with the definition of disability, reporting on individuals from diverse cultural or linguistic backgrounds, and various statistical limitations.

To identify knowledge gaps across available data sources, we conducted seven focus groups of individuals with disabilities. We asked them to articulate what they consider to be important areas in which to assess well-being. (See Appendix B for the focus group methodology and characteristics of participants.) We identified 11 high priority domains or areas of life, and 13 dimensions within each domain—such as choice, empowerment, and inclusion—that focus group participants said were important. We then reviewed existing data in each of the domains to report what is known about the status of people with disabilities, and to identify gaps in our knowledge. This review will assist us to identify current and potential indicators most relevant to individuals with disabilities that could be included in a national comprehensive indicator system and/or in a topical indicator system.

This review focuses on data that are currently published or easily available on websites. There are potentially useful questions included on some surveys but they are not aggregated or reported in a way that facilitates a comparison between people with and without disabilities. Even though some of these statistics could be available to researchers with computer programming skills, they are not included in our review.

 Disability data is collected in a large number of federal surveys. In Appendix C, we briefly describe each survey, the sample methodology and sample size, the dissemination instruments, relevant variables, definition of disability, frequency of data collection, ability to analyze data on people with disabilities from different racial and ethnic groups, and strengths and weaknesses in ability to provide valid, reliable, and comparable disability statistics across time.

A.     General Limitations of Existing Data

Data on individuals with disabilities have various limitations that affect its usefulness in measuring quality of life or in evaluating significant programs. These limitations are briefly described below, and discussed more fully in Appendix D.

Inconsistent definitions of disability: The definition of disability is quite complex and varies among different federal and state laws, public programs, insurance plans, and organizations. It is thus no surprise that surveys that were designed for varying purp