Log in

BY: KATHLEEN N. GILLESPIE, Ph.D., KENNETH B. HOMAN, Ph.D., and SHARON M. HOMAN, Ph.D.

Dr. Gillespie is associate professor, Department of Health Management and Policy, School of Public Health; Dr. K. Homan is professor, theological studies, College of Arts and Sciences; and Dr. S. Homan is professor, Department of Community Health, and director, Division of Biostatistics, School of Public Health, all at Saint Louis University, St. Louis.

"Never let your zeal outrun your charity. The former is but human, the latter is divine."
—Hosea Ballou (1771-1852), U.S. clergyman

Navigating and measuring the impact of Catholic health care community benefit is critical work. After all, Catholic health care organizations are called to strive for the divine goal of charity: to serve the underserved. As the quote by Rev. Ballou indicates, we may have the zeal to embark on many projects, but we must choose wisely those projects that provide the most good.

This article assesses the impact of evaluation in the context of the role that community benefits have in fulfilling their mission of providing community service and meeting the organization's charitable tax-exempt purpose. Also, this article outlines an evaluation framework for demonstrating the effectiveness and impact of community benefits and discusses the measurement issues in assessing impact.

Effective Evaluation — Navigating the Course of Community Benefit
How do we determine whether community benefit programs and activities have a positive impact on the well-being of persons and communities? What kind of evaluation system works best for navigating the terrain of community benefits? There are many "off the shelf" products and services that can be used. These products are much like the Global Positioning System that we used to direct our recent travel through southern Germany. With minimal information input, the easy-to-use GPS system accurately directed us, even allowing for changes en route. Although we had little knowledge of the landscape, roads or compass direction, we arrived at our planned destination.

Similarly, a GPS-type of evaluation system would work well for addressing a specific need at a particular point in time to accomplish a narrowly defined mission. However, the mission of community benefits is broad and our knowledge of "what works" and "where we are going" is at the frontier. Navigating a community benefits program to achieve impact is much more of an expedition. In fact, the following four core aspects of navigation that were critical to the success of the Corps of Discovery, the 1804 expedition led by Meriwether Lewis and William Clark, are vital to evaluating community benefits to achieve positive impact.

1) Community Collaboration is Required to Realize and Demonstrate Impact
First, successful navigation of the Lewis and Clark expedition depended on collaboration with native people. Collaboration, particularly at the grassroots community level, is as essential to community benefits impact as it was to reaching the Pacific Ocean. As historian Robert Archibald noted, "Without the goodwill and assistance of the Indian people whose lands we were claiming, even the stalwart crew of the Lewis and Clark expedition could not have survived. We can, and must, look into this meeting of different cultures as another kind of discovery by Lewis and Clark, the discovery that 'foreign' peoples, their traditions, their worldviews, were at least of equal value and importance to our familiar and restricted perspectives."¹ Community collaboration is essential to creating useful evaluation measures, gathering data and interpreting evaluation findings regarding outcomes and impact.

The definition of community as it relates to community benefit is multifaceted and includes "communities" within the community, such as staff, patients, business and civic organizations, partnering community organizations, advocacy groups, and area residents, particularly those living in poverty and at the margins of society. Members from these groups provide important insights into determining how to define and measure the impact of community benefit activities and programs.

2) Healthy People 2010 is the Best Map for Charting Success
Second, community benefit leaders need the best available common map to plan and chart the evaluation of community benefit with the goal of demonstrating and enhancing the impact of community benefits on the health of persons and communities. The captains of the Lewis and Clark expedition planned meticulously, used the best maps and gathered intelligence from informed persons to navigate the rapid and shifting Missouri River.

Healthy People 2010² is the best common map for charting the success and impact of community benefits and adapting the benefits services to the changing health needs of individuals and communities. Using Healthy People 2010, community benefits leaders can align their programs and services with specific goals and objectives that drive community health efforts at the national, state and local levels. The Healthy People 2010 roadmap provides well-defined indicators of success and established benchmarks that are shared by state and local health departments and community health programs. In collaboration with local public health organizations, community benefit leaders can develop strategies and action plans to address one or more of the leading Healthy People 2010 indicators (e.g., immunization, access to health care, injury and violence) and use the indicators to measure the impact on eliminating health disparities, promoting health and increasing quality of life.

3) A Scientific Approach is Critical to Successful Evaluation
The third core aspect of successful navigation is careful attention to the scientific side of the expedition, even when it is met with resistance or hostility. President Thomas Jefferson instructed Lewis, "Beginning at the mouth of the Missouri, you will take careful observations of latitude and longitude, and at all remarkable points on the river."³ Clark made daily notations, establishing reference points from visible landmarks. Lewis and Clark used scientific methods to remain accountable and committed to achieving measurable progress. Similarly, by using a scientific framework for evaluating community benefits, Catholic health care organizations will remain accountable and committed to achieving measurable impact on the health and well-being of persons and communities. The Centers for Disease Control and Prevention's Framework for Program Evaluation in Public Health⁴ was developed to guide public health professionals through a systematic process of evaluation governed by standards adopted from the Joint Committee on Standards for Educational Evaluation. These standards are used to assess the quality of evaluation activities to ensure evaluation methods are useful, practical, feasible, ethical and accurate.

4) Accountability is Measured at Organizational, Community and Project Levels
The fourth aspect of successful navigation is accountability. In the planning of a journey or a health initiative, we are obliged to ask, "Will it be worth it?" During the journey or project, we ask, "Is it worth it?" When the journey is over, we ask, "Was it worth it?" Years later, we ask again, "Was it worth it?" "What impact did it have?" Similar questions of accountability can be asked about community benefits. The Kellogg Foundation describes the three tiers, or levels, of evaluation — the organization, the community at large, and direct recipients of programs and services. "Was it worth it?" must be asked relative to each of these.

Measuring the Impact of Community Benefit
Measuring the impact of community benefits leads to greater accountability internally and externally. However, because community benefit programs are complex, it is challenging to measure impact. The first task is to establish the scale of a project (i.e, organizational impact, community impact of a cluster of related services, impact of a specific project), who is being measured (e.g., collaborators, patients, community children with asthma), and over what time period. If we focus only on evaluating short-term outcomes of community benefits, we may overlook important longer-term organizational and community impacts.

To measure impact, we need indicators (i.e., measures used to monitor progress) that are readily understood, relevant and related to available data. Qualitative and quantitative measures are appropriate. The indicators ought to arise from a group of internal and external community benefit team members. Focus groups and mind mapping can be used to facilitate stakeholder identification of key indicators for the organization, for important target areas (i.e., clusters) of community benefits, and for specific programs and activities.

At all three levels, we also need to evaluate the economic impact. Cost benefit and cost effectiveness measures are needed to assess the positive economic impact on society and accountability in meeting the charitable tax-exempt purpose of community benefits. Even though they are not designed to enhance revenue, community benefit activities should still be delivered in an efficient and effective manner if the organization is to be a good steward of scarce resources.

Organizational Impact
Sr. Patricia Talone, RSM, Ph.D., CHA's vice president for mission services, reminds us the focus of community benefit for Catholic health care is anchored in understandings of the dignity of the human person, the social nature of human persons, and "a deep and abiding sense of its identity as a healing ministry of the church."⁵ Further, community benefit for Catholic health care is not meant to be a justification of not-for-profit status. In theological terms, we are talking about evangelical witness — those activities and behaviors in the world that testify to the presence and activity of God. The Scriptures tell readers that the evangelical witness is that the blind see, the deaf hear, the lame walk and the mute speak. Community benefit testifies to the mission of Catholic health care.

The organizational focus on community benefit is extending the ministry of Jesus in service to others. Community benefit moves employees from mission to core values rooted in the mission to actions and behaviors that manifest both. It follows then that evaluative processes ought to probe the congruence of mission, core values and activities. Evaluation of community benefit at the organizational level serves the primary purpose of challenging us to be true to the mission by living out the mission in service to the poor and the vulnerable. It is a way of enhancing and protecting fundamental human dignity.

Thus, at the organizational tier of evaluation, it is important to measure the impact of community benefit on improving organizational capacity for living the mission. Rooted in core values of Catholic social teaching, community benefit programs are mission driven to:

• provide service to, and advocacy for, those people whose social condition put them at the margins of society and make them particularly vulnerable to discrimination.
• focus on prevention and health promotion.
• actively collaborate with community residents and organizations.
• integrate community benefit programs into the strategic and operational plans, with strong leadership support.
• demonstrate the impact of community benefits programs and services.

Self-assessment inventories can be developed to assess the impact of community benefit on strengthening the organizational capacity to accomplish this mission. Inventory items (scored from 5 = very high to 1 = very low) can be developed to measure the level of progress made during the past year, and the past three years, in designated progress areas. These areas might include developing stronger community relationships with local businesses to sponsor wellness services; improving infrastructure to support community-based prevention services; improving capacity for cultural sensitivity in the provision of health care services by staff training, hiring and supervision; developing board leadership; and increasing organizational learning related to community benefits and their theological undergirding. A 10- to 15-item inventory can be developed and used with key internal stakeholders to evaluate organizational impact.

Case studies and anticipatory research can also be used to track and analyze impact, as well as to improve impact in the future. Anticipatory research alerts health care leaders to possible future scenarios (e.g., cuts in public programs providing prenatal nutrition services).

Impact on Community in Targeted Health Areas
The second evaluation tier consists of the clusters, or sets, of related community benefit activities. As described by the American Hospital Association's community connections initiative, hospitals and health care organizations are valued and vital community resources.⁶ These resources can be evaluated to assess the long-term community impact. Clusters of similar community benefit include services that provide treatment (charity care, government-sponsored health care, subsidized care), community health services (assessment, health education, health improvement, wellness and prevention, immunizations), education of health professionals, research, cash and in-kind donations, community-building, and community benefit operations.

We can evaluate whether the collection of activities within a cluster of related services improves the health of the community, or a subset of the community. Healthy People 2010 objectives and indicators can be used to establish measures and evaluate the impact on a community of interest. For example, has partnering with area health care organizations to deliver cancer screening outreach and mobile van services contributed to elimination of health disparities related to breast cancer screening of African-American women? Using Healthy People 2010 indicators and local breast cancer screening targets, we can evaluate impact. We can determine the collective contribution of partnering health care and public health organizations, and media campaigns on increasing early screening and detection of breast cancer in this at-risk group.

Cluster evaluation of the impact of related community benefits services is much more complex than evaluating single services or programs. Adequate resources (7 to 15 percent of the budget for the cluster of services) are typically allocated, and external evaluators are often hired.

Impact of Community Benefit Projects and Activities on Participants
The third tier of evaluation is at the project level. Here our interest is in measuring the desired short- and long-term health outcomes among participants. The focus is on outcome evaluation rather than impact evaluation. For example, we might report rates of six and 12-month smoking cessation among participants in a hospital-based smoking cessation program. However, if good health outcomes of a prevention or health promotion program are achieved, it may be useful to estimate the potential impact on the larger population using the epidemiologic measure called the prevented risk fraction, which is the proportion of disease observed in the whole population attributable to exposure to the risk factor.

Economic Evaluation of Community Benefit Projects and Activities
At all three tiers of evaluation noted in this article, economic evaluation can be undertaken. By tracking costs at each level, the organization can compare the costs to the measures of impact. This can help the organization to decide which community benefit activities are the most effective and efficient use of resources. For example, suppose that through engagement with the community and review of Healthy People 2010, the organization has decided that smoking cessation should be a focus of community benefit activities. As a result, several projects have been undertaken. These include a computerized reminder system for providers at one of the organization's medical office buildings, financial assistance with obtaining nicotine patches and gum for clients of the hospital's clinic, and a smoking cessation program for patients recovering from myocardial infarctions that includes telephone follow-up. All of these activities are recommended in the CDC's Community Guide as effective interventions.⁷

At the organizational level, cost-effectiveness analysis can assist managers in determining how much time and how many resources to invest in each of the three activities. Using the number of persons served as the measure of outcome, the cost per person served can be calculated for the three activities. This may vary depending on the size of the project. For example, patients recovering from a heart attack are more likely to be receptive to smoking cessation than the general patient population. Expanding the smoking cessation program to a larger audience might increase the cost per continuing participant because there will be costs of reaching out to persons who either choose not to enter the program or quickly drop out. Comparing the cost per participant for the three programs can help community benefit directors choose which programs to expand.

The entire cluster of smoking cessation activities can also be assessed. Here the outcome measure should be something that is comparable across the three activities and that measures the effectiveness of the interventions. The number of quitters at some specified time (e.g., six months post-intervention) is a good indicator. The costs of the entire smoking cessation endeavor can be compared to the number of quitters. Then, cost per quitter is a measure of the economic efficiency of the smoking cessation effort.

There are numerous existing studies of smoking cessation programs. Using the Community Guide, a benchmark for cost per quitter of approximately $200 seems reasonable.⁸ If the organization's cost per quitter is much higher than the industry standard, then perhaps it should reassess its focus on smoking cessation. The community may benefit more from other endeavors using the same resources.

Finally, the impact on participants can be incorporated into an economic evaluation. This is often a more complex endeavor. The outcome measure should measure the improved health of the participants. Such measures include the prevented risk fraction noted above and quality- adjusted life years saved. The benefits of smoking cessation extend over a long period and accrue to the person and society. Typically, the health care organization will not be able to track these benefits, but the existing literature can be used to assign measures of benefit. For example, much is known about the extension of life span due to quitting smoking, so a reliable estimate of the number of life years saved due to smoking cessation can be made. This type of analysis is complex and requires specialized expertise. It is most useful outside the organization. For example, when seeking funding from a state legislature or agency, this type of cost-effectiveness analysis allows policy makers to compare the organization's projects to others, such as in the fields of education and housing.

Conclusion
The four core aspects of navigation that were important to the Lewis and Clark expedition across America are important in evaluating community benefit in Catholic health care. Since CHA introduced the 1989 document Social Accountability Budget: A Process for Planning and Reporting Community Service in a Time of Fiscal Constraint, community benefits leaders in Catholic health care organizations have adopted standardized models and policies to account for community benefits. To sustain their community orientation and reinforce their community benefit role, Catholic health care organizations need to evaluate impact as well as the short- and long-term outcomes. We recommend that community benefit leaders create evaluation systems that are collaborative, tie institutional objectives to the Healthy People 2010 objectives, use the CDC's public health evaluation framework, and address impact at the organizational, community and project levels.

NOTES

1. Robert Archibald, "Lewis and Clark New Perspectives: Introduction," Gateway Heritage: The Quarterly Magazine of the Missouri Historical Society 24, nos.2 and 3 (Fall 2003-Winter 2004): 1.
2. U.S. Department of Health and Human Services, Healthy People 2010 2nd ed., (Washington, DC: U.S. Government Printing Office, 2000), www.health.gov/healthypeople.
3. James Denny, "Lewis and Clark New Perspectives: Conquering the Lower Missouri River, The Opening Chapter of the Lewis and Clark Expedition," Gateway Heritage: The Quarterly Magazine of the Missouri Historical Society 24, nos. 2 and 3 (Fall 2003-Winter 2004): 18-29.
4. CDC, "Framework for Program Evaluation in Public Health," Morbidity and Mortality Weekly Report 48, no. RR-11 (1999): 1-40.
5. Sr. Patricia Talone, "The Theology of Community Benefit," Health Progress 86, no. 4 (July-August 2005): 20.
6. AHA, Community Connections, www.caringforcommunities.org/hospitalsinaction/index.html.
7. CDC, The Community Guide, www.thecommunityguide.org.
8. CDC. "Tobacco Use Prevention and Control — Economic Evaluation Findings," in The Community Guide, www.thecommunityguide.org/tobacco/tobac-econ.pdf.

Copyright © 2008 by the Catholic Health Association of the United States.
For reprint permission, contact Betty Crosby or call (314) 253-3477.