A robust ethics service can provide opportunities for improved patient care, provider satisfaction, and strengthen organizational culture through identifying complex clinical dynamics that can impact length of stay, staff turnover, and, ultimately, cost. However, without appropriate evaluation plans designed in tandem with measurements and expected outcomes, we calculate numbers instead of assessing impact. Those designing and/or evaluating ethics consultation, education, and policy development underutilize logic models, often key components of solid evaluation plans. I will discuss four types of evaluation, in particular the use of logic models in evaluation planning, with special attention to application for clinical and organizational ethics.

As hospital leaders contend with competing pressures about decreasing length of stay but increasing satisfaction (for patients, staff, and physicians), measures of success vary by the discipline or interested party. In the ethics consultation literature, desirable patient outcomes take the form of decreases in non- beneficial treatments, decreased length of stay, fewer days in the intensive care unit, or decreased costs. Mark Repenshek offers three metrics to assess the value of a clinical ethics consultation service.¹ Batten argued that outcomes such as health care cost, clinical indicators in the intensive care unit, and patient satisfaction should not be used to evaluate the worth or success of a clinical ethics consultation service. He concluded that these are all outside of the ethics consultant's control and cannot be measured as ethics consultation outcomes.² We can see this clearly in the Andereck et al study of 384 patients in the intensive care unit, which concluded that not only were proactive ethics consultations ineffective in reducing overall length of hospital stay, ICU days, non-beneficial treatments, or hospital costs, but also such consultations were not effective in increasing perceptions of quality of care by patients or clinicians.³ Craig and May warned that it is easy to mistake the goals of ethics consultation to other outcomes because such outcomes are more easily measured and are closely related to the goals of ethics consultation.⁴ I similarly argue that the cause-and-effect relationship between ethics recommendation and outcome remains muddied due to the transdisciplinary provision of health care, but this relationship can be tightened when clearer causal measures are utilized, such as earlier versus late ethics consultations and excess length of stay.⁵

Clinical and operational colleagues often utilize The Model of Improvement, developed by Associates in Process Improvement, to accelerate improvement ranging from decreased surgical site infections to moving resources across care sites. Three questions guide the Model for Improvement: What are we trying to accomplish? How will we know a change is an improvement? What change can we make that will result in an improvement?

These questions inform the PDSA (plan-do- study-act) model of quality improvement. Unlike other forms of research, where the purpose is to discover new knowledge, those who conduct quality improvement initiatives attempt to acquire data quickly and interpret data for action. For example, a quality improvement team might meet with nursing to figure out how to eliminate excess trips to the medication room (a prime opportunity for medication errors). The team might ask nurses to wear pedometers to track steps or collect log-in information from the medication room door to see how many times nurses are keying in to collect medications. Those data might be collected for 30 days and then the team gets together to evaluate the data to decide what intervention could be made to expedite medication administration. When looking at the three questions, the team would probably say that they are trying to improve nurse efficiency or decrease medication errors. Improvement could be in the form of improved nurse satisfaction with daily work or decreased errors in medication administration. The change made could be moving the medication room to a more central location or establishing set times for medication administration. How might we consider a similar 'win' for an ethics service such as improving patient safety or patient satisfaction? Or is that even the right question ethics should be asking? If the task of ethicists in Catholic health care is to "facilitate discernment and provide guidance for making just and moral decisions when answers aren't always clear,"⁶ we must articulate a set of metrics and measures of success that mirror the responsibility of an ethicist. For that, we must better understand the various forms of evaluation and the kinds of measures of success for each form.

Evaluation asks two main questions: What do you want to know? and How will you know it? To answer these questions, four interrelated aspects comprise a well-designed evaluation plan: needs or asset assessment; process evaluation; outcome evaluation; and impact evaluation. Needs assessments seek to answer the following questions: What are the characteristics, needs and priorities of the target population? What are potential barriers/ facilitators? What is most appropriate to do? Process evaluation asks: How is the program implemented? Are activities delivered as intended? Is there fidelity of implementation? Are persons being reached as intended? Outcome evaluation targets the following:

To what extent are desired changes occurring? Are goals being met? Who is benefiting/not benefiting? What seems to work or not work? What are the unintended outcomes? Finally, impact evaluation asks us to consider: To what extent can changes be attributed to the program? What are the net effects? What are final consequences? Is the program worth the resources it costs?

In the Striving for Excellence in Ethics document from CHA,⁷ these four evaluation forms can be found in the following four ethics standards: 2.c.i. client needs assessment; 2.c.iv. individual member self-evaluation; 3.d.vi. evaluation of the consultation; and 3.h.i. evaluating and assessing effectiveness of structures, processes and quality of outcomes of ethics consultation. For example, in the needs assessment, a strong evaluator would help guide a service in completing the CHA assessment tool to gauge if a standard is fully present and functional, or the degree to which the standard is an opportunity for improvement. Employing standardized tools helps us compare cohorts and provide comparisons to other institutions. Perhaps a measure of success would be that a needs assessment is completed every two years or within six months of a change in ethics leadership. A needs assessment cannot stand alone in terms of organizational and service improvement nor can arbitrary metrics like a 5% change in persons rating a standard at a '2' withstand scrutiny regarding inter-rater reliability or differential loss to follow-up. For example, if we assess the group at one period of time, but a whole different group completes the needs assessment at a future time, it is difficult to draw meaningful conclusions about improvement. A potential way to address such concerns and to attend to the interrelated areas of evaluation is the logic model.

A logic model indicates precisely how each activity will lead to desired changes and can assist in the planning, operationalizing, and sustaining of a robust ethics service. Logic models can enhance accountability by keeping stakeholders focused on outcomes by preventing mismatches between activities and effects. These frameworks enhance relationships through a shared effort of collaboration and offer a transparent road map to a shared definition of success. Logic models help us to know what and when to measure, allow a simultaneous focus on both process measures and outcome measures, and ultimately prioritize where we will spend our limited resources. One might consider a particular limitation to a logic model to be the amount of time the process could take. Unlike the PDSA approach, which forces rapid-cycle improvement, the logic model, because it incorporates many other aspects that have their own set of barriers and constraints, may appear to delay implementation and success. However, a logic model can outlive changes in leadership, funding, or personnel because the model requires continual examination and reflection. As such, the logic model becomes a time-saving and resource-saving instrument in the design and implementation of a multifaceted ethics service.

Given that few organizations are in the position to begin an ethics service from scratch, developing an evaluation model in the midst of service delivery might seem daunting. Fortunately, implementation is a perfect time to consider adding in a logic model as it can allow for mid-course corrections by providing an inventory of what you have and what you need to operate the program or by reducing and avoiding unintended effects.⁸ In many ways, ethicists should look to evaluation, especially the logic model, as the perfect complement to how we go about our work. An ethicist does not merely find the 'right' answer or opine without thought of consequence. An ethicist helps persons to discern the good and helps to form persons who are aimed towards the good. Evaluation serves that same function; and a logic model offers a transparent roadmap of how we will move towards the good, whatever that means to the various stakeholders and their competing interests, which sounds like a pretty typical task for an ethicist.

MARY E. HOMAN, DrPH, MA, MSHCE
Assistant Professor
Center for Bioethics and Medical Humanities
Institute for Health & Equity 
Medical College of Wisconsin
Milwaukee
[email protected]

ENDNOTES

1. Mark Repenshek, "Assessing ROI for Clinical Ethics Consultation Services," Health Care Ethics USA 25, no. 3 (Summer 2017): 12 – 20.
2. Jason Batten, "Assessing Clinical Ethics Consultation: Processes and Outcomes," Medicine and Law 32, no. 2 (June 2013): 141.
3. William S. Andereck et al., "Seeking to Reduce Nonbeneficial Treatment in the ICU: An Exploratory Trial of Proactive Ethics Intervention," Critical Care Medicine 42, no. 4 (April 2014): 824 – 30, https://doi. org/10.1097/CCM.0000000000000034.
4. J M Craig and Thomas May, "Evaluating the Outcomes of Ethics Consultation," The Journal of Clinical Ethics 17, no. 2 (2006): 168.
5. Mary E. Homan, "Factors Associated with the Timing and Patient Outcomes of Clinical Ethics Consultation in a Catholic Health Care System," The National Catholic Bioethics Quarterly 18, no. 1 (Spring 2018): 71 – 92, https://doi.org/10.5840/ncbq20181818.
6. The Catholic Health Association, "It's Time to Consider a Career in Catholic Health Care Ethics" (The Catholic Health Association of the United States, n.d.), https:// www.chausa.org/docs/default-source/general-files/ ethics_career.pdf?sfvrsn=2.
7. Catholic Health Association of the United States and Ascension Health, Striving for Excellence in Ethics: A Resource for the Catholic Health Ministry (Saint Louis/ Washington, DC: The Catholic Health Association of the United States, 2014).
8. Center for Community Health and Development, "Chapter 2, Section 1: Developing a Logic Model or Theory of Change," in The Community Tool Box (Lawrence, KS: University of Kansas, 2017), https://ctb.ku.edu/en/ table-of-contents/overview/models-for-community- health-and-development/logic-model-development/main.